Manage Data

These symptoms involve the jobs that maintain data after ingest, such as merge.

Merge

Merge combines partitions to maintain query performance and storage efficiency.

Failures are gradual: merge falls behind, partition counts rise, and query latency degrades over days. Merge metrics carry the signal; no error is raised.

Merge is falling behind

Required access: Cluster

Confirm: sum by (target) (candidates{app="merge-controller"}) climbs and stays high while active_merges stays flat, and actual_partition_count pulls away from ideal_partition_count.

Merge combines small partitions into larger ones to reduce how many files a query reads. When merge can't keep up, partition counts climb and query latency degrades over days. Nothing raises an error. The metrics are the only signal.

No single metric reports merge lag. Confirming this takes two readings, both trends, not thresholds. Navigate to https://hostname.hydrolix.live/prometheus and run both:

1. Check whether work is queuing up faster than it's consumed.

   sum by (target) (candidates{app="merge-controller"})
   sum by (target) (active_merges{app="merge-controller"})
   

   candidates counts partition groups the controller has assembled and marked ready. It plateaus at merge_max_candidates, where the controller pauses construction. Read a flat line at that value as a full buffer, not a drained queue.

2. Check whether the backlog is changing the shape of the data.

   sum by (target, project_name, table_name) (actual_partition_count)
   sum by (target, project_name, table_name) (ideal_partition_count)
   

   A widening gap means partitions are accumulating faster than merge reduces them. Merge efficiency by project and table reports the same relationship as a ratio. Compare a target against its own history, not against 1.0.

Fix: Add memory when merge peers are OOM-killed persistently. A peer that restarts under memory pressure can't drain a backlog. Adding replicas repeats the failure. See Troubleshoot merge-peer OOM.

Add replicas when the peers are stable and candidates is saturated. The merge-peer default comes from the scale profile: 1-4 on prod, lower on dev and eval, higher on mega. Raise the ceiling for the profile in use:

spec:
  scale:
    merge-peer:
      replicas: 1-<max>

If candidates stays low while the partition gap grows, the bottleneck is construction, not consumption, and merge_max_candidates isn't the limit. The controller builds candidates from a buffer of at most merge_max_partitions partitions, and refills it only once it's empty.

Prevent: Alert on each target's efficiency relative to its own recent range, not on a fixed value. Autoscale merge peers on a merge metric, not CPU. Capacity then follows the queue. See Metric candidates and autoscaling.

Merge peers are OOM-killed

Required access: Cluster

Confirm: kubectl describe pod <pod-name> | grep -A 3 "Last State" reports OOMKilled. Identify the container first: merge peer and merge indexer take memory from different fields.

Fix: Merge: Troubleshoot merge-peer OOM.

Partition cleaner

The partition cleaner deletes files in object storage that the catalog no longer references. It's off by default and runs only on its schedule, so orphaned files can collect without raising an error.

Orphaned partition files stay in object storage

Required access: Cluster

Confirm: bulk_delete_success_count{app="partition-cleaner"} is flat or missing, while increase(reaper_orphaned_total[7d]) is above zero.

Starting in Hydrolix version 6.4, Reaper can delete a partition's catalog record and leave its files in object storage, counting each one in reaper_orphaned_total. Only the partition cleaner removes these files. See Decay and Reaper: Unconfirmed deletes.

To check whether the partition cleaner is removing them:

1. Check that kubectl get deployment partition-cleaner shows a running pod. The replica count defaults to 0.

2. Check that it isn't in dry-run mode, which is also the default. A dry run logs running in dry-run mode, would have deleted <n> files and deletes nothing.

3. Check that deletes succeed. Navigate to https://{hostname}/prometheus and compare successful and failed deletes since the last run:

   sum(increase(bulk_delete_success_count{app="partition-cleaner"}[7d]))
   sum(increase(bulk_delete_failure_count{app="partition-cleaner"}[7d]))
   

   Runs start only on the schedule, Monday at 12:00 AM UTC by default, so a newly enabled cleaner shows no deletes until then.

4. Search the partition-cleaner logs for partition cleanup failed. Each one names a table the cleaner stopped partway through. This error doesn't always show in bulk_delete_failure_count, so check the logs even when that metric is zero.

Fix: Enable the partition cleaner when reaper_orphaned_total increases. Set the replica count to 1 and set the partition_cleaner_dry_run tunable to false.

spec:
  partition_cleaner_dry_run: false
  scale:
    partition-cleaner:
      replicas: 1

If bulk_delete_failure_count rises, the bulk delete errors warning in the partition-cleaner logs groups the failures by cause. The cleaner uses the same storage credentials as Reaper, so a permissions or bucket policy problem that blocked Reaper blocks the cleaner too.

After bulk_delete_success_count rises, remove both settings from the spec to stop the weekly scan of every table's storage. See Partition Cleaner: Configuration for the schedule and grace period.

Back to all symptoms