Skip to content

Partition Cleaner

The partition cleaner deletes partitions no longer referenced by the Hydrolix catalog.

When Hydrolix optimizes storage usage, it sometimes merges data from multiple partitions or time periods into a single partition. During this optimization step, Hydrolix doesn't automatically delete unused partitions or their data. Instead, the catalog removes references or pointers to the unused partitions.

The partition cleaner deletes partitions that have been rendered redundant for at least 24 hours. It's off by default. Once enabled, its pod runs continuously and starts a cleanup run weekly on Monday at 12:00 AM UTC.

Starting in Hydrolix version 6.4, the partition cleaner also reclaims the files that reaper orphans when it can't confirm a partition's files were deleted. See Unconfirmed deletes.

Configuration⚓︎

Configure the partition cleaner using environment variables on the partition-cleaner deployment.

The partition cleaner is off by default, and two settings control that. The partition-cleaner replica count defaults to 0, so no pod runs, and PARTITION_CLEANER_DRY_RUN defaults to true, so a running pod logs what it would delete and takes no action.

To enable the partition cleaner, set the replica count to 1 and set PARTITION_CLEANER_DRY_RUN to false.

Each run lists the objects of every table in every storage, so the number of list requests grows with the number of tables and the files they contain. On a large cluster, these list requests can become costly. Enable the partition cleaner when a cluster has orphaned files to reclaim rather than leaving it on permanently.

Schedule configuration⚓︎

  • Environment variable: PARTITION_CLEANER_SCHEDULE
  • Format: UNIX cron notation
  • Default: weekly on Monday at 12:00 AM UTC
  • Example: 0 0 * * 1 (default)

Dry run⚓︎

  • Environment variable: PARTITION_CLEANER_DRY_RUN
  • Values: true (default) or false

Grace period⚓︎

  • Environment variable: PARTITION_CLEANER_GRACE_PERIOD
  • Format: Duration string (24h)
  • Default: 24 hours

Metrics⚓︎

The partition-cleaner exports metrics to monitor partition cleaner behavior:

Metric Name Purpose Example
bulk_delete_bytes How much data was deleted bulk_delete_bytes{app="partition-cleaner",bucket="hdx-test-test",storage="hdx_primary"} 1.520361e+06
bulk_delete_duration How long deletes take (percentiles) bulk_delete_duration{app="partition-cleaner",bucket="hdx-test-test",storage="hdx_primary",quantile="0.99"} 466
bulk_delete_duration_sum Total time spent on deletes bulk_delete_duration_sum{app="partition-cleaner",bucket="hdx-test-test",storage="hdx_primary"} 603
bulk_delete_duration_count Number of delete operations measured bulk_delete_duration_count{app="partition-cleaner",bucket="hdx-test-test",storage="hdx_primary"} 24
bulk_delete_success_count Number of deletes that worked bulk_delete_success_count{app="partition-cleaner",bucket="hdx-test-test",storage="hdx_primary"} 33
bulk_delete_failure_count Number of deletes that failed bulk_delete_failure_count{app="partition-cleaner",bucket="hdx-test-test",storage="hdx_primary"} 0

Issue: Orphaned partition files stay in object storage⚓︎

Required access: Cluster

Confirm: bulk_delete_success_count{app="partition-cleaner"} is flat or missing, while increase(reaper_orphaned_total[7d]) is above zero.

Starting in Hydrolix version 6.4, Reaper can delete a partition's catalog record and leave its files in object storage, counting each one in reaper_orphaned_total. Only the partition cleaner removes these files. See Decay and Reaper: Unconfirmed deletes.

To check whether the partition cleaner is removing them:

  1. Check that kubectl get deployment partition-cleaner shows a running pod. The replica count defaults to 0.

  2. Check that it isn't in dry-run mode, which is also the default. A dry run logs running in dry-run mode, would have deleted <n> files and deletes nothing.

  3. Check that deletes succeed. Navigate to https://{hostname}/prometheus and compare successful and failed deletes since the last run:

    PromQL: Partition Cleaner Deletes
    1
2
    sum(increase(bulk_delete_success_count{app="partition-cleaner"}[7d]))
sum(increase(bulk_delete_failure_count{app="partition-cleaner"}[7d]))

    Runs start only on the schedule, Monday at 12:00 AM UTC by default, so a newly enabled cleaner shows no deletes until then.

  4. Search the partition-cleaner logs for partition cleanup failed. Each one names a table the cleaner stopped partway through. This error doesn't always show in bulk_delete_failure_count, so check the logs even when that metric is zero.

Fix: Enable the partition cleaner when reaper_orphaned_total increases. Set the replica count to 1 and set the partition_cleaner_dry_run tunable to false.

Enable the Partition Cleaner
1
2
3
4
5
spec:
  partition_cleaner_dry_run: false
  scale:
    partition-cleaner:
      replicas: 1

If bulk_delete_failure_count rises, the bulk delete errors warning in the partition-cleaner logs groups the failures by cause. The cleaner uses the same storage credentials as Reaper, so a permissions or bucket policy problem that blocked Reaper blocks the cleaner too.

After bulk_delete_success_count rises, remove both settings from the spec to stop the weekly scan of every table's storage. See Partition Cleaner: Configuration for the schedule and grace period.

Differences between Decay and Reaper and the partition cleaner⚓︎

Both decay/reaper and the partition cleaner delete partitions from object storage, but they serve different purposes. Decay is part of the Hydrolix data lifecycle management system and selects old data for Reaper to delete based on configured data retention policies. The partition cleaner performs maintenance by removing orphaned partitions that exist in object storage but aren't referenced in the catalog. Partitions become orphaned through system errors like catalog write failures and failed merge operations, and through Reaper dropping a catalog record whose file deletes weren't confirmed.

  • Decay and reaper manage data retention. These services also delete data from the catalog.
  • Partition cleaner maintains storage integrity. This service deletes data solely from object storage.

Differences between partition cleaner and periodic service⚓︎

The periodic service manages operational files generated during cluster operations(service logs, rejected data files, Keycloak backups) and runs its own cleaning vacuums to delete these items from object storage.

Periodic service cleans up ancillary operational data in the primary storage bucket. It doesn't look at ingested data or partitions.