Decay and Reaper

Data lifecycle management operates as a two-step process involving two services:

• Decay identifies partitions containing data that has exceeded the configured retention period and marks them as "deactivated" in the Hydrolix catalog. Once deactivated, these partitions are hidden from new queries but remain available to complete any in-flight queries. The decay service runs continuously, checking partition ages based on the primary timestamp of the data itself.

• Reaper permanently deletes partitions that have been deactivated for a specified grace period. It removes these partitions from object storage and from the catalog. When Reaper can't confirm that a partition's files were deleted, it removes the partition's catalog record anyway and leaves any remaining files for the partition cleaner. The reaper service is also used by other Hydrolix services like merge-cleanup to handle partition deletion.

Configure data retention policies at the table level to manage how long data should remain alive before Decay and Reaper delete it.

Decay

Decay is a cronjob that identifies partitions that contain old data and marks them for deletion based on configured data retention policies. It prevents unbounded storage consumption by deactivating partitions that exceed configured age thresholds, and sending deletion tasks to the Reaper service for physical removal from the catalog and object storage.

What Decay does

Partition deactivation

1. Examines all tables with age.max_age_days configured (tables with value 0 are skipped)
2. Calculates cutoff time as current_time - age.max_age_days
3. Identifies active partitions where both max_timestamp and min_timestamp are older than the cutoff
4. Marks matching partitions as inactive in the Catalog, which makes them unavailable to future queries
5. Sets deactivation_reason to decay in partition metadata for audit purposes

Important notes:

• Age is calculated relative to the partition's primary datetime values, not the ingestion time.
• Decay only processes unlocked partitions. It skips partitions that are locked by Merge.
• Deactivated partitions continue serving any in-flight queries that were active at deactivation time.

Reaper queue management

1. Examines all tables with reaper.max_age_days configured (tables with value 0 are skipped)
2. Calculates cutoff time as current_time - reaper.max_age_days
3. Identifies inactive, unlocked partitions where the modified timestamp (deactivation time) is older than the cutoff
4. Creates ReapEvent objects for each partition and sends them to the Reaper queue in RabbitMQ

Important notes:

• Reaper age is calculated from the deactivation date set by Decay, not the original data timestamp.

Decay configuration

Decay Kubernetes tunables

Parameter	Description	Default
decay_enabled	Enable/disable the Decay CronJob	true
decay_schedule	CRON schedule (daily at midnight UTC)	0 0 * * *
decay_batch_size	Entries fetched per catalog request for deactivation phase	5000
decay_max_deactivate_iterations	Max deactivation loops per table. Default is unlimited	None
decay_reap_batch_size	Entries fetched per catalog request for reaping phase	5000
decay_max_reap_iterations	Max reaping loops per table. Default is unlimited	None

Decay environment variables

• BATCH_SIZE - Maps to decay_batch_size
• REAP_BATCH_SIZE - Maps to decay_reap_batch_size
• DECAY_MAX_ITERATIONS - Maps to decay_max_deactivate_iterations
• REAP_MAX_ITERATIONS - Maps to decay_max_reap_iterations
• REAPER_QUEUE - Queue ARN for reaper jobs

Decay table-level settings

Parameter	Description	Default
age.max_age_days	Days data remains active/queryable before deactivation	0 (indefinite)

Review Data Retention Policies for details on how to configure Time-to-Live (TTL) policies.

Manual decay execution

Run Decay manually by creating a job with kubectl.

kubectl create job --from=cronjob/decay {decay-manual-job-name} -n $HDX_KUBERNETES_NAMESPACE

In K9s, use command :cronjobs, select your decay job and select t for trigger.

The decay pod relating to the manual trigger will appear in the list of pods.

Decay metrics

Decay doesn't have dedicated Prometheus metrics. Its database operations are tracked with database metrics labelled method="deactivate_entries" and db="catalog".

Metric Name	Type	Description
query_latency_summary	Summary	Latency summary for deactivating partitions (p50, p90, p99)
query_latency_histo	Histogram	Latency histogram for deactivating partitions
query_count	Counter	Count of deactivation queries executed
query_failure	Counter	Count of failed deactivation queries

Reaper

The Reaper service deletes partitions that have been marked for deletion by Decay. It's also used by other services like Merge-Cleanup to handle partition deletion. It runs as a separate, continuous service consuming the reaper message queue from RabbitMQ.

What Reaper does

Reaper reads ReapEvents from RabbitMQ in batches. For each batch, the service:

• groups the events by storage.
• deletes each partition's files in a bulk request per storage.
• removes the confirmed partitions from the catalog in a single batch operation.

Unconfirmed deletes

This behavior was introduced in Hydrolix version 6.4.0.

Reaper removes a partition only when the bulk delete response lists every one of the partition's files as deleted. One unlisted file leaves the whole partition unconfirmed, even when the rest were deleted.

What happens to an unconfirmed partition depends on the response.

A rate-limiting or ambiguous response, such as a 429 or a timed-out request, defers the partition and opens a backoff window for that storage. Nothing is removed from the catalog, and the event returns to the queue for a later attempt.

For any other error, such as an access-denied or internal error from the storage, reaper removes the partition's catalog record and leaves any remaining files in object storage. Queries no longer reference the partition, but its files stay until the partition cleaner removes them. Each drop increments reaper_orphaned_total.

When reaper_orphaned_total is greater than zero, enable the partition cleaner: set its replica count to 1 and set PARTITION_CLEANER_DRY_RUN to false.

Reaper configuration

Reaper table-level settings

Parameter	Description	Default
reaper.max_age_days	Days after deactivation before physical deletion	1

Review Data Retention Policies for details on how to configure Time-to-Live (TTL) policies.

Reaper environment variables

Variable	Description	Default
REAPER_THROTTLE_COOLDOWN	Initial backoff window after a rate-limiting response from a storage	60s
REAPER_THROTTLE_COOLDOWN_MAX	Upper bound for the backoff window as it doubles	15m

How to check the Reaper queue

Navigate to the RabbitMQ pod shell and run command

rabbitmqctl --node rabbit@rabbitmq-0.rabbit list_queues

Reaper metrics

Reaper has dedicated Prometheus metrics. All of them carry the labels storage_id, project_id, and table_id.

Metric Name	Type	Description
reaper_orphaned_total	Counter	Catalog records deleted without confirmed deletion of the corresponding object store files. These files stay in storage until the partition cleaner removes them.
reaper_throttled_total	Counter	Reap events deferred after a rate-limiting response from the storage
reaper_backoff_skipped_total	Counter	Reap events deferred without contacting the storage because its backoff window was open
reaper_failure	Counter	Count of reaper failures
reaper_malformed_event_total	Counter	Reap events dropped from the queue because they don't name a valid partition. Nothing is deleted and the catalog record is left alone.
reaper_stale_storage_total	Counter	Reap events where the storage ID was missing from the published configuration
reaper_storage_not_found_total	Counter	Reap events treated as already reaped because the storage reported not-found, or because its host stayed unresolvable across retries
reaper_reap_all_not_found_total	Counter	Reap events where the reaper reached the storage and every file delete reported not-found. Expected on redelivery, but a sustained rate can indicate a storage path mismatch.
reaper_reap_duration	Summary	Duration in milliseconds of a reaper's reap cycle (p50, p90, p99)

Use reaper_orphaned_total to measure the backlog the partition cleaner needs to reclaim. Don't use reaper_failure for this: it counts both the failures that drop a catalog record and the delayed retries for an unresolvable storage host, where nothing has been dropped. A rise in reaper_failure isn't evidence that any files are orphaned.

A rise in reaper_throttled_total or reaper_backoff_skipped_total means the storage is rejecting delete requests and the reaper is backing off, not that data is at risk. Reaping falls behind while the backoff windows are open, and deferred events stay in the queue.

When Reaper deletes catalog entries, they're tracked with database metrics labelled method="delete_deactivated_entry" and db="catalog".

Metric Name	Type	Description
query_latency_summary	Summary	Latency for deleting catalog entries
query_latency_histo	Histogram	Latency histogram for catalog deletions
query_count	Counter	Count of deletion queries
query_failure	Counter	Count of failed deletions