Data Lifecycle
The Data Lifecycle feature allows you to set data retention policies controlling how long partitions are available for query and how long to wait before deleting old, inactive data
Set these data retention policies in your table settings configuration.
Data lifecycle management operates as a two-step process. Two services carry out the steps:
-
Decay identifies partitions containing data that has exceeded the configured retention period and marks them as "deactivated" in the Hydrolix catalog. Once deactivated, these partitions are hidden from new queries but remain available to complete any in-flight queries. The decay service runs continuously, checking partition ages based on the primary timestamp of the data itself.
-
Reaper permanently deletes partitions that have been deactivated for a specified grace period. It removes these partitions from object storage and from the catalog. The reaper service is also used by other Hydrolix services like merge-cleanup to handle partition deletion.
An optional, scheduled service, the partition cleaner deletes orphaned partition files left in object storage. See Unconfirmed deletes.
| Component | Description |
|---|---|
| Decay | Checks partition age. When an active partition exceeds the configured maximum active age, the partition is made inactive. When an inactive partition exceeds the configured maximum inactive age, sends a partition deletion task to the queue. |
| Reaper | Retrieves partition delete tasks from the queue. Deletes the indicated partition's files from the storage bucket, then removes the partition from the catalog. When it can't confirm the file deletes, removes the catalog record and leaves any remaining files for the partition cleaner. |
| Partition cleaner | Deletes files that remain in the storage bucket with no catalog record referencing them, including the files Reaper leaves behind. |
| Queues | One queue that contains partition deletion tasks and a separate dead-letter queue, containing expired jobs that weren't completed in a certain period of time. |
| Hydrolix Database Storage Bucket | Contains the partitions that comprise the database. Part of the core infrastructure. |
| Catalog | Contains metadata regarding data stored in Hydrolix. Part of the core infrastructure. |