Data Lifecycle
The Data Lifecycle feature allows you to set data retention policies controlling how long partitions are available for query and how long to wait before deleting old, inactive data
Set these data retention policies in your table settings configuration.
Data lifecycle management operates as a two-step process involving two services:
-
Decay identifies partitions containing data that has exceeded the configured retention period and marks them as "deactivated" in the Hydrolix catalog. Once deactivated, these partitions are hidden from new queries but remain available to complete any in-flight queries. The decay service runs continuously, checking partition ages based on the primary timestamp of the data itself.
-
Reaper permanently deletes partitions that have been deactivated for a specified grace period. It removes these partitions from both the catalog and object storage. The reaper service is also used by other Hydrolix services like merge-cleanup to handle partition deletion.
| Component | Description | Scale to 0 |
|---|---|---|
| Decay | Checks partition age. When an active partition exceeds the configured maximum active age, the partition is made inactive. When an inactive partition exceeds the configured maximum inactive age, sends a partition deletion task to the queue. | Yes |
| Reaper | Retrieves partition delete tasks from the queue. Removes the indicated partition from the catalog, then from the storage bucket. | Yes |
| Queues | One queue that contains partition deletion tasks and a separate dead-letter queue, containing expired jobs that were not completed within a certain period of time. | Yes |
| Hydrolix Database Storage Bucket | Contains the partitions that comprise the database. Part of the core infrastructure. | No |
| Catalog | Contains metadata regarding data stored in Hydrolix. Part of the core infrastructure. | No |