Data Lifecycle Diagrams

Hydrolix stores table data as individual partitions in object storage, each with a corresponding metadata row in a catalog database.

These diagrams show the lifecycles for data in both the catalog and the object store. They show which services create partitions, mark them inactive, remove storage objects, and manage the corresponding metadata entries.

To configure, see Set Data Retention Policies and the operational description of the Decay and Reaper services.

Catalog lifecycle

Every partition in object storage gets a corresponding row in the catalog, and the services in this diagram are the ones that create, deactivate, and remove those rows. Intake, the merge controller, and the alter service add rows, the merge controller and decay mark rows inactive, and reap collection collects inactive rows for the reaper to delete.

---
config:
  themeVariables:
    fontSize: 18px
  flowchart:
    padding: 20
---
flowchart TB
  classDef default stroke:#00A99D,stroke-width:2px

  catalog[(Catalog)]

  subgraph lifecycle-begin[Partition Creation]
    intake[Intake
           Services]
    merge-controller[Merge
                     Services]
    insert[Manual
           Ingest]
    alter[Alter
          Service]
  end

  subgraph lifecycle-end[Partition Removal and Metadata Maintenance]
    decay[Decay
          Job]
    reapsweep[Reap
              Collection Job]
    mergecleanup[Merge
                 Cleanup Job]
    reaper[Reaper
           Service]
  end

  catalog ~~~ decay
  catalog ~~~ mergecleanup
  catalog ~~~ reapsweep
  catalog ~~~ reaper

  intake --creates catalog entries
            for new partitions--> catalog
  merge-controller --marks original entries
                     inactive and inserts
                     new, merged entries--> catalog
  insert --creates catalog entries
            for new partitions--> catalog
  alter --marks original entries
          inactive and inserts
          new, altered entries--> catalog
  decay --marks old
          partitions
          inactive--> catalog
  mergecleanup --collects
                 merged
                 inactive--> catalog
  reapsweep --collects
              old
              inactive--> catalog
  reaper --deletes inactive entries --> catalog

  classDef dashed font-style:italic,stroke-dasharray: 3 3

Catalog and partition deletion

Deleting a partition means removing both its catalog row and its files in object storage.

The merge cleanup and reap collection cron jobs find inactive rows and publish reap events into a RabbitMQ queue. The reaper consumes those events from the queue, deletes the files from object storage, and deletes the catalog rows.

The partition cleaner is an optional, scheduled service that removes files in object storage with no metadata entry in a catalog row. This catches partitions the reaper missed.

---
config:
  themeVariables:
    fontSize: 18px
  flowchart:
    padding: 20
    nodeSpacing: 80
    rankSpacing: 130
---
flowchart TB
  classDef default stroke:#00A99D,stroke-width:2px

  catalog[("Catalog<br/>(PostgreSQL)")]
  rabbitmq@{ shape: das, label: "Persistent Queue\nRabbitMQ\n(stateful set)" }
  storage@{ shape: lin-cyl, label: "Object\nStorage" }
  reapsweep["Reap Collection<br/>(cron job)"]
  mergecleanup["Merge Cleanup<br/>(cron job)"]
  reaper["Reaper<br/>(deployment)"]
  cleaner["Partition Cleaner<br/>(deployment)"]

  %% Invisible edges keep the catalog in the top rank. The reaper edge comes
  %% first so the layout engine keeps the reaper in the row when it breaks the
  %% RabbitMQ read/write cycle. The repeated cleaner edges pull the catalog to
  %% the right so its edge labels don't overlap.
  catalog ~~~ reaper
  catalog ~~~ reapsweep
  catalog ~~~ mergecleanup
  catalog ~~~ cleaner
  catalog ~~~ cleaner
  catalog ~~~ cleaner
  catalog ~~~ cleaner

  %% Edge order sets the left-to-right order of the row: cron jobs, then
  %% reaper, then cleaner. The reaper's RabbitMQ edges come before its object
  %% storage edge so RabbitMQ lands on the left.
  reapsweep --collects<br/>old<br/>inactive<br/>(read)--> catalog
  mergecleanup --collects<br/>merged<br/>inactive,<br/>unlocks<br/>(read/write)--> catalog
  reapsweep --publishes<br/>reap events<br/>(write)--> rabbitmq
  mergecleanup --publishes<br/>reap events<br/>(write)--> rabbitmq
  reaper --deletes catalog entries corresponding to deleted object store files<br/>(write)--> catalog
  rabbitmq --consumes
             reap
             events<br/>(read)--> reaper
  reaper --writes to
           retry
           queue<br/>(write)--> rabbitmq
  reaper --deletes partitions<br/>(write)--> storage
  cleaner --verifies partition entries<br/>(read)--> catalog
  cleaner --deletes partitions the catalog no longer references<br/>(read/write)--> storage

Partition states

Partitions are written by intake, merge, alter services, or a client manually inserting data with INSERT INTO. Every partition is in one of the following states:

Active?	Locked	Data age relative to age.max_age_days	Description
True	No	under age.max_age_days	Written by intake, merge, ALTER, or INSERT INTO. Queryable and eligible for merge.
True	No	over age.max_age_days	Past retention but not yet identified by Decay, or the table has age.max_age_days set to 0. Still queryable.
True	Yes	any	Owned by the lock holder: a merge reservation, an ALTER job, or a manual lock on a problem partition. Queryable. Decay skips it.
False	No	any	This partition has been deactivated by decay or replaced with a new partition by merge or alter. Reaper deletes it once it has been inactive for reaper.max_age_days.
False	Yes	any	Locked in preparation either for removal (partitions replaced by a merge and waiting for Merge Cleanup, or problem partitions parked for inspection) or activation (new partitions from an ALTER job that hasn't committed yet).

Problematic partitions can be removed from the system by locking the partition and setting the active flag to False. The data remain in the partition for inspection, but the query system doesn't consult the partition, since it's inactive.

Partition state diagram

The query system uses partitions that are active whether locked or not. This diagram shows how a partition moves between the states in the table, and which service or job drives each transition. States with a bold outline are queryable.

---
title: Partition Lifecycle State Diagram
config:
  themeVariables:
    fontSize: 18px
---
stateDiagram-v2
  classDef onbrand stroke:#00A99D,stroke-width:2px

  old: active (old)

  [*]:::onbrand --> active:::usable
  active --> old:::usable : older than max age

  alter:::onbrand: locked (altering)
  active --> alter:::usable : ALTER job
  alter --> inactive:::onbrand : ALTER job committed

  merged : inactive (merged)
  merging: locked (merging)
  active --> merging:::usable : eligible for compaction
  merging --> merged:::onbrand : marked by merge-controller
  merged --> inactive : found by merge-cleanup

  old --> inactive : marked by decay
  inactive --> [*]:::onbrand : deleted by reaper

  classDef usable stroke:#00A99D,stroke-width:4px