Performance Features

Hydrolix performance improvements fall into two groups:

• Features that reduce query latency
• Features that reduce the clock time, CPU, and disk space a cluster uses to store data

Several features serve both goals and appear in both sections, each described from the relevant angle.

For capacity planning, see Scaling. To ignore Kubernetes requests and limits, disable operator management of particular resources, or load balance with pools of services, see Resource Tuning.

To measure the effect of any of these changes, see Query Performance Debugging and the Metrics Reference.

Query performance

These features reduce query latency, either by narrowing the data a query reads or by shaping the data so there's less to compute at query time.

• Adaptive Indexing indexes high-cardinality string columns with Bloom filters instead of dictionaries, so equality (=) and IN queries on those columns skip the dictionary load. Disabled by default.
• Full-Text Search speeds up filtering on string columns that contain long text values such as log messages and URLs. Both pattern matching (LIKE, ILIKE, MATCH) and equality (=) queries benefit. Disabled by default.
• Merge combines small partitions into larger ones, which reduces the number of partitions a query evaluates.
• Partition Bloom Filters reject partitions that can't hold a value before query peers read their data or indexes, which reduces the partitions that must be evaluated for equality (=) or IN queries.
• Query Efficiency covers the query patterns that let Hydrolix use partition pruning and indexes. It also covers antipatterns.
• Query Head Pools isolate query workloads from each other, so one heavy workload doesn't slow the rest.
• Query Options control pool selection, rate limiting, and circuit breakers for individual queries.
• Summary Tables aggregate data as it arrives, so dashboards and reports read precomputed rollups rather than raw data.

Storage and write performance

These features reduce ingest latency, how much data a cluster stores, or both.

• Adaptive Indexing drops dictionary indexes for high-cardinality string columns and writes smaller Bloom filters in their place. This reduces the disk space required for these indexes. Disabled by default.
• Data Lifecycle Management sets how long a table keeps data, which bounds total storage.
• Decay and Reaper apply the data lifecycle retention policies. They deactivate aged partitions and later delete them.
• Merge rewrites many small partitions into fewer large ones, which lowers per-partition storage overhead and the amount of metadata the catalog holds.
• Partition Cleaner deletes partitions the catalog no longer references, including the ones merge leaves behind.
• Stream Settings control how many partitions to keep open and for how long while ingesting data, which data counts as late-arriving and can be rejected from storage, and how much work the indexer does for each partition it writes.
• Summary Tables store aggregated rows, so a retention policy can keep summaries and let raw data age out.
• Transform Structure controls which columns Hydrolix indexes and stores. Turning off index for a column, or suppressing a field, cuts both index build time and stored bytes.
• Vacuum removes rejected data, cluster logs, and other operational leftovers on a daily schedule.