Adaptive Indexing

Adaptive indexing was introduced in Hydrolix version 6.3 and is disabled by default.

Adaptive indexing provides an alternative to the indexer when writing partitions for string columns. Creating the column with a Bloom filter index instead of a dictionary index reduces storage and improves query performance when a dictionary index grows too large as the result of a high-cardinality column.

This feature is distinct from partition Bloom filters which the query system uses to skip partitions entirely.

How it works⚓︎

Adaptive indexing uses a Bloom filter index rather than a dictionary index for high-cardinality string columns with large indexes. This reduces query peer memory consumption and indexing storage requirements, and avoids loading a large dictionary for equality (=) and IN queries on those columns.

Hydrolix builds an index for every column that a transform marks as indexed. Indexed string columns get a dictionary index by default. The indexer collects the column's distinct values into a dictionary and builds an index that maps each dictionary entry to the blocks holding those values.

A dictionary index is most useful when values repeat. When a string column holds mostly distinct values, and the set of distinct values is large, its dictionary can approach the size of the compressed column data. Loading the index can dominate the memory a query peer needs to open the partition.

Adaptive indexing provides an alternative to the indexer: write the column with a Bloom filter rather than a dictionary, reducing query peer memory usage and query latency for columns with large dictionaries. Additionally, block Bloom filter indexes are most effective for queries pulling most of their data from object storage rather than a cache.

Query operation⚓︎

Consider the following query:

Example Query Using WHERE Filter
SELECT timestamp, app, component, message
FROM hydro.logs
WHERE error = 'connection reset by peer'
  AND timestamp > NOW() - INTERVAL 1 HOUR;

At query time, for a column indexed with block Bloom filters, Hydrolix hashes the queried value (connection reset by peer) from the predicate (WHERE error = 'connection reset by peer') and checks that hash against each block's Bloom filter.

The Bloom filter returns one of two answers: the value is definitely absent, or the value might be present. A definite absent result lets the query peer skip the block.

Bloom filters never return false negatives. They return a small rate of false positives, which means the query peer reads a small number of blocks that hold no matching rows and discards those rows during evaluation. Both outcomes produce correct results.

Adaptive indexing and partition Bloom filters complement each other.

partition Bloom filters skips whole partitions before any index or dictionary is read; this is partiton pruning
block Bloom filter indexes narrow which data blocks to read inside a partition.

Requirements⚓︎

Column requirements⚓︎

Adaptive indexing evaluates indexed string columns. All other columns keep their index types:

DictKeyIndex used for bool, int8, uint8, int32, uint32, int64, uint64, uuid, ip
RangeIndex used for datetime, datetime64, epoch
DictKeyIndex used for string columns by default
BloomKeyIndex used for string columns when a DictKeyIndex would be inefficient and adaptive indexing is enabled
FullTextIndex used for string columns when full text indexing is enabled

Partition requirements⚓︎

The indexer evaluates which index to use while writing a partition. It replaces the column's dictionary index with a Bloom filter index only when these conditions hold for the data in that partition:

The partition is high-cardinality.
The partition's dictionary index is at least as large as the dict_bytes_threshold setting

Because the index type depends on the data in each partition, a column that uses a dictionary index in some partitions might use block Bloom filter indexes in others.

Query requirements⚓︎

Equality (=) and IN predicates benefit from block Bloom filter indexes.

Query with Equality PredicateQuery with IN Predicate

SELECT timestamp, component FROM my_project.my_table WHERE error = 'connection reset by peer';

SELECT timestamp, component FROM my_project.my_table WHERE error IN ('connection reset by peer', 'connection timed out');

Configuration⚓︎

Hydrolix disables adaptive indexing by default. The adaptive_encoding_enabled setting of the partition_bloom tunable turns it on.

Enable Adaptive Indexing
spec:
  partition_bloom:
    adaptive_encoding_enabled: true

Adaptive indexing and partition Bloom filters are independent features

Adaptive indexing is configured under the same partition_bloom tunable as partition Bloom filters, but the two features are independent. They can be enabled and disabled separately.

Two more settings in the same tunable control which columns qualify and how accurate their filters are.

Setting	Default	Description
`adaptive_encoding_enabled`	`false`	Whether the indexer can replace string dictionary indexes with block Bloom filter indexes.
`dict_bytes_threshold`	`536870912`	Minimum dictionary size, in bytes, for a column to qualify for block Bloom filter indexes. The default is `512 MiB`.
`block_false_rate_pct`	`1`	The target false-positive rate of each block Bloom filter, as an integer percentage. The default is 1%.

Change settings with care

Hydrolix recommends keeping the default values for dict_bytes_threshold and block_false_rate_pct.

Lowering dict_bytes_threshold qualifies string columns with smaller indexes for block Bloom filter indexes. A column that loses a dictionary it could have loaded cheaply gives up exact block lookups for a risk of false positives, which could slow queries the dictionary index served well.

Lowering block_false_rate_pct makes each filter more accurate and larger, which adds index bytes to every block. Raising it reduces filter sizes but risks more unnecessary blocks being read during queries.

Contact Hydrolix Support before changing these values on a production cluster.

Disable block Bloom filter indexes⚓︎

Stop using adaptive indexing in either of two ways:

Disable the feature. On v6.3, set adaptive_encoding_enabled to false. New writes won't use BloomKeyIndex for any indexed string column.
Roll back to a version earlier than v6.3. Earlier versions don't recognize adaptive_encoding_enabled and never write block Bloom filter indexes.

Neither approach rewrites the indexes for existing partitions, so any column already written with a block Bloom filter index keeps it. On v6.3, queries will use those indexes. Earlier versions can't read the index type, so queries on those columns perform a full partition scan, but still return correct results.

Merge considerations⚓︎

Disabling the feature or reverting to a previous Hydrolix version doesn't rewrite existing partition indexes immediately, but merge does over time. The merge service compacts multiple partitions into a single new partition and index using the index types compatible with the cluster version and configuration at the time of merge. When adaptive_encoding_enabled is false, data from any partition still eligible for merge is rewritten without a block Bloom filter index. The reverse is also true: with the feature enabled, merging partitions may produce a partition with a block Bloom filter index.

Partitions not eligible for merge keep their original index. See Merge pools for the age, size, and width criteria that determine eligibility.

Metrics⚓︎

These metrics show whether queries are using block Bloom filter indexes.

Metric	Type	Instrumentation site	Purpose
`hdx_partition_bloom_index_columns_queried`	Histogram	Query peer	Columns whose block Bloom filter index is consulted for one partition.
`hdx_bloom_index_columns_queried`	Query runtime stat	Query peer	Columns whose block Bloom filter index is consulted across a whole query, which equals the number of dictionary loads avoided. Available in the `query_detail_runtime_stats` column of `hydro.logs`.

Limitations⚓︎

Only indexed string columns qualify
Only equality (=) and IN benefit from a block Bloom filter index. Other predicates such as range, pattern (LIKE), and negation (!=, NOT IN) read all of the column's blocks.
The indexer still builds the dictionary in memory with adaptive indexing enabled and discards it when it finalizes the partition. Therefore, CPU usage and clock time don't reduce when writing the partition.