Partition Bloom Filters

This feature was introduced in Hydrolix version 6.2

Queries using equality (=) and IN predicates can perform faster on high-cardinality columns using partition Bloom filters.

These filters identify and skip partitions that can't contain a value for a column. Hydrolix builds these filters automatically for eligible columns when it writes a partition, then uses them at query time to reject partitions before reading them.

How it works

A Hydrolix partition contains three files:

• manifest.hdx: Partition-level metadata. Contains the partition Bloom filters. Written in v4 format as mentioned in Partition format version.
• index.hdx: The partition indexes.
• data.hdx: The compressed column row data, organized into blocks.

When Hydrolix writes a partition, it builds one Bloom filter per eligible column, which increases write time and manifest.hdx file size. This filter summarizes the distinct values the column holds in that partition. At query time, for an equality (=) or IN predicate, Hydrolix probes the matching column's filter for each candidate partition. The filter returns one of two answers: the value is definitely absent, or the value might be present. A definite absent result lets Hydrolix skip the partition without reading its column data or indexes, sometimes referred to as partition pruning.

Bloom filters never return false negatives, so a skip is always safe. They can return a small rate of false positives. In that case, Hydrolix reads the partition index data from the index.hdx file. If the value isn't actually present, the lookup returns no matching blocks and the partition data in data.hdx isn't scanned. The false positive results in extra index lookup work (CPU cost and clock time) but never a wrong result.

Because partition Bloom filters live in manifest.hdx, a manifest written in v4 format is slightly larger than the v3 manifest for the same data. How much larger depends on the number of eligible columns in the partition, and the max_bytes_per_column setting caps the filter size for each column.

Supported column types

Hydrolix writes partition Bloom filters for all indexed columns of the following data types, regardless of cardinality:

• Strings
• UInt128, which are UUID and IP data types

What queries benefit

Partition pruning helps most with equality and IN queries on high-cardinality columns, where the queried value appears in only a small fraction of partitions. In those cases Hydrolix rejects most partitions before reading them, which is where the query-time savings come from.

SELECT * FROM my_table WHERE trace_id = '9f2c1a7e-...';

Range, comparison, and pattern (LIKE) predicates don't benefit, because a Bloom filter can answer only exact-value membership.

Partition format version

Column Bloom filters are part of the v4 on-disk manifest format, which is used as of v6.2 and is compatible with v6.1 versions and above. With the feature enabled, Hydrolix writes new partitions in this format. With it disabled, Hydrolix writes new partitions in the previous v3 format without the filters. Existing partitions keep whichever format they were written in.

Rollback considerations

If a cluster has partition Bloom filters enabled, don't roll back earlier than v6.1.

Hydrolix v6.2 writes partition data in v4 format, and v6.1 is the oldest release that reads v4 partitions correctly, though it skips the Bloom filters. Earlier versions can't read v4 partitions.

To stop creating v4 partitions, disable the feature (see Configuration). New writes revert to v3, and partitions already in v4 keep that format until merge rewrites them.

Merged partition format

Disabling the feature or reverting to a previous Hydrolix version doesn't rewrite existing partitions immediately, but merge does over time. The merge service compacts multiple partitions into a single new partition written in the format specified when it runs. When partition_bloom.enabled is false, data from any v4 partition still eligible for merge is rewritten in v3 format. The reverse is also true: with the feature enabled, merging v3 partitions produces v4 output.

Partitions not eligible for merge keep the format they were written in. See Merge pools for the age, size, and width criteria that determine eligibility.

Configuration

Enabled by default

Partition Bloom filters are enabled by default.

The partition_bloom tunable controls partition Bloom filters. Hydrolix enables them by default. Set enabled to false to turn the feature off, after which the indexer writes new partitions in v3 format without filters.

The default configuration for the partition_bloom tunable is:

The tunable also holds settings that control the size and accuracy of each filter.

Hydrolix v6.2

Setting	Default	Description
enabled	true	Whether the indexer writes partition Bloom filters
false_rate_pct	10	Target false-positive rate, as a percentage
max_bytes_per_column	131072	Maximum filter size for each column, in bytes
max_hash_funcs	7	Maximum number of hash functions used to build each filter

Change settings with care

Hydrolix recommends keeping the default values for false_rate_pct, max_bytes_per_column, and max_hash_funcs.

These settings affect false positive rates, disk usage, computational complexity, and query latency.

false_rate_pct and max_bytes_per_column control filter size. Smaller filters skip fewer partitions, which sends queries to partitions that hold no matching data and increases query latency. Larger filters skip more partitions, but they increase manifest size and write-time cost for every partition.

max_hash_funcs sets the maximum number of hash functions per Bloom filter. Raising this lowers the false positive rate, but increases query latency.

Contact Hydrolix Support before changing these values on a production cluster.

Metrics

See the hdx_partition_bloom_* metrics on the Indexer metrics page for metrics emitted at write time.

See the hdx_partition_bloom_* metrics on the Query metrics page for metrics emitted at query time.

Limitations

• Only equality (=) and IN predicates benefit
• Only supported column types (string, uint128) with indexing enabled get filters

Risks

• Filters increase the size of the manifest.hdx file in each partition and add a cost to partition write time
• If the manifest size grows significantly, it can increase query latency. Reduce filter size with the max_bytes_per_column setting.