Introduction and Concepts

Hydrolix always shards data by time and can optionally shard by a single column's values.

During ingestion, Hydrolix writes data into partitions which only contain data within a time window. At query time, Hydrolix reads only the partitions whose time windows overlap with the time range of the query. To configure how much time a partition can span, see hot_data_max_minutes_per_partition and cold_data_max_minutes_per_partition in Stream Settings.

A cluster can enable additional column-based sharding to improve query performance. Hydrolix hashes each entry for a user-designated "shard key" column and writes rows with different hashes to separate partitions. When a query filters on the shard key column, Hydrolix prunes partitions in the time range that don't match the hash of the filter value.

How sharding works

The ingestion system groups incoming data by time window and, when sharding is enabled, by shard key. It writes each group to its own partition and adds a catalog entry for the partition that records its shard key hash. When the query system requests partition metadata from the catalog, it uses the shard key hash to prune partitions that can't match the query, which reduces the amount of data it reads.

Shard key

Enable sharding in table configuration. If a table shards its data on the column region, a query like the following would take advantage of sharding and perform more quickly:

SELECT count()
FROM example.logs
WHERE timestamp >= '2026-10-06 10:00:00'
  AND timestamp < '2026-10-06 11:00:00'
  AND region = 'eu-west'
  AND status >= 500

Before reading any data, Hydrolix asks the catalog for the partitions that match both conditions:

1. Time: only partitions that overlap 10:00 to 11:00 are considered.
2. Shard key: of those, only the partitions with region = 'eu-west' are read. Partitions with other values for the region column are skipped.

The status >= 500 filter is then applied to rows in the remaining partitions.

The query engine performs a catalog lookup for all partitions whose shard key hash matches a filter value and all partitions without a shard key. Whether unique shard key values hash to the same value depends on the sharding algorithm.

Hydrolix always reads partitions lacking a shard key, so queries include data ingested prior to sharding being enabled. Other partitions can't contain relevant data, so Hydrolix prunes them, reducing the amount of data it reads for a query.

Sharding diagrams

Every ingest service such as intake head writes partitions by grouping incoming data by time first, into a time window, then optionally subdividing by a shard key. It then writes a partition from the resulting groups.

These diagrams illustrate the time windowing and sharding logic. Each diagram shows a single ingest replica. A running cluster may contain multiple replicas, each of which may write partitions with data in the same time range.

No ShardingStrict Shard KeyPerformance Shard Key

In this example, there is no shard key in use and the ingestion system receives data for two different time windows. Either sharding isn't enabled or the table never had a shard key. The illustration demonstrates time windowing, no shards, and at least one partition per time window.

---
title: No Shard Key or Shard Key Not Enabled
config:
  themeVariables:
    fontSize: 18px
---
flowchart TB
  classDef default stroke:#00A99D,stroke-width:2px
  classDef nodegroup fill:none,stroke:#8B9FAD,stroke-width:2px,stroke-dasharray:5 4,color:#4A5F73

  incoming@{ shape: sm-circ, label: "Incoming Data"}

  subgraph buckets["Time Windows"]
    p0["timestamp [0-300)"]
    p1["timestamp [300-600)"]
  end

  subgraph partitions[Partitions]
    partition_p0["timestamp [0-300)"]
    partition_p1["timestamp [300-600)"]
  end

  incoming --bucket by timestamp--> p0 --write partition--> partition_p0
  incoming --bucket by timestamp--> p1 --write partition--> partition_p1

  class buckets,shards,partitions nodegroup

In this example, ingest receives data in a single time window.

Incoming data is subdivided according to the values of the shard key column and each shard is written as one or more partitions.

---
title: Strict Shard Key
config:
  themeVariables:
    fontSize: 18px
---
flowchart TB
  classDef default stroke:#00A99D,stroke-width:2px
  classDef nodegroup fill:none,stroke:#8B9FAD,stroke-width:2px,stroke-dasharray:5 4,color:#4A5F73

  incoming@{ shape: sm-circ, label: "Incoming Data"}

  subgraph buckets["Time Windows"]
    p0["timestamp [0-300)"]
  end

  subgraph shards[Shards]
    shard_p0_region0["hash('eu-central-1')"]
    shard_p0_region1["hash('ap-south-1')"]
    shard_p0_region2["hash('us-east-2')"]
  end

  subgraph partitions[Partitions]
    partition_p0_region0["`contains rows where
                           region = 'eu-central-1'`"]
    partition_p0_region1["`contains rows where
                           region = 'ap-south-1'`"]
    partition_p0_region2["`contains rows where
                           region = 'us-east-2'`"]
  end

  incoming --bucket by timestamp--> p0
  p0 --shard on
       region--> shard_p0_region0 --write partition--> partition_p0_region0
  p0 --shard on
       region--> shard_p0_region1 --write partition--> partition_p0_region1
  p0 --shard on
       region--> shard_p0_region2 --write partition--> partition_p0_region2

  class buckets,partitions nodegroup

The column in this case is a cloud region. A cloud region column has few distinct values, which makes it a good fit for strict mode. Individual partition sizes for each shard are determined by the count of records associated with each region in this example.

In this example, ingestion receives data in a single time window.

Incoming data is subdivided by computing the hash modulus of the shard key value. As in all other sharding scenarios, each shard is written by intake as its own partition.

In performance mode there are at most 7 shards. Hashing with CRC32C and taking the result modulo a prime number (7) spreads distinct values roughly evenly across the buckets. Use this algorithm when sharding on a high-cardinality column.

---
title: Performance Shard Key
config:
  themeVariables:
    fontSize: 18px
---
flowchart TB
  classDef default stroke:#00A99D,stroke-width:2px
  classDef nodegroup fill:none,stroke:#8B9FAD,stroke-width:2px,stroke-dasharray:5 4,color:#4A5F73

  incoming@{ shape: sm-circ, label: "Incoming Data"}

  subgraph buckets["Time Windows"]
    p0["timestamp [0-300)"]
  end

  subgraph shards[Shards]
    shard_p0_mod0["hash(hostname) % 7 = 0"]
    shard_p0_mod1["hash(hostname) % 7 = 1"]
    shard_p0_mod2["hash(hostname) % 7 = 2"]
  end

  subgraph partitions[Partitions]
    partition_p0_mod0["hash(hostname) % 7 = 0"]
    partition_p0_mod1["hash(hostname) % 7 = 1"]
    partition_p0_mod0["`contains rows where
                        hash(hostname) % 7 = 0`"]
    partition_p0_mod1["`contains rows where
                        hash(hostname) % 7 = 1`"]
    partition_p0_mod2["`contains rows where
                        hash(hostname) % 7 = 2`"]
  end

  incoming --bucket by timestamp--> p0
  p0 --shard on hash(hostname) % 7--> shard_p0_mod0 --write partition--> partition_p0_mod0
  p0 --shard on hash(hostname) % 7--> shard_p0_mod1 --write partition--> partition_p0_mod1
  p0 --shard on hash(hostname) % 7--> shard_p0_mod2 --write partition--> partition_p0_mod2

  class buckets,partitions nodegroup

The column here is a hostname, which has many possible values, making it a poor fit for strict mode and a good fit for performance mode.

Risks and limitations

• Fragmentation: Shard with care. Use performance mode for high-cardinality columns to avoid creating excessive partition directories. Small shards degrade system performance.
• Shard key can't be changed: Once set,shard_key can't be changed. You can disable sharding or switch the algorithm.
• SQL Transforms and shard key: Hydrolix computes the shard key hash before applying SQL transforms, so a column derived in a SQL transform can't be used as a shard key.

Next steps

See Configuration and Sharding summary tables.