Configuration

Configure sharding using the Config API on

• Create table
• Update table
• Patch table

Configure the following in the settings object of the request body:

Setting	Type	Default	Description
enable_sharding	boolean	null	Enables or disables column-based sharding. See enable sharding. When true, both shard_key and shard_key_algo must be set.
shard_key	string	null	The column to shard by. Must be a string column. Can't be changed or cleared once set.
shard_key_algo	string	null	The sharding algorithm: strict or performance. Can't be cleared once set, but can switch between the two. See Algorithms

Shard key can't be changed

Once set, the value of shard_key can't be changed. However, you can disable sharding or switch the algorithm.

Algorithms

Hydrolix supports two sharding algorithms:

• strict creates a single shard and corresponding partition directory for each distinct value in the column. The shard key is the guaranteed unique hash (CRC32C) column value. This offers the best query performance for low-cardinality columns as each partition stores only that column. Beware that this option may create too many directories for high-cardinality columns.
• performance creates a fixed set of 7 shards and partition directories. The shard key is computed from the column value is hashed using CRC32C mod 7. This reduces the number of partition directories and cloud storage LIST API calls. This choice is better suited for high-cardinality columns like customer_id or session_id. The tradeoff is that queries filtering on a single value may read partitions containing other values that hash to the same bucket.

	strict (default)	performance
Hash	CRC32C of the value	CRC32C of the value % 7
Directories	one per distinct value	7 fixed (7 is prime, for spread)
Ideal for	low cardinality columns	high cardinality columns

You can switch between strict and performance on an existing table. New partitions use the currently configured algorithm, and queries check both hash formats.

{
   "settings": {
      "shard_key_algo": "strict"
   }
}

Sharding splits each time window into more, smaller partitions. The more shards a table’s data is split into, and the less data the table ingests per time window, the smaller each partition gets. Small partitions degrade system performance instead of improving query speed.

Sort keys and performance mode

When sorting incoming data to write a partition, Hydrolix uses the table's sort_keys first, and then other columns, in order of cardinality. A sort key isn't required when using sharding, but putting the shard key column first in sort_keys is recommended, especially with performance.

Sharding groups incoming rows into partitions and sort_keys places rows with the same values together in the partition.

{
   "name": "my_table",
   "settings": {
      "shard_key": "customer_id",
      "shard_key_algo": "performance",
      "enable_sharding": true,
      "sort_keys": ["customer_id", "locale"]
   }
}

Sharding lets queries prune partitions, and sorting an indexed column lets them skip blocks inside the partitions they still read. Inside each partition, the column index records which blocks hold each value, and Hydrolix reads only the blocks that match the filter. Sorting by the shard key groups each value's rows into a few adjacent blocks. Without that sort, the values that share a performance bucket are mixed throughout the partition, so most blocks hold some rows for the filtered value and few can be skipped.

Enable sharding

To enable sharding for a table, set shard_key to a string column name, choose a sharding algorithm, and set enable_sharding to true.

{
   "name": "mytable",
   "settings": {
      "shard_key": "region",
      "shard_key_algo": "performance",
      "enable_sharding": true
   }
}

Sharding can subsequently be disabled and re-enabled.

Set all 3 shard settings

Hydrolix accepts a shard_key and shard_key_algo without enable_sharding, but doesn't shard data by that column until enable_sharding is true.