Skip to content

Table Settings

Tables use a number of options to define how data is stored:

Table Settings Overview
{
   "name": "mytable",
   "settings": {
      "sort_keys": [],
      "shard_key": null,
      "shard_key_algo": null,
      "enable_sharding": null,
      "max_future_days": 0,
      "expected_tb_per_day": 1
   }
}

Reject Future Data⚓︎

The max_future_days option filters out rows with a primary timestamp in the future. By default, Hydrolix uses a value of 0, and rejects any events with a timestamp greater than now().

If your application stores future primary timestamps, you can configure a value above 0 that meets your needs.

If you prefer not to reject any incoming future data, you can set max_future_days to null.

For example, the following settings example accepts primary timestamps up to 14 days in the future:

Accept Future Timestamps
1
2
3
4
5
6
{
   "name": "mytable",
   "settings": {
      "max_future_days": 14
   }
}

Ingestion Sort Priority⚓︎

Hydrolix sorts data during ingestion by column cardinality. For most use cases, the default sorting is sufficient. However, some query patterns benefit from custom sorting.

To change the sort priority:

  1. Determine a column or columns you would like to sort by.
  2. Add the column name to the sort_keys array.

Hydrolix sorts incoming data first by the declared order of the sort_keys array, then by other columns in order of cardinality.

Consider an application where many queries sort first by customer_id, then by locale. If Hydrolix stores data sorted by these columns, iteration over these queries can benefit from spatial locality. The following settings example sorts data first by customer_id, then by locale:

Custom Sort Keys
1
2
3
4
5
6
{
   "name": "mytable",
   "settings": {
      "sort_keys": ["customer_id", "locale"]
   }
}

Sorting isn't retroactive

Changes to the sort_keys field only affect data ingested after applying the change. Hydrolix doesn't change the sort order of existing data.

Sharding⚓︎

See sharding to distribute data across multiple nodes, improving query performance and scalability using the fields:

  • shard_key
  • shard_key_algo
  • enable_sharding