Table Settings
Tables use a number of options to define how data is stored:
| Table Settings Overview | |
|---|---|
Reject Future Data⚓︎
The max_future_days option filters out rows with a primary timestamp in the future. By default, Hydrolix uses a value of 0, and rejects any events with a timestamp greater than now().
If your application stores future primary timestamps, you can configure a value above 0 that meets your needs.
If you prefer not to reject any incoming future data, you can set max_future_days to null.
For example, the following settings example accepts primary timestamps up to 14 days in the future:
Ingestion Sort Priority⚓︎
Hydrolix sorts data during ingestion by column cardinality. For most use cases, the default sorting is sufficient. However, some query patterns benefit from custom sorting.
To change the sort priority:
- Determine a column or columns you would like to sort by.
- Add the column name to the
sort_keysarray.
Hydrolix sorts incoming data first by the declared order of the sort_keys array, then by other columns in order of cardinality.
Consider an application where many queries sort first by customer_id, then by locale. If Hydrolix stores data sorted by these columns, iteration over these queries can benefit from spatial locality. The following settings example sorts data first by customer_id, then by locale:
| Custom Sort Keys | |
|---|---|
Sorting isn't retroactive
Changes to the sort_keys field only affect data ingested after applying the change. Hydrolix doesn't change the sort order of existing data.
Sharding⚓︎
See sharding to distribute data across multiple nodes, improving query performance and scalability using the fields:
shard_keyshard_key_algoenable_sharding
Related information⚓︎
- Reduce Columnar Database Costs by Analyzing Cardinality - How to analyze column cardinality to optimize storage and query performance
- Solving the Challenge of High-Cardinality Data with SQL Transformation - Techniques for handling high-cardinality columns without degrading system performance