Skip to content

Storage Mapping

Control the storage location of data by column values. This feature can be useful if you have multiple storage locations.

Storage mapping⚓︎

Use storage mapping to shard a single table into partitions stored in multiple storage locations. This feature tells Hydrolix to select a storage location based on literal values within a user-specified mapping column. Provide a list of values that maps to each storage bucket for the given column.

The following example shards data using the column named US State. Hydrolix selects a storage location for data according to the following rules:

  • Rows where the US State column contains values New York or Colorado map to 8bc2f07d-cdfc-storage-2
  • Rows where the US State column contains values Oregon or New Hampshire map to 8bc2f07d-cdfc-storage-3
  • All other rows map to the default storage bucket, 8bc2f07d-cdfc-storage-1
Table Mapping Example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
"my_table": {
  "name": "<table_name>",
  ...
  "storage_map": {
    "default_storage_id": "8bc2f07d-cdfc-storage-1",
    "column_name": "US State",
    "column_value_mapping": {
    "8bc2f07d-cdfc-storage-2": [ "New York", "Colorado" ],
    "8bc2f07d-cdfc-storage-3": [ "Oregon", "New Hampshire" ]
    }
  }
}

You can also configure storage mappings in the Hydrolix UI:

How to configure storage mappings on a table's bucket settings in the Hydrolix UI

  1. Log into the UI at https://{myhost}.hydrolix.live.
  2. Click Data in the left nav.
  3. Select the project and table you want to configure.
  4. In Advanced Options, select the for bucket settings and Edit.
  5. Input the mapping column under Column Name.
  6. Click Add mapping.
  7. In the right nav, click Add mapping to configure a storage mapping.
  8. In the Storage ID dropdown, select the ID of the storage where you'd like to store a subset of data.
  9. In Values text entry box, enter the values you would like to map to the storage bucket you just selected. Press space after entering a value to persist it to the list of mapping values.
  10. Configure additional mappings by clicking Add mapping and repeating the previous 3 steps.
  11. Click Save changes to persist your storage mapping settings for the table.

Use cases⚓︎

  • Compliance: Keeping data in a particular region may be a compliance requirement. For example, for GDPR.
  • Security: You can segment data by customer. Data for a subset of customers may require additional layers of security.

Tradeoffs⚓︎

Separating data into multiple storage buckets can impact system performance depending on your query patterns and resources.