Skip to content

Column Value Mapping

Control the storage location of data by column values. This feature can be useful if you have multiple storage locations.

Don't use spread list with column value mapping

Random selection using spread_list and explicit column_value_mapping are incompatible storage mapping features.

When both are used, spread list takes precedence and column value mapping configuration is ignored.

Why use the feature?⚓︎

Use column value mapping to shard a single table into partitions stored in specific storage locations.

This feature instructs the ingestion system to select storage based on literal values in one column. Configure the feature by supplying a list of literal values assigned to a specific storage location.

Example configuration⚓︎

The following example configuration shards data on cciso, a column containing an ISO-3166-1 alpha-2 code.

Example Column Value Mapping
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "storage_map": {
    "column_name": "cciso",
    "column_value_mapping": {
      "812eb81e-bd6d-466d-9d37-25415b5dd81a": ["US", "FR"],
      "a2b8a015-4e4e-468d-8479-0b5605166bea": ["DE", "IT"]
    },
    "default_storage_id": "812eb81e-bd6d-466d-9d37-25415b5dd81a"
  }
}
  • Rows where the cciso column contains values US or FR map to 812eb81e-bd6d-466d-9d37-25415b5dd81a
  • Rows where the cciso column contains values DE or IT map to a2b8a015-4e4e-468d-8479-0b5605166bea
  • All other rows map to the table's default storage 812eb81e-bd6d-466d-9d37-25415b5dd81a
  • If default_storage_id isn't set for a table, the cluster's primary storage is used, see storage settings

Use column value mapping⚓︎

Enable using UI⚓︎

You can also configure storage mappings in the Hydrolix UI:

How to configure storage mappings on a table's bucket settings in the Hydrolix UI

  1. Log into the UI at https://hostname.hydrolix.live.
  2. Click Data in the left nav.
  3. Select the project and table you want to configure.
  4. In Advanced Options, select the for bucket settings and Edit.
  5. Input the mapping column under Column Name.
  6. Click Add mapping.
  7. In the right nav, click Add mapping to configure a storage mapping.
  8. In the Storage ID dropdown, select the ID of the storage where you'd like to store a subset of data.
  9. In Values text entry box, enter the values you would like to map to the storage bucket you just selected. Press space after entering a value to persist it to the list of mapping values.
  10. Configure additional mappings by clicking Add mapping and repeating the previous 3 steps.
  11. Click Save changes to persist your storage mapping settings for the table.

Enable using API⚓︎

Define the default_storage_id for the table. Incoming columns without an explicit mapping are stored in the default table storage.

Collect lists of the values explicitly to be sent to each specific storage object ID.

Available methods for creating and modifying a table's settings to control the column value mapping:

  • Initially configure and activate the feature using table endpoints. The column_name must be defined using this endpoint.
  • Use the bucket_settings endpoints which limits management to the column_value_mapping dictionary and spread_list settings.

Use the table endpoints⚓︎

  1. Save the response from a GET table endpoint to a file.

  2. Modify the file with the storage UUIDs. Add this entry to the settings:

    Add a Storage Map Column Values Mapping
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    {
  "settings": {
    "storage_map": {
      "column_name": "cciso",
      "column_value_mapping": {
        "a2b8a015-4e4e-468d-8479-0b5605166bea": ["DE", "FR", "IT"]  // (1)!
      },
      "default_storage_id": "812eb81e-bd6d-466d-9d37-25415b5dd81a"
    }
  }
}
    1. Each key in the column_value_mapping must be a valid storage ID. See validation.

  3. Send the request to the PUT table settings endpoint.

  4. Confirm the presence of the column_name and column_value_mapping in the response.

    Expected column value mapping settings fragment in API response

    Table Using Column Value Mapping
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
    {
  "settings": {
    "storage_map": {
      "default_storage_id": "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
      "column_name": "cciso",                                        // (2)!
      "column_value_mapping": {
        "a2b8a015-4e4e-468d-8479-0b5605166bea": [                    // (3)!
          "DE",
          "FR",
          "IT"
        ]
      }
    }
  }
}
    1. The ingestion system selects the default storage for all column values not explicitly mapped elsewhere.
    2. Specify the column name to use for sharding to different storage locations.
    3. Explicitly map a list of values to a specific storage ID, here DE, FR, and IT create a single shard and are stored at the same storage location.
    Complete API response object
    Table Using Column Value Mapping
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
    {
  "project": "8a236d8f-aefc-4043-aa0b-1fdc0f21872f",
  "name": "tablename",
  "description": null,
  "uuid": "bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "created": "2026-02-10T16:03:35.611797Z",
  "modified": "2026-07-02T20:24:06.645850Z",
  "publish_task_id": 967118,
  "url": "https://hostname.hydrolix.live/config/v1/orgs/ae5e3698-b13a-4f8f-ab82-ad2fa391a1a8/projects/8a236d8f-aefc-4043-aa0b-1fdc0f21872f/tables/bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "type": "turbine",
  "primary_key": "timestamp",
  "settings": {
    "default_query_options": {},
    "rate_limit": null,
    "stream": {
      "token_auth_enabled": null,
      "token_list": [],
      "intake_head_url": "https://hostname.hydrolix.live/ingest/event?table=project.tablename&transform=main",
      "hot_data_max_age_minutes": 60,
      "hot_data_max_active_partitions": 12,
      "hot_data_max_rows_per_partition": 1048576,
      "hot_data_max_minutes_per_partition": 5,
      "hot_data_max_open_seconds": 20,
      "hot_data_max_idle_seconds": 10,
      "cold_data_max_age_days": 365,
      "cold_data_max_active_partitions": 168,
      "cold_data_max_rows_per_partition": 1048576,
      "cold_data_max_minutes_per_partition": 60,
      "cold_data_max_open_seconds": 60,
      "cold_data_max_idle_seconds": 30,
      "message_queue_max_rows": 500
    },
    "age": {
      "max_age_days": 0
    },
    "reaper": {
      "max_age_days": 1
    },
    "merge": {
      "enabled": true,
      "memory_coefficient": null
    },
    "autoingest": [
      {
        "enabled": false,
        "source": "",
        "source_region": "",
        "pattern": "",
        "max_rows_per_partition": 12288000,
        "max_minutes_per_partition": 60,
        "max_active_partitions": 50,
        "dry_run": false,
        "source_credential": null,
        "source_credential_id": null,
        "bucket_credential": null,
        "bucket_credential_id": null
      }
    ],
    "sort_keys": [],
    "shard_key": null,
    "enable_sharding": null,
    "shard_key_algo": null,
    "legacy_sharding": null,
    "max_future_days": 0,
    "max_request_bytes": 0,
    "storage_map": {
      "default_storage_id": "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
      "column_name": "cciso",                                        // (2)!
      "column_value_mapping": {
        "a2b8a015-4e4e-468d-8479-0b5605166bea": [                    // (3)!
          "DE",
          "FR",
          "IT"
        ]
      }
    }
  }
}
    1. The ingestion system selects the default storage for all column values not explicitly mapped elsewhere.
    2. Specify the column name to use for sharding to different storage locations.
    3. Explicitly map a list of values to a specific storage ID, here DE, FR, and IT create a single shard and are stored at the same storage location.

Change mappings with bucket settings endpoints⚓︎

The bucket settings endpoint allows management of table settings for column_value_mapping and spread_list only. This limits risk of accidental change to other table settings. The bucket settings endpoint can't change the default_storage_id or column_name.

  1. Create a settings fragment containing all desired mappings.

    Create a Storage Map Column Values Mapping Settings Fragment
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
    {
  "settings": {
    "storage_map": {
      "column_value_mapping": {
        "a2b8a015-4e4e-468d-8479-0b5605166bea": ["DE", "FR", "IT"],  // (1)!
        "812eb81e-bd6d-466d-9d37-25415b5dd81a": ["US"]
      }
    }
  }
}
    1. Each key in the column_value_mapping must be a valid storage ID. See validation.

  2. Send the request to the PATCH bucket settings endpoint.

  3. Confirm the presence of the storage_map.column_value_mapping in the response.

    Expected column value mapping settings fragment in API response

    Table Using Column Value Mapping
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
    {
  "settings": {
    "storage_map": {
      "default_storage_id": "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
      "column_name": "cciso",                                        // (2)!
      "column_value_mapping": {
        "a2b8a015-4e4e-468d-8479-0b5605166bea": [                    // (3)!
          "DE",
          "FR",
          "IT"
        ],
        "812eb81e-bd6d-466d-9d37-25415b5dd81a": [                    // (4)!
          "US"
        ]
      }
    }
  }
}
    1. The ingestion system selects the default storage for all column values not explicitly mapped elsewhere.
    2. Specify the column name to use for sharding to different storage locations.
    3. Explicitly map a list of values to a specific storage ID, here DE, FR, and IT create a single shard and are stored at the same storage location.
    4. It's acceptable to map values to the default storage.
    Complete API response object
    Table Using Column Value Mapping
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
    {
  "project": "8a236d8f-aefc-4043-aa0b-1fdc0f21872f",
  "name": "tablename",
  "description": null,
  "uuid": "bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "created": "2026-02-10T16:03:35.611797Z",
  "modified": "2026-07-02T20:58:10.983879Z",
  "publish_task_id": 967173,
  "url": "https://hostname.hydrolix.live/config/v1/orgs/ae5e3698-b13a-4f8f-ab82-ad2fa391a1a8/projects/8a236d8f-aefc-4043-aa0b-1fdc0f21872f/tables/bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "type": "turbine",
  "primary_key": "timestamp"
  "settings": {
    "default_query_options": {},
    "rate_limit": null,
    "stream": {
      "token_auth_enabled": null,
      "token_list": [],
      "hot_data_max_age_minutes": 60,
      "hot_data_max_active_partitions": 12,
      "hot_data_max_rows_per_partition": 1048576,
      "hot_data_max_minutes_per_partition": 5,
      "hot_data_max_open_seconds": 20,
      "hot_data_max_idle_seconds": 10,
      "cold_data_max_age_days": 365,
      "cold_data_max_active_partitions": 168,
      "cold_data_max_rows_per_partition": 1048576,
      "cold_data_max_minutes_per_partition": 60,
      "cold_data_max_open_seconds": 60,
      "cold_data_max_idle_seconds": 30,
      "message_queue_max_rows": 500,
      "intake_head_url": "https://hostname.hydrolix.live/ingest/event?table=project.tablename&transform=main"
    },
    "age": {
      "max_age_days": 0
    },
    "reaper": {
      "max_age_days": 1
    },
    "merge": {
      "enabled": true,
      "memory_coefficient": null
    },
    "autoingest": [
      {
        "enabled": false,
        "source": "",
        "source_region": "",
        "pattern": "",
        "max_rows_per_partition": 12288000,
        "max_minutes_per_partition": 60,
        "max_active_partitions": 50,
        "dry_run": false,
        "source_credential": null,
        "source_credential_id": null,
        "bucket_credential": null,
        "bucket_credential_id": null
      }
    ],
    "sort_keys": [],
    "shard_key": null,
    "enable_sharding": null,
    "shard_key_algo": null,
    "max_future_days": 0,
    "max_request_bytes": 0,
    "storage_map": {
      "default_storage_id": "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
      "column_name": "cciso",                                        // (2)!
      "column_value_mapping": {
        "a2b8a015-4e4e-468d-8479-0b5605166bea": [                    // (3)!
          "DE",
          "FR",
          "IT"
        ],
        "812eb81e-bd6d-466d-9d37-25415b5dd81a": [                    // (4)!
          "US"
        ]
      }
    }
  }
}
    1. The ingestion system selects the default storage for all column values not explicitly mapped elsewhere.
    2. Specify the column name to use for sharding to different storage locations.
    3. Explicitly map a list of values to a specific storage ID, here DE, FR, and IT create a single shard and are stored at the same storage location.
    4. It's acceptable to map values to the default storage.

Validation⚓︎

A bucket must exist and be usable when creating a storage object. This configuration time check prevents usage of invalid or unavailable buckets.

The Config API validates that storage objects used in the column_value_mapping exist when configuration is applied.

Use cases⚓︎

  • Compliance: Keeping data in a particular region may be a compliance requirement. For example, GDPR.
  • Security: You can segment data by customer. Data for a subset of customers may require additional layers of security.

Limitations⚓︎

  • Only one column may be defined as the column_name for the feature.
  • Both column_name and column_value_mapping must be specified to activate the feature.
  • The column value mapping feature won't be active if spread list is enabled for the table.
  • Separating data into multiple storage buckets can impact system performance depending on your query patterns and resources.