Skip to content

Spread List

The spread list feature randomly selects an available storage object when writing a partition.

Turn on the feature using a table-level configuration setting described in this page.

The merge system is fully compatible with spread list and doesn't change the storage location of partitions.

Don't use spread list with column value mapping

Random selection using spread_list and explicit column_value_mapping are incompatible storage mapping features.

When both are used, spread list takes precedence and column value mapping configuration is ignored.

Why use the feature?⚓︎

If you encounter object storage resource limits when listing, reading, or writing data, the spread_list feature can reduce the frequency at which these limits are triggered. By distributing the partitions across multiple storage locations, the resource demands for each storage location is lower.

This is especially useful for query time operations which can request many partitions at once.

Learn more about improving Query Efficiency.

Use spread list⚓︎

Each table must be independently configured to use the spread list feature.

Summary tables are an exception. They inherit the underlying table's spread_list table setting if they aren't configured with their own.

Enable using UI⚓︎

To use the spread list feature with an existing table, follow these steps in the Hydrolix UI.

  1. Log into the UI at https://hostname.hydrolix.live.
  2. Click Data in the left nav.
  3. Select the project and table you want to configure.
  4. In Advanced Options, select the for bucket settings and Edit.
  5. Select the toggle Use Spread List.
  6. Use the multi-select box to add all storage objects over which to spread partitions.

Enable using API⚓︎

Collect all of the storage object IDs over which to spread partitions.

Available methods for modifying a table's settings to enable spread list:

Use the bucket settings endpoints⚓︎

The bucket settings endpoint allows management of table settings for spread_list and column_value_mapping only. This limits risk of accidental change to other table settings. The bucket settings endpoint can't change the default_storage_id.

  1. Create a settings fragment with the storage UUIDs.

    Create a Storage Map Spread List Settings Fragment
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
    {
  "settings": {
    "storage_map": {
       "spread_list": [
          "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
          "a2b8a015-4e4e-468d-8479-0b5605166bea"
       ]
    }
  }
}
    1. Each entry in the spread_list list must be a valid storage ID. See validation.

  2. Send the request to the PATCH bucket settings endpoint.

  3. Confirm the presence of the storage_map.spread_list in the response.

    Expected spread list settings fragment in API response

    Table Using Spread List
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    {
  "settings": {
    "storage_map": {
      "column_name": null,
      "spread_list": [
        "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
        "a2b8a015-4e4e-468d-8479-0b5605166bea"
      ]
    }
  }
}
    1. The intake system selects a random storage UUID from this list to write a partition.
    Complete API response object
    Table Using Spread List
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
    {
  "project": "8a236d8f-aefc-4043-aa0b-1fdc0f21872f",
  "name": "tablename",
  "description": null,
  "uuid": "bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "created": "2026-02-10T16:03:35.611797Z",
  "modified": "2026-06-29T22:48:59.474207Z",
  "publish_task_id": 960610,
  "url": "https://hostname.hydrolix.live/config/v1/orgs/ae5e3698-b13a-4f8f-ab82-ad2fa391a1a8/projects/8a236d8f-aefc-4043-aa0b-1fdc0f21872f/tables/bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "type": "turbine",
  "primary_key": "timestamp"
  "settings": {
    "default_query_options": {},
    "rate_limit": null,
    "stream": {
      "token_auth_enabled": null,
      "token_list": [],
      "hot_data_max_age_minutes": 60,
      "hot_data_max_active_partitions": 12,
      "hot_data_max_rows_per_partition": 1048576,
      "hot_data_max_minutes_per_partition": 5,
      "hot_data_max_open_seconds": 20,
      "hot_data_max_idle_seconds": 10,
      "cold_data_max_age_days": 365,
      "cold_data_max_active_partitions": 168,
      "cold_data_max_rows_per_partition": 1048576,
      "cold_data_max_minutes_per_partition": 60,
      "cold_data_max_open_seconds": 60,
      "cold_data_max_idle_seconds": 30,
      "message_queue_max_rows": 500,
      "intake_head_url": "https://hostname.hydrolix.live/ingest/event?table=project.tablename&transform=main"
    },
    "age": {
      "max_age_days": 0
    },
    "reaper": {
      "max_age_days": 1
    },
    "merge": {
      "enabled": true,
      "memory_coefficient": null
    },
    "autoingest": [
      {
        "enabled": false,
        "source": "",
        "source_region": "",
        "pattern": "",
        "max_rows_per_partition": 12288000,
        "max_minutes_per_partition": 60,
        "max_active_partitions": 50,
        "dry_run": false,
        "source_credential": null,
        "source_credential_id": null,
        "bucket_credential": null,
        "bucket_credential_id": null
      }
    ],
    "sort_keys": [],
    "shard_key": null,
    "enable_sharding": null,
    "shard_key_algo": null,
    "max_future_days": 0,
    "max_request_bytes": 0,
    "storage_map": {
      "column_name": null,
      "spread_list": [
        "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
        "a2b8a015-4e4e-468d-8479-0b5605166bea"
      ]
    }
  }
}
    1. The intake system selects a random storage UUID from this list to write a partition.

Use the table endpoints⚓︎

The table endpoints allow management of all features for a table. This example demonstrates the use of the PUT endpoint which replaces all table configuration at once.

  1. Save the response from a GET table endpoint to a file.

  2. Modify the settings to contain the storage UUIDs over which to distribute partitions.

    Add a Storage Map Spread List
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
    {
  "settings": {
    "storage_map": {
       "spread_list": [
          "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
          "a2b8a015-4e4e-468d-8479-0b5605166bea"
       ]
    }
  }
}
    1. Each entry in the spread_list list must be a valid storage ID. See validation.

  3. Send the request to the PUT table settings endpoint.

  4. Confirm the presence of the storage_map.spread_list in the response.

    Expected spread list settings fragment in API response

    Table Using Spread List
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    {
  "settings": {
    "storage_map": {
      "column_name": null,
      "spread_list": [
        "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
        "a2b8a015-4e4e-468d-8479-0b5605166bea"
      ]
    }
  }
}
    1. The intake system selects a random storage UUID from this list to write a partition.
    Complete API response object
    Table Using Spread List
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
    {
  "project": "8a236d8f-aefc-4043-aa0b-1fdc0f21872f",
  "name": "tablename",
  "description": null,
  "uuid": "bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "created": "2026-02-10T16:03:35.611797Z",
  "modified": "2026-06-29T22:48:59.474207Z",
  "publish_task_id": 960610,
  "url": "https://hostname.hydrolix.live/config/v1/orgs/ae5e3698-b13a-4f8f-ab82-ad2fa391a1a8/projects/8a236d8f-aefc-4043-aa0b-1fdc0f21872f/tables/bc6f6d63-a284-4685-a5cf-0544cccf6437",
  "type": "turbine",
  "primary_key": "timestamp"
  "settings": {
    "default_query_options": {},
    "rate_limit": null,
    "stream": {
      "token_auth_enabled": null,
      "token_list": [],
      "hot_data_max_age_minutes": 60,
      "hot_data_max_active_partitions": 12,
      "hot_data_max_rows_per_partition": 1048576,
      "hot_data_max_minutes_per_partition": 5,
      "hot_data_max_open_seconds": 20,
      "hot_data_max_idle_seconds": 10,
      "cold_data_max_age_days": 365,
      "cold_data_max_active_partitions": 168,
      "cold_data_max_rows_per_partition": 1048576,
      "cold_data_max_minutes_per_partition": 60,
      "cold_data_max_open_seconds": 60,
      "cold_data_max_idle_seconds": 30,
      "message_queue_max_rows": 500,
      "intake_head_url": "https://hostname.hydrolix.live/ingest/event?table=project.tablename&transform=main"
    },
    "age": {
      "max_age_days": 0
    },
    "reaper": {
      "max_age_days": 1
    },
    "merge": {
      "enabled": true,
      "memory_coefficient": null
    },
    "autoingest": [
      {
        "enabled": false,
        "source": "",
        "source_region": "",
        "pattern": "",
        "max_rows_per_partition": 12288000,
        "max_minutes_per_partition": 60,
        "max_active_partitions": 50,
        "dry_run": false,
        "source_credential": null,
        "source_credential_id": null,
        "bucket_credential": null,
        "bucket_credential_id": null
      }
    ],
    "sort_keys": [],
    "shard_key": null,
    "enable_sharding": null,
    "shard_key_algo": null,
    "max_future_days": 0,
    "max_request_bytes": 0,
    "storage_map": {
      "column_name": null,
      "spread_list": [
        "812eb81e-bd6d-466d-9d37-25415b5dd81a",  // (1)!
        "a2b8a015-4e4e-468d-8479-0b5605166bea"
      ]
    }
  }
}
    1. The intake system selects a random storage UUID from this list to write a partition.

Verification steps⚓︎

After the spread list feature is enabled, newly arriving data is written to multiple storage locations. The exact delay after configuration is dependent on the time it takes the ingestion services to reload cluster configuration and also table-specific settings. See also stream settings.

Examine the table's catalog metadata with a query like this. Replace the project and tablename.

Report of Storage Locations Used Recently
1
2
3
4
5
6
7
8
SELECT
  storage_id,
  min_timestamp,
  max_timestamp,
  root_path
FROM project.`tablename#.catalog`
WHERE min_timestamp > (NOW() - INTERVAL 1 DAY)
ORDER BY min_timestamp, storage_id

Confirm the presence of all storage UUIDs once the spread list feature is enabled for the table.

Validation⚓︎

A bucket must exist and be usable when creating a storage object. This configuration time check prevents usage of invalid or unavailable buckets.

The Config API validates that storage objects used in the spread_list exist when configuration is applied.

Limitations⚓︎

Other storage mapping settings, like column_value_mapping won't be active if spread_list is enabled.

See Storage Settings for more about other storage settings you can enable.