Skip to content

Storage Settings

Set cluster-wide or table-specific default storage locations.

A Hydrolix storage object is defined by both an existing S3-compatible bucket and a path. Use the storage API endpoints to create, list, update, and delete storage objects.

Cloud storage configuration⚓︎

Configure a storage bucket by specifying the cloud vendor, bucket name, region, and endpoint in the storage object. The org field is the UUID of your Hydrolix organization. Examples are provided for each of the four Hydrolix-supported vendors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "name": "prod_s3",
  "description": "Production data on Amazon S3",
  "org": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "settings": {
    "bucket_name": "my-hydrolix-data",
    "bucket_path": "/",
    "region": "us-east-1",
    "cloud": "aws",
    "endpoint": "https://s3.us-east-1.amazonaws.com"
  }
}

Set cloud to aws. See AWS Services by Region for valid regions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "name": "prod_gcs",
  "description": "Production data on Google Cloud Storage",
  "org": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "settings": {
    "bucket_name": "my-hydrolix-data",
    "bucket_path": "/",
    "region": "us-central1",
    "cloud": "gcp",
    "endpoint": "https://storage.googleapis.com"
  }
}

Set cloud to gcp. See Google Cloud locations for valid regions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "name": "prod_linode",
  "description": "Production data on Linode Cloud Storage",
  "org": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "settings": {
    "bucket_name": "my-hydrolix-data",
    "bucket_path": "/",
    "region": "us-east",
    "cloud": "linode",
    "endpoint": "https://us-east-1.linodeobjects.com"
  }
}

Set cloud to linode. See Linode Region Availability for valid regions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "name": "prod_azure",
  "description": "Production data on Azure Storage",
  "org": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "settings": {
    "bucket_name": "myhydrolixdata",
    "bucket_path": "/",
    "region": "eastus",
    "cloud": "azure",
    "endpoint": "https://myhydrolixdata.blob.core.windows.net"
  }
}

Set cloud to azure. See Azure regions for valid regions.

Storage settings attributes⚓︎

Performance mode⚓︎

The io_perf_mode storage setting tunes the parallelism and striping behavior when reading data from a defined storage location. It supports three possible settings:

Keyword Integer Value Behavior
aggressive 0 (default) Smaller range requests, more read operations
moderate 1
relaxed 2 Larger range requests, fewer read operations

The io_perf_mode can be specified as a keyword or an integer. The default is aggressive.

Choose a performance mode setting based on cloud storage limits, such as IOPS or billing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
    "name": "hdx_primary",
    "description": "Primary storage bucket",
    "org": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
    "settings": {
        "bucket_name": "my-hydrolix-data",
        "bucket_path": "/",
        "region": "us-east-1",
        "cloud": "aws",
        "endpoint": "https://s3.us-east-1.amazonaws.com",
        "io_perf_mode": "moderate"
    }
}

Set default storage for a cluster⚓︎

The cluster-wide default storage serves as the backing store for many cluster operations like log storage and application backups, for example, Keycloak.

Use settings.is_default to assign a default storage bucket for a Hydrolix cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
    "name": "hdx_primary",
    "description": "Primary storage bucket",
    "org": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
    "settings": {
        "bucket_name": "my-hydrolix-data",
        "bucket_path": "/",
        "region": "us-east-1",
        "cloud": "aws",
        "endpoint": "https://s3.us-east-1.amazonaws.com",
        "is_default": true
    }
}

A cluster can't operate without a defined default storage. If you don't specify a default storage location and your cluster only contains a single storage definition, Hydrolix automatically assigns it as the default.

Set default storage for a table⚓︎

You can configure a default storage bucket for any table in your Hydrolix cluster.

This snippet assigns a unique ID of the storage location as the default for a table. All partitions for the table are written to the bucket and path defined in the storage object.

1
2
3
4
5
6
7
8
9
{
  "name": "cdn_logs",
  "settings": {
    ...
    "storage_map": {
      "default_storage_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"
    }
  }
}

Use Hydrolix UI to configure default table storage⚓︎

  1. Log into the UI, hosted at https://hostname.hydrolix.live.
  2. Click Data in the left sidebar.
  3. Click the name of the project that contains the table you want to configure.
  4. Click the name of the table that you want to configure.
  5. Under Advanced Options, click the to the right of bucket settings.
  6. In the dropdown, click Edit.
  7. In the right sidebar, under Default Storage ID, select the ID of the storage you would like to use as the default storage.
  8. Click Save changes to persist your default storage setting for the table.