Skip to content

Overview

The Hydrolix Config API offers control over Projects and Tables, Data Source Configuration, Dictionaries, Functions, Query Options, and many other functions of the Hydrolix cluster.

Use the Config API for controlling the Hydrolix cluster. Consult the OpenAPI reference for details of specific API endpoints.

This page provides an introduction to some common patterns used by the Config API.

The Config API supports versioned endpoints. The v2 version of the API

  • is the recommended version for use in Hydrolix v6.4 and later.
  • supports all operations present in the v1 endpoints, with small calling convention changes described in Config API v2 Endpoints.
  • changes endpoints by removing unnecessary path segments, making the URLs shorter.
  • receives all future development effort.

Use the v2 endpoints in all application development

The v1 endpoints remain available and usable after the release of Hydrolix v6.4, but don't support some newer features.

OpenAPI browsers⚓︎

Choose from these options:

  • Retrieve the OpenAPI spec at your cluster's /config/schema/ for local development
  • Browse with Redoc at your cluster's /config/redoc/
  • Browse with Swagger at your cluster's /config/swagger/
  • View recent OpenAPI definitions on this documentation site under Config API

Use the OpenAPI browser URLs from your cluster to guarantee a match between the running API version in the cluster and the OpenAPI specification.

Authentication and authorization⚓︎

Users authenticate to the Config API using the same authentication mechanisms supported for all APIs in the cluster. See User Authentication.

The Config API supports a fine-grained role based access control (RBAC) system. The RBAC system allows administrators to control authorized activity by managing users, roles, policies, permissions, and scopes.

An Audit Logging system produces a record of user activity.

User accounts can use the Login to Hydrolix endpoint to acquire an authorization token.

Use the Generate service account tokens endpoint to acquire an authorization token for a service account.

Acquire an authorization token⚓︎

If you're using a user account instead of a service account, POST credentials to the login endpoint.

Acquire a Bearer Token
1
2
3
4
curl -s -X POST "https://${HDX_HOSTNAME}/config/v1/login" \
  -H "Content-Type: application/json" \
  -d '{"username":"'${HDX_USER}'","password":"'${HDX_PASSWORD}'"}' \
  | jq -r '.auth_token.access_token'

This token expires after 24 hours. For long-running integrations, service account tokens are strongly preferred.

A Hydrolix administrator generates a service account token and provides it to the integrator.

Generate a Service Account Token (Admin Step)
1
2
3
4
5
curl -s -X POST "https://${HDX_HOSTNAME}/config/v1/service_accounts/${HDX_SA_UUID}/tokens/" \
  -H "Authorization: Bearer ${HDX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"expiry": "2027-03-31T00:00:00Z"}' \
  | jq -r '.token'

The response contains the token. Use it directly as Authorization: Bearer <token> in ingest requests. Default TTL is 365 days if expiry is omitted.

Either token type is a JWT that can be decoded to read the exp field using any standard JWT library.

Present an authorization token⚓︎

With an authorization token in $HDX_TOKEN, send a request to /config/v2/users/current/.

1
2
3
4
5
6
7
8
curl \
  --request GET \
  --url "https://hostname.hydrolix.live/config/v2/users/current/" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json" \
    | jq .

If the token is valid, the Config API returns an HTTP 200 and an informational block describing the user.

{
    "audit": true,
    "customers": [
        {
            "cloud": "aws",
            "hdx_deployment_id": null,
            "kubernetes": true,
            "name": "customername",
            "uuid": "67c14240-08a9-434f-a9b4-3fa91fb01fe6"
        },
        {
            "cloud": "aws",
            "hdx_deployment_id": null,
            "kubernetes": true,
            "name": "sample_project",
            "uuid": "b365e2a4-d8cf-4eef-9f1c-f13dd783c358"
        }
    ],
    "default_customer": "67c14240-08a9-434f-a9b4-3fa91fb01fe6",
    "email": null,
    "emailVerified": false,
    "enabled": true,
    "is_service_account": true,
    "name": "accountname",
    "orgs": [
        {
            "cloud": "aws",
            "kubernetes": true,
            "name": "Hydrolix",
            "type": "singletenant",
            "uuid": "db7cddc1-36fc-49df-b1fb-5a205b9611d8"
        }
    ],
    "roles": [
        {
            "id": 53,
            "name": "newsprojectonly",
            "policies": [
                {
                    "permissions": [
                        "select_sql",
                        "dictGet_sql",
                        "show_columns_sql",
                        "show_tables_sql",
                        "show_databases_sql",
                        "show_dictionaries_sql"
                    ],
                    "scope_deployment_id": "customername__news",
                    "scope_id": "8a236d8f-aefc-4043-aa0b-1fdc0f21872f",
                    "scope_name": "news",
                    "scope_type": "project"
                }
            ]
        }
    ],
    "uuid": "531ec32e-952c-425f-8349-064807aec521"
}

If the token is invalid, the Config API returns an HTTP 401 and the following error.

1
2
3
{
  "detail": "Invalid authentication credentials provided."
}

Credentials⚓︎

Credentials store authentication information for external services such as cloud storage providers (AWS, GCP, Azure), Kafka clusters, and SIEM data sources. The Config API manages credentials separately from the resources that use them, allowing credential reuse across multiple configurations.

Manage credentials⚓︎

The Config API provides endpoints to create, list, and retrieve credentials. The following examples demonstrate common credential management operations.

List available credential types⚓︎

Before creating a credential, use /config/v2/credentials/types/ to list the available credential types for your cloud provider:

1
2
3
4
5
6
7
8
curl \
  --request GET \
  --url "https://hostname.hydrolix.live/config/v2/credentials/types/" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json" \
    | jq .

Create a credential⚓︎

Use /config/v2/credentials/ to create a credential by providing the credential type and required details.

curl \
  --request POST \
  --url "https://hostname.hydrolix.live/config/v2/credentials/" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "aws-prod-storage",
    "org": "ae5e3698-b13a-4f8f-ab82-ad2fa391a1a8",
    "type": "aws_access_keys",
    "description": "Production AWS storage credentials",
    "details": {
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    }
  }' \
    | jq .

List credentials⚓︎

Use /config/v2/credentials/ to fetch a listing of all credentials for all customers visible to the requesting account.

1
2
3
4
5
6
7
8
curl \
  --request GET \
  --url "https://hostname.hydrolix.live/config/v2/credentials/" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json" \
    | jq .

Get a specific credential⚓︎

Use /config/v2/credentials/{id} to retrieve details about a specific credential by ID.

1
2
3
4
5
6
7
8
curl \
  --request GET \
  --url "https://hostname.hydrolix.live/config/v2/credentials/33e3610c-ef2d-40ca-b411-4377157183de/" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json" \
    | jq .

Unique credential names⚓︎

This feature was introduced in Hydrolix version 5.8.

Starting in Hydrolix v5.8, credential names must be unique across the entire cluster. The API returns an error if an attempt is made to create or update a credential with a duplicate name.

During upgrade to v5.8, a migration script automatically ensures existing credentials have unique names, potentially renaming duplicates. Hydrolix recommends checking any automation that references credentials by name after upgrading.

Reference⚓︎

See the Config API OpenAPI reference for complete credential endpoint documentation

Common patterns⚓︎

The Config API implements REST conventions. The published OpenAPI reference describes API behavior.

Pagination⚓︎

The Config API supports fully-featured, consistent pagination across endpoints.

Two patterns of pagination are in use

  • Page number for resources managed by most endpoints
  • Cursor style for catalog and data management, endpoints that can return millions of objects

For catalog and partition management endpoints, which can track millions of objects, a cursor-style pagination is necessary. These endpoints are primarily used inside the cluster for data management.

Page number⚓︎

Most endpoints in the Config API for managing resources use the common industry pattern of page number pagination. In each request, API clients request one page of results and communicate both the page size and page number to the server.

Page number pagination is used for all objects with unbounded size. These resources are always returned in paginated responses.

Requests can include HTTP query parameters page (default 1) and page_size (default 100, maximum 1000).

Page numbers begin at page 1, which is why the default is 1.

An example request: /config/v2/users/?page=2&page_size=300

This example annotated response shows the results returned by Hydrolix, now including important pagination context:

1
2
3
4
5
6
7
8
{
  "results": [ ... ],    # Array of items
  "count": int,          # total records
  "current": int,        # this page num
  "num_pages": int,      # total pages
  "next": int,           # next page number or 0 if no next
  "previous": int,       # prev page number or 0 if no prev
}

Cursor style⚓︎

All catalog and partition management endpoints use cursor pagination. These endpoints are crucial for managing the catalog and interacting with the underlying Hydrolix data storage.

The first request to any endpoint supporting cursor pagination should omit the cursor query parameter. Subsequent requests can set the cursor to the value in the next or previous to move through the result set. The page_size controls the maximum number of results in the response.

All requests can include the page_size (default 1000, max 10000) HTTP query parameter. After the first request, subsequent requests must include HTTP query parameter cursor. The cursor value indicates position and direction in the result set. Get catalog example:

curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v2/catalog/" \
  --data "cursor=cD0yMDI2LTA4LTI3KzEwJTNBMTAlM0E0NS4wOTkxNjUlMkIwMCUzQTAw" \
  --data "page_size=100" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json" \
    | jq .
{
  "next": "https://hostname.hydrolix.live/config/v2/catalog/?cursor=cD0yMDI2LTA5LTEzKzAxJTNBMDAlM0E0MC42MDYwOTQlMkIwMCUzQTAw&page_size=100",
  "previous": "https://hostname.hydrolix.live/config/v2/catalog/?cursor=cj0xJnA9MjAyNi0wOS0xNSswMiUzQTEwJTNBNDYuNzY2NDUyJTJCMDAlM0EwMA%3D%3D&page_size=100",
  "results": [
    {
      "data_path": "data/v2/current/42bc986dc5eec4d3/1789434000-1789437599-1789438245781377-7ee5870f26f04115.hdx",
      "root_path_name": "hydro/logs",
      "project": "9527cfe9-2722-48a6-8437-e9895d318689",
      "table": "9539aa26-92f7-4731-a8ed-36b96fead069",
        [ ... omitted ... ]
      "rows": 607193,
      "mem_size": 1200200505,
      "metadata": {
        "rows": 607193,
        "source": "merge",

        [ ... omitted ... ]

        "columns": 247,
        "uncompressed_mem_size": 3258631831
      },
      "shard_key": "42bc986dc5eec4d3",
      "storage_id": "812eb81e-bd6d-466d-9d37-25415b5dd81a",
      "lock": null
    }
  ]
}

Response values and parameters:

  • The cursor HTTP query parameter values are opaque Base64-encoded strings. If the value of previous is cj0xJnA9NDg3, supply a query parameter like cursor=cj0xJnA9NDg3 to return the previous set of results.
  • The page_size HTTP query parameter requests the specified count of items in the response results.
  • The results value contains an array of items in the response, here, catalog entries.
  • The next value in the response is a link to the next page of data or null if there's no next page.
  • The previous value in the response is a link to the prior page of data or null if there's no previous page.

Caching considerations⚓︎

Responses from the Config API include a header to prevent response caching:

Cache-Control: max-age=0, no-cache, no-store, must-revalidate

This limits risk for the possibly sensitive account or cluster configuration data from being cached by browsers or proxies.

OpenAPI reference⚓︎

The Config API OpenAPI reference offers implementers details of the API.