Projects and Tables

Hydrolix stores data in tables. You can group tables together in logical namespaces called projects. To reference your data, use the full path project.table. For example: monitoring.http_logs.

Projects⚓︎

Projects are equivalent to databases in a traditional RDBMS. You can create any number of projects, as long as each name is unique. Use short, lower case names with underscores to improve readability (for example, http_logs).

For example, an organization could contain three different projects:

Multiple projects in an organization: Systems Monitoring, Stock Trading, and IOT

These projects contain three independent sets of unrelated data in the same deployment.

You can manage projects through the Config API or the Hydrolix user interface.

Customer assignment changes in v6.4

A project belongs to a Customer object, which groups a cluster's projects, storages, and credentials. Customer was introduced in Hydrolix v6.1, where a background job assigned it.

Starting in Hydrolix v6.4, a project created through the API must specify a Customer, and that assignment can't be changed afterward. The account making the request must itself be assigned to that Customer. See Hierarchy Adoption for the full set of model enforcement changes.

Find a customer UUID⚓︎

Creating a project requires the uuid of its customer. List the customers to find it, or read the customers block of the /config/v2/users/current/ response, which returns only the customers the account belongs to.

List customers v2 (recommended)List customers v1

GET /config/v2/customers/

curl \
  --request GET \
  --fail-with-body --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  -- "${HDX_HYDROLIX_URL}/config/v2/customers/" \
  | jq -r '.results[] | [.uuid, .name] | @tsv'

GET /config/v1/customers/. See Get customers.

curl \
  --request GET \
  --fail-with-body --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  -- "${HDX_HYDROLIX_URL}/config/v1/customers/" \
  | jq -r '.results[] | [.uuid, .name] | @tsv'

Create a project with the API⚓︎

You must authenticate to use the API.

Log in with your username/password.
Find the uuid of the customer that owns the project.
Use the create project endpoint, providing a customer, name, and description.

Starting in Hydrolix v6.4, both endpoint versions require a customer. The v2 endpoint also requires an org in the request body, because it isn't a path segment. For a comparison of the two calling conventions, see Config API v2 Endpoints.

A project belongs to both hierarchies while the transition is underway, so creating one through the v2 endpoint names a Customer and an Org. In a future release, Customer objects become Orgs and only one will be required. See Organizations.

Request v2 (recommended)Request v1Response

POST /config/v2/projects/. See Create project.

{
  "name": "monitoring",
  "description": "Global monitoring of web services",
  "customer": "cccccccc-cccc-4ccc-8ccc-cccccccccccc",
  "org": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
}

POST /config/v1/orgs/{org_id}/projects/. See Create project.

{
  "name": "monitoring",
  "description": "Global monitoring of web services",
  "customer": "cccccccc-cccc-4ccc-8ccc-cccccccccccc"
}

{
    "uuid": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
    "name": "monitoring",
    "description": "Global monitoring of web services",
    "customer": "cccccccc-cccc-4ccc-8ccc-cccccccccccc",
    ...
}

The response contains the uuid of the new project. To interact with resources in the project, include the project uuid in subsequent requests: as a path parameter with the v1 endpoints, or as a body parameter with the v2 endpoints.

Project settings⚓︎

The settings object specifies project-level configuration. It describes options such as default configuration for queries and rate limiting.

Two project fields sit outside settings and control the customer relationship:

customer - the uuid of the customer that owns the project. Required when the project is created, and immutable afterward, starting in Hydrolix v6.4.
hdx_deployment_id - read-only, derived from the customer's deployment ID. Starting in Hydrolix v6.4, change it with PUT /customers/{id}/deployment_id/, available on both the v1 and v2 routers. It replaces PUT /projects/{id}/deployment_id/, which is removed.

Property	Type	Purpose	Default	Required
`default_query_options`	object	See Query Options Reference for descriptions of each option.	See Query Options Reference	No
`blob`	string	Don't use this API option.	`null`	No
`rate_limit`	object	Limits bytes per second ingest rate and max payload size. See Rate Limiting.	no limit	No

Special projects⚓︎

Some project names are reserved or forbidden. Reserved projects exist on every cluster and can't be renamed or deleted; users with sufficient permissions can read their tables. Forbidden names can't be used when creating a project and can't be changed.

Project name	Status	Contents
`hydro`	Reserved	Hydrolix operational data tables used by the cluster application.
`hdx`	Reserved	Query system diagnostic tables, including `active_queries`.
`system`	Reserved	Data reflecting the current state of the server.
`information_schema`	Reserved	Internal tables describing the data stored in the cluster.
`_local`	Reserved	Data for database server management.
`_replicated`	Reserved	Data for database server management.
`_temporary`	Reserved	Data for database server management.
`default`	Forbidden	Can't be used as a project name.
`query_stats`	Forbidden	Can't be used as a project name.

Tables⚓︎

A table and the associated transforms represents your data set. Hydrolix will store it as a compressed, sorted, two-dimensional data structure in a number of .hdx files in cloud storage. It's referenced by the project and you can have many tables in the same project.

The Create table endpoint allows you to define a name for your data, along with defaults and controls for:

Controls for stream ingest - hot/cold data parameters
Controls for autoingest - patterns and queues to read for notifications
Enable/disable background merge functionality to optimize your data storage
Data retention policies and removal of old data

Before you can ingest data you will need to define a transform (a write schema) for a table, describing the data types to use.

Tables are flexible by design

One table may have multiple ingest transforms. This increases the number of columns of a table.

Multiple transforms on a single table must share the same datetime column
The total number of columns is a union of all transforms
Ideal for very closely associated data sets arriving from different ingest methods

Manage tables using the Tables endpoints or the Web UI.

Create a table using the API⚓︎

You must authenticate to use the API.

Authenticate to the API.
Assign any storage or credential the table uses to the project's customer. Starting in Hydrolix v6.4, a table can't reference a storage or credential that doesn't belong to its customer.
Create the table.

An example create table API request/response exchange. The v2 endpoint takes the project as a body parameter instead of a path segment.

Request v2 (recommended)Request v1Response

POST /config/v2/tables/. See Create table.

{
  "name": "http_logs",
  "description": "web logs",
  "project": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"
}

POST /config/v1/orgs/{org_id}/projects/{project_id}/tables/. See Create table.

{
  "name": "http_logs",
  "description": "web logs"
}

{
    "project": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
    "name": "http_logs",
    "description": "web logs",
    "uuid": "dddddddd-dddd-4ddd-8ddd-dddddddddddd",
    "created": "2022-05-31T03:48:55.172580Z",
    "modified": "2022-05-31T03:48:55.172599Z",
    "settings": {
        "stream": {
            "hot_data_max_age_minutes": 3,
            "hot_data_max_active_partitions": 3,
            "hot_data_max_rows_per_partition": 12288000,
            "hot_data_max_minutes_per_partition": 1,
            "hot_data_max_open_seconds": 60,
            "hot_data_max_idle_seconds": 30,
            "cold_data_max_age_days": 3650,
            "cold_data_max_active_partitions": 50,
            "cold_data_max_rows_per_partition": 12288000,
            "cold_data_max_minutes_per_partition": 60,
            "cold_data_max_open_seconds": 300,
            "cold_data_max_idle_seconds": 60
        },
        "age": {
            "max_age_days": 0
        },
        "reaper": {
            "max_age_days": 1
        },
        "merge": {
            "enabled": true,
            "partition_duration_minutes": 60,
            "input_aggregation": 20000000000,
            "max_candidates": 20,
            "max_rows": 10000000,
            "max_partitions_per_candidate": 100,
            "min_age_mins": 1,
            "max_age_mins": 10080
        },
        "autoingest": {
            "enabled": false,
            "source": "",
            "pattern": "",
            "max_rows_per_partition": 12288000,
            "max_minutes_per_partition": 60,
            "max_active_partitions": 50,
            "input_aggregation": 1073741824,
            "dry_run": false
        },
        "sort_keys": [],
        "shard_key": null,
        "max_future_days": 0
    },
    "url": "https://hostname.hydrolix.live/config/v1/orgs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/projects/bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb/tables/dddddddd-dddd-4ddd-8ddd-dddddddddddd"
}

The response contains the uuid of the created table. The url field always uses the nested v1 form, whichever endpoint version created the table. To reference resources contained in a table, such as transforms, include the table uuid: as a path parameter with the v1 endpoints, or as a body parameter with the v2 endpoints.

Table settings⚓︎

The settings object specifies table-level configuration.

Property	Type	Purpose	Default	Required
`default_query_options`	object	Set query options for this table. See Query Options Precedence and Query Options Reference.	See Query Options Reference	No
`rate_limit`	object	Limits bytes per second ingest rate and max payload size. See Rate Limiting.	`null` (no limit)	No
`summary`	object	Set this option if you want to create a summary table.	`null`	No
`stream`	object	Set this option to configure stream ingest options for the table.		No
`age`	object	Use this setting to configure a data retention policy after which data will be deactivated.	See Data Lifecycle Management	No
`reaper`	object	Use this setting to configure a data retention policy after which data will be deleted.	See Data Lifecycle Management	No
`merge`	object	Enable/disable merge and configure the merge pools. See the Merge Pools documentation.	`"enabled": true`, all other nested options default to `null`	No
`autoingest`	array[object]	Enable and configure a continuous, batch ingest task for this table. Starting in Hydrolix v6.4, assign any credential named here to the table's customer before use. See also Batch Ingest.	`"enabled": false`	No
`sort_keys`	array[string]	Change the sort order of data as it's ingested and stored. See also Table Settings Reference.	`null`, Hydrolix sorts columns according to cardinality	No
`shard_key`	string	The column to shard by. Can't be changed once set. See also Table Settings Reference.	`null`, results in time-based sharding	No
`shard_key_algo`	string	Sharding algorithm: `strict` or `performance`. Can't be cleared once set. See also Table Settings Reference.	`null`	No
`enable_sharding`	boolean	Enables or disables sharding. Requires `shard_key` and `shard_key_algo` when `true`. See also Table Settings Reference.	`null`	No
`legacy_sharding`	boolean	Read-only. Indicates the table was migrated from pre-v5.10 sharding. See also Table Settings Reference.	`null`	No
`max_future_days`	integer	Retain rows with a timestamp less than this configured value of unit days. See also Table Settings Reference.	`0`	No
`max_request_bytes`	integer	Maximum allowed request size in bytes as measured by the content length of the request.	`0` (no maximum)	No
`storage_map`	object	Assigns a default storage bucket to a table. Starting in Hydrolix v6.4, assign every storage location named in this setting to the table's customer before use. See also Table Settings Reference.	`turbine`	No