Table

  • Belongs to a Project
  • Is a columnar structure
  • Is stored in cloud object storage
  • Requires a DateTime type as the primary column.
  • Needs to be referenced when ingesting and querying.

A Transform is required to ingest data into a table and a View is required to query data from the table. A table can have multiple views and multiple transforms.

One table may have mulple ingest transforms, as long as they all have a common datetime column that can be used for indexing. This means:

  • To add a new column to the table, it just needs to be specified in a new transform
  • Related data with different structure, like logs and metrics from the same system, can be stored in the same table, reducing the need for joins

The sample project used in tutorials and reference includes a variety of tables. Each table has several columns. The table metrics has columns timestamp, hostname, region, etc…

      

Create a Table

To create a Table two methods can be employed:

  • via the User Interface or
  • via the API using the create table endpoint.

A table is created within a project, it is therefore important to have created the project first, instructions to do this can be found here Project

Example Configuration settings

Click arrow to expand

{
    "name": "ExampleTable",
    "description": "This is an example table",
    "project": "9fb5550c-3306-11eb-adc1-0242ac120002"
}

At a basic level a table has the following properties:

Element Description
project the UUID for the project the table will reside within
name the name of the table
descripton a short description of the table
settings any additional settings that need to be set. (Not Required)

settings

Hydrolix provides a number of settings that can be configured to optimize the table and its behaviour when storing data within the Hydrolix platform.

The following Elements can be set within settings:

Element Description Availability
age Settings for how long data is viewable for Available
reaper Settings for how long data is physically retained for Available
stream Settings for streaming ingest Available
merge Settings for the merging Not Available
autoingest Settings for the automatic ingestion of data Not Available

age and reaper

These settings manage data time-to-live ttl. It is calculated relative to primary datetime value (not ingest datetime).

It’s design as a two step process. First we soft delete after age.max_age_days then reaper.max_age_days later it is physically removed. Soft delete means it cannot be queried.

Due to storage level time-interval partitioning, it may be possible at midnight boundaries, some data may take an extra 24 hours to expires.

In the example below, the data is viewable for 14 days and then physically deleted after 16 days (T-(14+2)).

Example Configuration settings

Click arrow to expand

	{
    "name": "my_name",
    "description": "my_desctiption",
    "project": "project_uuid",
	  "settings": {
		  "age": {
		    "max_age_days": 14
		  },
		  "reaper": {
			 "max_age_days": 2
		  }
	  }
  }

Element Description Default
age.max_age_days The total number of days the data is available to be viewed from today. For example a value of 10 would mean that data would only be available T-10 days. 0 (keep forever)
age.max_age_reaper The total number of days that data is available before being deleted. For example a value of 2 would mean that data would be delete T-2 days after the max_age_days value 0 (keep forever)

autoingest

Auto-ingest is used to continiously ingest files triggered by an Object Stores native event notification stream. See S3 Event Notification for a full description.

merge (Not Available)

stream

If you plan to stream data into Hydrolix via either HTTP or Kafka, you can control the behaviour of the stream through settings.stream attribute. See stream settings for a full desctiption.