Table

• Belongs to a Project
• Is a columnar structure
• Is stored in cloud object storage
• Requires a DateTime type as the primary column.
• Needs to be referenced when ingesting and querying.

A Transform is required to ingest data into a table and a View is required to query data from the table. A table can have multiple views and multiple transforms.

One table may have mulple ingest transforms, as long as they all have a common datetime column that can be used for indexing. This means:

• To add a new column to the table, it just needs to be specified in a new transform
• Related data with different structure, like logs and metrics from the same system, can be stored in the same table, reducing the need for joins

The sample project used in tutorials and reference includes a variety of tables. Each table has several columns. The table metrics has columns timestamp, hostname, region, etc…

Create a Table

To create a Table two methods can be employed:

• via the User Interface or
• via the API using the create table endpoint.

A table is created within a project, it is therefore important to have created the project first, instructions to do this can be found here Project

{
    "name": "ExampleTable",
    "description": "This is an example table",
    "project": "9fb5550c-3306-11eb-adc1-0242ac120002"
}

At a basic level a table has the following properties:

settings

Hydrolix provides a number of settings that can be configured to optimize the table and its behaviour when storing data within the Hydrolix platform.

The following Elements can be set within settings:

age and reaper

These settings manage data time-to-live ttl. It is calculated relative to primary datetime value (not ingest datetime).

It’s design as a two step process. First we soft delete after age.max_age_days then reaper.max_age_days later it is physically removed. Soft delete means it cannot be queried.

Due to storage level time-interval partitioning, it may be possible at midnight boundaries, some data may take an extra 24 hours to expires.

In the example below, the data is viewable for 14 days and then physically deleted after 16 days (T-(14+2)).

	{
    "name": "my_name",
    "description": "my_desctiption",
    "project": "project_uuid",
	  "settings": {
		  "age": {
		    "max_age_days": 14
		  },
		  "reaper": {
			 "max_age_days": 2
		  }
	  }
  }

autoingest

Auto-ingest is used to continiously ingest files triggered by an Object Stores native event notification stream. See S3 Event Notification for a full description.

merge (Not Available)

stream

If you plan to stream data into Hydrolix via either HTTP or Kafka, you can control the behaviour of the stream through settings.stream attribute. See stream settings for a full desctiption.