Projects and Tables

As with most databases, Hydrolix uses the same ideas or projects and tables to group and store common datasets together.

Projects

Your organization can have any number of projects, each of which is a collection of tables. You can think of a Hydrolix project as equivalent to a database in a typical RDBMS.

  • Organizes related data together, like a "database" in an RDBMS
  • Contains one or more tables
  • Has configuration attributes like users, roles, read and write access, etc..
  • Needs to be referenced when ingesting and querying.

For example, in an organization you could have three different Projects, "Systems Monitoring", "Stock Trading", and "IOT" for 3 different sets of unrelated data. They can all co-exist in the same deployment.

Projects can be managed via the web user interface or the Project API.

An example of creating a project can be found in the getting started section.

Create a Project

To create a project, a call is made to the create project API or it is created within the user interface.

When creating a project via the API, the following is required:

  • an acccess_token from login
  • UUID of the organization where the project should be created
  • a name for the project

The organization UUID is specified in the path as well as in the request document.
The response will carry a UUID for the project. This will be needed for project operations, like creating tables.

{
    "uuid": "the-project-uuid",
    "name": "firstproject",
    "description": "stuff",
    "created": "2020-05-04T23:08:11.488083Z",
    "modified": "2020-05-04T23:08:11.488104Z",
    "org": "the-org-uuid-from-request"
}

Tables

A Hydrolix project contains a number of tables, each one a two-dimensional data structure stored as compressed .hdx files within your own cloud resources.

Each Hydrolix table has a lot of important metadata and configuration associated with it which defines the way it addresses incoming data, handles queries, cleans up old data, and more. Understanding table-related concepts like ingestion transforms and query views is a key part of making Hydrolix work exactly how you need it to.

Tables

  • Belong to a Project
  • Is a columnar structure
  • Is stored in cloud object storage
  • Requires a DateTime type as the primary column.
  • Needs to be referenced when ingesting and querying.

A Transform is required to ingest data into a table and a View is required to query data from the table. A table can have multiple views and multiple transforms.

👍

Tables are flexible by design!

One table may have multiple ingest transforms, as long as they all have a common datetime column that can be used for indexing. This means:

  • To add a new column to the table, it just needs to be specified in a new transform
  • Related data with different structure, like logs and metrics from the same system, can be stored in the same table, reducing the need for joins

The sample project used in tutorials include a variety of tables. Each table has several columns. The table metrics has columns timestamp, hostname, region, etc...

Create a Table

To create a Table two methods can be employed:

  • via the User Interface or
  • via the API using the create table endpoint.

Note: A table is created within a project, it is therefore important to have created the project first, instructions to do this can be found above

Required:

  • an acccess_token from login
  • UUID of the organization where the table should be created
  • UUID of the project where the table should be created
  • a name for the table

The project UUID is specified in the path as well as in the request document. The organization UUID is required in the path.

The response will carry a UUID for the table. This will be needed for table operations, like creating transforms and views.

{
    "uuid": "the-table-uuid",
    "name": "myTable",
    "description": "my first table",
    "created": "2020-05-04T23:20:04.334094Z",
    "modified": "2020-05-04T23:20:04.334126Z",
    "settings": {
        "index": {}
    },
    "project": "the-project-uuid-from-request"
}

Did this page help you?