Hydrolix Components

The Hydrolix Stack can be separated into two groups of components.

  • Hydrolix Core - These components are required to maintain state. When these are turned off, the system is not operational. They include:

    • the object storage where the database is stored
    • the configuration server
    • database instances that are required to operate the service.
  • Hydrolix Compute - These are the workload specific components. They can be scaled either to 0 or to the level that is required to satisfy a specific workload. They cover areas such as:

    • streaming ingest
    • batch ingest
    • query

Components

The following lists the major components and their purpose. On an initial build, the infrastructure will use a minimal size using the following size resources.

Hydrolix Core

Component Description Scale to 0 Default Scale
Bastion Bastion Management compute node that allows access by Hydrolix to support the infrastructure and customer. No 1
Catalog Contains information for data stored in object storage. Yes 2
Config API API server that enables the command and control API. No 1
Deploy A serverless function that enables the creation of a Hydrolix environment. N/A N/A
Stream Checkpoint Keeps ongoing state information of incoming ingest streams. N/A N/A
Bucket (hdxcli-xxx) The primary storage mechanism for system and database configurations and data stored by the system. N/A N/A
User Interface Contains the logic for displaying the user interface Yes 1
Zookeeper Manages the Head and Peer clusters for streaming and query. Yes 1

Hydrolix Compute

Batch Intake

Batch Intake scans a bucket or bucket path and imports it into the Hydrolix format. Batch ingest required Hydrolix to be scaled to at least --minimal for authentication.

Component Description Scale to 0 Default Type (Scale)
Batch Intake API/Batch Lister Used to manage the import of a batch of data. N/A N/A
Batch Peers Workers that ingest and encode data into the HDX format. Yes 1
Queues Two queues that are used to manage the files being imported. N/A N/A

Stream Intake

These components are used for the streaming ingest of your data. Streaming Intake can utilize two mechanisms, pre-described data and self described data. More information on Streaming ingest can be found here - Ingesting

The stream head provides the API interface that data is sent too. The URL path to the Stream Head is as follows:

https://<host>/ingest>
Component Description Scale to 0 Default Type (Scale)
Stream Head Used to manage the import of stream of data and entry point of a data-stream Yes 1
Stream Peers Workers that ingest and encode data into the HDX format. Yes 1
Queues Two queues that are used to manage the files being imported. No 2

Query

These components are used for the retrieval of data from the datastore. The query head provides the API interface that can be used to make SQL queries. The URL path to the Query Head is as follows:

https://<host>/query?query=<sql>

More information on Query can be found here - Querying

Component Description Scale to 0 Default Type (Scale)
Query Head The API end-point for queries. Manages and aggregates answers from the query head for issue to the end-user. Yes 1
Query Peer Two queues that are used to manage the files being imported. Yes 1