AWS Component Types

The following diagram provides a high level view of the Hydrolix Stack as deployed by AWS CloudFormation.

image

Components

The following lists the major components and their purpose. On an initial build, the infrastructure will use a minimal size using the following size resources.

Hydrolix Core

ComponentDescriptionAWS ServiceScale to 0Default Type (Scale)
BastionBastion Management node that allows access by Hydrolix to support the infrastructure and customer.EC2Yest2.small (x1)
CatalogContains information for data stored within S3.Amazon RDS (Postgres)Nodb.r5.large (HA Pair)
Config APIAPI server that enables the command and control API.EC2Yest2.small (x1)
DeployA Lambda function that enables the creation of a Hydrolix environment.LambdaN/A-
Stream CheckpointKeeps ongoing state information of incoming ingest streams.DynamoDbN/A-
S3 Bucket (hdxcli-xxx)The primary storage mechanism for system and database configurations and data stored by the system.S3N/A-
User Interface (ui)Contains the logic for displaying the user interfaceEC2Yest2.micro (x1)
ZookeeperManages the Head and Peer clusters for streaming and query.EC2Yest2.micro (x3)

Hydrolix Compute

Batch Intake

Batch Intake provides a mechanism in which to scan an S3 bucket or directory and import it into the Hydrolix format. To enable batch intake it utilizes the Config API to ingest data, more information can be found here - [Ingesting]/(docs/loading-the-basics)

ComponentDescriptionAWS ServiceScale to 0Default Type (Scale)
Batch Intake API/Batch ListerUsed to manage the import of a batch of data.LambdaYes-
Batch PeersWorkers that ingest and encode data into the HDX format.EC2Yesr5n.2xlarge (x1)
QueuesTwo queues that are used to manage the files being imported.SQSNo-

Stream Intake

These components are used for the streaming ingest of your data. Streaming Intake can utilize two mechanisms, pre-described data and self described data. More information on Streaming ingest can be found here - Ingesting

The stream head provides the API interface that data is sent too. The URL path to the Stream Head is as follows:

https://<host>/ingest>
ComponentDescriptionAWS ServiceScale to 0Default Type (Scale)
Stream HeadUsed to manage the import of stream of data and entry point of a data-streamEC2Yesr5.xlarge (x1)
Stream PeersWorkers that ingest and encode data into the HDX format.EC2Yesr5.xlarge (x1)
QueuesTwo queues that are used to manage the files being imported.KinesisNo2 Shards

Kafka Intake

These components are used for the streaming ingest of your data via Kafka. Streaming Intake can utilize two mechanisms, pre-described data and self described data. More information on Streaming ingest can be found here - Ingesting and on Kafka here - Kafka streams

ComponentDescriptionAWS ServiceScale to 0Default Type (Scale)
Kafka PeerUsed to manage the import of stream of data and entry point of a data-streamEC2Yesr5.xlarge (x1)

Query

These components are used for the retrieval of data from the datastore. The query head provides the API interface that can be used to make SQL queries.
The URL path to the Query Head is as follows:

https://<host>/query?query=<sql>

More information on Query can be found here - Querying

ComponentDescriptionAWS ServiceScale to 0Default Type (Scale)
Query HeadThe API end-point for queries. Manages and aggregates answers from the query head for issue to the end-user.EC2Yesc5n.large (x1)
Query PeerTwo queues that are used to manage the files being imported.EC2Yesc5n.4xlarge (x1)

Observability Components

These components are optional and are provided for observability of the Hydrolix platform. The services capabilities are installed at initial build, however can be completely scaled to 0.

ComponentDescriptionAWS ServiceScale to 0Default Type (Scale)More Information
GrafanaA pre-integrated version of the Grafana analytics and visualisation tool.EC2Yest2.small(x0)grafana
PrometheusA pre-integrated version of the Prometheus system monitoring tool forEC2Yest2.medium(x0)System Metrics

Optional Components

These components are optional and are provided for ease of use. The services capabilities are installed at initial build, however can be completely scaled to 0.

ComponentDescriptionAWS ServiceScale to 0Default Type (Scale)More Information
SupersetA pre-integrated version of the Superset analytics and visualisation toolEC2Yest2.small(x0)superset