AWS Component Types
The following diagram provides a high level view of the Hydrolix Stack as deployed by AWS CloudFormation.
Components
The following lists the major components and their purpose. On an initial build, the infrastructure will use a minimal size using the following size resources.
Hydrolix Core
Component | Description | AWS Service | Scale to 0 | Default Type (Scale) |
---|---|---|---|---|
Bastion | Bastion Management node that allows access by Hydrolix to support the infrastructure and customer. | EC2 | Yes | t2.small (x1) |
Catalog | Contains information for data stored within S3. | Amazon RDS (Postgres) | No | db.r5.large (HA Pair) |
Config API | API server that enables the command and control API. | EC2 | Yes | t2.small (x1) |
Deploy | A Lambda function that enables the creation of a Hydrolix environment. | Lambda | N/A | - |
Stream Checkpoint | Keeps ongoing state information of incoming ingest streams. | DynamoDb | N/A | - |
S3 Bucket (hdxcli-xxx) | The primary storage mechanism for system and database configurations and data stored by the system. | S3 | N/A | - |
User Interface (ui) | Contains the logic for displaying the user interface | EC2 | Yes | t2.micro (x1) |
Zookeeper | Manages the Head and Peer clusters for streaming and query. | EC2 | Yes | t2.micro (x3) |
Hydrolix Compute
Batch Intake
Batch Intake provides a mechanism in which to scan an S3 bucket or directory and import it into the Hydrolix format. To enable batch intake it utilizes the Config API to ingest data, more information can be found here - [Ingesting]/(docs/loading-the-basics)
Component | Description | AWS Service | Scale to 0 | Default Type (Scale) |
---|---|---|---|---|
Batch Intake API/Batch Lister | Used to manage the import of a batch of data. | Lambda | Yes | - |
Batch Peers | Workers that ingest and encode data into the HDX format. | EC2 | Yes | r5n.2xlarge (x1) |
Queues | Two queues that are used to manage the files being imported. | SQS | No | - |
Stream Intake
These components are used for the streaming ingest of your data. Streaming Intake can utilize two mechanisms, pre-described data and self described data. More information on Streaming ingest can be found here - Ingesting
The stream head provides the API interface that data is sent too. The URL path to the Stream Head is as follows:
https://<host>/ingest>
Component | Description | AWS Service | Scale to 0 | Default Type (Scale) |
---|---|---|---|---|
Stream Head | Used to manage the import of stream of data and entry point of a data-stream | EC2 | Yes | r5.xlarge (x1) |
Stream Peers | Workers that ingest and encode data into the HDX format. | EC2 | Yes | r5.xlarge (x1) |
Queues | Two queues that are used to manage the files being imported. | Kinesis | No | 2 Shards |
Kafka Intake
These components are used for the streaming ingest of your data via Kafka. Streaming Intake can utilize two mechanisms, pre-described data and self described data. More information on Streaming ingest can be found here - Ingesting and on Kafka here - Kafka streams
Component | Description | AWS Service | Scale to 0 | Default Type (Scale) |
---|---|---|---|---|
Kafka Peer | Used to manage the import of stream of data and entry point of a data-stream | EC2 | Yes | r5.xlarge (x1) |
Query
These components are used for the retrieval of data from the datastore. The query head provides the API interface that can be used to make SQL queries.
The URL path to the Query Head is as follows:
https://<host>/query?query=<sql>
More information on Query can be found here - Querying
Component | Description | AWS Service | Scale to 0 | Default Type (Scale) |
---|---|---|---|---|
Query Head | The API end-point for queries. Manages and aggregates answers from the query head for issue to the end-user. | EC2 | Yes | c5n.large (x1) |
Query Peer | Two queues that are used to manage the files being imported. | EC2 | Yes | c5n.4xlarge (x1) |
Observability Components
These components are optional and are provided for observability of the Hydrolix platform. The services capabilities are installed at initial build, however can be completely scaled to 0.
Component | Description | AWS Service | Scale to 0 | Default Type (Scale) | More Information |
---|---|---|---|---|---|
Grafana | A pre-integrated version of the Grafana analytics and visualisation tool. | EC2 | Yes | t2.small(x0) | grafana |
Prometheus | A pre-integrated version of the Prometheus system monitoring tool for | EC2 | Yes | t2.medium(x0) | System Metrics |
Optional Components
These components are optional and are provided for ease of use. The services capabilities are installed at initial build, however can be completely scaled to 0.
Component | Description | AWS Service | Scale to 0 | Default Type (Scale) | More Information |
---|---|---|---|---|---|
Superset | A pre-integrated version of the Superset analytics and visualisation tool | EC2 | Yes | t2.small(x0) | superset |
Updated 5 months ago