Core
The Hydrolix Platform is deployed with a number of "Core" and base components that are required to operate the system. The main core components are the Bastion, Database bucket and the Catalog.
The Database bucket
The Database bucket contains the platform configuration, table and database configurations, platform tuneables, logs and the data files encoded in the HDX file format themselves. Database files are stored as time defined partitions containing the raw data and the indexes. The bucket name will be in the format of hdxcli-xxxxxx.
For example an AWS deployed service has the following directory structure:
[email protected]:~$ aws s3 ls hdxcli-xxxxyy99
PRE cf_templates/
PRE config/
PRE db/
PRE hdxinf/
PRE logs/
PRE results/
PRE secrets/
The Catalog, Audit and Ingest State Databases.
Hydrolix uses a Database instance to manage state information that is needed to provide the "glue" for the stateless components. It contains metadata on the data partitions that are stored within the Database Storage bucket and information on the jobs and tasks that have been executed as part of the ingest load.
The main database is called the Catalog which can be queried through the Query infrastructure using the reserved view for each table #.catalog.
query-peer :) select * from sample.`my_data#.catalog` limit 1
SELECT *
FROM sample.`my_data#.catalog`
LIMIT 1
Query id: 2627bd78-bc1b-4e4d-bff1-1d30789d9d69
┌─partition────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────min_timestamp─┬───────max_timestamp─┬─manifest_size─┬─data_size─┬─index_size─┬─rows─┬─mem_size─┬─root_path─────────────────────────────────────────────────────────────────┬─shard_key─┐
│ b78ff71c-639e-4da4-b194-732630e6b5bb/55a880fb-a606-4bec-88a6-277c6bc9ec03/data/v2/current/1230768000-1435622400-e9a81367ca4523ea.hdx │ 2009-01-01 00:00:00 │ 2015-06-30 00:00:00 │ 880 │ 15815 │ 29 │ 5000 │ 141845 │ b78ff71c-639e-4da4-b194-732630e6b5bb/55a880fb-a606-4bec-88a6-277c6bc9ec03 │ │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────┴─────────────────────┴───────────────┴───────────┴────────────┴──────┴──────────┴───────────────────────────────────────────────────────────────────────────┴───────────┘
Query Metadata
To know more about table metadata take a look here - Querying table metadata
Other Components
In additional to the above there are some other components that are included in the service
| Component | Description | Scale to 0 |
|---|---|---|
| Keycloak | Manages RBAC and Access control for services. | No |
| Operator | Kubernetes Operator that helps to manage the Hydrolix scale and infrastructure. | No |
| Validator | Service for testing Hydrolix Transforms used in the ingestion of data. | Yes |
| Version | A simple end-point to display the current running version of the Hydrolix platform | No |
| Vector | Deployed on each Node and used to collect Logs from each of the services. | No |
Updated 7 months ago