The Hydrolix Platform is deployed with a number of "Core" and base components that are required to operate the system. The main core components are the Bastion, Database bucket and the Catalog.

253

The Database bucket

The Database bucket contains the platform configuration, table and database configurations, platform tuneables, logs and the data files encoded in the HDX file format themselves. Database files are stored as time defined partitions containing the raw data and the indexes. The bucket name will be in the format of hdxcli-xxxxxx.

For example an AWS deployed service has the following directory structure:

[email protected]:~$ aws s3 ls hdxcli-xxxxyy99
                           PRE cf_templates/
                           PRE config/
                           PRE db/
                           PRE hdxinf/
                           PRE logs/
                           PRE results/
                           PRE secrets/

The Catalog, Audit and Ingest State Databases.

Hydrolix uses a Database instance to manage state information that is needed to provide the "glue" for the stateless components. It contains metadata on the data partitions that are stored within the Database Storage bucket and information on the jobs and tasks that have been executed as part of the ingest load.

The main database is called the Catalog which can be queried through the Query infrastructure using the reserved view for each table #.catalog.

query-peer :) select * from sample.`my_data#.catalog` limit 1

SELECT *
FROM sample.`my_data#.catalog`
LIMIT 1

Query id: 2627bd78-bc1b-4e4d-bff1-1d30789d9d69

β”Œβ”€partition────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────min_timestamp─┬───────max_timestamp─┬─manifest_size─┬─data_size─┬─index_size─┬─rows─┬─mem_size─┬─root_path─────────────────────────────────────────────────────────────────┬─shard_key─┐
β”‚ b78ff71c-639e-4da4-b194-732630e6b5bb/55a880fb-a606-4bec-88a6-277c6bc9ec03/data/v2/current/1230768000-1435622400-e9a81367ca4523ea.hdx β”‚ 2009-01-01 00:00:00 β”‚ 2015-06-30 00:00:00 β”‚           880 β”‚     15815 β”‚         29 β”‚ 5000 β”‚   141845 β”‚ b78ff71c-639e-4da4-b194-732630e6b5bb/55a880fb-a606-4bec-88a6-277c6bc9ec03 β”‚           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ‘

Query Metadata

To know more about table metadata take a look here - Querying table metadata

Other Components

In additional to the above there are some other components that are included in the service

203
ComponentDescriptionScale to 0
KeycloakManages RBAC and Access control for services.No
OperatorKubernetes Operator that helps to manage the Hydrolix scale and infrastructure.No
ValidatorService for testing Hydrolix Transforms used in the ingestion of data.Yes
VersionA simple end-point to display the current running version of the Hydrolix platformNo
VectorDeployed on each Node and used to collect Logs from each of the services.No