System Components

Batch (Intake)Batch ingest from a remote storage bucket. Includes Auto ingest where automated bucket notifications are used to notify new files being created.
Cloud Storage BucketHydrolix Storage bucket containing the Database files (partitions), service metadata, logs and system data.
DecayAges out data within tables based on table settings.
Intake-api (Intake)API ingest end-point for batch, receives work from the turbine-api or from a notification queue.
Intake HeadStreaming Ingest components for HTTP ingest of data. Includes Streaming Summary components for the aggregation of incoming data.
Kafka (Intake)Kafka intake components for ingestion of data from a customer’s Kafka infrastructure.
KeycloakAccess and Authorization of components and users.
Kinesis (Intake)Kinesis intake components for ingestion of data from a customer’s Kinesis infrastructure.
MergeCompaction service that optimizes storage.
OperatorKubernetes Operator for the command and control of the Kubernetes infrastructure.
PostgresDatabase used for storage of “State” within the Hydrolix platform, includes metadata for tables and partitions, jobs and tasks state and auditing.
PrometheusPrometheus service for reporting metrics.
QueryQuery components for accessing projects and tables storing data.
RabbitMQQueueing service used by the Batch Ingest service.
ReaperTable partition service that removers access to “old” partitions.
RedpandaQueueing mechanism for Streaming and merge.
TraefikApplication layer (7) Load balancer.
Turbine-APIRestful API for control of data and infrastructure
UIUser interface.
ValidatorA debugging interface that allows the development and debugging of transforms.
VectorLogging component that retrieves logs from nodes and places them on a RedPanda Queue and within the Hydrolix storage bucket.
VersionSimple service that returns the current version of Hydrolix.
VacuumCleans up stored files within storage that are no longer needed.
ZookeeperManages available query-peer nodes.