System Components

Component	Description
Batch (Intake)	Batch ingest from a remote storage bucket. Includes Auto ingest where automated bucket notifications are used to notify new files being created.
Cloud Storage Bucket	Hydrolix Storage bucket containing the Database files (partitions), service metadata, logs and system data.
Decay	Ages out data within tables based on table settings.
Intake-api (Intake)	API ingest end-point for batch, receives work from the turbine-api or from a notification queue.
Kafka (Intake)	Kafka intake components for ingestion of data from a customer’s Kafka infrastructure.
Keycloak	Access and Authorization of components and users.
Kinesis (Intake)	Kinesis intake components for ingestion of data from a customer’s Kinesis infrastructure.
Merge	Compaction service that optimizes storage.
Operator	Kubernetes Operator for the command and control of the Kubernetes infrastructure.
Postgres	Database used for storage of “State” within the Hydrolix platform, includes metadata for tables and partitions, jobs and tasks state and auditing.
Prometheus	Prometheus service for reporting metrics.
Query	Query components for accessing projects and tables storing data.
RabbitMQ	Queueing service used by the Batch Ingest service.
Reaper	Table partition service that removers access to “old” partitions.
RedPanda	Queueing mechanism for Streaming and merge.
Stream Ingest (Intake)	Streaming Ingest components for HTTP ingest of data. Includes Streaming Summary components for the aggregation of incoming data.
Traefik	Application layer (7) Load balancer.
Turbine-API	Restful API for control of data and infrastructure https://docs.hydrolix.io/reference/accept-invite
UI	User interface.
Validator	A debugging interface that allows the development and debugging of transforms.
Vector	Logging component that retrieves logs from nodes and places them on a RedPanda Queue and within the Hydrolix storage bucket.
Version	Simple service that returns the current version of Hydrolix.
Vacuum	Cleans up stored files within storage that are no longer needed.
Zookeeper	Manages available query-peer nodes.