System Components
| Component | Description |
|---|---|
| Batch (Intake) | Batch ingest from a remote storage bucket. Includes Auto ingest where automated bucket notifications are used to notify new files being created. |
| Cloud Storage Bucket | Hydrolix Storage bucket containing the Database files (partitions), service metadata, logs and system data. |
| Decay | Ages out data within tables based on table settings. |
| Intake-api (Intake) | API ingest end-point for batch, receives work from the turbine-api or from a notification queue. |
| Intake Head | Streaming Ingest components for HTTP ingest of data. Includes Streaming Summary components for the aggregation of incoming data. |
| Kafka (Intake) | Kafka intake components for ingestion of data from a customer’s Kafka infrastructure. |
| Keycloak | Access and Authorization of components and users. |
| Kinesis (Intake) | Kinesis intake components for ingestion of data from a customer’s Kinesis infrastructure. |
| Merge | Compaction service that optimizes storage. |
| Operator | Kubernetes Operator for the command and control of the Kubernetes infrastructure. |
| Postgres | Database used for storage of “State” within the Hydrolix platform, includes metadata for tables and partitions, jobs and tasks state and auditing. |
| Prometheus | Prometheus service for reporting metrics. |
| Query | Query components for accessing projects and tables storing data. |
| RabbitMQ | Queueing service used by the Batch Ingest service. |
| Reaper | Table partition service that removers access to “old” partitions. |
| Redpanda | Queueing mechanism for Streaming and merge. |
| Traefik | Application layer (7) Load balancer. |
| Turbine-API | Restful API for control of data and infrastructure |
| UI | User interface. |
| Validator | A debugging interface that allows the development and debugging of transforms. |
| Vector | Logging component that retrieves logs from nodes and places them on a RedPanda Queue and within the Hydrolix storage bucket. |
| Version | Simple service that returns the current version of Hydrolix. |
| Vacuum | Cleans up stored files within storage that are no longer needed. |
| Zookeeper | Manages available query-peer nodes. |
Updated 6 months ago