Pod Descriptions

Describe the K8 platform deployment

Hydrolix operator automatically deploys several pod to ensure that the service is running properly. Here is the complete list of pods deployed by Hydrolix and their usage:

Pod NameDescription
alter-headMicro-service used to queue up work for the Alter Service
alter-indexerA sidecar process deployed with the alter-peer to provide HDX data extraction, data compression and indexing.
alter-peerThe Alter Service supports async modification of Hydrolix data
autoingestMicro-service used to poll for cloud object storage update notifications, and queue up work for batch-peers
batch-headMicro-service used to scan cloud object storage, and queue up work for batch-peers
batch-indexerA sidecar process deployed with the batch-peer to provide HDX data compression and indexing
batch-peerThe Batch Service supports one-off import of legacy files from cloud object storage. Also used for automatic ingest of newly-added cloud object storage files (Autoingest)
decayMicro-service used to Age out partitions which exceed a (configurable) time to live
indexerA sidecar process deployed alongside all of the ingest services (eg, batch-peer, kafka-peer, kinesis-peer, stream-peer, summary-peer) to provide HDX data compression and indexing
init-clusterMicro-service used only at cluster creation time
init-turbine-apiMicro-service used only at cluster creation time
intake-apiMicro-service used to queue up work for batch-heads
kafka-indexerA sidecar process deployed with the kafka-peer to provide HDX data compression and indexing
kafka-peerThe Kafka Service supports real-time ingestion from a customer's kafka data stream
keycloakOpen source component used for user authentication and authorization
kinesis-indexerA sidecar process deployed with the kinesis-peer to provide HDX data compression and indexing
kinesis-peerThe Kinesis Service supports real-time ingestion from a customer's AWS Kinesis data stream
load-sample-projectMicro-service used only at cluster creation time
merge-headMicro-service used to queue up work for the Merge Service
merge-indexerA sidecar process deployed with the merge-peer to provide HDX data extraction, data compression and indexing
merge-peerThe Merge Service combines small, related partitions into larger ones to improve storage and query efficiency
operatorMicro-service which manages the scale and deployment of various pods
otelOpen source component used for collecting distributed query traces
postgresStateful database used to store customer configuration data and provide a shared catalog of active Hydrolix partitions
prometheusOpen source component used for collecting system and application health metrics
promwaltzMicro-service which helps coordinate prometheus service discovery
query-headThe front-end of the Hydrolix Query Service, query heads parse and compile incoming SQL queries, identify candidate partitions, and distribute work across pools of query-peers
query-peerThe back-end of the Hydrolix Query Service, query peers download a filtered set of column indexes and data blocks from cloud object storage and returns partially-aggregated results to the appropriate query-head.
rabbitmqOpen source component used for managing service task queues (eg, merge, batch, and reaper)
reaperMicro-service used to permanently delete partitions which have been marked as inactive by other services (eg, decay or merge)
redpandaOpen source component which provides reliable buffering of incoming messages for stream peers and summary peers
stream-headThe front-end of the Hydrolix Stream Service, stream heads receive and decompress incoming messages, verify data formats, and push events to redpanda
stream-indexerA sidecar process deployed with the stream-peer to provide HDX data compression and indexing
stream-peerThe back-end of the Hydrolix Stream Service, stream peers read messages from redpanda, apply transformation policies, and push indexed partitions to cloud objects storage.
summary-indexerA sidecar process deployed with the summary-peer to provide HDX data compression and indexing
summary-peerThe Summary Service enables continuous update of materialized views by applying alternative transformation policies to incoming events to generate compact (lossy) partitions
traefikOpen source component which provides authentication and load balancing across various endpoints (eg, config API, stream ingest, and query)
traefik-cfgMicro-service which manages traefik configuration updates
turbine-apiThe Config Service is responsible for managing project, table and user permissions.
uiMicro-service which supports the (optional) Hydrolix user interface
vacuumMicro-service used to identify orphaned or invalid partitions which may exist in object storage but are not referenced in the Hydrolix catalog
validatorMicro-service which enables pre-publication testing of new transformation policies
validator-indexerA sidecar process deployed with the validator to replicate the behavior of a Hydrolix stream peer
vectorOpen source component used for pushing log files to cloud object storage
versionMicro-service used to expose the current version of a Hydrolix cluster
zookeeperOpen source component used to track membership of individual query peer pools