Logging and Debugging a Cluster
Hydrolix operator automatically deploys several pod to ensure that the service is running properly. Here is the complete list of pods deployed by Hydrolix and their usage:
| Pod Name | Description |
|---|---|
| alter-head | Micro-service used to queue up work for the Alter Service . |
| alter-indexer | A sidecar process deployed with the alter-peer to provide HDX data extraction, data compression and indexing. |
| alter-peer | The Alter Service supports async modification of Hydrolix data. |
| autoingest | Micro-service used to poll for cloud object storage update notifications, and queue up work for batch-peers. |
| batch-head | Micro-service used to scan cloud object storage, and queue up work for batch-peers. |
| batch-indexer | A sidecar process deployed with the batch-peer to provide HDX data compression and indexing. |
| batch-peer | The Batch Service supports one-off import of legacy files from cloud object storage. Also used for automatic ingest of newly-added cloud object storage files (Autoingest). |
| decay | Micro-service used to Age out partitions which exceed a (configurable) time to live. |
| indexer | A sidecar process deployed alongside all of the ingest services (eg, batch-peer, kafka-peer, kinesis-peer, stream-peer, summary-peer) to provide HDX data compression and indexing. |
| init-cluster | Micro-service used only at cluster creation time. |
| init-turbine-api | Micro-service used only at cluster creation time. |
| intake-api | Micro-service used to queue up work for batch-heads. |
| intake-head | Process that receives and decompresses incoming messages, verifies data formats, applies transformation policies, and pushes indexed partitions to cloud object storage. Replaces stream-head, stream-peer, and summary-peer: refer to the migration guide to switch to intake-head. |
| kafka-indexer | A sidecar process deployed with the kafka-peer to provide HDX data compression and indexing. |
| kafka-peer | The Kafka Service supports real-time ingestion from a customer's kafka data stream. |
| keycloak | Open source component used for user authentication and authorization. |
| kinesis-coordinator | A sidecar process that enables dynamic coordination of Kinesis peer-to-shard mapping. |
| kinesis-indexer | A sidecar process deployed with the kinesis-peer to provide HDX data compression and indexing. |
| kinesis-peer | The Kinesis Service supports real-time ingestion from a customer's AWS Kinesis data stream. |
| load-sample-project | Micro-service used only at cluster creation time. |
| merge-head | Micro-service used to queue up work for the Merge Service. |
| merge-indexer | Sidecar process deployed with the merge-peer to provide HDX data extraction, data compression and indexing. |
| merge-peer | The Merge Service combines small, related partitions into larger ones to improve storage and query efficiency. |
| monitor-ingest | monitor-ingest produces one event per second as a synthetic heartbeat test. |
| operator | Micro-service which manages the scale and deployment of various pods. |
| otel | Open source component used for collecting distributed query traces. |
| postgres | Stateful database used to store customer configuration data and provide a shared catalog of active Hydrolix partitions. |
| prometheus | Open source component used for collecting system and application health metrics. |
| promwaltz | Micro-service which helps coordinate Prometheus service discovery. Restarts automatically on configuration changes. |
| query-head | The front-end of the Hydrolix Query Service, query heads parse and compile incoming SQL queries, identify candidate partitions, and distribute work across pools of query-peers. |
| query-peer | The back-end of the Hydrolix Query Service, query peers download a filtered set of column indexes and data blocks from cloud object storage and returns partially-aggregated results to the appropriate query-head. |
| rabbitmq | Open source component used for managing service task queues (e.g. merge, batch, and reaper). |
| reaper | Micro-service used to permanently delete partitions which have been marked as inactive by other services (e.g. decay or merge). |
| redpanda | Open source component which provides reliable buffering of incoming messages for stream peers and summary peers. |
| stream-head | The front-end of the Hydrolix Stream Service, stream heads receive and decompress incoming messages, verify data formats, and push events to Redpanda. Deprecated: refer to our migration guide to switch to the more efficient intake-head pods. |
| stream-indexer | Sidecar process deployed with the stream-peer to provide HDX data compression and indexing. |
| stream-peer | The back-end of the Hydrolix Stream Service, stream peers read messages from Redpanda, apply transformation policies, and push indexed partitions to cloud objects storage. Deprecated: refer to our migration guide to switch to the more efficient intake-head pods. |
| summary-indexer | A sidecar process deployed with the summary-peer to provide HDX data compression and indexing. |
| summary-peer | The Summary Service enables continuous update of materialized views by applying alternative transformation policies to incoming events to generate compact (lossy) partitions. Deprecated: refer to our migration guide to switch to the more efficient intake-head pods. |
| traefik | Open source component which provides authentication and load balancing across various endpoints (eg, config API, stream ingest, and query). |
| traefik-cfg | Micro-service which manages traefik configuration updates. Restarts automatically on configuration changes. |
| turbine-api | The Config Service is responsible for managing project, table and user permissions. |
| ui | Micro-service which supports the (optional) Hydrolix user interface. |
| vacuum | Micro-service used to identify orphaned or invalid partitions which may exist in object storage but are not referenced in the Hydrolix catalog. |
| validator | Micro-service which enables pre-publication testing of new transformation policies. |
| validator-indexer | A sidecar process deployed with the validator to replicate the behavior of a Hydrolix stream peer. |
| vector | Open source component used for pushing log files to cloud object storage. |
| version | Micro-service used to expose the current version of a Hydrolix cluster. |
| zookeeper | Open source component used to track membership of individual query peer pools. |
Updated 20 days ago