K8 Platform

Describe the K8 platform deployment

Hydrolix operator automatically deploys several pod to ensure that the service is running properly.
Here's the list of pod deployed by Hydrolix and their usage:

Pod Name

Description

batch-head

Call the intake-api service to list batch jobs created by users.
Then list the files included from the cloud storage, and create a task per files. The task is created in postgres database.

batch-peer

Calls postgres database to get the different files to index.
Download the file from cloud storage, generate a Hydrolix partition and send that partition to cloud storage.

intake-api

API used to generate batch jobs

keycloak

User and right management for Hydrolix platform

merge-head

Service which list the partitions that can be merged together.
It create a job which is queued in rabbitMQ.

merge-peer

Service which merge partition together by reading tasks from rabbitMQ.

operator

Hydrolix operator which manages the scale and the deployment of the different pods

postgres

Database which stores metadata information for the partition (catalog service).

prometheus

Prometheus server fetching the different Hydrolix metrics

query-head

Service receiving query from end users, responsible of listing the partition required.
Assign partition to the different query peer and get the response from each one.

query-peer

Download partition from cloud storage, execute the SQL Statement and provide response to query head.

rabbitmq

Distributed queueing system used by merge service, and reaper

redpanda

Kafka service to store streaming message retrieve by the stream-head.

stream-head

Service receiving HTTP POST request containing data to be indexed.
Verify the format and send the data into redpanda

stream-peer

Service reading redpanda queue and generating Hydrolix partition from the data.

traefik

traefik load balancing between the different service (query, streaming data, API)

turbine-api

API service to manage project, table and user /rights management.

UI

User Interface service.

version

A small service to provide the version of the Hydrolix infrastructure

zookeeper

Zookeeper to manage and orchestrate which query peer is online and what capacity is available to distribute the partition.


Did this page help you?