Scaling your Cluster

How to scale Hydrolix with Kubernetes

Basic Scaling

Altering the Scale of your cluster is most easily achieved through Kubernetes using the edit command.

Hydrolix has both Stateful and Stateless components and editing of these components need to take this into account.

Stateful components

ServiceDescription
postgresThe Core
redpandaIngest
prometheusReporting and Control

Stateless Components

ServiceDescriptionServiceDescription
alter-peerAlterquery-headQuery
batch-headIngestquery-peerQuery
batch-peerIngestreaperAge
decayAgestream-headIngest
intake-apiIngeststream-peerIngest
kafka-peerIngestturbine-apiQuery
keycloakThe CoretraefikThe Core
merge-headMergeversionThe Core
merge-peerMergezookeeperThe Core
operatorKubernetes Operator
kubectl edit hydrolixcluster --namespace="$CLIENT_ID"

🚧

Stateful Persistent Volume changes

You can only increase persistent volume storage, decreasing the PVC size will not work

Advanced and Pre-defined scaling options

Hydrolix provides the ability through Kubernetes to define your own scale as well as for the user to use some pre-defined scaling options.

To either use these options or define your own configuration, you can use the hkt tool to give you a basic template for services.

For example :

hkt hydrolix-cluster --env EMAIL_PASSWORD=$EMAIL_PASSWORD > hydrolixcluster.yaml

The output will look similar to the following:

apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: hdxcli-xxxyyyy
  namespace: hdxcli-xxxyyyy
spec:
  admin_email: [email protected]
  client_id: hdxcli-xxxyyyy
  cloud: gcp
  domain: hydrolix.net
  env:
    EMAIL_PASSWORD: XXXXXXYYYYYYYYZZZZZZZZ
  host: hydrolix
  owner: admin
  region: us-central1
  scale_profile: prod

The file can then be edited with your favourite text editor, adding and changing the components you require. For example altering the number of batch and alter-peers to 5 each.

.....
spec:
  .....
  owner: admin
  region: us-central1
  scale:
    alter-peer:
        replicas: 5
    batch-peer:
        replicas: 5
    .....

Once ready the configuration is applied as follows:

.....
  cloud: gcp
  domain: hydrolix.net
  env:
    EMAIL_PASSWORD: 
  host: hydrolix
  owner: admin
  region: us-central1
  scale_off : true  <--- To Turn everything off.
kubectl apply -f hydrolixcluster.yaml

πŸ“˜

Services with PVC Storage.

Some of the Services Hydrolix uses need to maintain state in order to provide a good level of High Availability and redundancy. The postgres, redpanda services use PVC storage and so when setting storage manually you should specify storage using the data_storage key.

Note PVC changes are also significant changes, and we recommend talking to us before making them.

❗️

Don't forget to apply your changes

kubectl apply -f hydrolixcluster.yaml

Predefined Scale

Hydrolix provides a short-cut to specify pre-defined scaling options. The scale for each setting is specified in the latter half of this page and can be found here - Predefined Scaling Options.

To specify the options they are placed in the top-level spec of your hydrolixcluster.yaml using the scale-profile key. The options to available are:

An example is shown below.

apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: hdxcli-xxxyyyy
  namespace: hdxcli-xxxyyyy
spec:
  admin_email: [email protected]
  client_id: hdxcli-xxxyyyy
  cloud: gcp
  domain: hydrolix.net
  env:
    EMAIL_PASSWORD: 
  host: hydrolix
  owner: admin
  region: us-central1
  scale_profile: prod  <--- For the Prod Profile

Turn off everything

If you want to turn everything off you can do that by adding the following into the top level spec:

scale_off : true

This will turn all stateless components off that can be turned off. This should be used with care.

Predefined Scaling Option Descriptions

Default/Minimal

A basic scale profile, focused around a minimal trial deployment.

ServiceDescriptionReplicasCPUMemoryStorageData Storage
alter-headAlter011Gi5Gi-
alter-peerAlter011Gi5Gi-
autoingestAuto Batch0500m512Mi1Gi-
batch-headIngest1500m1Gi5Gi-
batch-peerIngest1-1228Gi10Gi-
decayAge1500m1Gi5Gi-
indexerSidecar Indexer component for ingest peers0*18Gi10Gi*-
init-clusterInitialisation Job for cluster1500m1Gi5Gi-
init-turbine-apiInitialisation Job for cluster1500m1Gi5Gi-
intake-apiIngest1500m1Gi5Gi-
kafka-peerIngest028Gi10Gi-
keycloakThe Core112Gi5Gi-
kinesis-peerKinesis service placeholder028Gi10Gi-
load-sample-projectInitialisation Job for cluster1500m1Gi1Gi-
merge-headMerge1500m1Gi5Gi-
merge-peerMerge1-1218Gi10Gi-
operatorKubernetes Operator1500m1Gi5Gi-
postgresThe Core1416Gi5Gi100Gi
prometheusReporting and Control )111Gi5Gi50Gi
promwaltzSideCar discovery Service for Prometheus0*500m512Mi1Gi*-
query-headQuery1648Gi50Gi-
query-peerQuery3-121560Gi50Gi-
rabbitmqIngest114Gi5Gi10Gi
reaperAge121Gi5Gi-
redpandaIngest226Gi5Gi1Ti
stream-headIngest1-12410Gi10Gi-
stream-peerIngest1-12210Gi10Gi-
traefikThe Core211Gi5Gi-
turbine-apiQuery1500m1Gi5Gi-
uiReporting and Control1500m1Gi5Gi-
versionThe Core1500m1Gi1Gi-
zookeeperThe Core111Gi5Gi5Gi

*Replicas and Data Storage do not apply to these components and will show as default setting.

Prod

A suggested profile when looking at a Production account.

ServiceDescriptionReplicasCPUMemoryStorageData Storage
alter-headAlter011Gi5Gi-
alter-peerAlter021Gi5Gi-
autoingestAuto Batch0500m512Mi1Gi-
batch-headIngest1500m1Gi5Gi-
batch-peerIngest1-1228Gi10Gi-
decayAge1500m1Gi5Gi-
indexerSidecar Indexer component for ingest peers0*18Gi10Gi*-
init-clusterInitialisation Job for cluster1500m1Gi5Gi-
init-turbine-apiInitialisation Job for cluster1500m1Gi5Gi-
intake-apiIngest2500m1Gi5Gi-
kafka-peerIngest028Gi10Gi-
keycloakThe Core112Gi5Gi-
kinesis-peerKinesis service placeholder028Gi10Gi-
load-sample-projectInitialisation Job for cluster1500m1Gi1Gi-
merge-head`Merge1500m1Gi5Gi-
merge-peerMerge2-1218Gi10Gi-
operatorKubernetes Operator1500m1Gi5Gi-
postgresThe Core1616Gi5Gi100Gi
prometheusReporting and Control )111Gi1Gi50Gi
promwaltzSideCar discovery Service for Prometheus0*500m512Mi1Gi*-
query-headQuery2648Gi50Gi-
query-peerQuery3-121560Gi50Gi-
rabbitmqIngest314Gi5Gi10Gi
reaperAge121Gi5Gi-
redpandaIngest228Gi5Gi1Ti
stream-headIngest2-1248Gi10Gi-
stream-peerIngest1-1228Gi10Gi-
traefikThe Core211Gi5Gi-
turbine-apiQuery2500m1Gi5Gi-
uiReporting and Control1500m1Gi5Gi-
versionThe Core1500m1Gi1Gi-
zookeeperThe Core311Gi5Gi5

*Replicas and Data Storage do not apply to these components and will show as default setting.

mega

A suggested profile when looking at Production streaming traffic.

ServiceDescriptionReplicasCPUMemoryStorageData Storage
alter-headAlter011Gi5Gi-
alter-peerAlter021Gi5Gi-
autoingestAuto Batch0500m512Mi1Gi-
batch-headIngest1500m1Gi5Gi-
batch-peerIngest1-12210Gi10Gi-
decayAge1500m1Gi5Gi-
indexerSidecar Indexer component for ingest peers0*18Gi10Gi*-
init-clusterInitialisation Job for cluster1500m1Gi5Gi-
init-turbine-apiInitialisation Job for cluster1500m1Gi5Gi-
intake-apiIngest2500m1Gi5Gi-
kafka-peerIngest028Gi10Gi-
keycloakThe Core112Gi5Gi-
kinesis-peerKinesis service placeholder028Gi10Gi-
load-sample-projectInitialisation Job for cluster1500m1Gi1Gi-
merge-headMerge1500m1Gi5Gi-
merge-peerMerge15-4518Gi10Gi-
operatorKubernetes Operator1500m1Gi5Gi-
postgresThe Core1816Gi5Gi100Gi
prometheusReporting and Control )111Gi1Gi50Gi
promwaltzSideCar discovery Service for Prometheus0*500m512Mi1Gi*-
query-headQuery21555Gi50Gi-
query-peerQuery3-201555Gi50Gi-
rabbitmqIngest314Gi5Gi10Gi
reaperAge121Gi5Gi-
redpandaIngest528Gi5Gi1Ti
stream-headIngest4-1248Gi10Gi-
stream-peerIngest15-4528Gi10Gi-
traefikThe Core311Gi5Gi-
turbine-apiQuery2500m1Gi5Gi-
uiReporting and Control1500m1Gi5Gi-
versionThe Core1500m1Gi1Gi-
zookeeperThe Core311Gi5Gi5Gi

*Replicas and Data Storage do not apply to these components and will show as default setting.

Dev

A suggested profile for a Development account. Note has a very low resource specification.

ServiceDescriptionReplicasCPUMemoryStorageData Storage
alter-headAlter0500m1Gi1Gi-
alter-peerAlter0500m1Gi1Gi-
autoingestAuto Batch0500m512Mi1Gi-
batch-headIngest1500m1Gi1Gi-
batch-peerIngest114Gi1Gi-
decayAge1500m512Mi1Gi-
indexerSidecar Indexer component for ingest peers0*500m1Gi1Gi*-
init-clusterInitialisation Job for cluster1500m1Gi1Gi-
init-turbine-apiInitialisation Job for cluster1500m1Gi1Gi-
intake-apiIngest1500m1Gi1Gi-
kafka-peerIngest0500m1Gi1Gi-
keycloakThe Core111Gi1Gi-
kinesis-peerKinesis service placeholder0500m1Gi1Gi-
load-sample-projectInitialisation Job for cluster1500m1Gi1Gi-
merge-headMerge1500m1Gi1Gi-
merge-peerMerge1500m1Gi1Gi-
operatorKubernetes Operator1500m1Gi1Gi-
postgresThe Core111Gi1Gi5Gi
prometheusReporting and Control )111Gi1Gi5Gi
promwaltzSideCar discovery Service for Prometheus0*500m512Mi1Gi*-
query-headQuery111Gi1Gi-
query-peerQuery111Gi1Gi-
rabbitmqIngest1500m1Gi1Gi5Gi
reaperAge1500m512Mi1Gi-
redpandaIngest111Gi1Gi5Gi
stream-headIngest1500m1Gi1Gi-
stream-peerIngest1500m1Gi1Gi-
traefikThe Core1500m1Gi1Gi-
turbine-apiQuery111Gi1Gi-
uiReporting and Control1500m1Gi1Gi-
versionThe Core1500m1Gi1Gi-
zookeeperThe Core1500m1Gi1Gi1Gi

*Replicas and Data Storage do not apply to these components and will show as default setting.

Configure Pools

Hydrolix has the capability to create pools of services for specific workloads. Each Pool has a ring-fenced capacity so that a workload will not be affected by other processes or users querying data or ingesting data.

For example a query-pool can be created for a subset of users that is accessible only by them, with another pool being used by everyone else or a specific pool of kafka servers can be used for the ingesting of server logs, but another Kafka pool used for performance logs.

By having this capability it allows independent access and scaling of workload components without the need to overprovision a single architecture. In addition with query server components, Role Based Performance Control is also possible - so users can have access to different levels of performance based on their role within a company.

The components that support pooling are:

  • query-peer
  • merge-peer
  • kafka-peer
  • batch-peer
  • stream-peer

To create a pool you need to specify the pool name in the hydrolixcluster.yaml configuration file:

apiVersion: v1
items:
- apiVersion: hydrolix.io/v1
  kind: HydrolixCluster
  spec:
    pools:
    - name: demo-pool
      replicas: 1
      service: query-peer
      cpu: "1.0"

In the pool configuration you can specify the scale similarly to the scale configuration


Did this page help you?