Scale your Cluster

Basic Scaling

Scaling a components resources or replica count is done within the scale: section of your hydrolixcluster.yaml file. Hydrolix has both Stateful and Stateless components and editing of these components need to take this into account.

Rather than you scaling each component by hand, we provide pre-made scale profiles that work for various throughput levels i.e scale_profile: prod provide components scaled for 1-4 TB/daily workload.

You can always override a components profile by providing your own in the scale section of the

Stateful components

These components have a data_storage scale key.

Stateless Components

These components all have cpu, memory& storage scale keys along with replicas

To edit the file, you can use this command

kubectl edit hydrolixcluster --namespace="$HDX_KUBERNETES_NAMESPACE"

🚧

Stateful Persistent Volume changes

You can only increase persistent volume storage, decreasing the PVC size will not work

Scale Settings

The following settings can be used to set the resource values of a pod

Specifying Pods

Most pods are straight forward to be specified in the HydrolixCluster.Yaml. A component can be specified and the setting you wish to apply can be added.

For example:

scale:
    stream-head:
       cpu: 2
       memory: 10Gi

Some components have multiple containers within a pod. Typically these containerss will be seen as <service>-peer and a turbine container. For example stream peers have a container called stream-peer and a container called turbine. Turbine is the "Indexer" component that executes some transformation and indexes the content.
Any setting you use with the default name (e.g. stream-peer) will only define the stream-peer component. It will not edit the turbine component.

To specify the turbine component you should use the <component>-indexer value in the HydrolixCluster.yaml. For example for stream uses stream-indexer. A list of the different pods etc can be found here

Scale Profiles

Specify a scale_profile: key in your hydrolixcluster.yaml file with a value of prod or mega.

• prod - a fully resilient production deployment (1-4 TB/day)

• mega - a fully resilient large scale production deployment (10-50 TB/day)

Once you kubectl apply -f this change the system will be automatically scaled.

Scale Overriding

You will most likely need to override some of the components scale settings. You may need more of them (horizontally scaling), or you may need more resources (vertical scaling). Either way you have full control.

Lets say you needed to scale the batch-peer component, as your task requires 5 instances and more memory that the profile provided. Edit the hydrolixcluster.yaml and add the override

.....
spec:
  .....
  scale_profile: prod
  scale:
    batch-peer:
      memory: 5G
    	replicas: 5
    .....

Once ready the configuration is applied as follows:

kubectl apply -f hydrolixcluster.yaml

📘

Services with PVC Storage.

Some of the Services Hydrolix uses need to maintain state in order to provide a good level of High Availability and redundancy. The postgres, redpanda services use PVC storage and so when setting storage manually you should specify storage using the data_storage key.

Note PVC changes are also significant changes, and we recommend talking to us before making them.

❗️

Don't forget to apply your changes

kubectl apply -f hydrolixcluster.yaml

Predefined Scale

Hydrolix provides a shortcut to specify predefined scaling options. The scale for each setting is specified in the latter half of this page and can be found here - Predefined Scaling Profiles.

To specify the options they are placed in the top-level spec of your hydrolixcluster.yaml using the scale-profile key. The options available are:

• prod - a fully resilient production deployment (1-4 TB/day)

• mega - a fully resilient large scale production deployment (10-50 TB/day)

An example is shown below.

apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: hdxcli-xxxyyyy
  namespace: hdxcli-xxxyyyy
spec:
  admin_email: [email protected]
  kubernetes_namespace: hdxcli-xxxyyyy
  kubernets_profile: gcp
  env:
    EMAIL_PASSWORD: 
  hydrolix_url: https://host.hydrolix.net
  db_bucket_region: us-central1
  scale_profile: prod  <--- For the Prod Profile

Scale Profiles

📘

More Information

More information on Scale Profiles found here

Turn off everything

If you want to turn everything off you can do that by adding the following into the top level spec:
scale_off : true

.....
  kubernets_profile: gcp
  hydrolix_url: https://host.hydrolix.net
  env:
    EMAIL_PASSWORD: 
  db_bucket_region: us-central1
  scale_off : true  <--- To Turn everything off.

This will turn all stateless components off that can be turned off. This should be used with care.

Configuring Query Peer Pools

Hydrolix has the capability to create pools of services for specific workloads. Each Pool has a ring-fenced capacity so that a workload will not be affected by other processes or users querying data or ingesting data.

For example a query-pool can be created for a subset of users that is accessible only by them, with another pool being used by everyone else or a specific pool of kafka servers can be used for the ingesting of server logs, but another Kafka pool used for performance logs.

By having this capability it allows independent access and scaling of workload components without the need to overprovision a single architecture. In addition with query server components, Role Based Performance Control is also possible - so users can have access to different levels of performance based on their role within a company.

To create a pool you need to specify the pool name in the hydrolixcluster.yaml configuration file:

apiVersion: v1
items:
- apiVersion: hydrolix.io/v1
  kind: HydrolixCluster
  spec:
    pools:
    - name: demo-pool
      replicas: 1
      service: query-peer
      cpu: "1.0"

In the pool configuration you can specify the scale similarly to the scale configuration