Scale by Component

Overview

Use component-level scaling when predefined scale profiles or custom profiles don’t provide enough control. This method lets you tune resources directly on individual pods or containers.

Scale pod settings

Use these settings to set the resource values of a pod:

When specified, both request and limit are set for the specified resource (memory, CPU, storage, data_storage). Use overcommit or limit_cpu tunables for more flexibility. See Overcommit for more information.

Configure single pods

Modify single pod settings in the hydrolixcluster.yaml file.

For example, this setting modifies the intake-head pods to have two CPUs and 10GiB of RAM allocated:

scale:
  intake-head:
    cpu: 2
    memory: 10Gi

Override component scale settings

You can also override the default scale profile for a single service by specifying new values under scale:.
This example increases both replicas and memory for the batch-peer component:

spec:
  scale_profile: prod
  scale:
    batch-peer:
      memory: 5Gi
      replicas: 5

Apply the configuration:

kubectl apply -f hydrolixcluster.yaml

This override applies only to batch-peer. All other components continue using the selected profile.

Configure multi-container pods

Some Hydrolix services run as pods with multiple containers. Refer to the *-indexer name for the indexer container.

The turbine container is the indexer component that executes transforms and indexes content.

Settings applied to the default intake-head service don't apply to the turbine container.

Use the <component>-indexer name to specify the turbine component in your hydrolixcluster.yaml file.
For example, for intake, use intake-indexer.

See Scale Profiles for instructions on listing all pods using a query in the HKW service.

Stateful components

Some Hydrolix services maintain state and require persistent volumes.

For these services, use the data_storage key to configure the size of the PersistentVolumeClaim (PVC).

scale:
  postgres:
    data_storage: 100Gi

🚧

PVC changes can disrupt stateful services such as postgres. Review the impact before applying changes. Contact Hydrolix Support if you need guidance.