Stateful and Stateless Scaling

Overview

Hydrolix components fall into two categories: stateful and stateless.

Stateful components like postgres or zookeeper persist data on disk using Kubernetes PersistentVolumeClaims (PVCs).
Because their state is tied to storage, scaling them requires care to avoid quorum loss, orphaned PVCs, or inconsistent replicas.

Stateless components like intake-head or query-head do not persist data between restarts.
They can be scaled up or down freely without risk to stored data.

Understanding this distinction is critical when planning scaling changes.

Scaling Summary

Stateful components

These components have a data_storage scale key. Scaling them up and down affects Kubernetes PVC and StatefulSet changes:

Stateful storage considerations

• Scaling down can delete pods, but not PVCs. When replicas drop, pods terminate, but the PVCs persist unless cleared manually.
  This can cause orphaned volumes or leftover data.
• Scaling up creates new pods with new PVCs.
  Without redundancy, scaling up doesn't automatically copy the state, and those new replicas may be empty or inconsistent.
• For clustered databases like postgres, scaling too aggressively can break quorum and lose replicas.

Stateless components

Stateless pods like intake-head, batch-peer, and query-head, don't persist data between restarts. They can be scaled up or down freely
without risk to state.

The remaining components are stateless and all have the following scale keys:

• cpu
• memory
• storage
• replicas

See the scale profiles page for a list of all services in a cluster by Hydrolix version.

Configure scaling

Edit the Hydrolix configuration spec file in Kubernetes to add or override component scale profiles.

kubectl edit hydrolixcluster --namespace="$HDX_KUBERNETES_NAMESPACE"

⚠️

Stateful Persistent Volume changes

Persistent volume storage can only be increased, not decreased.
Kubernetes doesn't support shrinking PVCs because it could corrupt or truncate stored data.