Scale Clusters with Hydrolix

Overview

Hydrolix supports both manual and automatic scaling.
You can scale individual components, apply cluster-wide profiles, or enable the HDX Autoscaler to react to Prometheus metrics.
This page introduces each option and links to detailed guides.

Profiles and Overrides

• Scale profiles: Use pre-made profiles designed for different ingest workloads.
• Custom profiles: Create named overrides for specific services or pools.
• Scale profile overrides: Override profile settings for individual components.
• Scheduled overrides: Apply scaling changes on a schedule.

Component-Level Scaling

• Scale by component: Set resources or replica counts in the scale section of hydrolixcluster.yaml.
• Stateful and stateless scaling: Learn how scaling differs for storage-backed and stateless services.

Autoscaling

• HDX Autoscaler with Prometheus: Scale dynamically based on custom metrics.
• Cool-up and cool-down: Set minimum wait times between scale-up or scale-down events to prevent thrash.
• Scheduled overrides: Apply scaling changes on a schedule.
• Scale to zero: Reduce all replicas to zero except the operator.
• Scale to minimal: Keep the API and UI available while reducing most components.
• Tolerance dead-zone: Define a dead-zone around the target metric where no scaling occurs, to avoid constant up/down adjustments.

Resource tuning

• Overcommit: Ignore Kubernetes CPU and memory requests and limits when running in resource-constrained environments.