Scheduler

This feature was introduced in Hydrolix version 5.11.

The Hydrolix scheduler is an optional bin-packing kube-scheduler that places new pods to optimize node utilization. For an overview of how the scheduler and descheduler work together for cost-optimization, see Scheduler and Descheduler.

Enable the scheduler

Add the scheduler section to your HydrolixCluster spec and opt in specific services under scale. Opting in is safe as the scheduler only affects placement of new pods. Because the cluster autoscaler, hdx-scaler, and the descheduler also move pods around, starting with one or two services makes it easier to attribute any placement changes you observe before expanding the opt-in list.

spec:
  scheduler:
    enabled: true
  scale:
    intake-head:
      scheduler_name: hdx-binpack-scheduler
    merge-peer:
      scheduler_name: hdx-binpack-scheduler
    query-peer:
      scheduler_name: hdx-binpack-scheduler

Setting scheduler.enabled: true deploys the scheduler, but no pods use it until you add scheduler_name to specific services under scale. To opt in a service under a specific profile only, use scale.profile.<profile-name>.<service>.scheduler_name instead.

Scheduler defaults

When enabled with no other configuration, the scheduler uses the following defaults:

• Strategy: MostAllocated
• Resource weights: cpu: 1, memory: 1 (equal influence)
• Scheduler name: hdx-binpack-scheduler
• Replicas: 1 on smaller scale_profile values (for example, dev or eval), 2 on prod and larger profiles for high availability through leader election and a hot standby

Configuration reference

Field	Type	Default	Description
scheduler.enabled	bool	false	Deploy the bin-packing scheduler
scheduler.name	string	hdx-binpack-scheduler	Scheduler name referenced by schedulerName in pod specs
scheduler.strategy	string	MostAllocated	MostAllocated (bin-pack) or LeastAllocated (spread)
scheduler.resource_weights	dict	{cpu: 1, memory: 1}	Scoring weights per resource. Higher weight increases that resource's influence on scheduling decisions. See MostAllocated for a worked example
scheduler.verbosity	int	2	Scheduler log verbosity (--v flag). Higher integers log more events; 0 logs only critical events. Raise for more verbose debugging output
scale.<service>.scheduler_name	string	unset	Opt a service into the custom scheduler

Scoring strategies

The bin-packing scheduler offers two scoring strategies. MostAllocated (the default) packs pods onto the busiest nodes to minimize node count. Choose this strategy for cost-driven autoscaling clusters. LeastAllocated spreads pods across nodes for even resource distribution. Choose this strategy when pod isolation matters more than packing density.

MostAllocated

This strategy scores nodes higher when they already have high resource utilization, packing pods onto fewer nodes. This is the recommended strategy for cost savings.

spec:
  scheduler:
    enabled: true
    strategy: MostAllocated
    resource_weights:
      cpu: 1
      memory: 1

For memory-heavy workloads (services that consume much more memory than CPU, such as large caches or in-memory aggregations), increase the memory weight so the MostAllocated score prioritizes filling each node's memory before its CPU:

spec:
  scheduler:
    enabled: true
    strategy: MostAllocated
    resource_weights:
      cpu: 1
      memory: 3

LeastAllocated

This strategy scores nodes higher when they have more free resources, spreading pods across the cluster. Choose this strategy for pod isolation or even resource distribution.

spec:
  scheduler:
    enabled: true
    strategy: LeastAllocated
    resource_weights:
      cpu: 1
      memory: 1

The resource_weights field tunes LeastAllocated scoring the same way it tunes MostAllocated. See the MostAllocated working example for how to weight a single resource more heavily.

Use an external scheduler

If you run a custom Kubernetes scheduler managed outside the Hydrolix operator, you can point Hydrolix services at it rather than deploying the Hydrolix-managed scheduler. See the Kubernetes scheduler configuration documentation for detail on custom schedulers.

To point Hydrolix at an external scheduler:

1. Leave scheduler.enabled: false so the operator doesn't deploy the Hydrolix scheduler.
2. Set scheduler.name to the external scheduler's name.
3. Point each scale.<service>.scheduler_name to the same name.

When deploying with the CLI, also pass --hdx-scheduler-name none to skip RBAC creation for the Hydrolix scheduler.

Verify the scheduler

Confirm the scheduler deployment is ready

kubectl get deployment hdx-binpack-scheduler -n <hydrolix-namespace>

The value for the Ready field should be 1/1 replicas for smaller profiles such as dev and eval or 2/2 replicas on prod and larger profiles.

Confirm opted-in pods use the custom scheduler

kubectl get pod -n <hydrolix-namespace> -l app=intake-head \
  -o jsonpath='{.items[0].spec.schedulerName}'

Opted-in pods return hdx-binpack-scheduler. Pods from services that haven't opted in return empty or default-scheduler.

Check the scheduler that placed a pod

kubectl describe pod -n <hydrolix-namespace> -l app=<service>

Examine the Events: section at the bottom of the output. The From column on the Scheduled event identifies which scheduler placed the pod.

Opted-in serviceNon-opted-in service

Events:
  Type    Reason     Age   From                   Message
  ----    ------     ----  ----                   -------
  Normal  Scheduled  2m    hdx-binpack-scheduler  Successfully assigned hydrolix/intake-head-0 to ip-10-0-1-5

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2m    default-scheduler  Successfully assigned hydrolix/<service>-0 to ip-10-0-1-5

Disable the scheduler

Setting scheduler.enabled: false removes the scheduler deployment, ConfigMap, and ServiceAccount. Pods already running keep running, but any new pod whose scale.<service>.scheduler_name still references hdx-binpack-scheduler goes Pending, because the referenced scheduler no longer exists.

To disable the scheduler safely:

1. Remove scheduler_name from every opted-in service under scale.
2. Wait for the operator to reconcile - new pods now use the default scheduler.
3. Set scheduler.enabled: false.

To recover pods that are already Pending, re-enable the scheduler (scheduler.enabled: true) and they transition to Running.

Troubleshooting

Pods are Pending after disabling the scheduler

You left scale.<service>.scheduler_name set on one or more services after setting scheduler.enabled: false. Either re-enable the scheduler, or remove the scheduler_name field from each opted-in service under scale. See Disable the scheduler for the safe sequence.

The validator warns about a scheduler-name mismatch

Your scheduler.name and one or more scale.<service>.scheduler_name entries disagree. Set both to the same value. When pointing at an external scheduler, set scheduler.name to the external scheduler's name even though scheduler.enabled is false.