Scheduler

This feature was introduced in Hydrolix version 5.11.

The Hydrolix scheduler is an optional bin-packing kube-scheduler that places new pods to optimize node utilization. For an overview of how the scheduler and descheduler work together for cost-optimization, see Scheduler and Descheduler.

Enable the scheduler

Add the scheduler section to your HydrolixCluster spec and opt in specific services under scale. Opting in is safe as the scheduler only affects placement of new pods. Because the cluster autoscaler, hdx-scaler, and the descheduler also move pods around, starting with one or two services makes it easier to attribute any placement changes you observe before expanding the opt-in list.

spec:
  scheduler:
    enabled: true
  scale:
    intake-head:
      scheduler_name: hdx-binpack-scheduler
    merge-peer:
      scheduler_name: hdx-binpack-scheduler
    query-peer:
      scheduler_name: hdx-binpack-scheduler

Setting scheduler.enabled: true deploys the scheduler, but no pods use it until you add scheduler_name to specific services under scale. To opt in a service under a specific profile only, use scale.profile.<profile-name>.<service>.scheduler_name instead.

Scheduler defaults

When enabled with no other configuration, the scheduler uses the following defaults:

• Strategy: MostAllocated
• Resource weights: cpu: 1, memory: 1 (equal influence)
• Scheduler name: hdx-binpack-scheduler
• Replicas: 1 on smaller scale_profile values (for example, dev or eval), 2 on prod and larger profiles for high availability through leader election and a hot standby

Opt in all services at once

This feature was introduced in Hydrolix version 6.1.

Per-service opt-in sets scheduler_name on each service under scale. Set scheduler.apply_to_all: true to make the scheduler the default for every operator-created workload instead.

spec:
  scheduler:
    enabled: true
    apply_to_all: true

When apply_to_all is true, the operator adds the scheduler's name as the default schedulerName on every workload it creates. apply_to_all: true requires enabled: true. If the scheduler is disabled, the validator rejects the spec.

To keep specific workloads on the default scheduler, list them under scheduler.apply_to_all_exclude:

spec:
  scheduler:
    enabled: true
    apply_to_all: true
    apply_to_all_exclude:
      - descheduler
      - <workload-name>

The apply_to_all_exclude list defaults to ["descheduler"]. Leave the field out of the spec and the descheduler stays on the default scheduler with no further configuration.

An exclusion list replaces the default

Setting apply_to_all_exclude replaces the default ["descheduler"] rather than extending it. If you set the field without listing descheduler, the descheduler is pulled into apply_to_all and no longer runs on the default scheduler. To exclude your own workloads and keep the descheduler excluded, include descheduler in the list.

The scheduler is always excluded from apply_to_all, which prevents a circular dependency.

Scheduler assignment precedence

A workload can get its scheduler from more than one place in the spec. When more than one applies, the operator uses the most specific one. The order from highest to lowest precedence is:

1. spec.targeting.<service>.scheduler_name - a scheduler assigned to a specific service through targeting.
2. spec.targeting.*.scheduler_name - a scheduler assigned to all services through targeting.
3. spec.scale.<service>.scheduler_name - a scheduler assigned to a specific service under scale.
4. spec.scheduler.apply_to_all - the global opt-in.
5. The default scheduler.

Targeting entries take precedence over the matching entries under scale.

An explicit scheduler_name always takes precedence over the global apply_to_all opt-in. When apply_to_all is true and a service also names the same scheduler, that per-service entry has no effect, and the validator warns about it.

Configuration reference

Field	Type	Default	Description
scheduler.enabled	bool	false	Deploy the bin-packing scheduler
scheduler.name	string	hdx-binpack-scheduler	Scheduler name referenced by schedulerName in pod specs
scheduler.strategy	string	MostAllocated	MostAllocated (bin-pack) or LeastAllocated (spread)
scheduler.resource_weights	dict	{cpu: 1, memory: 1}	Scoring weights per resource. Higher weight increases that resource's influence on scheduling decisions. See MostAllocated for a worked example
scheduler.verbosity	int	2	Scheduler log verbosity (--v flag). Higher integers log more events; 0 logs only critical events. Raise for more verbose debugging output
scheduler.apply_to_all	bool	false	Make the scheduler the default schedulerName on every operator-created workload. Requires scheduler.enabled: true
scheduler.apply_to_all_exclude	list	[descheduler]	Workloads to exclude from apply_to_all. The scheduler's own workload is always excluded
scale.<service>.scheduler_name	string	unset	Opt a service into the custom scheduler

Scoring strategies

The bin-packing scheduler offers two scoring strategies. MostAllocated (the default) packs pods onto the busiest nodes to minimize node count. Choose this strategy for cost-driven autoscaling clusters. LeastAllocated spreads pods across nodes for even resource distribution. Choose this strategy when pod isolation matters more than packing density.

MostAllocated

This strategy scores nodes higher when they already have high resource utilization, packing pods onto fewer nodes. This is the recommended strategy for cost savings.

spec:
  scheduler:
    enabled: true
    strategy: MostAllocated
    resource_weights:
      cpu: 1
      memory: 1

For memory-heavy workloads (services that consume much more memory than CPU, such as large caches or in-memory aggregations), increase the memory weight so the MostAllocated score prioritizes filling each node's memory before its CPU:

spec:
  scheduler:
    enabled: true
    strategy: MostAllocated
    resource_weights:
      cpu: 1
      memory: 3

LeastAllocated

This strategy scores nodes higher when they have more free resources, spreading pods across the cluster. Choose this strategy for pod isolation or even resource distribution.

spec:
  scheduler:
    enabled: true
    strategy: LeastAllocated
    resource_weights:
      cpu: 1
      memory: 1

The resource_weights field tunes LeastAllocated scoring the same way it tunes MostAllocated. See the MostAllocated working example for how to weight a single resource more heavily.

Use an external scheduler

If you run a custom Kubernetes scheduler managed outside the Hydrolix operator, you can point Hydrolix services at it rather than deploying the Hydrolix-managed scheduler. See the Kubernetes scheduler configuration documentation for detail on custom schedulers.

To point Hydrolix at an external scheduler:

1. Leave scheduler.enabled: false so the operator doesn't deploy the Hydrolix scheduler.
2. Set scheduler.name to the external scheduler's name.
3. Point each scale.<service>.scheduler_name to the same name.

When deploying with the CLI, also pass --hdx-scheduler-name none to skip RBAC creation for the Hydrolix scheduler.

Verify the scheduler

Confirm the scheduler deployment is ready

kubectl get deployment hdx-binpack-scheduler -n <hydrolix-namespace>

The value for the Ready field should be 1/1 replicas for smaller profiles such as dev and eval or 2/2 replicas on prod and larger profiles.

Confirm opted-in pods use the custom scheduler

kubectl get pod -n <hydrolix-namespace> -l app=intake-head \
  -o jsonpath='{.items[0].spec.schedulerName}'

Opted-in pods return hdx-binpack-scheduler. Pods from services that haven't opted in return empty or default-scheduler.

Check the scheduler that placed a pod

kubectl describe pod -n <hydrolix-namespace> -l app=<service>

Examine the Events: section at the bottom of the output. The From column on the Scheduled event identifies which scheduler placed the pod.

Opted-in serviceNon-opted-in service

Events:
  Type    Reason     Age   From                   Message
  ----    ------     ----  ----                   -------
  Normal  Scheduled  2m    hdx-binpack-scheduler  Successfully assigned hydrolix/intake-head-0 to ip-10-0-1-5

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2m    default-scheduler  Successfully assigned hydrolix/<service>-0 to ip-10-0-1-5

Disable the scheduler

Setting scheduler.enabled: false removes the scheduler deployment, ConfigMap, and ServiceAccount. Pods already running keep running, but any new pod whose scale.<service>.scheduler_name still references hdx-binpack-scheduler goes Pending, because the referenced scheduler no longer exists.

To disable the scheduler safely:

1. Remove scheduler_name from every opted-in service under scale, and set scheduler.apply_to_all: false if you enabled the global opt-in.
2. Wait for the operator to reconcile - new pods now use the default scheduler.
3. Set scheduler.enabled: false.

To recover pods that are already Pending, re-enable the scheduler (scheduler.enabled: true) and they transition to Running.

Troubleshooting

Pods are Pending after disabling the scheduler

You left scale.<service>.scheduler_name set on one or more services after setting scheduler.enabled: false.

Either re-enable the scheduler, or remove the scheduler_name field from each opted-in service under scale. See Disable the scheduler for the safe sequence.

The validator warns about a scheduler-name mismatch

Your scheduler.name and one or more scale.<service>.scheduler_name entries disagree.

Set both to the same value. When pointing at an external scheduler, set scheduler.name to the external scheduler's name even though scheduler.enabled is false.

The validator warns about redundant per-service scheduler names

You set scheduler.apply_to_all: true and also set scale.<service>.scheduler_name on one or more services.

When apply_to_all is true, per-service entries that name the same scheduler have no effect. Remove them, or add those services to scheduler.apply_to_all_exclude if you meant to keep them on the default scheduler.