Skip to content

Select HDX Scaler Implementation

Hydrolix provides a custom autoscaling service, HDX Scaler, capable of horizontal (pod count) and vertical (container resources) scaling based on Prometheus metrics and resource utilization. See Autoscaling Prometheus Metrics for more detail.

If you are enabling autoscaling for the first time, see enable the autoscaler.

There are two implementations:

  • hdx-scaler-go, written in Go is the default HDX Scaler service in v6.4, introduced in v5.11
  • hdx-scaler, written in Python was the original HDX Scaler service

Both hdx-scaler and hdx-scaler-go run as singleton pods which is enforced by the Hydrolix Kubernetes operator. Both scalers may be enabled at the same time, which is useful during migration. See Deploy both scaler services.

Upgrading to v6.4 changes default⚓︎

The default changed in Hydrolix version 6.4.

The default HDX Scaler is engine: go in Hydrolix v6.4. Clusters upgrading to v6.4 which don't specify an HDX Scaler engine now use hdx-scaler-go.

These changes take effect on upgrade:

  • hdx-scaler-go runs by default on every cluster. Before v6.4, neither HDX Scaler ran by default.
  • The default engine becomes go.
  • Any hdxscalers entry without an explicit engine key now uses hdx-scaler-go.

Services that set engine keep the same engine after upgrade.

Control the HDX Scaler implementation manually

To select the HDX Scaler implementation manually, use the instructions in Continue using Python engine and manual migration to switch incrementally.

Continue using Python engine⚓︎

To keep a service on hdx-scaler on or after v6.4, pin its engine and ensure there is a Python scaler replica running before upgrading:

Pin a Service to the Python HDX Scaler
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
spec:
  scale:
    hdx-scaler:
      replicas: 1
    intake-head:
      replicas: 1-3
      hdxscalers:
        - metric: http_source_outstanding_reqs
          port: 27182
          target_value: 5
          engine: python

To pin an entire cluster, set the cluster-wide default:

Pin the Whole Cluster to the Python HDX Scaler
1
2
3
4
5
6
spec:
  hdxscaler:
    engine: python
  scale:
    hdx-scaler:
      replicas: 1

Migration steps⚓︎

These steps stage the migration one service at a time, comparing the two engines before cutting over. They apply to clusters on v6.3 and earlier, and to v6.4 clusters that pinned services to engine: python and now want a measured cutover.

Manual migration is useful for two cases:

  • keeping services on the Python scaler
  • comparing the two engines before committing to hdx-scaler-go

There are three main sections:

Preparation⚓︎

  1. Prerequisites: Upgrade the cluster to v5.11 or above
  2. Deploy both scalers: Enable both scalers in the same cluster

Monitor and switch⚓︎

  1. Enable hdx-scaler-go for one service in dry-run mode: Enable hdx-scaler-go for one service in dry-run mode. hdx-scaler continues scaling every service.
  2. Compare hdx-scaler-go decisions against hdx-scaler: Examine hdx-scaler-go decision log for parity.
  3. Cut the service over to hdx-scaler-go: hdx-scaler-go starts scaling the service.
  4. Repeat per service or move the whole cluster at once: Repeat service-by-service or for the whole cluster.

Confirmation⚓︎

  1. Confirm hdx-scaler is disabled: Verify that no services remain using hdx-scaler.

An optional rollback step is available at any time to return individual services or the entire cluster to using hdx-scaler.

Configuration⚓︎

Control the migration by using the engine field. This field can be set for the entire cluster or per service.

Set the engine field

1
2
3
spec:
  hdxscaler:
    engine: go

1
2
3
4
5
6
7
8
9
spec:
  scale:
    intake-head:
      replicas: 1-5
      hdxscalers:
        - metric: http_source_outstanding_reqs
          port: 27182
          target_value: 5
          engine: go

You'll set this field to the following values in order during migration:

  1. shadow: Use hdx-scaler to scale the cluster and compute hdx-scaler-go decisions without making changes to the cluster.
  2. go: Use hdx-scaler-go to scale the cluster.
  3. (optional) python: Use hdx-scaler to scale the cluster. hdx-scaler-go stops applying the affected specs. See Rollback.

Prerequisites⚓︎

Confirm the cluster is on v5.11 or later.

Deploy both scaler services⚓︎

Set both scalers to a single replica in the HydrolixCluster spec and apply:

1
2
3
4
5
6
spec:
  scale:
    hdx-scaler:
      replicas: 1
    hdx-scaler-go:
      replicas: 1

Both scalers deploy. On v6.3 and earlier, the scaler engine defaults to python, and hdx-scaler continues scaling every service.

On v6.4 and later the default is go. Set engine: python at the cluster or service level to hold services on hdx-scaler while staging the comparison.

hdx-scaler vs hdx-scaler-go: Kubernetes object configuration

When enabled, hdx-scaler-go writes its configuration to an HPAConfig or VPAConfig object while hdx-scaler writes to a single ConfigMap object.

HPAConfig and VPAConfig are Hydrolix custom resources, which are new object types that extend the Kubernetes API and can be inspected like any other resource, for example, using kubectl describe hpaconfig intake-head.

A ConfigMap is a built-in Kubernetes object that holds configuration as key-value data and can be inspected with kubectl describe configmaps hdxscaler.

Enable hdx-scaler-go for one service in dry-run mode⚓︎

Add engine: shadow, which enables a dry-run mode for hdx-scaler-go, to a target service hdxscalers entry and apply:

1
2
3
4
5
6
7
8
9
spec:
  scale:
    intake-head:
      replicas: 1-3
      hdxscalers:
        - metric: http_source_outstanding_reqs
          port: 27182
          target_value: 5
          engine: shadow

hdx-scaler continues scaling the cluster.

Verification

Assuming the example config above for intake-head, expect the following results.

  1. Connect to the transitioning cluster
  2. :hpaconfig or :vpaconfig
  3. Press e while highlighting intake-head
  4. You should see the following fields/values at a minimum: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  app: intake-head
  dryRun: true
  metrics:
    name: http_source_outstanding_reqs
    port: 27182
  scaling:
    maxReplicas: 3
    minReplicas: 1
    targetValue: 5

This configuration indicates that hdx-scaler-go is in dry run mode.

  1. Connect to the transitioning cluster
  2. :configmaps
  3. Press e while highlighting hdxscaler
  4. You should see the following fields/values at a minimum: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
data:
  hdxscaler.yaml: |
    autoscalers:
    - app: intake-head
      dry_run: false
      max: 3
      metric: http_source_outstanding_reqs
      min: 1
      path: metrics
      port: 27182
      target_value: 5

This configuration indicates that hdx-scaler is in live mode.

Compare hdx-scaler-go decisions against hdx-scaler⚓︎

Watch the hdx-scaler-go custom resource status and metrics while a representative load runs:

1
2
kubectl get hpaconfig -o wide
kubectl get vpaconfig -o wide

Check the MESSAGE column.

Decision-making messages like:

  • metric above target
  • metric below target
  • metric within tolerance

indicate that hdx-scaler-go is reading the target (such as the metric http_source_outstanding_reqs), applying calculations, and reaching a verdict.

Messages like the following indicate errors:

  • metric(s) unavailable: hdx-scaler-go can't reach or scrape the metric and therefore can't make scaling-related decisions.
  • deployment not found: The target deployment isn't available.

Additionally:

  • cooldown active: Indicates a recent scale-up or scale-down, resulting in a timer gating further scaling activities. This message should eventually transition to a decision-making message.

Cut over to hdx-scaler-go⚓︎

When the dry run behavior looks right, flip the entry to engine: go and apply:

1
2
3
4
5
6
7
8
9
spec:
  scale:
    intake-head:
      replicas: 1-3
      hdxscalers:
        - metric: http_source_outstanding_reqs
          port: 27182
          target_value: 5
          engine: go
Verify hdx-scaler-go is enabled

Assuming the example config above for intake-head, expect the following results.

  1. Connect to the transitioning cluster
  2. :hpaconfig or :vpaconfig
  3. Press e while highlighting intake-head
  4. You should see the following fields/values at a minimum: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  app: intake-head
  dryRun: false
  metrics:
    name: http_source_outstanding_reqs
    port: 27182
  scaling:
    maxReplicas: 3
    minReplicas: 1
    targetValue: 5

This configuration indicates that hdx-scaler-go is in live mode.

  1. Run the shell command kubectl describe hpaconfig intake-head
  2. You should see the following fields/values at a minimum:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Spec:
  App: intake-head
  Dry Run: false
  Metrics:
    Name: http_source_outstanding_reqs
    Port: 27182
  Scaling:
    Max Replicas: 3
    Min Replicas: 1
    Target Value: 5

This configuration indicates that hdx-scaler-go is in live mode.

Repeat per service or move the whole cluster at once⚓︎

Either repeat Steps 2–5 for each remaining service or enable hdx-scaler-go for the entire cluster:

1
2
3
spec:
  hdxscaler:
    engine: go

The engine field set at the per-service level overrides the cluster-wide setting. This allows specific services to be scaled by hdx-scaler. When all services are using hdx-scaler-go, remove the hdx-scaler entry:

1
2
3
4
spec:
  scale:
    hdx-scaler:      #<-- remove
      replicas: 1    #<-- remove

Confirm hdx-scaler is disabled⚓︎

Verify no services are using hdx-scaler
  1. Connect to the transitioning cluster
  2. :configmaps
  3. Press e while highlighting hdxscaler
  4. Assuming no other services are using hdx-scaler, you should see the following fields/values: 
1
2
3
data:
  hdxscaler.yaml: |
    autoscalers: []

This configuration indicates that hdx-scaler is disabled for all services.

  1. Run the shell command kubectl describe configmaps hdxscaler
  2. You should see the following fields/values indicating that no services are using hdx-scaler:
1
2
3
4
5
6
7
Name:         hdxscaler

Data
====
hdxscaler.yaml:
----
autoscalers: []

This configuration indicates that hdx-scaler is disabled for all services.

Rollback (available at any step)⚓︎

Set the engine field back to one of the following:

  • engine: shadow: Uses hdx-scaler to scale the service or cluster. hdx-scaler-go returns to dry-run mode.
  • engine: python: Uses hdx-scaler to scale the service or cluster. hdx-scaler-go stops applying the affected specs.

Both values need at least one hdx-scaler replica. On v6.4 and later, hdx-scaler defaults to 0 replicas. To rollback, set the replica:

Give the Python Scaler a Replica
1
2
3
4
spec:
  scale:
    hdx-scaler:
      replicas: 1

With no replica in place, the Hydrolix Kubernetes tool warns that the rolled-back specs won't be applied. See Validation warnings for engine and replica mismatches.

When a service transitions from using go to python or shadow, the operator cleans up the stale HPAConfig and VPAConfig custom resources in its next reconciliation.