Skip to content

Migrate from hdx-scaler to hdx-scaler-go

Hydrolix provides a custom autoscaling service capable of horizontal (pod count) and vertical (container resources) scaling based on Prometheus metrics and resource utilization. See Autoscaling Prometheus Metrics for more detail.

v5.11 introduced hdx-scaler-go, the canonical Hydrolix autoscaling service. If you are turning on autoscaling for the first time, see autoscaling-prometheus-metrics. Use this page to migrate a cluster that's already running the pre-v5.11 hdx-scaler for autoscaling.

Both hdx-scaler and hdx-scaler-go run as a singleton pod which is enforced by the Hydrolix Kubernetes operator. Both scalers may be enabled at the same time during migration. See Migration.

Migration steps⚓︎

There are three main sections:

Preparation⚓︎

  1. Prerequisites: Upgrade the cluster to v5.11 or above
  2. Deploy both scalers: Enable both scalers in the same cluster

Monitor and switch⚓︎

  1. Enable hdx-scaler-go for one service in dry-run mode: Enable hdx-scaler-go for one service in dry-run mode. hdx-scaler continues scaling every service.
  2. Compare hdx-scaler-go decisions against hdx-scaler: Examine hdx-scaler-go decision log for parity.
  3. Cut the service over to hdx-scaler-go: hdx-scaler-go starts scaling the service.
  4. Repeat per service or move the whole cluster at once: Repeat service-by-service or for the whole cluster.

Confirmation⚓︎

  1. Confirm hdx-scaler is disabled: Verify that no services remain using hdx-scaler.

An optional rollback step is available at any time to return individual services or the entire cluster to using hdx-scaler.

Configuration⚓︎

Control the migration by using the engine field. This field can be set for the entire cluster or per service.

Set the engine field

1
2
3
spec:
  hdxscaler:
    engine: go

1
2
3
4
5
6
7
8
9
spec:
  scale:
    intake-head:
      replicas: 1-5
      hdxscalers:
        - metric: http_source_outstanding_reqs
          port: 27182
          target_value: 5
          engine: go

You'll set this field to the following values in order during migration:

  1. shadow: Use hdx-scaler to scale the cluster and compute hdx-scaler-go decisions without making changes to the cluster.
  2. go: Use hdx-scaler-go to scale the cluster.
  3. (optional) python: Use hdx-scaler to scale the cluster. Disables hdx-scaler-go. See Rollback.

Prerequisites⚓︎

Confirm the cluster is on v5.11 or later.

Deploy both scaler services⚓︎

Set both scalers to a single replica in the HydrolixCluster spec and apply:

1
2
3
4
5
6
spec:
  scale:
    hdx-scaler:
      replicas: 1
    hdx-scaler-go:
      replicas: 1

Both scalers deploy. The scaler engine defaults to python, so hdx-scaler continues scaling every service.

hdx-scaler vs hdx-scaler-go: Kubernetes object configuration

When enabled, hdx-scaler-go writes its configuration to an HPAConfig or VPAConfig object while hdx-scaler writes to a single ConfigMap object.

HPAConfig and VPAConfig are Hydrolix custom resources, which are new object types that extend the Kubernetes API and can be inspected like any other resource, for example, using kubectl describe hpaconfig intake-head.

A ConfigMap is a built-in Kubernetes object that holds configuration as key-value data and can be inspected with kubectl describe configmaps hdxscaler.

Enable hdx-scaler-go for one service in dry-run mode⚓︎

Add engine: shadow, which enables a dry-run mode for hdx-scaler-go, to a target service hdxscalers entry and apply:

1
2
3
4
5
6
7
8
9
spec:
  scale:
    intake-head:
      replicas: 1-3
      hdxscalers:
        - metric: http_source_outstanding_reqs
          port: 27182
          target_value: 5
          engine: shadow

hdx-scaler continues scaling the cluster.

Verification

Assuming the example config above for intake-head, expect the following results.

  1. Connect to the transitioning cluster
  2. :hpaconfig or :vpaconfig
  3. Press e while highlighting intake-head
  4. You should see the following fields/values at a minimum: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  app: intake-head
  dryRun: true
  metrics:
    name: http_source_outstanding_reqs
    port: 27182
  scaling:
    maxReplicas: 3
    minReplicas: 1
    targetValue: 5

This configuration indicates that hdx-scaler-go is in dry run mode.

  1. Connect to the transitioning cluster
  2. :configmaps
  3. Press e while highlighting hdxscaler
  4. You should see the following fields/values at a minimum: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
data:
  hdxscaler.yaml: |
    autoscalers:
    - app: intake-head
      dry_run: false
      max: 3
      metric: http_source_outstanding_reqs
      min: 1
      path: metrics
      port: 27182
      target_value: 5

This configuration indicates that hdx-scaler is in live mode.

Compare hdx-scaler-go decisions against hdx-scaler⚓︎

Watch the hdx-scaler-go custom resource status and metrics while a representative load runs:

1
2
kubectl get hpaconfig -o wide
kubectl get vpaconfig -o wide

Check the MESSAGE column.

Decision-making messages like:

  • metric above target
  • metric below target
  • metric within tolerance

indicate that hdx-scaler-go is reading the target (such as the metric http_source_outstanding_reqs), applying calculations, and reaching a verdict.

Messages like the following indicate errors:

  • metric(s) unavailable: hdx-scaler-go can't reach or scrape the metric and therefore can't make scaling-related decisions.
  • deployment not found: The target deployment isn't available.

Additionally:

  • cooldown active: Indicates a recent scale-up or scale-down, resulting in a timer gating further scaling activities. This message should eventually transition to a decision-making message.

Cut over to hdx-scaler-go⚓︎

When the dry run behavior looks right, flip the entry to engine: go and apply:

1
2
3
4
5
6
7
8
9
spec:
  scale:
    intake-head:
      replicas: 1-3
      hdxscalers:
        - metric: http_source_outstanding_reqs
          port: 27182
          target_value: 5
          engine: go
Verify hdx-scaler-go is enabled

Assuming the example config above for intake-head, expect the following results.

  1. Connect to the transitioning cluster
  2. :hpaconfig or :vpaconfig
  3. Press e while highlighting intake-head
  4. You should see the following fields/values at a minimum: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  app: intake-head
  dryRun: false
  metrics:
    name: http_source_outstanding_reqs
    port: 27182
  scaling:
    maxReplicas: 3
    minReplicas: 1
    targetValue: 5

This configuration indicates that hdx-scaler-go is in live mode.

  1. Run the shell command kubectl describe hpaconfig intake-head
  2. You should see the following fields/values at a minimum:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Spec:
  App: intake-head
  Dry Run: false
  Metrics:
    Name: http_source_outstanding_reqs
    Port: 27182
  Scaling:
    Max Replicas: 3
    Min Replicas: 1
    Target Value: 5

This configuration indicates that hdx-scaler-go is in live mode.

Repeat per service or move the whole cluster at once⚓︎

Either repeat Steps 2–5 for each remaining service or enable hdx-scaler-go for the entire cluster:

1
2
3
spec:
  hdxscaler:
    engine: go

The engine field set at the per-service level overrides the cluster-wide setting, so you can continue to have specific scaled by hdx-scaler. When everything is using hdx-scaler-go, remove the hdx-scaler entry:

1
2
3
4
spec:
  scale:
    hdx-scaler:      #<-- remove
      replicas: 1    #<-- remove

Confirm hdx-scaler is disabled⚓︎

Verify no services are using hdx-scaler
  1. Connect to the transitioning cluster
  2. :configmaps
  3. Press e while highlighting hdxscaler
  4. Assuming no other services are using hdx-scaler, you should see the following fields/values: 
1
2
3
data:
  hdxscaler.yaml: |
    autoscalers: []

This configuration indicates that hdx-scaler is disabled for all services.

  1. Run the shell command kubectl describe configmaps hdxscaler
  2. You should see the following fields/values indicating that no services are using hdx-scaler:
1
2
3
4
5
6
7
Name:         hdxscaler

Data
====
hdxscaler.yaml:
----
autoscalers: []

This configuration indicates that hdx-scaler is disabled for all services.

Rollback (available at any step)⚓︎

Set the engine field back to one of the following:

  • engine: shadow: Uses hdx-scaler to scale the service or cluster. hdx-scaler-go returns to dry-run mode.
  • engine: python: Uses hdx-scaler to scale the service or cluster. Disables hdx-scaler-go.

When a service transitions from using go to python or shadow, the stale HPAConfig and VPAConfig custom resources are cleaned up by the operator in its next reconciliation.