Skip to content

Scheduled Overrides

Scheduled overrides automate temporary overrides for the scale of components in a cluster.

Overrides are specified within the Hydrolix spec configuration and take effect when the container's current time falls within the schedule block specified for an override. The HDX custom resource definition itself is never mutated by the contents of the overrides block. Instead, overrides live only in the operator memory.

Requirements⚓︎

Each override must contain the fields timezone, patch, and exactly one of weekly, window, or cron.

Field name Type Description
timezone string IANA TZ used with the specified schedule. Defaults to UTC.
weekly Block containing the following fields:
days (format: A list containing elements that are 3-letter strings for each day of the week: Sun, Mon, Tue, Wed, Thu, Fri, Sat)
start (type: string, format: \"HH:MM\")
end (type: string, format: \"HH:MM\")		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
window Block containing the following fields:
start (type: ISO-8601 datetime)
end (type: ISO-8601 datetime)		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
cron Block containing the following fields:
cron expression (type: cron string)
duration (type: string)		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
patch Dictionary. Must be valid, properly-formatted fields from the list of Hydrolix tunables The HDX spec definition to apply while active.

Multiple overrides can be active at the same time. When there are duplicate keys updated by multiple overrides, the last key takes precedence.

Valid scheduled override example⚓︎

Weekly Override for High Traffic Days
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days:
          - Mon
          - Tues
          - Wed
          - Thurs
          - Fri
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999
Weekly Override for High Traffic Days in Condensed List Format
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days: ['Mon', 'Tues', 'Wed', 'Thu', 'Fri']
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999
Holiday Window Scale Down
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  overrides:
    winter-scale-down:
      timezone: America/Los_Angeles
      window:
        start: "1999-12-24T18:00:00Z"
        end: "2000-01-02T09:00:00Z"
      patch:
        scale_off: "true"
        celebrate: true
Nightly Ingest Scale Down
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
spec:
  overrides:
    nightly-intake-scale-down:
      timezone: America/Los_Angeles
      cron:
        expression: "0 2 * * 1-5"
        duration: "4h"
      patch:
        pools:
          hydrologs-intake-head:
            replicas: "0"

While particularly pertinent to automated scaling, scheduled overrides can be applied to any Hydrolix tunable. See Hydrolix tunables for additional tunable examples.

Limitation⚓︎

A scheduled override can't be updated while the cluster is in the active time window for that override. For example, take the following override:

Active Window Override
1
2
3
4
5
6
7
8
9
  overrides:
    test-override:
      patch:
        howdy: doody
        scale_off: "true"
      timezone: America/Los_Angeles
      window:
        start: "2025-07-29T09:58:00Z"
        end: "2025-07-29T17:00:00Z"

Updating this override prior to its start time would ensure the changes are applied as expected within the specified time window.

However, if current time for the cluster is "2025-07-29T10:00:00Z", which is in the active window, updating the override doesn't apply the changes.

Workaround⚓︎

To ensure override changes apply as expected, update the start time to a point in the future. For this example, update the window configuration:

Workaround: Start Set to Future Datetime
1
2
3
window:
  start: "2025-07-29T10:01:00Z"
  end: "2025-07-29T17:00:00Z"

Verification⚓︎

Hydrolix clusters use a configurable validating webhook to verify the correctness of the spec configuration of the hdx Kubernetes object, including any scheduled overrides. The webhook is responsible for the following types of validation:

Verification of required fields⚓︎

The validating webhook enforces that these required fields are present:

  • timezone
  • patch
  • exactly one of cron, weekly, or window

Bad scheduled override config missing required field: timezone⚓︎

1
2
3
4
5
6
7
8
9
spec:
    overrides:
        bad-override-missing-timezone:
            weekly:
                days: ["Sun"]
                start: "16:00"
                end: "22:00"
            patch:
                scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
    message: 'bad-override-missing-timezone: `timezone` field is required'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "message": "Handler 'on_change' failed temporarily: bad-override-missing-tz: `timezone` field is required", 
 "timestamp": "2025-09-04T01:09:12.305876+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "${HDX_NAMESPACE}"}, 
  "severity": "error"
}

Bad scheduled override config with too many schedule blocks⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
spec:
  overrides:
      bad-override-:
        timezone: UTC
        weekly:                         <-- invalid to have both
          days: ["Sun"]
          start: "16:00"
          end: "17:00"
        cron:                           <-- invalid to have both
          expression: 1
          duration: fortnight
        patch:
          scale_off: true

Value of status.kopf.progress.on_change.message in hdx Manifest

1
    message: 'bad-override-: exactly one schedule block required'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "message": "Handler 'on_change' failed temporarily: bad-override-: exactly one schedule block required",
  "timestamp": "2025-09-05T17:35:43.981979+00:00",
  "object": {
    "apiVersion": "hydrolix.io/v1",
    "kind": "HydrolixCluster",
    "name": "hdx",
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7",
    "namespace": "docs-sandbox"
  },
  "severity": "error"
}

Validation of syntax and values⚓︎

Each scheduled override block type requires different keys. The validating webhook enforces the required keys and validates that types and values are correct.

Bad scheduled override config with invalid cron field name: days⚓︎

The days field is invalid for cron block type, only valid for the weekly type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  overrides:
      bad-override-invalid-cron-field:
        timezone: UTC
        cron:
          days: ["Sun"]                 <-- invalid
          expression: 1
          duration: fortnight
        patch:
          scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
message: 'Cron.__init__() got an unexpected keyword argument ''days'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() got an unexpected keyword argument 'days'", "timestamp": "2025-09-04T01:33:15.516580+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

Bad scheduled override config with missing cron setting: expression⚓︎

1
2
3
4
5
6
7
8
spec:
  overrides:
      bad-override-missing-cron-field:
        timezone: UTC
        cron:
          duration: fortnight
        patch:
          scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
message: 'Cron.__init__() missing 1 required positional argument: ''expression'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() missing 1 required positional argument: 'expression'", 
  "timestamp": "2025-09-04T01:48:17.580770+00:00", 
  "object": 
  {
    "apiVersion": "hydrolix.io/v1", 
    "kind": "HydrolixCluster", 
    "name": "hdx", 
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
    "namespace": "docs-sandbox"
  }, 
  "severity": "error"
}

Bad scheduled override config with invalid values for window fields⚓︎

1
2
3
4
5
6
7
8
9
spec:
  overrides:
      bad-override-invalid-values:
        timezone: UTC
        window:
          start: preferably-soon        <-- invalid
          end: in-the-future            <-- invalid
        patch:
          scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
message: 'Invalid isoformat string: ''preferably-soon'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "message": "Timer 'overrides' failed with an exception. Will retry.", 
  "exc_info": "ValueError: Invalid isoformat string: 'preferably-soon'", 
  "timestamp": "2025-09-05T17:27:30.827272+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

Override status⚓︎

All overrides have their details and whether they're active or inactive written to status.overrides in the Hydrolix cluster configuration.

List of All Overrides and Their Status
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
status:
  overrides:
    invalid-override:
      errors:
        - Missing {'end'} in window schedule at .overrides.invalid-override.window
      is_active: false
      is_valid: false
      schedule_type: window
      state: invalid
      timezone: UTC

    valid-upcoming-override:
      is_active: false
      is_valid: true
      next_window:
        end: "2026-04-29T06:00:00-07:00"
        start: "2026-04-29T02:00:00-07:00"
      schedule_type: cron
      state: upcoming
      timezone: America/Los_Angeles

    valid-active-override:
      current_window:
        end: "2026-04-28T18:52:00-07:00"
        start: "2026-04-28T14:52:00-07:00"
      is_active: true
      is_valid: true
      next_window:
        end: "2026-04-29T18:52:00-07:00"
        start: "2026-04-29T14:52:00-07:00"
      schedule_type: cron
      state: active
      timezone: America/Los_Angeles

Additionally, the operator writes the list of currently active override names to status.activeOverrides. Once a cluster's clock time lies within the schedule blocks for an override, check the hdx configuration to find a list of active overrides:

List of Active Overrides in Hdx Configuration
1
2
3
4
status:
  activeOverrides:
    winter-scale-down
    nightly-intake-scale-down

When there are duplicate keys updated by multiple overrides, the last key takes precedence. In that example, any keys updated by both the nightly-intake-scale-down and winter-scale-down overrides use the value set by nightly-intake-scale-down.

Override metrics⚓︎

Troubleshoot delayed application or removal of overrides using two Prometheus gauges that show schedule override status. See Prometheus Operator metrics for a list of operator metrics.

Metric Description
o6r_hdx_active_overrides Number of overrides currently in their active time window. Derived from the schedule alone, independent of whether the cluster's configuration has converged.
o6r_hdx_override_drift_seconds Number of seconds the cluster's configuration has disagreed with the override schedule. Returns 0 when the two are synchronized.

Read together, the two gauges distinguish a healthy schedule from a stuck reconciliation:

o6r_hdx_active_overrides o6r_hdx_override_drift_seconds Meaning
Any value 0 The schedule and the cluster agree.
0 Climbing An override's window has closed, but the override is still applied.
1 or more Climbing An override's window has opened, but the override hasn't been applied.

When an override starts or ends normally, the drift gauge stays at 0. The gauge reports how long the cluster's configuration has been out of sync with the schedule, updated every 15 seconds, so a stuck reconciliation shows a steadily climbing value. Alert on a sustained non-zero value rather than on a single sample to guard against transient failures.

The o6r_hdx_override_drift_seconds gauge can also show drift that's happened for reasons unrelated to overrides. Check o6r_hdx_synced to see if this is the case.

o6r_hdx_override_drift_seconds o6r_hdx_synced Meaning
0 1 The schedule and the cluster agree.
Climbing 0 The operator can't apply any resources, not only overrides. Check the operator logs.
Climbing 1 The operator can't validate the new effective spec. The previous configuration is still syncing normally. Check the operator logs.
0 0 The operator has just started, or a sync is failing with no recent spec change.