Skip to content

Scheduled Overrides

Scheduled overrides automate temporary overrides for the scale of components in a cluster.

Overrides are specified within the Hydrolix spec configuration and take effect when the container's current time falls within the schedule block specified for an override. The HDX custom resource definition itself is never mutated by the contents of the overrides block. Instead, overrides live only in the operator memory.

Requirements⚓︎

Each override must contain the fields timezone, patch, and exactly one of weekly, window, or cron.

Field name Type Description
timezone string IANA TZ used with the specified schedule. Defaults to UTC.
weekly Block containing the following fields:
days (format: A list containing elements that are 3-letter strings for each day of the week: Sun, Mon, Tue, Wed, Thu, Fri, Sat)
start (type: string, format: \"HH:MM\")
end (type: string, format: \"HH:MM\")		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
window Block containing the following fields:
start (type: ISO-8601 datetime)
end (type: ISO-8601 datetime)		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
cron Block containing the following fields:
cron expression (type: cron string)
duration (type: string)		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
patch Dictionary. Must be valid, properly-formatted fields from the list of Hydrolix tunables The HDX spec definition to apply while active.

Multiple overrides can be active at the same time. When there are duplicate keys updated by multiple overrides, the last key takes precedence.

Valid scheduled override example⚓︎

Weekly Override for High Traffic Days
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days:
          - Mon
          - Tues
          - Wed
          - Thurs
          - Fri
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999
Weekly Override for High Traffic Days in Condensed List Format
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days: ['Mon', 'Tues', 'Wed', 'Thu', 'Fri']
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999
Holiday Window Scale Down
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  overrides:
    winter-scale-down:
      timezone: America/Los_Angeles
      window:
        start: "1999-12-24T18:00:00Z"
        end: "2000-01-02T09:00:00Z"
      patch:
        scale_off: "true"
        celebrate: true
Nightly Ingest Scale Down
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
spec:
  overrides:
    nightly-intake-scale-down:
      timezone: America/Los_Angeles
      cron:
        expression: "0 2 * * 1-5"
        duration: "4h"
      patch:
        pools:
          hydrologs-intake-head:
            replicas: "0"

While particularly pertinent to automated scaling, scheduled overrides can be applied to any Hydrolix tunable. See Hydrolix tunables for additional tunable examples.

Limitation⚓︎

A scheduled override can't be updated while the cluster is in the active time window for that override. For example, take the following override:

Active Window Override
1
2
3
4
5
6
7
8
9
  overrides:
    test-override:
      patch:
        howdy: doody
        scale_off: "true"
      timezone: America/Los_Angeles
      window:
        start: "2025-07-29T09:58:00Z"
        end: "2025-07-29T17:00:00Z"

Updating this override prior to its start time would ensure the changes are applied as expected within the specified time window.

However, if current time for the cluster is "2025-07-29T10:00:00Z", which is in the active window, updating the override doesn't apply the changes.

Workaround⚓︎

To ensure override changes apply as expected, update the start time to a point in the future. For this example, update the window configuration:

Workaround: Start Set to Future Datetime
1
2
3
window:
  start: "2025-07-29T10:01:00Z"
  end: "2025-07-29T17:00:00Z"

Verification⚓︎

Hydrolix clusters use a configurable validating webhook to verify the correctness of the spec configuration of the hdx Kubernetes object, including any scheduled overrides. The webhook is responsible for the following types of validation:

Verification of required fields⚓︎

The validating webhook enforces that these required fields are present:

  • timezone
  • patch
  • exactly one of cron, weekly, or window

Bad scheduled override config missing required field: timezone⚓︎

1
2
3
4
5
6
7
8
9
spec:
    overrides:
        bad-override-missing-timezone:
            weekly:
                days: ["Sun"]
                start: "16:00"
                end: "22:00"
            patch:
                scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
    message: 'bad-override-missing-timezone: `timezone` field is required'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "message": "Handler 'on_change' failed temporarily: bad-override-missing-tz: `timezone` field is required", 
 "timestamp": "2025-09-04T01:09:12.305876+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "${HDX_NAMESPACE}"}, 
  "severity": "error"
}

Bad scheduled override config with too many schedule blocks⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
spec:
  overrides:
      bad-override-:
        timezone: UTC
        weekly:                         <-- invalid to have both
          days: ["Sun"]
          start: "16:00"
          end: "17:00"
        cron:                           <-- invalid to have both
          expression: 1
          duration: fortnight
        patch:
          scale_off: true

Value of status.kopf.progress.on_change.message in hdx Manifest

1
    message: 'bad-override-: exactly one schedule block required'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "message": "Handler 'on_change' failed temporarily: bad-override-: exactly one schedule block required",
  "timestamp": "2025-09-05T17:35:43.981979+00:00",
  "object": {
    "apiVersion": "hydrolix.io/v1",
    "kind": "HydrolixCluster",
    "name": "hdx",
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7",
    "namespace": "docs-sandbox"
  },
  "severity": "error"
}

Validation of syntax and values⚓︎

Each scheduled override block type requires different keys. The validating webhook enforces the required keys and validates that types and values are correct.

Bad scheduled override config with invalid cron field name: days⚓︎

The days field is invalid for cron block type, only valid for the weekly type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  overrides:
      bad-override-invalid-cron-field:
        timezone: UTC
        cron:
          days: ["Sun"]                 <-- invalid
          expression: 1
          duration: fortnight
        patch:
          scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
message: 'Cron.__init__() got an unexpected keyword argument ''days'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() got an unexpected keyword argument 'days'", "timestamp": "2025-09-04T01:33:15.516580+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

Bad scheduled override config with missing cron setting: expression⚓︎

1
2
3
4
5
6
7
8
spec:
  overrides:
      bad-override-missing-cron-field:
        timezone: UTC
        cron:
          duration: fortnight
        patch:
          scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
message: 'Cron.__init__() missing 1 required positional argument: ''expression'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() missing 1 required positional argument: 'expression'", 
  "timestamp": "2025-09-04T01:48:17.580770+00:00", 
  "object": 
  {
    "apiVersion": "hydrolix.io/v1", 
    "kind": "HydrolixCluster", 
    "name": "hdx", 
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
    "namespace": "docs-sandbox"
  }, 
  "severity": "error"
}

Bad scheduled override config with invalid values for window fields⚓︎

1
2
3
4
5
6
7
8
9
spec:
  overrides:
      bad-override-invalid-values:
        timezone: UTC
        window:
          start: preferably-soon        <-- invalid
          end: in-the-future            <-- invalid
        patch:
          scale_off: "true"

Value of status.kopf.progress.on_change.message in hdx Manifest

1
message: 'Invalid isoformat string: ''preferably-soon'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "message": "Timer 'overrides' failed with an exception. Will retry.", 
  "exc_info": "ValueError: Invalid isoformat string: 'preferably-soon'", 
  "timestamp": "2025-09-05T17:27:30.827272+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

Override status⚓︎

All overrides have their details and whether they're active or inactive written to status.overrides in the Hydrolix cluster configuration.

List of All Overrides and Their Status
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
status:
  overrides:
    invalid-override:
      errors:
        - Missing {'end'} in window schedule at .overrides.invalid-override.window
      is_active: false
      is_valid: false
      schedule_type: window
      state: invalid
      timezone: UTC

    valid-upcoming-override:
      is_active: false
      is_valid: true
      next_window:
        end: "2026-04-29T06:00:00-07:00"
        start: "2026-04-29T02:00:00-07:00"
      schedule_type: cron
      state: upcoming
      timezone: America/Los_Angeles

    valid-active-override:
      current_window:
        end: "2026-04-28T18:52:00-07:00"
        start: "2026-04-28T14:52:00-07:00"
      is_active: true
      is_valid: true
      next_window:
        end: "2026-04-29T18:52:00-07:00"
        start: "2026-04-29T14:52:00-07:00"
      schedule_type: cron
      state: active
      timezone: America/Los_Angeles

Additionally, the operator writes the list of currently active override names to status.activeOverrides. Once a cluster's clock time lies within the schedule blocks for an override, check the hdx configuration to find a list of active overrides:

List of Active Overrides in Hdx Configuration
1
2
3
4
status:
  activeOverrides:
    winter-scale-down
    nightly-intake-scale-down

When there are duplicate keys updated by multiple overrides, the last key takes precedence. In that example, any keys updated by both the nightly-intake-scale-down and winter-scale-down overrides use the value set by nightly-intake-scale-down.