Skip to content

Scheduled Overrides

Scheduled overrides⚓︎

You can automate temporary overrides for the scale of the various components in a cluster using scheduled overrides.

Overrides are specified within the Hydrolix spec configuration and take effect when the container's current time falls within the schedule block specified for an override. The HDX custom resource definition itself is never mutated by the contents of the overrides block. Instead, overrides live only in the operator memory.

Requirements⚓︎

Each override must contain the fields timezone, patch, and exactly one of either weekly, window, or cron.

Field name Type Description
timezone string IANA TZ used with the specified schedule. Defaults to UTC.
weekly Block containing the following fields:
days (format: A list containing elements that are 3-letter strings for each day of the week: Sun, Mon, Tue, Wed, Thu, Fri, Sat)
start (type: string, format: \"HH:MM\")
end (type: string, format: \"HH:MM\")

 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
window Block containing the following fields:
start (type: ISO-8601 datetime)
end (type: ISO-8601 datetime)		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
cron Block containing the following fields:
cron expression (type: cron string)
duration (type: string)		 One of three possible schedule blocks. Conditionally required if no other schedule block is set.
patch Dictionary. Must be valid, properly-formatted fields from the list of Hydrolix tunables The HDX spec definition to apply while active.

Multiple overrides can be active at the same time. When there are duplicate keys updated by multiple overrides, the last key in the config will apply.

Valid scheduled override example⚓︎

Weekly Override for High Traffic Days
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days:
          - Mon
          - Tues
          - Wed
          - Thurs
          - Fri
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999
Weekly Override for High Traffic Days in Condensed List Format
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days: ['Mon', 'Tues', 'Wed', 'Thu', 'Fri']
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999
Holiday Window Scale Down
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  overrides:
    winter-scale-down:
      timezone: America/Los_Angeles
      window:
        start: "1999-12-24T18:00:00Z"
        end: "2000-01-02T09:00:00Z"
      patch:
        scale_off: "true"
        celebrate: true
Nightly Ingest Scale Down
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
spec:
  overrides:
    nightly-intake-scale-down:
      timezone: America/Los_Angeles
      cron:
        expression: "0 2 * * 1-5"
        duration: "4h"
      patch:
        pools:
          hydrologs-intake-head:
            replicas: "0"

While particularly pertinent to automated scaling, scheduled overrides can be applied to any Hydrolix tunable. See Hydrolix tunables for additional tunable examples.

Limitation⚓︎

You can't update a scheduled override if the cluster is within the active time window for the override. For example, take the following override:

Active Window Override
1
2
3
4
5
6
7
8
9
  overrides:
    test-override:
      patch:
        howdy: doody
        scale_off: "true"
      timezone: America/Los_Angeles
      window:
        start: "2025-07-29T09:58:00Z"
        end: "2025-07-29T17:00:00Z"

Updating this override prior to its start time would ensure the changes are applied as expected within the specified time window.

However, if current time for the cluster is "2025-07-29T10:00:00Z", which lies within the window, updating the override would not apply the changes.

Workaround⚓︎

To make sure any override changes apply as expected, update the start time to a point in the future. In this example, it would be sufficient to update the window configuration to

Workaround: Start Set to Future Datetime
1
2
3
window:
  start: "2025-07-29T10:01:00Z"
  end: "2025-07-29T17:00:00Z"

Verification⚓︎

Hydrolix clusters use a configurable validating webhook to verify the correctness of the spec configuration of the hdx Kubernetes object, including any scheduled overrides. The webhook is responsible for the following types of validation:

Verification of required fields⚓︎

The validating webhook enforces that these required fields are present:

  • timezone
  • patch
  • exactly one of cron, weekly, or window

Bad scheduled override config missing required field: timezone⚓︎

1
2
3
4
5
6
7
8
9
spec:
    overrides:
        bad-override-missing-timezone:
            weekly:
                days: ["Sun"]
                start: "16:00"
                end: "22:00"
            patch:
                scale_off: "true"

1
    message: 'bad-override-missing-timezone: `timezone` field is required'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "message": "Handler 'on_change' failed temporarily: bad-override-missing-tz: `timezone` field is required", 
 "timestamp": "2025-09-04T01:09:12.305876+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "${HDX_NAMESPACE}"}, 
  "severity": "error"
}

Bad scheduled override config with too many schedule blocks⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
spec:
  overrides:
      bad-override-:
        timezone: UTC
        weekly:                         <-- invalid to have both
          days: ["Sun"]
          start: "16:00"
          end: "17:00"
        cron:                           <-- invalid to have both
          expression: 1
          duration: fortnight
        patch:
          scale_off: true

1
message: 'bad-override-: exactly one schedule block required'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "message": "Handler 'on_change' failed temporarily: bad-override-: exactly one schedule block required",
  "timestamp": "2025-09-05T17:35:43.981979+00:00",
  "object": {
    "apiVersion": "hydrolix.io/v1",
    "kind": "HydrolixCluster",
    "name": "hdx",
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7",
    "namespace": "docs-sandbox"
  },
  "severity": "error"
}

Validation of syntax and values⚓︎

Each scheduled override block type requires different keys. The validating webhook enforces the required keys and validates that types and values are correct.

Bad scheduled override config with invalid cron field name: days⚓︎

The days field is invalid for cron block type, only valid for the weekly type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  overrides:
      bad-override-invalid-cron-field:
        timezone: UTC
        cron:
          days: ["Sun"]                 <-- invalid
          expression: 1
          duration: fortnight
        patch:
          scale_off: "true"

1
message: 'Cron.__init__() got an unexpected keyword argument ''days'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() got an unexpected keyword argument 'days'", "timestamp": "2025-09-04T01:33:15.516580+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

Bad scheduled override config with missing cron setting: expression⚓︎

1
2
3
4
5
6
7
8
spec:
  overrides:
      bad-override-missing-cron-field:
        timezone: UTC
        cron:
          duration: fortnight
        patch:
          scale_off: "true"

1
message: 'Cron.__init__() missing 1 required positional argument: ''expression'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() missing 1 required positional argument: 'expression'", 
  "timestamp": "2025-09-04T01:48:17.580770+00:00", 
  "object": 
  {
    "apiVersion": "hydrolix.io/v1", 
    "kind": "HydrolixCluster", 
    "name": "hdx", 
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
    "namespace": "docs-sandbox"
  }, 
  "severity": "error"
}

Bad scheduled override config with invalid values for window fields⚓︎

1
2
3
4
5
6
7
8
9
spec:
  overrides:
      bad-override-invalid-values:
        timezone: UTC
        window:
          start: preferably-soon        <-- invalid
          end: in-the-future            <-- invalid
        patch:
          scale_off: "true"

1
message: 'Invalid isoformat string: ''preferably-soon'''

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "message": "Timer 'overrides' failed with an exception. Will retry.", 
  "exc_info": "ValueError: Invalid isoformat string: 'preferably-soon'", 
  "timestamp": "2025-09-05T17:27:30.827272+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

The operator writes the list of currently active override names to .status.activeOverrides in the Hydrolix cluster configuration. Once a cluster's clock time lies within the schedule blocks for an override, check the hdx configuration to find a list of active overrides:

Active Overrides in Hdx Configuration
1
2
3
4
Status:
  Active Overrides:
    winter-scale-down
    nightly-intake-scale-down

Recall that when there are duplicate keys updated by multiple overrides, the last key in the config will apply. In the example above, any keys updated by both the nightly-intake-scale-down and winter-scale-down overrides will conform to the value set by nightly-intake-scale-down.