Scheduled Overrides

Scheduled overrides

You can automate temporary overrides for the scale of the various components in a cluster using scheduled overrides.

Overrides are specified within the Hydrolix spec configuration and take effect when the container's current time falls within the schedule block specified for an override. The HDX custom resource definition itself is never mutated by the contents of the overrides block. Instead, overrides live only in the operator memory.

Requirements

Each override must contain the fields timezone, patch, and exactly one of either weekly, window, or cron.

Field name	Type	Description
`timezone`	string	IANA TZ used with the specified schedule. Defaults to UTC.
`weekly`	Block containing the following fields: `days` (format: A list containing elements that are 3-letter strings for each day of the week: `Sun`, `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`) `start` (type: string, format: `"HH:MM"`) `end` (type: string, format: `"HH:MM"`)	One of three possible schedule blocks. Conditionally required if no other schedule block is set.
`window`	Block containing the following fields: `start` (type: ISO-8601 datetime) `end` (type: ISO-8601 datetime)	One of three possible schedule blocks. Conditionally required if no other schedule block is set.
`cron`	Block containing the following fields: `cron expression` (type: cron string `duration` (type: string)	One of three possible schedule blocks. Conditionally required if no other schedule block is set.
`patch`	Dictionary. Must be valid, properly-formatted fields from the list of Hydrolix tunables	The HDX spec definition to apply while active.

Multiple overrides can be active at the same time. When there are duplicate keys updated by multiple overrides, the last key in the config will apply.

Valid scheduled override examples

spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days:
          - Mon
          - Tues
          - Wed
          - Thurs
          - Fri
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999

spec:
  overrides:
    weekday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days: ['Mon', 'Tues', 'Wed', 'Thu', 'Fri']
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999

spec:
  overrides:
    winter-freeze:
      timezone: America/Los_Angeles
      window:
        start: "1999-12-24T18:00:00Z"
        end: "2000-01-02T09:00:00Z"
      patch:
        scale_off: "true"
        celebrate: true

spec:
  overrides:
    nightly-intake-scale-down:
      timezone: America/Los_Angeles
      cron:
        expression: "0 2 * * 1-5"
        duration: "4h"
      patch:
        pools:
          hydrologs-intake-head:
            replicas: "0"

While particularly pertinent to automated scaling, scheduled overrides can be applied to any Hydrolix tunable. See Hydrolix tunables for additional tunable examples.

Limitation

You can't update a scheduled override if the cluster is within the active time window for the override. For example, take the following override:

  overrides:
    test-override:
      patch:
        howdy: doody
        scale_off: "true"
      timezone: America/Los_Angeles
      window:
        start: "2025-07-29T09:58:00Z"
        end: "2025-07-29T17:00:00Z"

Updating this override prior to its start time would ensure the changes are applied as expected within the specified time window.

However, if current time for the cluster is "2025-07-29T10:00:00Z", which lies within the window, updating the override would not apply the changes.

Workaround

To make sure any override changes apply as expected, update the start time to a point in the future. In this example, it would be sufficient to update the window configuration to

window:
  start: "2025-07-29T10:01:00Z"
  end: "2025-07-29T17:00:00Z"

Verification

Hydrolix clusters use a configurable validating webhook to verify the correctness of the spec configuration of the hdx Kubernetes object, including any scheduled overrides. The webhook is responsible for the following types of validation:

Verification of required fields

The validating webhook enforces that these required fields are present:

timezone
patch
exactly one of cron, weekly, or window

Bad scheduled override config missing required field: `timezone`

spec:
  overrides:
      bad-override-missing-timezone:
        weekly:
          days: ["Sun"]
          start: "16:00"
          end: "22:00"
        patch:
          scale_off: "true"

message: 'bad-override-missing-timezone: `timezone` field is required'

{
  "message": "Handler 'on_change' failed temporarily: bad-override-missing-tz: `timezone` field is required", 
 "timestamp": "2025-09-04T01:09:12.305876+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "${HDX_NAMESPACE}"}, 
  "severity": "error"
}

Bad scheduled override config with too many schedule blocks

spec:
  overrides:
      bad-override-:
        timezone: UTC
        weekly:                         <-- invalid to have both
          days: ["Sun"]
          start: "16:00"
          end: "17:00"
        cron:                           <-- invalid to have both
          expression: 1
          duration: fortnight
        patch:
          scale_off: true

message: 'bad-override-: exactly one schedule block required'

{
  "message": "Handler 'on_change' failed temporarily: bad-override-: exactly one schedule block required",
  "timestamp": "2025-09-05T17:35:43.981979+00:00",
  "object": {
    "apiVersion": "hydrolix.io/v1",
    "kind": "HydrolixCluster",
    "name": "hdx",
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7",
    "namespace": "docs-sandbox"
  },
  "severity": "error"
}

Validation of syntax and values

Each scheduled override block type requires different keys. The validating webhook enforces the required keys and valiadtes that types and values are correct.

Bad scheduled override config with invalid cron field name: `days`

The days field is invalid for cron block type, only valid for the weekly type.

spec:
  overrides:
      bad-override-invalid-cron-field:
        timezone: UTC
        cron:
          days: ["Sun"]                 <-- invalid
          expression: 1
          duration: fortnight
        patch:
          scale_off: "true"

message: 'Cron.__init__() got an unexpected keyword argument ''days'''

{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() got an unexpected keyword argument 'days'", "timestamp": "2025-09-04T01:33:15.516580+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

Bad scheduled override config with missing cron setting: `expression`

spec:
  overrides:
      bad-override-missing-cron-field:
        timezone: UTC
        cron:
          duration: fortnight
        patch:
          scale_off: "true"

message: 'Cron.__init__() missing 1 required positional argument: ''expression'''

{
  "message": "Handler 'on_change' failed with an exception. Will retry.", 
  "exc_info": "TypeError: Cron.__init__() missing 1 required positional argument: 'expression'", 
  "timestamp": "2025-09-04T01:48:17.580770+00:00", 
  "object": 
  {
    "apiVersion": "hydrolix.io/v1", 
    "kind": "HydrolixCluster", 
    "name": "hdx", 
    "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
    "namespace": "docs-sandbox"
  }, 
  "severity": "error"
}

Bad scheduled override config with invalid values for window fields

spec:
  overrides:
      bad-override-invalid-values:
        timezone: UTC
        window:
          start: preferably-soon        <-- invalid
          end: in-the-future            <-- invalid
        patch:
          scale_off: "true"

message: 'Invalid isoformat string: ''preferably-soon'''

{
  "message": "Timer 'overrides' failed with an exception. Will retry.", 
  "exc_info": "ValueError: Invalid isoformat string: 'preferably-soon'", 
  "timestamp": "2025-09-05T17:27:30.827272+00:00", 
  "object": 
          {
            "apiVersion": "hydrolix.io/v1", 
            "kind": "HydrolixCluster", 
            "name": "hdx", 
            "uid": "06f69fc8-7006-4e36-9493-2071a2356dc7", 
            "namespace": "docs-sandbox"
          }, 
  "severity": "error"
}

The operator writes the list of currently active override names to .status.activeOverrides in the Hydrolix cluster configuration. Once a cluster's clock time lies within the schedule blocks for an override, check the hdx configuration to find a list of active overrides:

Status:
  Active Overrides:
    winter-freeze
    nightly-intake-scale-down

When duplicate keys exist in multiple scheduled overrides, they are applied in descending order as listed by .status.activeOverrides. As a result, the latest-applied active override wins. In the Example of multiple overrides, any keys updated by both the winter-freeze and nightly-intake-scale-down overrides will conform to the value set by nightly-intake-scale-down.