Scheduled Overrides

Scheduled overrides

You can automate temporary overrides for the scale of the various components in a cluster using scheduled overrides.

Overrides are specified within the Hydrolix spec configuration and take effect when the container's current time falls within the schedule block specified for an override. The HDX custom resource definition itself is never mutated by the contents of the overrides block. Instead, overrides live only in the operator memory.

Requirements

Each override must contain the fields timezone, patch, and exactly one of either weekly, window, or cron.

Field name	Type	Description
`timezone`	string	IANA TZ used with the specified schedule. Defaults to UTC.
`weekly`	Block containing the following fields: `days` (format: `[Sun..Sat]`) `start` (type: string, format: `"HH:MM"`) * `end` (type: string, format: `"HH:MM"`)	One of three possible schedule blocks. Conditionally required if no other schedule block is set.
`window`	Block containing the following fields: `start` (type: ISO-8601 datetime) `end` (type: ISO-8601 datetime)	One of three possible schedule blocks. Conditionally required if no other schedule block is set.
`cron`	Block containing the following fields: `cron expression` (type: cron string `duration` (type: string)	One of three possible schedule blocks. Conditionally required if no other schedule block is set.
`patch`	Dictionary. Must be valid, properly-formatted fields from the list of Hydrolix tunables	The HDX spec definition to apply while active.

Multiple overrides can be active at the same time. When there are duplicate keys updated by multiple overrides, the last key in the config will apply.

Example scheduled override

spec:
  overrides:
    sunday-gameday-scale-postgres-horizontally:
      timezone: America/New_York
      weekly:
        days: ["Sun"]
        start: "16:00"
        end: "22:00"
      patch:
        scale:
          postgres:
            replicas: 999

spec:
  overrides:
    winter-freeze:
      timezone: America/Los_Angeles
      window:
        start: "1999-12-24T18:00:00Z"
        end: "2000-01-02T09:00:00Z"
      patch:
        scale_off: "true"
        celebrate: true

spec:
  overrides:
    nightly-intake-scale-down:
      timezone: America/Los_Angeles
      cron:
        expression: "0 2 * * 1-5"
        duration: "4h"
      patch:
        pools:
          hydrologs-intake-head:
            replicas: "0"

While particularly pertinent to automated scaling, scheduled overrides can be applied to any Hydrolix tunable. See Hydrolix tunables for additional tunable examples.

Limitation

You can't update a scheduled override if the cluster is within the active time window for the override. For example, take the following override:

  overrides:
    test-override:
      patch:
        howdy: doody
        scale_off: "true"
      timezone: America/Los_Angeles
      window:
        start: "2025-07-29T09:58:00Z"
        end: "2025-07-29T17:00:00Z"

Updating this override prior to its start time would ensure the changes are applied as expected within the specified time window.

However, if current time for the cluster is "2025-07-29T10:00:00Z", which lies within the window, updating the override would not apply the changes. To make sure any override changes apply as expected, update the start time to a point in the future. In this example, it would be sufficient to update the window configuration to

window:
  start: "2025-07-29T10:01:00Z"
  end: "2025-07-29T17:00:00Z"

Verification

The operator writes the list of currently active override names to .status.activeOverrides in the Hydrolix cluster configuration. Once a cluster's clock time lies within the schedule blocks for an override, check the hdx configuration to find a list of active overrides:

Status:
  Active Overrides:
    winter-freeze
    nightly-intake-scale-down

Recall that when there are duplicate keys updated by multiple overrides, the last key in the config will apply. In the example above, any keys updated by both the nightly-intake-scale-down and winter-freeze overrides will conform to the value set by nightly-intake-scale-down.