Skip to content

Pod Anti-Affinity Rules

This feature was introduced in Hydrolix version 5.9.4.

The Hydrolix operator automatically applies preferred pod anti-affinity rules to several infrastructure components. These rules instruct the Kubernetes scheduler to spread pods across nodes, reducing the risk of a single node failure taking down an entire component.

Components with default anti-affinity⚓︎

The following components receive pod anti-affinity rules by default.

Component Rule type Application
ZooKeeper Preferred Applied to all replica counts
RabbitMQ Preferred Applied to all replica counts
Redpanda Preferred Applied to all replica counts
Traefik Preferred Applied to all replica counts
Keycloak Required Applied only when replicas > 1; omitted for single-replica deployments to avoid scheduling conflicts

Preferred, not required⚓︎

With the exception of Keycloak, anti-affinity rules are preferred, not required. The scheduler only schedules pods on the same node if no other option is available. This ensures that pods are always scheduled, even on smaller, resource-constrained clusters.

Keycloak's required anti-affinity rule means that scaling to multiple replicas requires at least as many nodes as replicas. If the cluster cannot satisfy the rule, Keycloak pods will remain in Pending state until a suitable node is available.

Default pod anti-affinity expression⚓︎

The default pod anti-affinity rules target "topologyKey": "kubernetes.io/hostname" as their goal is to spread components across nodes.

default pod_anti_affinity rule as applied to Zookeeper
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
  spec:
    targeting:
      zookeeper:
        pod_anti_affinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
              weight: 50

How default rules merge with spec.targeting⚓︎

When building a pod spec, targeting rules are resolved in a fixed priority order. The operator default is applied last, only where the user has not already supplied a value.

Priority order (highest to lowest):

  1. Pool-specific selector
  2. Component-specific selector (for example, zookeeper)
  3. * (global selector)
  4. Operator default

Example 1: User rules and operator default coexist⚓︎

This cluster spec defines a global node_affinity rule under *, which is applied to all components. Additionally, there is no pod_anti_affinity override set under any selector, so operator defaults are applied to all components that have them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
  spec:
    targeting:
      '*':
        node_affinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-role
                    operator: In
                    values: [hydrolix]

Example 2: Component selector overrides global selector⚓︎

If you define the same field at both the global and component-specific level, the component-specific value takes precedence. In this example, Zookeeper receives three targeting settings:

  • the global node_selector (node-role: hydrolix)
  • the component-specific node_selector, which overrides the global
  • the default pod anti-affinity rule (no value explicitly defined in the spec)

Other components receive the node-role defined at the global level and the default pod anti-affinity rule.

1
2
3
4
5
6
7
8
  spec:
    targeting:
      '*':
        node_selector:
          node-role: hydrolix
      zookeeper:
        node_selector:
          node-role: hydrolix-zookeeper

Override pod anti-affinity logic⚓︎

To replace the operator default for a component, supply your own pod anti-affinity setting under that component's targeting key.

In this example, the default hostname-spread rule is overridden by a zone-spread rule for the Zookeeper component. The user has defined a pod_anti_affinity rule that uses topology key topology.kubernetes.io/zone, overriding the default rule.

All other components retain the default pod anti-affinity rules.

Override Zookeeper Pod Anti-Affinity Default
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
  spec:
    targeting:
      zookeeper:
        pod_anti_affinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values: [zookeeper]
                topologyKey: topology.kubernetes.io/zone

Global selector applies to all components

If you define pod_anti_affinity under *, then that rule will apply to all components, including those that have no built-in default.

It's safer to use component-specific levels to override built-in defaults without affecting other components.

Remove a default⚓︎

To opt out of the operator default entirely, set pod_anti_affinity: {} for the relevant selector.

1
2
3
4
  spec:
    targeting:
      zookeeper:
        pod_anti_affinity: {}

Do not opt out under the global selector

Opting out of built-in pod anti-affinity rules breaks the intended availability protections for the Keycloak pod. Restarts and upgrades require Keycloak pods to co-exist on separate nodes. Use component-level selectors to remove pod anti-affinity defaults.

External resources⚓︎