Skip to content

Assigning Pods to Nodes

Use the targeting key in the Hydrolix spec to influence the Kubernetes scheduler.

How Kubernetes assigns pods to nodes⚓︎

The Kubernetes scheduler assigns pods to nodes based on resource availability and configurable placement constraints.

Available constraint mechanisms:

  • Node name: Direct node selection using the nodeName field. The pod runs only on the named node.
  • Node selector: A key-value label match. The scheduler places a pod only on nodes whose labels match the specified pairs.
  • Node affinity: A more expressive constraint system that supports hard requirements (requiredDuringSchedulingIgnoredDuringExecution) and soft preferences (preferredDuringSchedulingIgnoredDuringExecution).
  • Pod affinity: Constrains pod placement based on labels of pods already running on a node. For example, pod affinity can ensure that two frequently communicating services are scheduled in the same availability zone.
  • Pod anti-affinity: Constrains pod placement to avoid co-locating certain pods on the same node or topology domain.
  • Pod topology spread constraints: Controls how pods spread across failure domains such as regions, zones, and nodes.

For more detail, see Kubernetes: Assigning Pods to Nodes.

Use targeting in Hydrolix⚓︎

To make use of the Kubernetes scheduling mechanisms, use the spec.targeting key in the HydrolixCluster manifest.

Scope any mechanism using rules that apply to all pods, a service, or a pool. See Targeting priority for resolution of overlapping rules.

Automatic pod anti-affinity rules for availability

The Hydrolix operator automatically applies pod anti-affinity rules to several infrastructure components. These rules instruct the scheduler to spread pods across nodes, reducing the risk of a single node failure taking down an entire component.

Examples in this page demonstrate and explain the use of Kubernetes scheduling mechanisms in a Hydrolix cluster.

Node name⚓︎

Configure Node Name
1
2
3
4
5
6
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    query-head:
      node_name: gke-hydrolix-demo-4a2b59be-clnb

This example pins the query-head pod to a specific node by name. The pod is only scheduled on the node gke-hydrolix-demo-4a2b59be-clnb and remains pending if that node is unavailable.

See also Kubernetes: Node Name.

Node selector⚓︎

Configure Node Selector
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    '*':
      node_selector:
        node.kubernetes.io/instance-type: g6-dedicated-16
    init-turbine-api:
      node_selector:
        cloud.google.com/machine-family: n2

In this example, every pod must be scheduled on a node with the label node.kubernetes.io/instance-type: g6-dedicated-16. If there aren't enough nodes with the matching label, some pods remain in a pending state.

See also Kubernetes: Node Selector.

Node affinity⚓︎

Configure Node Affinity
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    intake-head:
      node_affinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                    - us-central1-f

This example restricts all intake-head pods to nodes in the us-central1-f availability zone. Because this uses requiredDuringSchedulingIgnoredDuringExecution, pods remain pending if no nodes in that zone are available.

See also Kubernetes: Node Affinity.

Pod affinity⚓︎

Configure Pod Affinity
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    intake-head:
      pod_affinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: app
                  operator: In
                  values:
                    - query-head
            topologyKey: topology.kubernetes.io/zone

This example requires that intake-head pods be scheduled in the same availability zone as pods with the label app: query-head. This is useful when two services communicate frequently and benefit from low-latency, same-zone placement.

See also Kubernetes: Pod Affinity.

Pod anti-affinity⚓︎

Configure Pod Anti-affinity
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    postgres:
      pod_anti_affinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: security
                    operator: In
                    values:
                      - S2
              topologyKey: topology.kubernetes.io/zone

This example expresses a soft preference to avoid scheduling postgres pods in the same availability zone as pods with the label security: S2. Because this uses preferredDuringSchedulingIgnoredDuringExecution with a weight, the scheduler tries to honor the constraint but can still co-locate the pods if necessary. See Pod Anti-affinity Rules for more information and additional examples.

See Kubernetes: Pod Anti-affinity

Pod topology spread constraints⚓︎

Configure Pod Topology Spread Constraints
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    batch-head:
      topology_spread_constraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: batch-head

This example distributes batch-head pods evenly across availability zones, allowing a maximum skew of 1 pod between any two zones. If an even spread isn't possible, DoNotSchedule prevents the scheduler from placing additional pods.

See Kubernetes: Pod Topology Spread Constraints.

Targeting priority⚓︎

Targeting keys fall into three categories, applied in order of increasing priority:

  1. Default ('*'): Global defaults applied to all services. Lowest priority.
  2. Service name (for example, intake-head): Overrides the global defaults for every pod belonging to that service, including all of its pools.
  3. Pool name (for example, pool:hydrologs-intake-head): Overrides both the global and service-level configuration for pods in that specific pool only. Highest priority.
Default Targeting
1
2
3
4
5
6
7
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    '*':
      node_selector:
        node.kubernetes.io/instance-type: g6-dedicated-16
Service Targeting
1
2
3
4
5
6
7
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    query-peer:
      node_selector:
        node.kubernetes.io/instance-type: g6-dedicated-32
Pool Targeting
1
2
3
4
5
6
7
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
  targeting:
    pool:my-custom-query-peer-pool:
      node_selector:
        node.kubernetes.io/instance-type: g6-dedicated-50

When the operator reconciles a pooled service, it merges these entries with the highest priority winning. A value set at a more specific level replaces the same value from a less specific level.

The following example demonstrates targeting priority with intake-head services.

Intake head priority example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
spec:
  targeting:
    # All intake-head pods across all pools: us-central1-f
    intake-head:
      node_affinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                    - us-central1-f

    # hydrologs-intake-head: us-central1-b (overrides the service-level configuration)
    pool:hydrologs-intake-head:
      node_affinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                    - us-central1-b

The intake-head entry schedules all intake-head pods, across every pool, into us-central1-f. The pool:hydrologs-intake-head entry overrides that for a single pool, placing its pods in us-central1-b. All other intake-head pools remain in us-central1-f.

Apply targeting to the operator⚓︎

The spec.targeting configuration is automatically applied to Hydrolix service workloads during the operator's reconciliation loop. However, the operator itself runs as a separate Kubernetes Deployment.

To apply the same targeting rules to the operator's own pod, pass the cluster configuration file when generating the operator manifests:

Use HKW to Apply Targeting Rules to All Deployments
1
2
curl "https://hkw.hydrolix.live/{VERSION}/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}" > operator.yaml
kubectl apply -f operator.yaml