Skip to content

Pod Priority Classes

Overview⚓︎

Hydrolix uses Kubernetes priority and preemption for pod scheduling and eviction. If Kubernetes can't schedule a pod, it preempts lower-priority pods in favor of higher-priority pending pods requesting scheduling.

Every Hydrolix workload is assigned a default priority class. The operator applies these assignments during deployment.

Terminology⚓︎

Term Description
PriorityClass A non-namespaced Kubernetes object that maps a priority class name to an integer priority value.
Priority Indicates the importance of a pod relative to other pods. Higher values mean higher priority.
Preemption Policy Determines whether lower-priority pods are evicted to accommodate higher-priority pods. Allowed values are Never and PreemptLowerPriority, default for all custom Hydrolix priority classes.

Priority classes⚓︎

Hydrolix uses these priority classes:

Name (priorityClassName) Value Preemption policy Source
system-node-critical 2_000_001_000 PreemptLowerPriority Kubernetes built-in
system-cluster-critical 2_000_000_000 PreemptLowerPriority Kubernetes built-in
hdx-critical 10_000 PreemptLowerPriority Custom Hydrolix priority class
vector 10_000 PreemptLowerPriority Custom Hydrolix priority class
hdx-highest 9_900 PreemptLowerPriority Custom Hydrolix priority class
hdx-high 9_800 PreemptLowerPriority Custom Hydrolix priority class
hdx-medium 9_700 PreemptLowerPriority Custom Hydrolix priority class
hdx-low 9_600 PreemptLowerPriority Custom Hydrolix priority class
hdx-lowest 9_500 PreemptLowerPriority Custom Hydrolix priority class

All custom Hydrolix priority classes use PreemptLowerPriority, meaning higher-priority pods can evict lower-priority pods when resources are constrained.

Determine priority assignment⚓︎

List the set of priority classes used by Hydrolix using the following command.

1
kubectl get priorityclasses

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
NAME                      VALUE        GLOBAL-DEFAULT   AGE     PREEMPTIONPOLICY
hdx-critical              10000        false            237d    PreemptLowerPriority
hdx-high                  9800         false            237d    PreemptLowerPriority
hdx-highest               9900         false            237d    PreemptLowerPriority
hdx-low                   9600         false            237d    PreemptLowerPriority
hdx-lowest                9500         false            237d    PreemptLowerPriority
hdx-medium                9700         false            237d    PreemptLowerPriority
system-cluster-critical   2000000000   false            2y10d   PreemptLowerPriority
system-node-critical      2000001000   false            2y10d   PreemptLowerPriority
vector                    10000        false            2y10d   PreemptLowerPriority

To see a listing of all pods by priority class, run the following command:

1
kubectl get pods -o custom-columns="PRIORITY_CLASS:.spec.priorityClassName,NAME:.metadata.name" --sort-by=.spec.priorityClassName

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
PRIORITY_CLASS         NAME
hdx-critical           turbine-api-df54f757-cbv6m
hdx-critical           grafana-f4fb568cf-6vklr
hdx-critical           zookeeper-0
hdx-critical           prometheus-0
hdx-critical           pushgateway-5bd8f76b97-h5jdg
hdx-critical           postgres-0
hdx-critical           elasticsearch-es-default-0
hdx-critical           traefik-cc98df5dd-cw7bw
hdx-critical           hdx-scaler-77fd5ffcd-vfp94
hdx-critical           rabbitmq-0
hdx-critical           operator-ddb48f48c-ftvjj
hdx-critical           superset-75f9fb8f96-ck955
hdx-critical           kibana-kb-7987947cbf-4gblk
hdx-critical           tooling-67c5b99b7-fnrrw
hdx-critical           keycloak-5f856c5965-sszpm
hdx-critical           hdx-scaler-go-555b4567d-w7vjs
hdx-high               job-purge-29823960-vqpqq
hdx-high               merge-peer-5d5bb694b-gqkvs
hdx-high               turbine-api-worker-5d958d5b66-hs2ph
hdx-high               usagemeter-29824880-67grw
hdx-high               intake-api-5ff7487586-jl8xl
hdx-high               init-turbine-api-v6-4-0-b3a335f1-c99fq
hdx-high               mcp-hydrolix-6cfccccc8d-hk7km
hdx-high               merge-cleanup-29824880-ncx9n
hdx-high               task-monitor-29824884-zfbrb
hdx-high               check-bucket-access-vxorgpgg5-9t9pp
hdx-high               task-monitor-29824882-tm7rn
hdx-high               init-cluster-v6-4-0-3a8acc70-pkthn
hdx-high               hydrologs-intake-head-5b65ff778-tjntl
hdx-high               stale-job-monitor-29824880-nj2b6
hdx-high               batch-head-7c5cfff6b4-dztrr
hdx-high               decay-29823840-vp4rz
hdx-high               cert-expiry-check-29823900-crxm6
hdx-high               prune-locks-29823870-ggkl8
hdx-highest            intake-head-59c94d5c4c-ffrw9
hdx-highest            intake-peer-5885697496-6fg86
hdx-highest            query-peer-8494c79777-5bh92
hdx-highest            query-head-74d9b5fdc6-fnftz
hdx-highest            merge-controller-7c4cbf5b77-f79hh
hdx-highest            intake-head-df-78cf6c4854-cvr6h
hdx-highest            intake-router-599584d47f-r54lf
hdx-low                periodic-service-74b8c85d65-5f652
hdx-low                batch-peer-7bbfb78bff-9d64v
hdx-low                ui-7c948cf48b-qjzgd
hdx-low                partition-cleaner-6457586b48-6qz8t
hdx-low                merge-peer-ii-dfb69c565-wg5wd
hdx-lowest             merge-peer-iii-6f48fb4b8c-v5gj9
hdx-lowest             validator-57585fdb5b-xjppj
hdx-lowest             version-55bcf45874-z6f78
hdx-lowest             reaper-69bb58fd54-wdtw9
hdx-medium             kibana-gateway-6998c967c6-4xxgs
hdx-medium             reloader-b5787984f-mrwd2
hdx-medium             acme-d6f97578c-cm4n4
system-node-critical   vector-52hcr
system-node-critical   vector-fj8kx
system-node-critical   vector-j2wbz
system-node-critical   vector-qw2nk
system-node-critical   vector-v4pbc
system-node-critical   vector-xv9rk

Non-Hydrolix workloads in the same namespace⚓︎

Hydrolix pods preempt unprotected workloads

If you run additional Kubernetes workloads in the same namespace as Hydrolix, all Hydrolix pods have priority classes that allow them to preempt pods that lack a priority class or have a lower-priority value.

During resource constraints, such as traffic spikes while new nodes are provisioning, the Kubernetes scheduler can evict your non-Hydrolix pods to make room for Hydrolix services.

To protect non-Hydrolix workloads sharing the namespace, assign priority classes to those workloads. You can use any Hydrolix priority class such as hdx-high, or create your own. See the Kubernetes documentation on PriorityClass for instructions.

For instructions on overriding default Hydrolix priority class assignments, see Override pod priority classes.