Manage and Scale Clusters

Covers the operator, which reconciles the cluster spec; the validating webhook, which checks spec changes; and Traefik, which terminates TLS at the cluster edge.

Failures appear as spec changes with no effect, or clients unable to connect. Check here first when ingest and queries fail together.

Clients can't reach the cluster at all

Confirm: The requesting IP is absent from the allowlist in hydrolixcluster.yaml.

A cluster with IP access controls rejects any address outside its allowlist, so every client fails the same way rather than one failing differently.

Fix. Add the address to the allowlist in the cluster spec:

kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
# add the IPs, then
kubectl apply -f hydrolixcluster.yaml

See Configure IP access for the spec fragments, and for what a public cluster allows compared with a limited-access one.

A component won't scale

Confirm: kubectl logs -l app=operator --tail=100 ends without errors and includes Resuming is processed: 1 succeeded; 0 failed. Its absence indicates the operator isn't reconciling.

The operator reconciles the cluster spec into running pods. When it stops reconciling, a replica change in hydrolixcluster.yaml has no effect and no error is raised.

Fix. Work through three causes in order.

The operator isn't reconciling. Restart it:

kubectl rollout restart deployment operator

There's no room to schedule new pods. kubectl describe nodes shows whether the cluster can place them, and a node that can't be added stalls a scale-up regardless of the spec.

Scaling is turned off cluster-wide. scale_off: true in hydrolixcluster.yaml makes the scaler ignore every service:

kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
# set scale_off: false, then
kubectl apply -f hydrolixcluster.yaml

Prevent. See Scale profiles for the replica and resource defaults each profile sets, so a manual change isn't competing with a profile.

A spec change is refused

Confirm: kubectl describe validatingwebhookconfigurations shows Failure Policy: Fail on hdx-validator-<namespace>. In Fail mode the webhook rejects invalid changes instead of warning.

Fix: Hydrolix Spec Configuration Validator: Fail mode.

TLS fails at the cluster edge

Confirm: kubectl -n <namespace> get secret traefik-tls -o yaml returns no secret, or the certificate it holds is expired or carries the wrong chain order.

Fix: Certificate Troubleshooting.

Back to all symptoms