Symptoms and Fixes
Each entry pairs a symptom with the evidence that confirms it: a log line, a metric, a query result, or a command whose output tells you whether this is what you're seeing. Follow a symptom to read the full evidence and the fix.
Not every entry is an error
Entries marked record a retried attempt, a canceled step, or a client that disconnected, while the operation itself succeeded. Confirm the outcome before acting on the log line.
Access and Permissions⚓︎
The Config API authenticates every request and decides what the caller can see. Authentication failures all return the same 401 Unauthorized with Invalid authentication credentials provided, so the message alone won't tell you which of these it is. The turbine-api pod logs carry the underlying error.
| Symptom | Confirm |
|---|---|
| A valid federated token has no matching Hydrolix user | The turbine-api logs show User not found. No external user has both an issuer matching the... |
Every request from one token issuer returns 401 |
The turbine-api pod can't reach the issuer's jwks_uri. Requests from other issuers still succeed... |
| A row policy doesn't restrict the rows a user sees | The roles attached to the row policy don't overlap the roles attached to the user. A policy reaches a... |
| A service account token stops working | The turbine-api logs show Token claimset invalid, or service account token has been revoked.... |
| A valid token is rejected even though the issuer is configured | audience is absent from the issuer entry in turbine_api_trusted_token_issuers, and the identity... |
Data Sources⚓︎
A data source sends data in from outside the cluster, with a token, a table name, and a transform mapping its fields to table columns. CDNs, log forwarders, and cloud services are all data sources. Most post to the streaming ingest API, but some use another method, such as Mux over Kinesis.
| Symptom | Confirm |
|---|---|
| An integration's requests fail with an HTTP error | 401 means the token is invalid, missing, or expired. 404 means the path, or the table and transform... |
| An integration sends data but nothing arrives | The sender reports no error but the table stays empty. Table health reports an Ingest Latency... |
Load Data⚓︎
Data reaches a table by streaming, batch, or manual ingest, and each path ends with an indexer writing partitions. For the systems that send the data, see Data Sources.
| Symptom | Confirm |
|---|---|
| Ingested data doesn't appear in the table | Intake head logs carry Error level messages. Common causes: the primary column's datetime format, a... |
| Ingest returns HTTP 429 | sum by (service) (rate(http_source_request_count{status_code="429"}[30m])) returns a non-zero rate... |
| Ingest returns HTTP 503 | sum by (service) (rate(http_source_request_count{status_code="503"}[30m])) returns a non-zero rate... |
| Clients time out reaching the ingest endpoint | kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP... |
| Intake pods are OOM-killed | kubectl get events --field-selector reason=OOMKilling names intake-head, kinesis-peer... |
Manage and Scale Clusters⚓︎
Covers the operator, which reconciles the cluster spec; the validating webhook, which checks spec changes; and Traefik, which terminates TLS at the cluster edge.
| Symptom | Confirm |
|---|---|
| Clients can't reach the cluster at all | The requesting IP is absent from the allowlist in hydrolixcluster.yaml. |
| A component won't scale | kubectl logs -l app=operator --tail=100 ends without errors and... |
| A spec change is refused | kubectl describe validatingwebhookconfigurations shows Failure Policy: Fail... |
| TLS fails at the cluster edge | kubectl -n <namespace> get secret traefik-tls -o yaml returns no secret, or the certificate it holds is... |
Manage Data⚓︎
Merge combines partitions to maintain query performance and storage efficiency.
| Symptom | Confirm |
|---|---|
| Merge is falling behind | sum by (target) (active_merges{app="merge-controller"}) and candidate counts stay high while... |
| Merge peers are OOM-killed | kubectl describe pod <pod-name> \| grep -A 3 "Last State" reports OOMKilled. Identify the container... |
Observability⚓︎
Intake pool monitoring posts a timestamp to the hydro.monitor table once a second per ingest pool. The gap between timestamps indicates whether a pool is keeping up. The service is disabled by default.
| Symptom | Confirm |
|---|---|
| Intake latency spikes without an obvious cause | kubectl top pods -n <namespace> \| grep intake shows pods at their CPU or memory limits. |
No rows arrive in hydro.monitor |
kubectl get hdx -n <namespace> -o yaml \| grep monitor_ingest shows the service disabled... |
| Heartbeats stop arriving from a pool | The pool's most recent hydro.monitor timestamp is older than the threshold for its submission interval. |
| A pool is missing from monitoring results | The pool's service type isn't monitored, or the pool is on the exemption list. |
Query and Explore⚓︎
The query head receives a query and delegates partition work to query peers, which fetch partitions from storage and return results. Peer membership comes from either ZooKeeper or the hdx-query-discovery sidecar, depending on cluster configuration, which is why several confirmations check peer membership rather than the query pods.
| Symptom | Confirm |
|---|---|
| Queries return HTTP 503 | kubectl get deployment/query-head -o wide or kubectl get deployment/query-peer -o wide shows fewer... |
| Queries are rejected as unauthorized | The cluster spec requires authentication or TLS and the client sends neither. |
| A query is cut off by a circuit breaker | The query returns a circuit breaker error naming the limit crossed. |
| Clients time out reaching the query endpoint | kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP... |
| Queries fail with a database exception | The query returns a database exception, not a syntax or authorization error. |
| Queries lose the database connection | The query head logs CatalogError, or kubectl describe statefulset postgres shows no ready replica. |
| Queries time out in the database | kubectl get deployment/zookeeper -o wide shows fewer than one available replica. |
The query head logs NETWORK_ERROR writing to a client |
The message names a socket write, such as Connection reset by peer, while writing to socket... |
| Queries fail on a project or table name | The project and table exist and the query names them exactly. |
Queries fail with HdxBadPartitionError |
The error names the partition that couldn't be read. A query peer raises this for any failure reading... |
Query peers log errors after a successful LIMIT query |
The query head logs success for the same query the peers logged errors for. The head cancels its peers... |
| The query head can't reach a query peer | The error names HdxPeerSource. Connection refused means the peer isn't listening, connect timed out... |
| Logs show network errors but the query succeeded | Query stats report query_attempts greater than 1. The query head retries... |