Symptoms and Fixes
This page contains troubleshooting symptoms, grouped by product area. A symptom is an observable problem, such as an HTTP 429 from ingest or a query timeout. Each entry provides either a fix or more detail about it.
To use this page⚓︎
Select a symptom to open its full detail and its fix.
Confirm first: the Confirm column tells you whether a symptom is the one you're getting, using a log line, a metric, a query result, or a command and its output. Several symptoms produce similar failures.
Required access: the permissions an entry's steps need.
Cluster The steps need permissions on the cluster itself: to run kubectl, read pod logs or the cluster's metrics, or change the Hydrolix cluster manifest. If you're a managed customer, contact Hydrolix Support.
User The steps don't need any permissions on the cluster. A Hydrolix user can follow them through the API or a client.
When nothing matches, see System Components to identify the component involved. For more assistance, contact Hydrolix Support.
Not every entry is an error
Entries marked record a retried attempt, a canceled step, or a client that disconnected, while the operation itself succeeded. Confirm the outcome before acting on the log line.
Access and Permissions⚓︎
The Config API authenticates every request and decides what the caller can see. Authentication failures all return the same 401 Unauthorized with Invalid authentication credentials provided, so the message alone won't tell you which of these it is. The turbine-api pod logs carry the underlying error.
| Symptom | Confirm |
|---|---|
| A valid federated token has no matching Hydrolix user | The turbine-api logs show User not found. No external user has both an issuer matching the... |
Every request from one token issuer returns 401 |
The turbine-api pod can't reach the issuer's jwks_uri. Requests from other issuers still succeed... |
| A row policy doesn't restrict the rows a user sees | The roles attached to the row policy don't overlap the roles attached to the user. A policy reaches a... |
| A service account token stops working | The turbine-api logs show Token claimset invalid, or service account token has been revoked.... |
| A valid token is rejected even though the issuer is configured | audience is absent from the issuer entry in turbine_api_trusted_token_issuers, and the identity... |
Data Sources⚓︎
A data source sends data in from outside the cluster, with a token, a table name, and a transform mapping its fields to table columns. CDNs, log forwarders, and cloud services are all data sources. Most post to the streaming ingest API, but some use another method, such as Mux over Kinesis.
| Symptom | Confirm |
|---|---|
| An integration's requests fail with an HTTP error | 401 means the token is invalid, missing, or expired. 404 means the path, or the table and transform... |
| An integration sends data but nothing arrives | The sender reports no error but the table stays empty. Table health reports an Ingest Latency... |
Load Data⚓︎
Data reaches a table by streaming, batch, or manual ingest, and each path ends with an indexer writing partitions. For the systems that send the data, see Data Sources.
| Symptom | Confirm |
|---|---|
| Ingested data doesn't appear in the table | Intake head logs carry Error level messages. Common causes: the primary column's datetime format, a... |
| Ingest returns HTTP 400 | The response body names the problem, such as no destination table specified... |
| Ingest returns HTTP 429 | sum by (service) (rate(http_source_request_count{status_code="429"}[30m])) returns a non-zero rate... |
| Ingest returns HTTP 503 | sum by (service) (rate(http_source_request_count{status_code="503"}[30m])) returns a non-zero rate... |
| Ingest clients get a connection timeout | kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP... |
| Intake pods are OOM-killed | kubectl get events --field-selector reason=OOMKilling names intake-head, kinesis-peer... |
Manage and Scale Clusters⚓︎
Covers the operator, which reconciles the cluster spec; the validating webhook, which checks spec changes; and Traefik, which terminates TLS at the cluster edge.
| Symptom | Confirm |
|---|---|
| Clients can't reach the cluster at all | The requesting IP is absent from the allowlist in hydrolixcluster.yaml. |
| A component won't scale | kubectl logs -l app=operator --tail=100 ends without errors and... |
| A spec change is refused | kubectl describe validatingwebhookconfigurations shows Failure Policy: Fail... |
| TLS fails at the cluster edge | kubectl -n <namespace> get secret traefik-tls -o yaml returns no secret, or the certificate it holds is... |
Manage Data⚓︎
These symptoms involve the jobs that maintain data after ingest, such as merge.
| Symptom | Confirm |
|---|---|
| Merge is falling behind | sum by (target) (candidates{app="merge-controller"}) climbs and stays high while active_merges... |
| Merge peers are OOM-killed | kubectl describe pod <pod-name> \| grep -A 3 "Last State" reports OOMKilled. Identify the container... |
| Orphaned partition files stay in object storage | bulk_delete_success_count{app="partition-cleaner"} is flat or missing... |
Observability⚓︎
Intake pool monitoring posts a timestamp to the hydro.monitor table once a second per ingest pool. The gap between timestamps indicates whether a pool is keeping up. The service is disabled by default.
| Symptom | Confirm |
|---|---|
| Intake latency spikes without an obvious cause | kubectl top pods -n <namespace> \| grep intake shows pods at their CPU or memory limits. |
No rows arrive in hydro.monitor |
kubectl get hdx -n <namespace> -o yaml \| grep monitor_ingest shows the service disabled... |
| Heartbeats stop arriving from a pool | The pool's most recent hydro.monitor timestamp is older than the threshold for its submission interval. |
| A pool is missing from monitoring results | The pool's service type isn't monitored, or the pool is on the exemption list. |
Query and Explore⚓︎
The query head receives a query and delegates partition work to query peers, which fetch partitions from storage and return results. Peer membership comes from either ZooKeeper or the hdx-query-discovery sidecar, depending on cluster configuration, which is why several confirmations check peer membership rather than the query pods.
| Symptom | Confirm |
|---|---|
| Queries return HTTP 503 | kubectl get deployment/query-head -o wide or kubectl get deployment/query-peer -o wide shows fewer... |
| Queries are rejected as unauthorized | The cluster spec requires authentication or TLS and the client sends neither. |
| A query is cut off by a circuit breaker | The query returns a circuit breaker error naming the limit crossed. |
| Query clients get a connection timeout | kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP... |
| Queries fail with a database exception | The query returns a database exception, not a syntax or authorization error. |
| Queries lose the database connection | The query head logs CatalogError, or kubectl describe statefulset postgres shows no ready replica. |
| Queries fail with a database timeout | kubectl get statefulset/zookeeper shows fewer than one ready replica. |
| Queries fail on a project or table name | The project and table exist and the query names them exactly. |
Queries fail with HdxBadPartitionError |
The error names the partition that couldn't be read. A query peer raises this for any failure reading... |
| The query head can't reach a query peer | The error names HdxPeerSource. Connection refused means the peer isn't listening, connect timed out... |
The query head logs NETWORK_ERROR writing to a client |
The message names a socket write, such as Connection reset by peer, while writing to socket... |
Query peers log errors after a successful LIMIT query |
The query head logs success for the same query the peers logged errors for. The head cancels its peers... |
| Logs show network errors but the query succeeded | Query stats report query_attempts greater than 1. The query head retries... |