Query and Explore
The query head receives a query and delegates partition work to query peers, which fetch partitions from storage and return results. Peer membership comes from either ZooKeeper or the hdx-query-discovery sidecar, depending on cluster configuration, which is why several confirmations check peer membership rather than the query pods.
Failures fall into three layers: connection, database, and query execution. The error identifies the layer.
Queries return HTTP 503⚓︎
Confirm: kubectl get deployment/query-head -o wide or kubectl get deployment/query-peer -o wide shows fewer than one available replica.
Fix: Query Debugging: HTTP 503 Service Temporarily Unavailable.
Queries are rejected as unauthorized⚓︎
Confirm: The cluster spec requires authentication or TLS and the client sends neither.
Fix: Query Debugging: Client authorization error.
A query is cut off by a circuit breaker⚓︎
Confirm: The query returns a circuit breaker error naming the limit crossed.
Fix: Query Debugging: Query circuit breaker errors.
Clients time out reaching the query endpoint⚓︎
Confirm: kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP is absent from the allowlist.
Fix: Query Debugging: Client connection timeout.
Queries fail with a database exception⚓︎
Confirm: The query returns a database exception, not a syntax or authorization error.
Fix: Query Debugging: Database exceptions.
Queries lose the database connection⚓︎
Confirm: The query head logs CatalogError, or kubectl describe statefulset postgres shows no ready replica.
Queries already assigned to peers can still finish while the catalog is unreachable. New queries fail, because the query head can't plan without the catalog.
Fix: Query Debugging: Database lost connection.
Queries time out in the database⚓︎
Confirm: kubectl get deployment/zookeeper -o wide shows fewer than one available replica.
Fix: Query Debugging: Database timeout.
Queries fail on a project or table name⚓︎
Confirm: The project and table exist and the query names them exactly.
Fix: Query Debugging: Syntax and user query errors.
Queries fail with HdxBadPartitionError⚓︎
Confirm: The error names the partition that couldn't be read. A query peer raises this for any failure reading partition data from object storage, network errors included.
The error covers every read failure on the peer-to-storage hop, so a storage outage and a genuinely corrupt partition surface the same way. The named partition is the starting point.
Detail: Query System Network Errors: Query peer to storage.
The query head can't reach a query peer⚓︎
Confirm: The error names HdxPeerSource. Connection refused means the peer isn't listening, connect timed out means it never finished connection setup, and Timeout exceeded while reading from socket means it was too slow to respond. The first two arrive as NETWORK_ERROR, the third as SOCKET_TIMEOUT.
The query engine can re-wrap these, so the outer code isn't a reliable way to tell them apart. The text after the code is, and that determines whether to look at peer availability or at peer load.
Detail: Query System Network Errors: Query head to peer.
Not an error⚓︎
These look like failures in a log but don't mean anything is wrong. Check them before opening an investigation.
The query head logs NETWORK_ERROR writing to a client⚓︎
Confirm: The message names a socket write, such as Connection reset by peer, while writing to socket or I/O error: Broken pipe, while writing to socket. The client closed the connection, so the cluster is reporting rather than failing.
A client that disconnects mid-result leaves this behind. The query head then cancels the peers it had connected, so the same event can produce peer errors too. Treat it as an error only when clients report failures they didn't cause.
Detail: Query System Network Errors: Client to query head.
Query peers log errors after a successful LIMIT query⚓︎
Confirm: The query head logs success for the same query the peers logged errors for. The head cancels its peers once it has enough rows to satisfy the query.
Expected behavior, not an error. On LIMIT and the latest-N-rows optimization the
query head stops peers early, and a canceled peer logs an error even though the
query returned a correct result.
Detail: Query System Network Errors: Query success.
Logs show network errors but the query succeeded⚓︎
Confirm: Query stats report query_attempts greater than 1. The query head retries NETWORK_ERROR, ATTEMPT_TO_READ_AFTER_EOF, and SOCKET_TIMEOUT, three attempts by default, so these appear in logs for queries that then succeed.
These errors are recoverable per attempt, not per query. The query head retries and
often succeeds, so their presence in a log isn't by itself evidence that anything
failed. Check query_attempts in the query stats before investigating further.