Query and Explore
The query head receives a query and delegates partition work to query peers, which fetch partitions from storage and return results. Peer membership comes from either ZooKeeper or the hdx-query-discovery sidecar, depending on cluster configuration, which is why several confirmations check peer membership rather than the query pods.
Failures fall into three layers: connection, database, and query execution. The error identifies the layer.
Queries return HTTP 400⚓︎
Required access: User
Confirm: The response body is JSON with an error field that describes the problem.
Query head returns HTTP 400 for every query error, including failed authentication. Use the message
in the response body to find the cause. The X-ClickHouse-Exception-Code response header also holds
an error code for ClickHouse exceptions. A proxy in front of query head, such as http-proxy, can
return its own status instead.
If the query sets hdx_query_streaming_result=true, the response returns a 200 even if an error
occurs after results start streaming. The response ends early instead. The client reports an
incomplete transfer, and query head logs Query error after response started streaming.
Fix: find the message in the error field.
Authentication failed: the cluster doesn't recognize the token. The message comes from ClickHouse and mentions passwords even when the request sent a token. Get a new token. See Authenticate to the API.User token expired: get a new token, as in the previous item.api login failed with provided username/password: the username or password is wrong, or the request had no credentials. See Queries are rejected as unauthorized.UNKNOWN_TABLEordb='x' does not exist: the table or project name is wrong. See Queries fail on a project or table name.Syntax error: the query isn't valid SQL. The message gives the position where parsing failed.- Any other message: see Queries fail with a database exception.
Queries return HTTP 503⚓︎
Required access: Cluster
Confirm: kubectl get deployment/query-head -o wide or kubectl get deployment/query-peer -o wide shows fewer than one available replica.
Fix: Query Debugging: HTTP 503 Service Temporarily Unavailable.
Queries are rejected as unauthorized⚓︎
Required access: Cluster
Confirm: The cluster spec requires authentication or TLS and the client sends neither.
Fix: Query Debugging: Client authorization error.
A query is cut off by a circuit breaker⚓︎
Required access: User
Confirm: The query returns a circuit breaker error naming the limit crossed.
Fix: Query Debugging: Query circuit breaker errors.
Query clients get a connection timeout⚓︎
Required access: Cluster
Confirm: kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP is absent from the allowlist.
Fix: Query Debugging: Client connection timeout.
Queries fail with a database exception⚓︎
Required access: User
Confirm: The query returns a database exception, not a syntax or authorization error.
Fix: Query Debugging: Database exceptions.
Queries lose the database connection⚓︎
Required access: Cluster
Confirm: The query head logs CatalogError, or kubectl describe statefulset postgres shows no ready replica.
Queries already assigned to peers can still finish while the catalog is unreachable. New queries fail, because the query head can't plan without the catalog.
Fix: Query Debugging: Database lost connection.
Queries fail with a database timeout⚓︎
Required access: Cluster
Confirm: kubectl get statefulset/zookeeper shows fewer than one ready replica.
Fix: Query Debugging: Database timeout.
Queries fail on a project or table name⚓︎
Required access: User
Confirm: The project and table exist and the query names them exactly.
Fix: Query Debugging: Syntax and user query errors.
Queries fail with HdxBadPartitionError⚓︎
Required access: Cluster
Confirm: The error names the partition that couldn't be read. A query peer raises this for any failure reading partition data from object storage, network errors included.
The error covers every read failure on the peer-to-storage hop, so a storage outage and a genuinely corrupt partition surface the same way. The named partition is the starting point.
Detail: Query System Network Errors: Query peer to storage.
The query head can't reach a query peer⚓︎
Required access: Cluster
Confirm: The error names HdxPeerSource. Connection refused means the peer isn't listening, connect timed out means it never finished connection setup, and Timeout exceeded while reading from socket means it was too slow to respond. The first two arrive as NETWORK_ERROR, the third as SOCKET_TIMEOUT.
The query engine can re-wrap these, so the outer code isn't a reliable way to tell them apart. The text after the code is, and that determines whether to look at peer availability or at peer load.
Detail: Query System Network Errors: Query head to peer.
Not an error⚓︎
These look like failures in a log but don't mean anything is wrong. Check them before opening an investigation.
The query head logs NETWORK_ERROR writing to a client⚓︎
Required access: Cluster
Confirm: The message names a socket write, such as Connection reset by peer, while writing to socket or I/O error: Broken pipe, while writing to socket. The client closed the connection, so the cluster is reporting rather than failing.
A client that disconnects mid-result leaves this behind. The query head then cancels the peers it had connected, so the same event can produce peer errors too. Treat it as an error only when clients report failures they didn't cause.
Detail: Query System Network Errors: Client to query head.
Query peers log errors after a successful LIMIT query⚓︎
Required access: Cluster
Confirm: The query head logs success for the same query the peers logged errors for. The head cancels its peers once it has enough rows to satisfy the query.
Expected behavior, not an error. On LIMIT and the latest-N-rows optimization the
query head stops peers early, and a canceled peer logs an error even though the
query returned a correct result.
Detail: Query System Network Errors: Query success.
Logs show network errors but the query succeeded⚓︎
Required access: User
Confirm: Query stats report query_attempts greater than 1. The query head retries NETWORK_ERROR, ATTEMPT_TO_READ_AFTER_EOF, and SOCKET_TIMEOUT, three attempts by default, so these appear in logs for queries that then succeed.
These errors are recoverable per attempt, not per query. The query head retries and
often succeeds, so their presence in a log isn't by itself evidence that anything
failed. Check query_attempts in the query stats before investigating further.