Skip to content

Query Debugging

Endpoint errors⚓︎

Issue: Client connection timeout⚓︎

Check traefik count⚓︎

Check Traefik service count is \< 1.

1
kubectl get deployment/traefik -o wide

Fix traefik count⚓︎

Scale Traefik via hydrolixcluster.yaml.

1
2
traefik:
   replicas: <Scale>

For more information, see Scale Profiles.

Check IP allowlist⚓︎

Check that the IP allowlist contains the requesting IP address.

Download the cluster configuration:

1
kubectl get hydrolixcluster -o yaml

Fix IP allowlist⚓︎

  1. Add the IP to the allowlist:

    1
       kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml

  2. Add the IPs.

  3. Apply the changes to your cluster:

    1
       kubectl apply -f hydrolixcluster.yaml

Find more information on IP allowlists, see Configure IP Access.

Issue: Client authorization error⚓︎

Check auth required⚓︎

Check if Query endpoint authentication is enabled.

Download your cluster configuration:

1
kubectl get hydrolixcluster -o yaml

In the configuration, look for enable_query_auth:

1
2
spec:
  enable_query_auth: true

Fix auth required⚓︎

If enabled, ensure the user is using the correct username and password or bearer token. For more information, see Enable Query Authentication.

If query authentication isn't enabled, enable it.

Check TLS required⚓︎

Check if TLS is enabled.

Download your cluster configuration:

1
kubectl get hydrolixcluster -o yaml

In the configuration, look for use_tls:

1
2
spec:
    use_tls: true

Fix TLS required⚓︎

If TLS is enabled, check the client is connecting using a secure TLS connection.

If TLS isn't enabled, enable it.

If using native protocol connection, secure uses port 9440 and non-secure uses port 9000.

Issue: HTTP 503 Service Temporarily Unavailable⚓︎

For more information, see Scale Profiles.

Check query head count⚓︎

Check query-head count is at a minimum of 1.

1
kubectl get deployment/query-head -o wide

Fix query head count⚓︎

Scale query-head via hydrolixcluster.yaml:

1
2
query-head:
   replicas: <scale number>

Check query peer count⚓︎

Check query-peer count is at a minimum of 1.

1
kubectl get deployment/query-peer -o wide

Fix query peer count⚓︎

Scale query-peer via hydrolixcluster.yaml:

1
2
query-peer:
   replicas: <Scale>

Pools⚓︎

For pools, check that the pool has query-peers.

Issue: Database exceptions⚓︎

For more information, see Scale Profiles.

  • DB::Exception: HdxStorageError: No peers available to run query in pool
  • DB::Exception: Database '_local' doesn't exist
  • DB::NetException: Net Exception: No route to host

Issue: Database timeout⚓︎

DB::NetException: Timeout: connect timed out

Check Zookeeper is running⚓︎

Check ZooKeeper instance count is 3:

1
kubectl get deployment/zookeeper -o wide

Fix Zookeeper is running⚓︎

Scale ZooKeeper with hydrolixcluster.yaml.

1
2
zookeeper:
   replicas: <Scale>

Issue: Database lost connection⚓︎

DB::NetException: Error: Lost connection to the database server. (version 3.x.x)>. (STD_EXCEPTION)

Check query head can reach PostgreSQL⚓︎

This error is uncommon and should only happen when a query head isn’t used for days or weeks.

Fix query head can reach PostgreSQL⚓︎

Retry the query.

If this doesn’t resolve the error, check that the PostgreSQL instance is suitably scaled. Increase CPU or memory if required.

  1. Download the cluster configuration.

    1
       kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml

  2. Update the scale.

  3. Apply the configuration to your cluster.

    1
       kubectl apply -f hydrolixcluster.yaml

Check PostgreSQL is running⚓︎

Check that PostgreSQL is running and in a healthy state.

1
kubectl describe statefulset postgres

Fix PostgreSQL is running⚓︎

Update the scale of PostgreSQL

  1. Download the cluster configuration.

    1
       kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml

  2. Update the scale.

  3. Apply the configuration to your cluster.

    1
       kubectl apply -f hydrolixcluster.yaml

If PostgreSQL is failing, even if resolved, this is a Severity 1 incident. Notify Hydrolix as soon as possible.

Syntax and user query errors⚓︎

Check project and table exist⚓︎

The table or project is unknown.

1
DB::Exception: Table _local.XXXX does not exist

Fix project and table exist⚓︎

Specify the project and table.

1
SELECT x FROM project.table WHERE ...

Check project and table names⚓︎

If the table and/or project has a hyphen in the name. For example, my-table.my-project

1
Expected one of: token, Dot, UUID, alias, AS, identifier, FINAL, SAMPLE, table, table function...

Fix project and table names⚓︎

Add backticks (`) around the table and project names.

1
select x from `my-project`.` my-table` where...

Other errors⚓︎

When a query error occurs with syntax or execution errors, the system can be overly verbose. Errors can be large and contain a lot of information. A good place to start when debugging is the first couple of lines, as they often contain the reason for the error. This example shows an unknown table and database has been requested:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
query-peer :) select thisdoesntexist from some.table

SELECT thisdoesntexist
FROM some.table

Query id: f8f0d160-cc58-4466-9c4a-011b5067a152

[query-head-7b65c8b674-dsnlv] 2022.08.31 17:00:10.197969 [ 10 ] {f8f0d160-cc58-4466-9c4a-011b5067a152} <Error> executeQuery: Code: 81. DB::Exception: Database some doesn't exist. (UNKNOWN_DATABASE) (version 22.3.2.1) (from 0.0.0.0:0) (in query: select thisdoesntexist from some.table), Stack trace (when copying this message, always include the lines below):

0. StackTrace::StackTrace() @ 0xac3418c in /usr/bin/turbine_server
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xabe9fad in /usr/bin/turbine_server
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x1129bbd7 in /usr/bin/turbine_server
1. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x1129d9a8 in /usr/bin/turbine_server
1. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x111e5e8c in /usr/bin/turbine_server
1. DB::JoinedTables::getLeftTableStorage() @ 0x11711c1e in /usr/bin/turbine_server
...

Query circuit breaker errors⚓︎

Hydrolix can apply circuit breakers to a query that help protect the infrastructure from abuse. Learn more about circuit breakers.