Query Debugging

Endpoint errors

Issue: Client connection timeout

Check

Check Traefik service count is \< 1.

kubectl get deployment/traefik -o wide

Fix

Scale Traefik via hydrolixcluster.yaml.

traefik:
   replicas: <Scale>

For more information, see Scale Profiles.

Check

Check that the IP allowlist contains the requesting IP address.

Download the cluster configuration:

kubectl get hydrolixcluster -o yaml

Fix

1. Add the IP to the allowlist:

      kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
   

2. Add the IPs.

3. Apply the changes to your cluster:

      kubectl apply -f hydrolixcluster.yaml
   

Find more information on IP allowlists, see Configure IP Access.

Issue: Client authorization error

Check

Check if Query endpoint authentication is enabled.

Download your cluster configuration:

kubectl get hydrolixcluster -o yaml

In the configuration, look for enable_query_auth:

spec:
  enable_query_auth: true

Fix

If enabled, ensure the user is using the correct username and password or bearer token. For more information, see Enable Query Authentication.

If query authentication isn't enabled, enable it.

Check

Check if TLS is enabled.

Download your cluster configuration:

kubectl get hydrolixcluster -o yaml

In the configuration, look for use_tls:

spec:  
    use_tls: true

Fix

If TLS is enabled, check the client is connecting using a secure TLS connection.

If TLS isn't enabled, enable it.

If using native protocol connection, secure uses port 9440 and non-secure uses port 9000.

Issue: HTTP 503 Service Temporarily Unavailable

For more information, see Scale Profiles.

Check

Check query-head count is at a minimum of 1.

kubectl get deployment/query-head -o wide

Fix

Scale query-head via hydrolixcluster.yaml:

query-head:
   replicas: <scale number>

Check

Check query-peer count is at a minimum of 1.

kubectl get deployment/query-peer -o wide

Fix

Scale query-peer via hydrolixcluster.yaml:

query-peer:  
   replicas: <Scale>

Pools

For pools, check that the pool has query-peers.

Issue: Database exceptions

For more information, see Scale Profiles.

• DB::Exception: HdxStorageError: No peers available to run query in pool
• DB::Exception: Database '_local' doesn't exist
• DB::NetException: Net Exception: No route to host

Check

Check query-peer instance count is a minimum of 1.

kubectl get deployment/query-peer -o wide

Fix

Scale query-peer using hydrolixcluster.yaml.

query-peer:
   replicas: <Scale>

Check

Check that the ZooKeeper instance count is 3.

kubectl get deployment/zookeeper -o wide

Fix

Scale ZooKeeper with hydrolixcluster.yaml.

zookeeper:
   replicas: <Scale>

Issue: Database timeout

DB::NetException: Timeout: connect timed out

Check

Check ZooKeeper instance count is 3:

kubectl get deployment/zookeeper -o wide

Fix

Scale ZooKeeper with hydrolixcluster.yaml.

zookeeper:
   replicas: <Scale>

Issue: Database lost connection

DB::NetException: Error: Lost connection to the database server. (version 3.x.x)>. (STD_EXCEPTION)

Check

This error is uncommon and should only happen when a query head isn’t used for days or weeks.

Fix

Retry the query.

If this doesn’t resolve the error, check that the PostgreSQL instance is suitably scaled. Increase CPU or memory if required.

1. Download the cluster configuration.

      kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
   

2. Update the scale.

3. Apply the configuration to your cluster.

      kubectl apply -f hydrolixcluster.yaml
   

Check

Check that PostgreSQL is running and in a healthy state.

kubectl describe statefulset postgres

Fix

Update the scale of PostgreSQL

1. Download the cluster configuration.

      kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
   

2. Update the scale.

3. Apply the configuration to your cluster.

      kubectl apply -f hydrolixcluster.yaml
   

If PostgreSQL is failing, even if resolved, this is a Severity 1 incident. Notify Hydrolix as soon as possible.

Syntax and user query errors

Check

The table or project is unknown.

DB::Exception: Table _local.XXXX does not exist

Fix

Specify the project and table.

SELECT x FROM project.table WHERE ...

Check

If the table and/or project has a hyphen in the name. For example, my-table.my-project

Expected one of: token, Dot, UUID, alias, AS, identifier, FINAL, SAMPLE, table, table function...

Fix

Add backticks (`) around the table and project names.

select x from `my-project`.` my-table` where...

Other errors

When a query error occurs with syntax or execution errors, the system can be overly verbose. Errors can be large and contain a lot of information. A good place to start when debugging is the first couple of lines, as they often contain the reason for the error. This example shows an unknown table and database has been requested:

query-peer :) select thisdoesntexist from some.table

SELECT thisdoesntexist
FROM some.table

Query id: f8f0d160-cc58-4466-9c4a-011b5067a152

[query-head-7b65c8b674-dsnlv] 2022.08.31 17:00:10.197969 [ 10 ] {f8f0d160-cc58-4466-9c4a-011b5067a152} <Error> executeQuery: Code: 81. DB::Exception: Database some doesn't exist. (UNKNOWN_DATABASE) (version 22.3.2.1) (from 0.0.0.0:0) (in query: select thisdoesntexist from some.table), Stack trace (when copying this message, always include the lines below):

0. StackTrace::StackTrace() @ 0xac3418c in /usr/bin/turbine_server
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xabe9fad in /usr/bin/turbine_server
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x1129bbd7 in /usr/bin/turbine_server
1. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x1129d9a8 in /usr/bin/turbine_server
1. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x111e5e8c in /usr/bin/turbine_server
1. DB::JoinedTables::getLeftTableStorage() @ 0x11711c1e in /usr/bin/turbine_server
...

Query circuit breaker errors

Hydrolix can apply circuit breakers to a query that help protect the infrastructure from abuse. Learn more about circuit breakers.