Query
Endpoint errors
Issue: Client connection timeout
Check
Check Traefik service count is < 1.
kubectl get deployment/traefik -o wide
Fix
Scale Traefik via hydrolixcluster.yaml
.
traefik:
replicas: <Scale>
For more information, see Scale Profiles.
Check
Check that the IP allowlist contains the requesting IP address.
Download the cluster configuration:
kubectl get hydrolixcluster -o yaml
Fix
- Add the IP to the allowlist:
kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
- Add the IPs.
- Apply the changes to your cluster:
kubectl apply -f hydrolixcluster.yaml
Find more information on IP allowlists, see Configure IP Access.
Issue: Client authorization error
Check
Check if Query endpoint authentication is enabled.
Download your cluster configuration:
kubectl get hydrolixcluster -o yaml
In the configuration, look for enable_query_auth
:
spec:
enable_query_auth: true
Fix
If enabled, ensure the user is using the correct username and password or bearer token. For more information, see Enable Query Authentication.
If query authentication is not enabled, enable it.
Check
Check if TLS is enabled.
Download your cluster configuration:
kubectl get hydrolixcluster -o yaml
In the configuration, look for use_tls
:
spec:
use_tls: true
Fix
If TLS is enabled, check the client is connecting using a secure TLS connection.
If TLS is not enabled, enable it.
If using native protocol connection, secure uses port 9440 and non-secure uses port 9000.
Issue: HTTP 503 Service Temporarily Unavailable
For more information, see Scale Profiles.
Check
Check query-head count is at a minimum of 1.
kubectl get deployment/query-head -o wide
Fix
Scale query-head via hydrolixcluster.yaml
:
query-head:
replicas: <scale number>
Check
Check query-peer count is at a minimum of 1.
kubectl get deployment/query-peer -o wide
Fix
Scale query-peer via hydrolixcluster.yaml
:
query-peer:
replicas: <Scale>
Pools
For pools, check that the pool has query-peers.
Issue: Database exceptions
For more information, see Scale Profiles.
DB::Exception: HdxStorageError: No peers available to run query in pool
DB::Exception: Database '_local' doesn't exist
DB::NetException: Net Exception: No route to host
Check
Check query-peer instance count is a minimum of 1.
kubectl get deployment/query-peer -o wide
Fix
Scale query-peer using hydrolixcluster.yaml
.
query-peer:
replicas: <Scale>
Check
Check that the ZooKeeper instance count is 3.
kubectl get deployment/zookeeper -o wide
Fix
Scale ZooKeeper with hydrolixcluster.yaml
.
zookeeper:
replicas: <Scale>
Issue: Database timeout
DB::NetException: Timeout: connect timed out
Check
Check ZooKeeper instance count is 3:
kubectl get deployment/zookeeper -o wide
Fix
Scale ZooKeeper with hydrolixcluster.yaml
.
zookeeper:
replicas: <Scale>
Issue: Database lost connection
DB::NetException: Error: Lost connection to the database server. (version 3.x.x)>. (STD_EXCEPTION)
Check
This error is uncommon and should only happen when a query head isn’t used for days or weeks.
Fix
Retry the query.
If this doesn’t resolve the error, check that the PostgreSQL instance is suitably scaled. Increase CPU or memory if required.
- Download the cluster configuration.
kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
- Update the scale.
- Apply the configuration to your cluster.
kubectl apply -f hydrolixcluster.yaml
Check
Check that PostgreSQL is running and in a healthy state.
kubectl describe statefulset postgres
Fix
Update the scale of PostgreSQL
- Download the cluster configuration.
kubectl get hydrolixcluster -o yaml > hydrolixcluster.yaml
- Update the scale.
- Apply the configuration to your cluster.
kubectl apply -f hydrolixcluster.yaml
If PostgreSQL is failing, even if resolved, this is a Severity 1 incident. Notify Hydrolix as soon as possible.
Syntax and user query errors
Check
The table or project is unknown.
DB::Exception: Table _local.XXXX does not exist
Fix
Specify the project and table.
SELECT x FROM project.table WHERE ...
Check
If the the table and/or project has a hyphen in the name. For example, my-table.my-project
Expected one of: token, Dot, UUID, alias, AS, identifier, FINAL, SAMPLE, table, table function...
Fix
Add backticks (`) around the table and project names.
select x from `my-project`.` my-table` where...
Other errors
When a query error occurs with syntax or execution errors, the system can be overly verbose. Errors can be large and contain a lot of information. A good place to start when debugging is the first couple of lines, as they often contain the reason for the error. This example shows an unknown table and database has been requested:
query-peer :) select thisdoesntexist from some.table
SELECT thisdoesntexist
FROM some.table
Query id: f8f0d160-cc58-4466-9c4a-011b5067a152
[query-head-7b65c8b674-dsnlv] 2022.08.31 17:00:10.197969 [ 10 ] {f8f0d160-cc58-4466-9c4a-011b5067a152} <Error> executeQuery: Code: 81. DB::Exception: Database some doesn't exist. (UNKNOWN_DATABASE) (version 22.3.2.1) (from 0.0.0.0:0) (in query: select thisdoesntexist from some.table), Stack trace (when copying this message, always include the lines below):
0. StackTrace::StackTrace() @ 0xac3418c in /usr/bin/turbine_server
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xabe9fad in /usr/bin/turbine_server
2. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x1129bbd7 in /usr/bin/turbine_server
3. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x1129d9a8 in /usr/bin/turbine_server
4. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x111e5e8c in /usr/bin/turbine_server
5. DB::JoinedTables::getLeftTableStorage() @ 0x11711c1e in /usr/bin/turbine_server
...
Query circuit breaker errors
Hydrolix can apply circuit breakers to a query that help protect the infrastructure from abuse. Learn more about circuit breakers.
Error | Circuit Breaker Information |
---|---|
DB::Exception: HdxStorageError Maximum number of rows exceeded for query: XXX rows (maximum is YYY) | hdx_query_max_rows |
DB::Exception: Limit for result exceeded, max bytes: XXX B, current bytes: YYY | hdx_query_max_result_bytes |
DB::Exception: Limit for result exceeded, max rows: XX, current rows: YYY thousand | hdx_query_max_result_rows |
DB::Exception: h::db::HdxStorageError: <HdxStorageError Maximum time range exceeded for query: XXX seconds (maximum is YYY) | hdx_query_max_timerange_sec |
DB::Exception: h::db::HdxStorageError: <HdxStorageError hdx_query_timerange_required is set to true. Your query needs a time range filter in a WHERE clause | hdx_query_timerange_required |
DB::Exception: h::db::HdxStorageError: <HdxStorageError Maximum number of partitions exceeded for query: XXX partitions (maximum is YYY) | hdx_query_max_partitions |
DB::Exception: Timeout exceeded: elapsed XXX seconds, maximum: YYY. | hdx_query_max_execution_time |
DB::Exception: Limit for number of columns to read exceeded. Requested: 2, maximum: 1. | hdx_query_max_columns_to_read |
DB::Exception: Memory limit (for query) exceeded: would use 16.00 KiB (attempt to allocate chunk of 4096 bytes), maximum: 1.00 KiB. (MEMORY_LIMIT_EXCEEDED) | hdx_query_max_memory_usage |
Updated about 1 month ago