Skip to content

Configure Query Head Pooling

Use these procedures to enable query head pooling and set up its components. For an overview of how the feature works, see Query head pools.

To set up query head pooling in a basic configuration:

  1. Create at least one additional query head pool.
  2. Enable http_proxy if you have HTTP clients.
  3. Enable clickprox if you have native TCP clients.
  4. Enable the query_head_pooling_enabled tunable.
  5. Configure routing rules.
  6. Update client applications to pass the database parameter.

Enable query head pooling⚓︎

Query head pooling is disabled by default. These tunables control it:

Tunable Description Default
query_head_pooling_enabled Enables per-pool routing rules in Traefik and pool-aware proxy configuration. Requires http_proxy.enabled to have any observable effect on HTTP routing. false
http_proxy.enabled Enables the HTTP proxy. Required to route HTTP traffic through the proxy. false
query_head_native_routing_enabled Routes native TCP connections through ClickProx. true

For more detail on how these tunables interact, see Query head pools.

To route all default-pool HTTP traffic through the proxy, see HTTP proxy.

To support both HTTP and native TCP clients, enable both http_proxy and clickprox:

Enable Query Head Pooling
1
2
3
4
5
6
7
spec:
  http_proxy:
    enabled: true  # required for HTTP client routing
  clickprox:
    enabled: true  # required for native TCP client routing
  query_head_pooling_enabled: true
  query_head_native_routing_enabled: true

Configure ClickProx⚓︎

Configure ClickProx with the clickprox tunable:

ClickProx Configuration
1
2
3
4
5
6
spec:
  clickprox:
    enabled: true
    port: 9000
    metrics_port: 9090
    drain_timeout: 30
Field Description
enabled Enable or disable ClickProx
port Port for incoming native TCP connections
metrics_port Port for Prometheus metrics
drain_timeout Seconds to wait for in-flight connections before shutting down

Create query head pools⚓︎

Create query head pools by adding a pool definition to hydrolixcluster.yaml.

Set the service field to query-head:

Query Head Pool
1
2
3
4
5
6
7
8
spec:
  pools:
    analytics-query-head:
      name: analytics-query-head
      service: query-head
      cpu: 4
      memory: 16Gi
      replicas: 2

Configure routing rules⚓︎

Incoming connections and HTTP requests are routed to a query head pool based on the client's database parameter.

  • The query-head pool is always the default pool. Routing rules can select any pool, including the default.
  • Each routing rule can specify a default_query_peer pool, used when the query doesn't specify hdx_query_pool_name.
  • Each routing rule can also set a priority. A lower number is higher priority.

Clients must use the database parameter

The query head pool routing feature only works if clients present the database parameter. See examples in route by database.

Update applications to include the database parameter with HTTP requests and native protocol connections. It's only used for query head pool selection.

Route by database⚓︎

Route by Database
1
2
3
4
5
6
7
8
9
spec:
  pools:
    analytics-query-head:
      name: analytics-query-head
      service: query-head
      routing:
        database: hydro.logs
        priority: 10
        default_query_peer: analytics-query-peer

Select a query head pool using clickhouse-client by setting the --database parameter.

1
2
3
4
5
6
printf "%s\n" "SELECT COUNT(*) FROM hydro.logs WHERE timestamp > NOW() - INTERVAL 1 HOUR " \
  | clickhouse-client \
    --host "${HDX_HOST}" --port 9440 \
    --user "__api_token__" --password "${HDX_TOKEN}" \
    --secure \
    --database hydro.logs  # (1)!
  1. The --database parameter matches a known routing rule and the analytics-query-head is used.
1
2
3
4
5
6
printf "%s\n" "SELECT COUNT(*) FROM hydro.logs WHERE timestamp > NOW() - INTERVAL 1 HOUR " \
  | clickhouse-client \
    --host "${HDX_HOST}" --port 9440 \
    --user "__api_token__" --password "${HDX_TOKEN}" \
    --secure \
    --database any-other-string  # (1)!
  1. The --database parameter doesn't match any rules. The default query head pool is used.

Select a query head pool using curl by setting the database HTTP query parameter.

1
2
3
4
5
6
printf "%s\n" "SELECT COUNT(*) FROM hydro.logs WHERE timestamp > NOW() - INTERVAL 1 HOUR " \
  | curl \
    --data-binary @- \
    --fail-with-body --silent \
    --header "Authorization: Bearer ${HDX_TOKEN}" \
    -- "${HDX_HYDROLIX_URL}:8088/?database=hydro.logs"  # (1)!
  1. The --database parameter matches a known routing rule and the analytics-query-head is used.
1
2
3
4
5
6
printf "%s\n" "SELECT COUNT(*) FROM hydro.logs WHERE timestamp > NOW() - INTERVAL 1 HOUR " \
  | curl \
    --data-binary @- \
    --fail-with-body --silent \
    --header "Authorization: Bearer ${HDX_TOKEN}" \
    -- "${HDX_HYDROLIX_URL}:8088/?database=any-other-string"  # (1)!
  1. The --database parameter doesn't match any rules. The default query head pool is used.