Skip to content

HTTP Proxy

The HTTP proxy improves performance and stability by caching SELECT and WITH queries and routing traffic through a dedicated service pod. It's optional, and disabled by default.

Routing behavior⚓︎

HTTP proxy uses port 9444 which always accepts query traffic.

  • If the proxy is enabled, traffic goes to the http-proxy pod
  • If the proxy is disabled, it routes directly to query heads

Starting in Hydrolix version 6.3, setting replicas to 0 in the http-proxy scale profile disables the proxy and routes traffic to the query heads.

Route default-pool HTTP through the proxy⚓︎

By default, ClickHouse HTTP traffic goes directly to the query head. Enable all_clickhouse_http_via_chproxy_enabled to route this traffic through the proxy instead. Use this to apply caching to queries that don't use a named pool, for example.

Route Default-Pool HTTP Through Proxy
1
2
3
4
spec:
  http_proxy:
    enabled: true
  all_clickhouse_http_via_chproxy_enabled: true

This flag has no effect when http_proxy.enabled is false; traffic continues to route directly to the query head.

Behavior change in Hydrolix version 6.1

Before v6.1, all_clickhouse_http_via_chproxy_enabled had no effect unless query_head_pooling_enabled was also true. If you have all_clickhouse_http_via_chproxy_enabled: true in your cluster configuration without query_head_pooling_enabled, default-pool HTTP traffic will now route through the proxy where it previously went directly to the query head.

Grafana configuration⚓︎

To configure Grafana, set the Hydrolix Grafana plugin to use the HTTP connection to port 9444.

Authentication and TLS⚓︎

Credentials from the Grafana client pass through unchanged during authentication to the query head. Starting in Hydrolix version 5.11, the proxy rejects requests without credentials before they reach the cache.

Mutual TLS to Redis or Valkey⚓︎

You need a client certificate and key only if your Redis/Valkey server requires client certificates. To enable mutual TLS between the proxy and a Redis/Valkey cache:

  1. Set up the Redis/Valkey-based cache storage.
  2. Set use_tls=true.
  3. Configure the HTTP proxy with a client certificate and key.

Setting use_tls=true without HTTP_PROXY_REDIS_TLS_CERT and HTTP_PROXY_REDIS_TLS_KEY gives server-only TLS: the proxy encrypts the connection but presents no client certificate.

The proxy verifies the Redis/Valkey server certificate against the system trust store in the http-proxy image. You can't supply a custom certificate authority (CA). A server certificate issued by a private CA fails verification unless you set insecure_skip_verify: true, which disables server verification.

Avoid insecure_skip_verify in production

Setting insecure_skip_verify: true disables verification of the Redis/Valkey server's identity. Traffic stays encrypted, but anyone who can intercept it can impersonate the server and read or change cached query results. Instead, use a server certificate from a publicly trusted CA, or put a TLS-terminating proxy in front of Redis/Valkey.

The cluster terminates TLS for client traffic before it reaches the proxy pod, and the proxy communicates over plain HTTP internally.

Cache options⚓︎

The HTTP proxy supports two cache backends: file-system and Redis/Valkey.

File-system cache⚓︎

In file_system mode, this cache type is best for:

  • Local development
  • Single-replica clusters
  • Simple setups without external dependencies

Cached responses are stored on the local disk of the proxy pod. Only the pod that served the query can return the cached result.

If you set max_size, keep it below the http-proxy pod's ephemeral storage limit. When you don't set it, the default is half of that limit. To check the limit, run:

kubectl get deployment http-proxy -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}request={.resources.requests.ephemeral-storage}{"\t"}limit={.resources.limits.ephemeral-storage}{"\n"}{end}'

The cache is ephemeral; replacing the pod clears the cache.

Redis or Valkey cache⚓︎

Use redis mode for either Redis or Valkey. This cache type is best for:

  • Production environments
  • Multi-replica or distributed clusters
  • Scenarios that need a consistent shared cache

Cached responses are stored in an external Redis/Valkey service. All proxy replicas share the same cache data.

The proxy doesn't enforce a size limit on a Redis/Valkey cache, so http_proxy.cache.max_size has no effect in redis mode. Set maxmemory and an eviction policy on the Redis/Valkey service itself.

Prerequisites⚓︎

  • (Optional) To use Redis or Valkey for caching, you must provision an instance and provide the endpoint during configuration
  • (Optional) TLS support requires a terminating proxy in front of Redis, Valkey, or a certificate from a publicly trusted CA.
    • For TLS with authentication, prepare Kubernetes secrets containing:
      • HTTP_PROXY_REDIS_USERNAME
      • HTTP_PROXY_REDIS_PASSWORD
      • HTTP_PROXY_REDIS_TLS_CERT (client certificate for mutual TLS)
      • HTTP_PROXY_REDIS_TLS_KEY (client certificate key for mutual TLS)
  • (Optional) For Prometheus metrics, ensure Prometheus is installed and can scrape pod metrics.

Enable HTTP proxy⚓︎

To enable HTTP proxy, edit the hydrolixcluster.yaml configuration file and add this line to the spec: section:

1
2
3
4
hydrolixcluster:
  spec:
    http_proxy:
      enabled: true

Configure cache options⚓︎

To configure the cache option, select either File-System or Redis/Valkey cache setup and modify it to match your settings:

1
2
3
4
5
6
7
8
9
hydrolixcluster:
  spec:
    http_proxy:
      enabled: true
      cache:
        mode: file_system
        dir: /tmp/http-proxy/cache
        max_size: 500M
        expire: 2m
hydrolixcluster:
  spec:
    http_proxy:
      enabled: true
      cache:
        mode: redis
        addresses:
          - valkey-primary:6379
        pool_size: 20
        db_index: 0
        use_tls: true
        insecure_skip_verify: false
        expire: 1m

Verify pod creation⚓︎

After enabling HTTP proxy, verify the new pod is running in your cluster:

kubectl get pods | grep http-proxy

You should see a pod named http-proxy-xxxxx with a Running status.

If the pod doesn't appear, check the operator logs or restart the operator pod.

Confirm routing behavior⚓︎

The cluster automatically routes traffic to the proxy when enabled.

  • Traffic to port 9444 goes through the http-proxy pod
  • If disabled, that traffic goes directly to the query heads

To test routing, run the following command:

curl -i -u <user>:<password> "https://<your-domain>:9444/query?query=SELECT%201"

Look for the X-Cache header:

X-Cache: HIT | MISS | N/A

Check cache behavior⚓︎

Run the same query twice.

  • The first response should return X-Cache: MISS.
  • The second response should return X-Cache: HIT if caching is working.

Query result caching⚓︎

When the HTTP proxy is enabled, it caches query results per user. The proxy returns a cached result until it expires, so it doesn't send identical queries to the query head again. The default expire value is 1m, and you can change it with http_proxy.cache.expire. You can't turn caching off in the cluster configuration, but a request can opt out, as described in Control caching for a request.

Cacheable queries⚓︎

The proxy caches a response only when all of these conditions are true:

  • The query starts with SELECT or WITH. The check isn't case-sensitive, and the proxy skips leading whitespace and SQL comments (-- and /* */) first.
  • The response is an HTTP 200 that completes normally. The proxy never caches errors or cancelled queries. Starting in Hydrolix version 6.4, it also never caches responses that end early.
  • The request isn't a multipart/form-data POST with external data.

Cached results can be out of date

A cached result can be up to one cache expire interval old, or as old as the hdx_http_proxy_ttl value if a request sets one.

Separately, the proxy doesn't detect non-deterministic functions such as now(). It runs the query once and caches the result, including the value the function returned. Later requests get that same value until the entry expires.

Cache response headers⚓︎

The proxy sets the X-Cache response header to show how it handled a query.

Value Meaning
HIT The proxy served the response from the cache.
MISS The proxy fetched the response from the query head. It stored the response only if the status was 200 and the query completed normally.
N/A The response was larger than the cache's maximum payload size, so the proxy didn't cache it.

If the response has no X-Cache header, the request skipped the cache. This happens when the query isn't a SELECT or WITH query, when the request is a multipart/form-data POST with external data, or when the request opts out with hdx_http_proxy_enabled=false or no_cache=1.

Cached and newly stored responses also include Cache-Control: max-age=<seconds remaining>.

Control caching for a request⚓︎

Add these parameters to the request query string to change how the proxy caches a single query. Starting in Hydrolix version 6.4, the last value wins if you repeat a parameter. Hydrolix versions 5.10 through 6.3 use the first value.

Parameter Effect
hdx_http_proxy_ttl=<ttl> Overrides the time to live (TTL) of the stored result. See hdx_http_proxy_ttl.
hdx_http_proxy_enabled=false Bypasses the cache for this request. See hdx_http_proxy_enabled.
no_cache=1 Bypasses the cache for this request. Same effect as hdx_http_proxy_enabled=false. See no_cache.
cache_namespace=<string> Adds the string to the cache key. See cache_namespace.

This example bypasses the cache for one query:

Bypass the Cache for One Query
curl -i -u <user>:<password> "https://<your-domain>:9444/?hdx_http_proxy_enabled=false" \
  --data-binary "SELECT count() FROM my_project.my_table"

This example caches a result for five minutes:

Set a Custom TTL for One Query
curl -i -u <user>:<password> "https://<your-domain>:9444/?hdx_http_proxy_ttl=5m" \
  --data-binary "SELECT count() FROM my_project.my_table"

Cache key⚓︎

As of Hydrolix version 6.4, a request returns HIT only when all of these values match a stored entry:

  • The query text, byte for byte, after the proxy strips leading whitespace and comments. Inline SETTINGS, letter case, and internal whitespace all count. Two queries that differ only in leading comments share a cache entry.
  • The Accept-Encoding request header.
  • The URL parameters default_format, database, compress, enable_http_compression, extremes, max_result_rows, result_overflow_mode, hdx_query_output_format, and cache_namespace.
  • A hash of all param_* query parameter values.
  • A hash of the credentials the proxy uses to reach the query head.

Because the proxy passes each caller’s own credentials to the query head, it keeps a separate cache entry for each set of credentials.

Concurrent identical queries⚓︎

The proxy coalesces identical queries that arrive at the same time. The first query runs on the query head, and the duplicates wait for its result instead of running again. If the first query fails with an error that the proxy can't retry, each waiting duplicate gets an HTTP 500 response that contains the first query's error message. If the query head returns 503 or 408, the duplicates retry instead.

Coalescing applies only while the first query is still running. After it finishes, identical queries get a normal HIT until the entry expires.

Cache entries for Grafana dashboards⚓︎

The credentials the Hydrolix Grafana plugin sends, and the way it writes time filters, determine whether dashboard users share cache entries. The plugin's credentials settings describe the authentication types.

With Forward OAuth Identity, the proxy hashes each user's own token into the cache key, so users never share entries or coalesce. A user also gets a new entry after a token refresh, because the new token produces a new hash.

With User Account or Service Account, every user sends the same credentials, so requests can share an entry and coalesce, but only when the query text is byte-identical. The plugin expands time macros such as $__timeFilter, $__fromTime, and $__toTime to absolute Unix timestamps in seconds, and the _ms variants use milliseconds. With a relative time range such as Last 1 hour, two users match only when their panels render in the same second. Absolute time ranges produce identical query text and match reliably.

Add secrets for authentication (Redis mode only)⚓︎

These secrets are only required if your Redis/Valkey deployment uses authentication or TLS.

To use Redis/Valkey with TLS and authentication, modify this command to add your secrets. client.crt and client.key are the proxy's own client certificate and private key, which a Redis/Valkey server that verifies client certificates accepts. They aren't a CA bundle.

1
2
3
4
5
6
7
8
kubectl patch secret curated --type='json' \
  -p='[{"op": "add", "path": "/data/HTTP_PROXY_REDIS_USERNAME", "value": "'$(echo -n "youruser" | base64)'" }]'

kubectl patch secret curated --type='json' \
  -p='[{"op": "add", "path": "/data/HTTP_PROXY_REDIS_PASSWORD", "value": "'$(echo -n "yourpass" | base64)'" }]'

kubectl patch secret curated --type='json' -p='[{"op": "add", "path": "/data/HTTP_PROXY_REDIS_TLS_CERT", "value": "'$(cat ./client.crt | base64)'" }]'
kubectl patch secret curated --type='json' -p='[{"op": "add", "path": "/data/HTTP_PROXY_REDIS_TLS_KEY", "value": "'$(cat ./client.key | base64)'" }]'