Query System Observability
This page illustrates data flow for logging and metrics from the query system and where these are collected.
Observability data flow⚓︎
Most nodes in the diagram link to relevant documentation.
---
config:
flowchart:
padding: 20
---
flowchart TB
subgraph client-software[Client Applications]
clickhouse-http-clients[ClickHouse HTTP
Clients]
mysql-clients[MySQL
Clients]
clickhouse-clients[ClickHouse
Clients]
http-client[HTTP-capable
Clients]
end
subgraph query-system[Hydrolix Query System]
http-query-api[HTTP Query
API server]
query-head-h[ClickHouse
HTTP]
query-head-c[ClickHouse
Native]
query-head-m[MySQL
Server]
end
clickhouse-clients --> query-head-c
mysql-clients --> query-head-m
clickhouse-http-clients --> query-head-h
http-client -- HTTP header hdx_query_debug=true --> http-query-api
query-system -- sends system
logs (vector) --> hydrologs
query-system -- Prometheus
scrapes --> metrics
query-system -- records recent
queries in a table --> active-queries[ active queries ]
query-system -. optionally writes
logs (vector) .-> storage
http-query-api -- HTTP header
X-Hdx-Query-Stats --> http-client
subgraph observability-systems[Observability Systems]
storage[Primary Storage]
hydrologs[Hydrologs]
active-queries[Active Queries]
metrics[Prometheus Metrics]
end
click hydrologs href "https://docs.hydrolix.io/latest/self-managed/observability/hydrologs/"
click active-queries href "https://docs.hydrolix.io/latest/exploring-data/query-troubleshooting/active-queries/"
click metrics href "https://docs.hydrolix.io/latest/exploring-data/query-troubleshooting/query-metrics/"
click http-query-api href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/http-query-api/"
click query-head-h href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/clickhouse/clickhouse-http/"
click query-head-c href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/clickhouse/"
click query-head-m href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/mysql-client/"
click mysql-clients href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/mysql-client/"
click clickhouse-clients href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/clickhouse/clickhouse-client/"
Where to find observability data⚓︎
- Clients of the HTTP Query API can set
hdx_query_debugat the query level for additional query performance debugging information. - The query head in the Hydrolix query system maintains a system table for Active Queries in the last five minutes.
- The cluster sends query head and peer logs to the
hydro.logsHydrolix table. See Hydrologs and Hydrologs Query Examples. Clusters may also send to other log destinations. - The Prometheus Integration in each cluster collects application metrics. See Query Metrics.