Skip to content

Query System Observability

This page illustrates data flow for logging and metrics from the query system and where these are collected.

Observability data flow⚓︎

Most nodes in the diagram link to relevant documentation.

---
config:
  themeVariables:
    fontSize: 18px
  flowchart:
    padding: 20
---
flowchart TB
  classDef default stroke:#00A99D,stroke-width:2px

  subgraph client-software[Client Applications]
      clickhouse-http-clients[ClickHouse HTTP
                              Clients]
      mysql-clients[MySQL
                    Clients]
      clickhouse-clients[ClickHouse
                         Clients]
      http-client[HTTP-capable
                  Clients]
  end

  subgraph query-system[Hydrolix Query System]
    http-query-api[HTTP Query
                   API server]
    query-head-h[ClickHouse
                 HTTP]
    query-head-c[ClickHouse
                 Native]
    query-head-m[MySQL
                 Server]
  end

  clickhouse-clients --> query-head-c
  mysql-clients --> query-head-m
  clickhouse-http-clients --> query-head-h

  http-client -- HTTP header hdx_query_debug=true --> http-query-api

  query-system -- sends system
                    logs (vector) --> hydrologs
  query-system -- Prometheus
                    scrapes --> metrics
  query-system -- records recent
                    queries in a table --> active-queries[ active queries ]
  query-system -. optionally writes
                    logs (vector) .-> storage
  http-query-api -- HTTP header
                    X-Hdx-Query-Stats --> http-client

  subgraph observability-systems[Observability Systems]
    storage[Primary Storage]
    hydrologs[Hydrologs]
    active-queries[Active Queries]
    metrics[Prometheus Metrics]
  end

  classDef nodegroup fill:none,stroke:#8B9FAD,stroke-width:2px,stroke-dasharray:5 4,color:#4A5F73
  class client-software,query-system,observability-systems nodegroup

click hydrologs href "https://docs.hydrolix.io/latest/self-managed/observability/hydrologs/"
click active-queries href "https://docs.hydrolix.io/latest/exploring-data/query-troubleshooting/active-queries/"
click metrics href "https://docs.hydrolix.io/latest/exploring-data/query-troubleshooting/query-metrics/"

click http-query-api href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/http-query-api/"
click query-head-h href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/clickhouse/clickhouse-http/"
click query-head-c href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/clickhouse/"
click query-head-m href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/mysql-client/"

click mysql-clients href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/mysql-client/"
click clickhouse-clients href "https://docs.hydrolix.io/latest/exploring-data/query-interfaces/clickhouse/clickhouse-client/"

Where to find observability data⚓︎