Query Metrics
Query metrics overview⚓︎
Prometheus in Hydrolix⚓︎
The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information.
You can query, view, and actively monitor this information using a stack's Grafana instance, or you can access it with your own monitoring platform. See Prometheus Integration for more information about setting up Prometheus with an external server.
Use multiple components with a metric⚓︎
If more than one component uses a given metric, querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query.
For example, process_open_fds{service="stream-peer"}.
Query metrics⚓︎
These metrics track activity specific to querying.
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempts to the storage service. |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to the storage service, in seconds. |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time for the storage service, in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram of HTTP response times from the storage service, in seconds. |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service. |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempts to the storage service. |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status codes from the storage service. |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit). |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the cache. |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the network. |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram of file evictions from the cache. |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram of CPU usage in seconds. |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram of I/O times in seconds. |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram of wait times for file descriptors, in seconds. |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per-query catalog partition count. |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per-query partition read count. |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per-core partition usage count. |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total. |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of cores used total. |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time-range distribution. |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of columns read. |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding HDX blocks, in seconds. |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening HDX partitions, in seconds. |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading HDX partitions, in seconds. |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partitions skipped due to no matching columns. |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition-block read counts. |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available. |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match). |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup times, in seconds. |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocks, in percentage. |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of total skipped index blocks. |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error). |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram of query I/O wait times, in seconds. |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram of query CPU wait times, in seconds. |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting HDX blocks to ClickHouse, in seconds. |
query_health |
Histogram | Head/Query peer | Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error). |
query_peer_availability |
Histogram | Head/Query peer | Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers). |
query_attempts_total |
Histogram | Head/Query peer | Histogram of total query attempts. |
query_response_seconds |
Histogram | Head/Query peer | Histogram of total query response times, in seconds. |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of total query rows read. |
query_read_bytes |
Histogram | Head/Query peer | Histogram of total query read bytes. |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of total query rows written. |
For a complete list of the metrics used by Hydrolix, including Prometheus, RabbitMQ, and others, see All Metrics.