Overview
This page contains most metrics used in Hydrolix for the following:
Use find on page to locate specific metrics.
Custom Hydrolix metrics⚓︎
Custom general metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
bytes_written |
Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created |
Counter | Batch peer, Stream peer | Count of partitions created. |
upload_duration |
Summary | Any intake peer | Time spent uploading a file, in milliseconds. |
Custom query metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempts to the storage service. |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to the storage service, in seconds. |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time for the storage service, in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram of HTTP response times from the storage service, in seconds. |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service. |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempts to the storage service. |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status codes from the storage service. |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit). |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the cache. |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the network. |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram of file evictions from the cache. |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram of CPU usage in seconds. |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram of I/O times in seconds. |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram of wait times for file descriptors, in seconds. |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per-query catalog partition count. |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per-query partition read count. |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per-core partition usage count. |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total. |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of cores used total. |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time-range distribution. |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of columns read. |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding HDX blocks, in seconds. |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening HDX partitions, in seconds. |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading HDX partitions, in seconds. |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partitions skipped due to no matching columns. |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition-block read counts. |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available. |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match). |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup times, in seconds. |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocks, in percentage. |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of total skipped index blocks. |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error). |
hdx_partition_cache_data_read_bytes |
Histogram | Query peer | Histogram of bytes read from the local data cache. Tracks data.hdx bytes served from ephemeral storage |
hdx_partition_net_data_read_bytes |
Histogram | Query peer | Histogram of data.hdx bytes read from cloud object storage. Complementing hdx_partition_cache_data_read_bytes for cache hit ratio analysis |
hdx_partition_bloom_skip |
Histogram | Query peer | Histogram of Bloom pre-filter outcomes per partition (bucket=0 not skipped, bucket=1 skipped), for measuring how many partitions the pre-filter removes from a scan (query acceleration). |
hdx_partition_bloom_false_positive |
Histogram | Query peer | Histogram of Bloom false positives per partition (bucket=0 no, bucket=1 yes), where the filter reported a possible match but the index found no matching blocks; for tracking pre-filter accuracy, since a high rate erodes the benefit. |
hdx_partition_bloom_columns_checked |
Histogram | Query peer | Histogram of columns for which a partition Bloom filter is consulted during query. |
hdx_partition_bloom_check_seconds |
Histogram | Query peer | Histogram of time spent probing Bloom filters per partition, in seconds, for weighing pre-filter overhead against the partitions it skips. |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram of query I/O wait times, in seconds. |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram of query CPU wait times, in seconds. |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting HDX blocks to ClickHouse, in seconds. |
query_health |
Histogram | Head/Query peer | Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error). |
query_peer_availability |
Histogram | Head/Query peer | Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers). |
query_attempts_total |
Histogram | Head/Query peer | Histogram of total query attempts. |
query_response_seconds |
Histogram | Head/Query peer | Histogram of total query response times, in seconds. |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of total query rows read. |
query_read_bytes |
Histogram | Head/Query peer | Histogram of total query read bytes. |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of total query rows written. |
peer_coord_active |
Gauge | Query head | Coordinators currently alive. |
peer_coord_pending_partitions |
Gauge | Query head | Partitions currently waiting for dispatch. |
peer_coord_inflight_partitions |
Gauge | Query head | Partitions currently in dispatched batches. |
peer_coord_connected_peers |
Gauge | Query head | Peers currently connected across all coordinators. |
peer_coord_free_peers |
Gauge | Query head | Peers currently idle and available for the next batch. |
peer_coord_connection_failures_total |
Counter | Query head | Count of peer connection failures. |
peer_coord_partitions_completed_total |
Counter | Query head | Count of partitions completed. |
peer_coord_partitions_errored_total |
Counter | Query head | Count of partition errors. |
peer_coord_batch_processing_seconds |
Histogram | Query head | Wall time from batch dispatch to completion, in seconds. |
hdx_partition_data_cache_path |
Histogram | Query peer | Data cache outcome for one partition. 0 = cold, nothing was cached yet, so data is fetched from object storage and then written to the cache for later queries. 1 = all cached, served entirely from local disk. 2 = partial, some byte ranges came from the cache and the rest were fetched. 3 = no cache, the caching path didn't apply, so data is fetched and not cached, which is what happens for partitions older than data_cache_max_partition_age_seconds. Repeated cold results for the same partitions suggest the cache is too small for the working set. |
hdx_partition_data_cache_actual_http_requests |
Histogram | Query peer | Byte-range HTTP requests actually issued to object storage to read one partition. Reading a partition is split into several byte ranges, so this counts requests per partition rather than per query. |
hdx_partition_data_cache_nocache_http_requests |
Histogram | Query peer | Byte-range HTTP requests that reading one partition would have needed with no cache. Subtract the actual count from this to get the requests the cache avoided. |
hdx_cache_capacity_bytes |
Gauge | Query peer | Configured local file cache quota for the pod. Reports 0 when the cache is disabled. |
hdx_cache_used_bytes |
Gauge | Query peer | Bytes currently occupied by the local file cache. Divide by hdx_cache_capacity_bytes for the cache fill ratio. |
Custom batch metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
processed_count |
Counter | Batch peer | Count of items processed. |
processed_failure |
Counter | Batch peer | Count of processing failures. |
processing_duration_histo |
Histogram | Batch peer | Histogram of Batch processing durations (milliseconds). |
processing_duration_summary |
Summary | Batch peer | Summary of Batch processing durations (milliseconds). |
rows_read |
Counter | Batch peer | Count of rows read. |
Custom batch controller metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
batch_peer_idle_duration_seconds |
Histogram | Batch controller | Time peers spend waiting for work after completing a job. High values indicate controller bottlenecks. |
batch_work_request_duration_seconds |
Histogram | Batch controller | Time to process a work request. |
batch_job_completion_duration_seconds |
Histogram | Batch controller | Time to process job completion. |
batch_connected_peers |
Gauge | Batch controller | Number of currently connected batch peers. |
batch_active_leases |
Gauge | Batch controller | Number of active job leases (jobs currently being processed). |
batch_jobs_created_total |
Counter | Batch controller | Total jobs created. |
batch_jobs_completed_total |
Counter | Batch controller | Jobs completed by status. |
batch_job_creation_duration_seconds |
Histogram | Batch controller | Time to create and enqueue a new job. |
batch_work_requests_total |
Counter | Batch controller | Work requests by outcome. |
batch_bytes_indexed |
Counter | Batch controller | Total bytes indexed by batch peers. |
batch_job_processing_duration_seconds |
Histogram | Batch controller | Time peers spend processing jobs (reported by peers on completion). |
batch_queue_depth |
Gauge | Batch controller | Current queue depth. |
batch_cache_size |
Gauge | Batch controller | Current cache size. |
batch_persistence_queue_depth |
Gauge | Batch controller | Async persistence queue depth. High values indicate storage write bottleneck. |
batch_lease_acquisitions_total |
Counter | Batch controller | Lease acquisition attempts. |
batch_lease_expirations_total |
Counter | Batch controller | Leases that expired (worker timeout). |
batch_state_transitions_total |
Counter | Batch controller | Job state machine transitions. |
batch_persistence_action_duration_seconds |
Histogram | Batch controller | Time for persistence operations. |
batch_storage_requests_total |
Counter | Batch controller | Storage API calls by operation. |
batch_storage_errors_total |
Counter | Batch controller | Storage errors by type. |
batch_storage_rate_limits_total |
Counter | Batch controller | Rate limit responses from object store when writing. |
batch_storage_inflight |
Gauge | Batch controller | Currently in-flight storage operations. |
batch_storage_duration_seconds |
Histogram | Batch controller | Storage operation latency. |
Custom merge metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
merge_duration_summary |
Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_duration_histo |
Histogram | Merge peer | Merge processing duration, in milliseconds. Buckets double from 1000 to 1024000, covering 1 second to about 17 minutes. |
merge_sdk_duration_summary |
Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_sdk_duration_histo |
Histogram | Merge peer | Merge SDK processing duration, in milliseconds. Buckets double from 1000 to 1024000, covering 1 second to about 17 minutes. |
merge_candidate_histo |
Histogram | Merge peer | Partitions per merge candidate. |
merge_candidate_inactive |
Counter | Merge peer | Merge candidates skipped due to an inactive partition within the candidate. |
merge_success |
Counter | Merge peer | Count of merge successes. |
merge_failure |
Counter | Merge peer | Count of merge failures. |
Custom streaming metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in the backlog (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds (requires intake_head_index_backlog_enabled = true). |
http_source_outstanding_reqs |
Gauge | Intake head | Number of outstanding ingest event requests. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of HTTP requests. |
http_source_request_duration_ns |
Histogram | Stream head | Histogram of HTTP request durations in nanoseconds. |
http_source_request_error_count |
Counter | Stream head | Count of HTTP request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Custom AWS Kinesis metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations, in nanoseconds. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of Kinesis checkpoint operation errors. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors reading from Kinesis. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis, in milliseconds. |
kinesis_source_operation_count |
Counter | Stream peer | Count of Kinesis operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of Kinesis operation durations, in nanoseconds. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
Custom Kafka metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration, in nanoseconds. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration, in nanoseconds. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed from Kafka. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed from Kafka. |
Custom DNS metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
dns_num_ips_in_cache |
Histogram | Any intake peer | The size of the IP pool used in the DNS system. |
dns_lookup_time |
Histogram | Any intake peer | Milliseconds per lookup. |
dns_ttl |
Histogram | Any intake peer | TTLs received per lookup. |
Custom PostgreSQL metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
pgx_pool_total_acquire_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool. |
pgx_pool_total_acquire_duration_ns_count |
Counter | CatalogDB | Total duration in nanoseconds of all successful acquires from the pool. |
pgx_pool_total_acquire_cancel_count |
Counter | CatalogDB | Cumulative count of acquires from the pool that were canceled by a context. |
pgx_pool_total_acquire_empty_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool that waited for a resource because the pool was empty. |
pgx_pool_total_conns_opened_count |
Counter | CatalogDB | Cumulative count of new connections opened. |
pgx_pool_total_destroyed_max_lifetime_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnLifetime. |
pgx_pool_total_destroyed_max_idle_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnIdleTime. |
pgx_pool_current_size |
Gauge | CatalogDB | Total number of resources currently in the pool. |
pgx_pool_current_constructing |
Gauge | CatalogDB | Number of connections currently being constructed. |
pgx_pool_current_acquired |
Gauge | CatalogDB | Number of connections currently acquired. |
pgx_pool_current_idle |
Gauge | CatalogDB | Number of currently idle connections in the pool. |
pgx_pool_max |
Gauge | CatalogDB | Maximum size of the pool. |
Custom partition cleaner metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
bulk_delete_duration |
Summary | Partition cleaner | Time spent performing bulk deletes. |
bulk_delete_success_count |
Counter | Partition cleaner | Files deleted. |
bulk_delete_failure_count |
Counter | Partition cleaner | Files unable to be deleted. |
bulk_delete_bytes |
Counter | Partition cleaner | Bytes deleted from storage. |
Custom scaler metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
hdxscaler_goroutines |
Gauge | hdx-scaler-go | Current number of goroutines. |
hdxscaler_kubelet_fetch_total |
Counter | hdx-scaler-go | Total kubelet stats/summary fetches by result (hit, miss, error). |
hdxscaler_kubelet_fetch_duration_seconds |
Histogram | hdx-scaler-go | Kubelet stats/summary API call latency (cache misses only). |
hdxscaler_kubelet_cache_entries |
Gauge | hdx-scaler-go | Number of entries in the kubelet stats cache. |
hdxscaler_sync_pods_total |
Counter | hdx-scaler-go | Total ListPods API calls per app. |
hdxscaler_event_cache_entries |
Gauge | hdx-scaler-go | Number of entries in the event watcher cache. |
hdxscaler_restart_count_entries |
Gauge | hdx-scaler-go | Number of entries in the OOM restart count tracker. |
hdxscaler_scrapes_total |
Counter | hdx-scaler-go | Total number of scrape requests. |
Age metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
reaper_stale_storage_total |
Counter | Reaper | Count of reap events where the storage ID was missing from the published configuration. |
reaper_storage_not_found_total |
Counter | Reaper | Count of reap events where the underlying storage returned a not-found error on a list or delete. |
query_count (deprecated) |
Counter | Reaper | Count of Reaper calls to the catalog. |
query_failure (deprecated) |
Counter | Reaper | Count of failed Reaper catalog calls. |
query_latency_summary (deprecated) |
Summary | Reaper | Latency of calls to the catalog. |
Alter metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_count (deprecated) |
Counter | Alter | Count of Alter calls to the catalog. |
query_failure (deprecated) |
Counter | Alter | Count of failed Alter catalog calls. |
query_latency_summary (deprecated) |
Summary | Alter | Latency of calls to the catalog. |
ClickProx metrics⚓︎
| Metric Name | Type | Components | Purpose | Labels |
|---|---|---|---|---|
clickprox_frontend_connections_total |
Gauge | ClickProx | Current number of client connections to ClickProx. | |
clickprox_backend_connections_total |
Gauge | ClickProx | Current number of back-end connections per query head pool. | pool |
clickprox_exception_packets_sent_total |
Counter | ClickProx | Count of ClickHouse exception packets sent to clients. | type pool |
clickprox_backend_connection_failures_total |
Counter | ClickProx | Count of failed back-end connection attempts. | pool |
clickprox_dropped_connections_total |
Counter | ClickProx | Count of connections dropped due to errors. | pool direction |
clickprox_routing_decisions_total |
Counter | ClickProx | Count of routing decisions by matching criterion. | criterion value pool |
Descheduler metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
descheduler_pods_evicted_total |
Counter | Descheduler | Total pods evicted since start. |
descheduler_pods_evicted |
Gauge | Descheduler | Pods evicted in last descheduler loop. |
descheduler_loop_duration_seconds |
Histogram | Descheduler | Descheduler loop duration. |
descheduler_strategy_duration_seconds |
Histogram | Descheduler | Strategy execution duration per loop. |
Intake spill metrics⚓︎
| Metric Name | Type | Components | Description |
|---|---|---|---|
hdx_sink_spill_catalog_accepted_count |
Counter | Intake head | Count of spilled catalog entries accepted into the indexing pipeline for processing. |
hdx_sink_spill_catalog_duration_ns |
Histogram | Intake head | Duration in nanoseconds taken to spill catalog adds. |
hdx_sink_spill_catalog_failure_count |
Counter | Intake head | Count of spill failures for catalog adds resulting in lost data. |
hdx_sink_spill_catalog_race_lost_count |
Counter | Intake head | Count of lost races attempting to add a spilled data entry to the catalog. |
hdx_sink_spill_raw_accepted_count |
Counter | Intake head | Count of spilled raw data entries accepted into the indexing pipeline for processing. |
hdx_sink_spill_raw_duration_ns |
Histogram | Intake head | Duration in nanoseconds taken to spill raw data buckets. |
hdx_sink_spill_raw_failure_count |
Counter | Intake head | Count of spill failures for raw data resulting in lost data. |
Merge metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
merge_success (deprecated) |
Counter | Merge peer | Count of merge successes. |
merge_failure (deprecated) |
Counter | Merge peer | Count of merge failures. |
query_count (deprecated) |
Counter | Merge peer | Count of Merge calls to the catalog. |
query_failure (deprecated) |
Counter | Merge peer | Count of failed Merge catalog calls. |
query_latency_summary (deprecated) |
Summary | Merge peer | Latency of calls to the catalog. |
merge_duration_summary (deprecated) |
Summary | Merge peer | Merge processing duration, in milliseconds. |
Merge controller metrics⚓︎
| Metric Name | Type | Components | Description | Labels |
|---|---|---|---|---|
partition_distribution |
Histogram | Merge Controller | Histogram of partition memory size. | project_id project_name table_id table_name target |
segments |
Gauge | Merge Controller | Current count of known segments. A segment is a group of buckets that share a merge target. | project_id project_name table_id table_name target |
active_merges |
Gauge | Merge Controller | Current count of active merge operations. | project_id project_name table_id table_name target |
partitions |
Gauge | Merge Controller | Current count of partitions awaiting segmentation. | project_id project_name table_id table_name target |
partitions_dispatched |
Counter | Merge Controller | Count of partitions dispatched for merging. Determine "partitions dispatched for merge per second" by combining with rate function. | project_id project_name table_id table_name target |
candidates |
Gauge | Merge Controller | Current count of constructed candidates waiting dispatch. | project_id project_name table_id table_name target |
candidates_dispatched |
Counter | Merge Controller | Count of candidates dispatched for merging. Determine "candidates dispatched for merge per second" by combining with rate function. | project_id project_name table_id table_name target |
partitions_per_candidate |
Histogram | Merge Controller | Distribution of partitions within a dispatched candidate. | project_id project_name table_id table_name target |
candidate_mem_size |
Histogram | Merge Controller | Distribution of calculated memory requirements for dispatched candidates. | project_id project_name table_id table_name target |
duplicate_partitions |
Counter | Merge Controller | Count of partitions sourced that are already being tracked. | project_id project_name table_id table_name target |
memory_coefficient |
Gauge | Merge Controller | Current value of the memory coefficient being applied to sourced partitions. | project_id project_name table_id table_name target |
expired_segments |
Counter | Merge Controller | Count of segments expired for moving out of the target range. | project_id project_name table_id table_name target |
bucket_duration |
Histogram | Merge Controller | Distribution of time in milliseconds of a bucket's age upon closing. The basis label gives the reason for closing: full = the bucket reached max size, idle_ttl = the bucket wasn't added to within the idle timeout, age_ttl = the bucket was open longer than the max age, segment_ttl = the bucket was part of a segment that fell out of the target range. |
project_id project_name table_id table_name target basis |
connected_clients |
Gauge | Merge Controller | Count of currently connected merge-peers. | pool_id |
query_latency |
Histogram | Merge Controller | Distribution of response times in milliseconds for catalog operations. | project_id table_id method |
admin_query_latency |
Histogram | Merge Controller | Distribution of response times in milliseconds for admin catalog operations. | project_id table_id method |
state_duration |
Histogram | Merge Controller | Distribution of time in milliseconds spent in each reactor state. | project_id table_id state |
merge_duty_cycle |
Gauge | Merge Controller | Overall work rate of the merge-controller, as a value between 0 and 1. | project_id project_name table_id table_name target |
row_weighted_efficiency |
Gauge | Merge Controller | Merge target efficiency as a value between 0.0 and 1.0, where a value near 1.0 means the target's data sits in close to the fewest partitions its size allows. Efficiency is calculated for each hour, storage location, and shard key group, then combined as the sum of ideal partition counts divided by the sum of actual partition counts, so the groups holding the most partitions have the most influence. Despite the name, the weighting is by partition count rather than by row count. |
project_id project_name table_id table_name target |
mem_weighted_efficiency |
Gauge | Merge Controller | Merge target efficiency as a value between 0.0 and 1.0, weighted by memory size. Each hour, storage location, and shard key group contributes in proportion to the bytes it holds, so a small badly packed group has little influence on the result. |
project_id project_name table_id table_name target |
harmonic_efficiency |
Gauge | Merge Controller | Merge target efficiency as a value between 0.0 and 1.0, calculated as the harmonic mean of the per group efficiencies. Every group counts equally and low values weigh heavily, so a single badly merged group visibly lowers the result. |
project_id project_name table_id table_name target |
ideal_partition_count |
Gauge | Merge Controller | Given the current required memory for all partitions in the target, what's the ideal number of partitions if each partition was at the target size. | project_id project_name table_id table_name target |
actual_partition_count |
Gauge | Merge Controller | Current count of partitions in the target. | project_id project_name table_id table_name target |
RabbitMQ metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
rabbitmq_queue_messages |
Gauge | RabbitMQ | Sum of ready and unacknowledged messages (queue depth). |
rabbitmq_queues |
Gauge | RabbitMQ | Count of RabbitMQ queues. |
erlang_vm_statistics_bytes_received_total |
Counter | RabbitMQ | Total number of bytes received through RabbitMQ ports. |
erlang_vm_statistics_bytes_output_total |
Counter | RabbitMQ | Total number of bytes output through RabbitMQ ports. |
Prometheus⚓︎
Prometheus batch metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
processed_count |
Counter | Batch peer | Count of items processed. |
processed_failure |
Counter | Batch peer | Count of processing failures. |
processing_duration_histo |
Histogram | Batch peer | Histogram of Batch processing durations (milliseconds). |
processing_duration_summary |
Summary | Batch peer | Summary of Batch processing durations (milliseconds). |
rows_read |
Counter | Batch peer | Count of rows read. |
Prometheus DNS metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
dns_num_ips_in_cache |
Histogram | Any intake peer | The size of the IP pool used in the DNS system. |
dns_lookup_time |
Histogram | Any intake peer | Milliseconds per lookup. |
dns_ttl |
Histogram | Any intake peer | TTLs received per lookup. |
Prometheus general metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
bytes_written |
Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created |
Counter | Batch peer, Stream peer | Count of partitions created. |
process_cpu_seconds_total |
Counter | Batch peer, Stream head, Stream peer | Total user and system CPU time spent in seconds. |
promhttp_metric_handler_requests_total |
Counter | Batch peer, Stream head, Stream peer | Total number of scrapes by HTTP status code. |
process_max_fds |
Gauge | Batch peer, Stream head, Stream peer | Maximum number of open file descriptors. |
process_open_fds |
Gauge | Batch peer, Stream head, Stream peer | Number of open file descriptors. |
process_resident_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Resident memory size in bytes. |
process_start_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Virtual memory size in bytes. |
process_virtual_memory_max_bytes |
Gauge | Batch peer, Stream head, Stream peer | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight |
Gauge | Batch peer, Stream head, Stream peer | Current number of scrapes being served. |
upload_duration |
Summary | Any intake peer | Time spent uploading a file, in milliseconds. |
Prometheus Go metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
go_memstats_alloc_bytes_total |
Counter | Batch peer, Stream head, Stream peer | Total number of bytes allocated, even if freed. |
go_memstats_frees_total |
Counter | Batch peer, Stream head, Stream peer | Total number of frees. |
go_memstats_lookups_total |
Counter | Batch peer, Stream head, Stream peer | Total number of pointer lookups. |
go_memstats_mallocs_total |
Counter | Batch peer, Stream head, Stream peer | Total number of mallocs. |
go_goroutines |
Gauge | Batch peer, Stream head, Stream peer | Number of goroutines that currently exist. |
go_info |
Gauge | Batch peer, Stream head, Stream peer | Information about the Go environment. |
go_memstats_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes allocated and still in use. |
go_memstats_buck_hash_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used by the profiling bucket hash table. |
go_memstats_gc_cpu_fraction |
Gauge | Batch peer, Stream head, Stream peer | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes that are in use. |
go_memstats_heap_objects |
Gauge | Batch peer, Stream head, Stream peer | Number of allocated objects. |
go_memstats_heap_released_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Number of seconds since 1970 of last garbage collection. |
go_memstats_mcache_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system. |
go_threads |
Gauge | Batch peer, Stream head, Stream peer | Number of OS threads created. |
go_gc_duration_seconds |
Summary | Batch peer, Stream head, Stream peer | A summary of the pause duration of garbage collection cycles. |
Prometheus stream ingest metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog growing too big. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog growing too big. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing. Only produced when intake_head_index_backlog_enabled is true. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of errors in Kinesis checkpoint operations. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors in Kinesis source reads. |
kinesis_source_operation_count |
Counter | Stream peer | Count of operations on Kinesis. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in sink backlog waiting to be indexed. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in sink backlog waiting to be indexed. Only produced when intake_head_index_backlog_enabled is true. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis source. |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds. Only produced when intake_head_index_backlog_enabled is true. |
http_source_request_duration_ns |
Histogram | Stream head | A histogram of HTTP request durations in nanoseconds. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of duration of operations on Kinesis. |
Prometheus Kafka metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration. |
Prometheus merge metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
merge_candidate_inactive (deprecated) |
Counter | Merge peer | Merge candidates skipped due to an inactive partition within the candidate. |
merge_success (deprecated) |
Counter | Merge peer | Count of merge successes. |
merge_failure (deprecated) |
Counter | Merge peer | Count of merge failures. |
merge_duration_histo (deprecated) |
Histogram | Merge peer | Merge processing duration, in milliseconds. Buckets double from 1000 to 1024000, covering 1 second to about 17 minutes. |
merge_sdk_duration_histo (deprecated) |
Histogram | Merge peer | Merge SDK processing duration, in milliseconds. Buckets double from 1000 to 1024000, covering 1 second to about 17 minutes. |
merge_candidate_histo (deprecated) |
Histogram | Merge peer | Partitions per merge candidate. |
merge_duration_summary (deprecated) |
Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_summary (deprecated) |
Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
Prometheus operator metrics⚓︎
| Metric | Type | Components | Description |
|---|---|---|---|
o6r_up |
Gauge | Hydrolix Operator | Returns 1 if the Operator is running, otherwise 0. |
o6r_reconcile_seconds_created |
Gauge | Hydrolix Operator | Total seconds spent in the Operator reconcile loop. |
o6r_reconcile_seconds_total |
Counter | Hydrolix Operator | Cumulative total seconds spent in the Operator reconcile loop. |
o6r_hdx_ready |
Gauge | Hydrolix Operator | Reports the hydrolixcluster's current status through a status label. The series whose label matches the current status is 1 and all others are 0. Possible values are Ready, Not Ready, Upgrading, Scaled Off, and Invalid Spec. |
o6r_hdx_synced |
Gauge | Hydrolix Operator | Returns 1 if the Operator is successfully syncing resources, otherwise 0. |
o6r_hdx_warnings |
Gauge | Hydrolix Operator | Number of validation warnings for a hydrolixcluster. |
o6r_hdx_ignored_diffs |
Gauge | Hydrolix Operator | Number of resources with pending changes ignored by the Operator. |
o6r_hdx_htns |
Gauge | Hydrolix Operator | Number of HTNs in use in a hydrolixcluster. |
o6r_hdx_active_overrides |
Gauge | Hydrolix Operator | Number of scheduled overrides currently in their active time window. Derived from the schedule alone, independent of whether the cluster has converged. |
o6r_hdx_override_drift_seconds |
Gauge | Hydrolix Operator | Number of seconds the applied configuration has disagreed with the scheduled override configuration. Returns 0 when the two are in sync. |
Prometheus PostgreSQL metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
pgx_pool_total_acquire_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool. |
pgx_pool_total_acquire_duration_ns_count |
Counter | CatalogDB | Total duration in nanoseconds of all successful acquires from the pool. |
pgx_pool_total_acquire_cancel_count |
Counter | CatalogDB | Cumulative count of acquires from the pool that were canceled by a context. |
pgx_pool_total_acquire_empty_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool that waited for a resource because the pool was empty. |
pgx_pool_total_conns_opened_count |
Counter | CatalogDB | Cumulative count of new connections opened. |
pgx_pool_total_destroyed_max_lifetime_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnLifetime. |
pgx_pool_total_destroyed_max_idle_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnIdleTime. |
pgx_pool_current_size |
Gauge | CatalogDB | Total number of resources currently in the pool. |
pgx_pool_current_constructing |
Gauge | CatalogDB | Number of connections currently being constructed. |
pgx_pool_current_acquired |
Gauge | CatalogDB | Number of connections currently acquired. |
pgx_pool_current_idle |
Gauge | CatalogDB | Number of currently idle connections in the pool. |
pgx_pool_max |
Gauge | CatalogDB | Maximum size of the pool. |
Prometheus query metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempts to the storage service. |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to the storage service, in seconds. |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time for the storage service, in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram of HTTP response times from the storage service, in seconds. |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service. |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempts to the storage service. |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status codes from the storage service. |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit). |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the cache. |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the network. |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram of file evictions from the cache. |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram of CPU usage in seconds. |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram of I/O times in seconds. |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram of wait times for file descriptors, in seconds. |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per-query catalog partition count. |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per-query partition read count. |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per-core partition usage count. |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total. |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of cores used total. |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time-range distribution. |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of columns read. |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding HDX blocks, in seconds. |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening HDX partitions, in seconds. |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading HDX partitions, in seconds. |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partitions skipped due to no matching columns. |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition-block read counts. |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available. |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match). |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup times, in seconds. |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocks, in percentage. |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of total skipped index blocks. |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error). |
hdx_partition_cache_data_read_bytes |
Histogram | Query peer | Histogram of bytes read from the local data cache. Tracks data.hdx bytes served from ephemeral storage |
hdx_partition_net_data_read_bytes |
Histogram | Query peer | Histogram of data.hdx bytes read from cloud object storage. Complementing hdx_partition_cache_data_read_bytes for cache hit ratio analysis |
hdx_partition_bloom_skip |
Histogram | Query peer | Histogram of Bloom pre-filter outcomes per partition (bucket=0 not skipped, bucket=1 skipped), for measuring how many partitions the pre-filter removes from a scan (query acceleration). |
hdx_partition_bloom_false_positive |
Histogram | Query peer | Histogram of Bloom false positives per partition (bucket=0 no, bucket=1 yes), where the filter reported a possible match but the index found no matching blocks; for tracking pre-filter accuracy, since a high rate erodes the benefit. |
hdx_partition_bloom_columns_checked |
Histogram | Query peer | Histogram of columns for which a partition Bloom filter is consulted during query. |
hdx_partition_bloom_check_seconds |
Histogram | Query peer | Histogram of time spent probing Bloom filters per partition, in seconds, for weighing pre-filter overhead against the partitions it skips. |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram of query I/O wait times, in seconds. |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram of query CPU wait times, in seconds. |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting HDX blocks to ClickHouse, in seconds. |
query_health |
Histogram | Head/Query peer | Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error). |
query_peer_availability |
Histogram | Head/Query peer | Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers). |
query_attempts_total |
Histogram | Head/Query peer | Histogram of total query attempts. |
query_response_seconds |
Histogram | Head/Query peer | Histogram of total query response times, in seconds. |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of total query rows read. |
query_read_bytes |
Histogram | Head/Query peer | Histogram of total query read bytes. |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of total query rows written. |
peer_coord_active |
Gauge | Query head | Coordinators currently alive. |
peer_coord_pending_partitions |
Gauge | Query head | Partitions currently waiting for dispatch. |
peer_coord_inflight_partitions |
Gauge | Query head | Partitions currently in dispatched batches. |
peer_coord_connected_peers |
Gauge | Query head | Peers currently connected across all coordinators. |
peer_coord_free_peers |
Gauge | Query head | Peers currently idle and available for the next batch. |
peer_coord_connection_failures_total |
Counter | Query head | Count of peer connection failures. |
peer_coord_partitions_completed_total |
Counter | Query head | Count of partitions completed. |
peer_coord_partitions_errored_total |
Counter | Query head | Count of partition errors. |
peer_coord_batch_processing_seconds |
Histogram | Query head | Wall time from batch dispatch to completion, in seconds. |
hdx_partition_data_cache_path |
Histogram | Query peer | Data cache outcome for one partition. 0 = cold, nothing was cached yet, so data is fetched from object storage and then written to the cache for later queries. 1 = all cached, served entirely from local disk. 2 = partial, some byte ranges came from the cache and the rest were fetched. 3 = no cache, the caching path didn't apply, so data is fetched and not cached, which is what happens for partitions older than data_cache_max_partition_age_seconds. Repeated cold results for the same partitions suggest the cache is too small for the working set. |
hdx_partition_data_cache_actual_http_requests |
Histogram | Query peer | Byte-range HTTP requests actually issued to object storage to read one partition. Reading a partition is split into several byte ranges, so this counts requests per partition rather than per query. |
hdx_partition_data_cache_nocache_http_requests |
Histogram | Query peer | Byte-range HTTP requests that reading one partition would have needed with no cache. Subtract the actual count from this to get the requests the cache avoided. |
hdx_cache_capacity_bytes |
Gauge | Query peer | Configured local file cache quota for the pod. Reports 0 when the cache is disabled. |
hdx_cache_used_bytes |
Gauge | Query peer | Bytes currently occupied by the local file cache. Divide by hdx_cache_capacity_bytes for the cache fill ratio. |
Scheduler metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
scheduler_schedule_attempts_total |
Counter | Scheduler | Scheduling outcomes (result=scheduled/unschedulable/error). |
scheduler_pending_pods |
Gauge | Scheduler | Pods in scheduler queues (activeQ/backoffQ/unschedulable). |
scheduler_scheduling_attempt_duration_seconds |
Histogram | Scheduler | End-to-end scheduling latency per attempt. |
scheduler_pod_scheduling_attempts |
Histogram | Scheduler | Successful scheduling attempts per pod. |
scheduler_queue_incoming_pods_total |
Counter | Scheduler | Pods entering the scheduling pipeline. |
scheduler_plugin_evaluation_total |
Counter | Scheduler | Plugin evaluation calls. |
scheduler_framework_extension_point_duration_seconds |
Histogram | Scheduler | Extension point latency (Filter/Score/Bind). |
scheduler_plugin_execution_duration_seconds |
Histogram | Scheduler | Per-plugin execution latency (NodeResourcesFit etc.). |
scheduler_inflight_events |
Gauge | Scheduler | Inflight scheduling events. |
scheduler_cache_size |
Gauge | Scheduler | Scheduler cache entries (nodes/pods/assumed). |
scheduler_goroutines |
Gauge | Scheduler | Goroutine count. |
scheduler_preemption_attempts_total |
Counter | Scheduler | Preemption attempts. |
scheduler_unschedulable_pods |
Gauge | Scheduler | Not emitted when 0 unschedulable pods exist. |
Ingest metrics⚓︎
General metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
process_cpu_seconds_total |
Counter | Batch peer, Stream head, Stream peer | Total user and system CPU time spent in seconds. |
promhttp_metric_handler_requests_total |
Counter | Batch peer, Stream head, Stream peer | Total number of scrapes by HTTP status code. |
process_max_fds |
Gauge | Batch peer, Stream head, Stream peer | Maximum number of open file descriptors. |
process_open_fds |
Gauge | Batch peer, Stream head, Stream peer | Number of open file descriptors. |
process_resident_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Resident memory size in bytes. |
process_start_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Virtual memory size in bytes. |
process_virtual_memory_max_bytes |
Gauge | Batch peer, Stream head, Stream peer | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight |
Gauge | Batch peer, Stream head, Stream peer | Current number of scrapes being served. |
Go metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
go_memstats_alloc_bytes_total |
Counter | Batch peer, Stream head, Stream peer | Total number of bytes allocated, even if freed. |
go_memstats_frees_total |
Counter | Batch peer, Stream head, Stream peer | Total number of frees. |
go_memstats_lookups_total |
Counter | Batch peer, Stream head, Stream peer | Total number of pointer lookups. |
go_memstats_mallocs_total |
Counter | Batch peer, Stream head, Stream peer | Total number of mallocs. |
go_goroutines |
Gauge | Batch peer, Stream head, Stream peer | Number of goroutines that currently exist. |
go_info |
Gauge | Batch peer, Stream head, Stream peer | Information about the Go environment. |
go_memstats_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes allocated and still in use. |
go_memstats_buck_hash_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used by the profiling bucket hash table. |
go_memstats_gc_cpu_fraction |
Gauge | Batch peer, Stream head, Stream peer | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes that are in use. |
go_memstats_heap_objects |
Gauge | Batch peer, Stream head, Stream peer | Number of allocated objects. |
go_memstats_heap_released_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Number of seconds since 1970 of last garbage collection. |
go_memstats_mcache_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system. |
go_threads |
Gauge | Batch peer, Stream head, Stream peer | Number of OS threads created. |
go_gc_duration_seconds |
Summary | Batch peer, Stream head, Stream peer | A summary of the pause duration of garbage collection cycles. |
Indexer metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_row_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of rows processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_byte_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of bytes processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_value_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of values processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_error_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of errors in indexing and uploading to storage. |
indexer_rows_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total rows indexed (written to partitions). |
indexer_bytes_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total bytes indexed (written to partitions). |
indexer_partitions_rejected_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Histogram of partitions not written. Values: 0 = raw data parsing failed, 1 = raw data or transform schema mismatch, 3 = error writing partition file, 4 = other error during indexing. |
indexer_partitions_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total partitions created. |
indexer_partition_write_seconds_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Time from receiving indexing query to writing partition file, in seconds. |
hdx_partition_bloom_columns_total_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Number of columns whose partitions are written with Bloom filters. Useful for monitoring Bloom filter coverage. String and uint128 (ip, uuid) columns use partition Bloom filters. |
hdx_partition_bloom_bytes_total_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total Bloom filter bytes per partition at write time, for monitoring the storage overhead Bloom filters add. |
hdx_partition_bloom_build_seconds_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Time to build Bloom filters per partition at write time, in seconds, for monitoring Bloom build cost during indexing. |
Redpanda metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
internal_event_queue_byte_count{mode="sink"} |
Counter | Stream head | Byte count sent to Redpanda. |
internal_event_queue_row_count{mode="sink"} |
Counter | Stream head | Row count sent to Redpanda. |
internal_event_queue_row_count{mode="source"} |
Counter | Stream Summary | Row count received from Redpanda. |
internal_event_queue_value_count{mode="source"} |
Counter | Stream Summary | Value count received from Redpanda. |
Storage metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
object_store_http_status_code_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful. |
object_store_http_error_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of HTTP request errors. For example, timeouts and connection errors. |
object_store_http_bytes_tx |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes transmitted to object storage (request body only). |
object_store_http_bytes_rx |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes received from object storage (response body only). |
net_http_status_code_bucket |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | HTTP status code histogram count from storage. |
object_store_http_histo |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A histogram of object storage interaction latencies. |
object_store_http_summary |
Summary | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A summary of object storage interaction latencies. |
Stream and intake head metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Stream summary metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_count |
Counter | Summary Peer | Count of calls to the catalog. |
query_failure |
Counter | Summary Peer | Count of failed catalog calls. |
query_latency_summary |
Counter | Summary Peer | Latency of calls to catalog. |
Traefik metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
traefik_service_requests_total |
Counter | Traefik | HTTP Traefik request information. |
traefik_service_request_duration_seconds_count/sum/bucket |
Counter | Traefik | Response time of Traefik to client. |
http_source_request_duration_ns_count/sum/bucket |
Counter | Traefik | Response time from Stream Head. |