Overview
This page contains most metrics used in Hydrolix for the following:
Use find on page to locate specific metrics.
Custom Hydrolix metrics⚓︎
Custom general metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
bytes_written |
Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created |
Counter | Batch peer, Stream peer | Count of partitions created. |
upload_duration |
Summary | Any intake peer | Time spent uploading a file, in milliseconds. |
Custom query metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempts to the storage service. |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to the storage service, in seconds. |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time for the storage service, in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram of HTTP response times from the storage service, in seconds. |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service. |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempts to the storage service. |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status codes from the storage service. |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit). |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the cache. |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the network. |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram of file evictions from the cache. |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram of CPU usage in seconds. |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram of I/O times in seconds. |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram of wait times for file descriptors, in seconds. |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per-query catalog partition count. |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per-query partition read count. |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per-core partition usage count. |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total. |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of cores used total. |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time-range distribution. |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of columns read. |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding HDX blocks, in seconds. |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening HDX partitions, in seconds. |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading HDX partitions, in seconds. |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partitions skipped due to no matching columns. |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition-block read counts. |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available. |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match). |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup times, in seconds. |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocks, in percentage. |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of total skipped index blocks. |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error). |
hdx_partition_cache_data_read_bytes |
Histogram | Query peer | Histogram of bytes read from the local data cache. Tracks data.hdx bytes served from ephemeral storage |
hdx_partition_net_data_read_bytes |
Histogram | Query peer | Histogram of data.hdx bytes read from cloud object storage. Complementing hdx_partition_cache_data_read_bytes for cache hit ratio analysis |
hdx_partition_bloom_skip |
Histogram | Query peer | Histogram of bloom pre-filter outcomes per partition (bucket=0 not skipped, bucket=1 skipped), for measuring how many partitions the pre-filter removes from a scan (query acceleration). |
hdx_partition_bloom_false_positive |
Histogram | Query peer | Histogram of bloom false positives per partition (bucket=0 no, bucket=1 yes), where the filter reported a possible match but the index found no matching blocks; for tracking pre-filter accuracy, since a high rate erodes the benefit. |
hdx_partition_bloom_columns_checked |
Histogram | Query peer | Histogram of bloom filter columns probed per partition during query pre-filtering, for confirming the pre-filter is engaging on a query. |
hdx_partition_bloom_check_seconds |
Histogram | Query peer | Histogram of time spent probing bloom filters per partition, in seconds, for weighing pre-filter overhead against the partitions it skips. |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram of query I/O wait times, in seconds. |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram of query CPU wait times, in seconds. |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting HDX blocks to ClickHouse, in seconds. |
query_health |
Histogram | Head/Query peer | Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error). |
query_peer_availability |
Histogram | Head/Query peer | Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers). |
query_attempts_total |
Histogram | Head/Query peer | Histogram of total query attempts. |
query_response_seconds |
Histogram | Head/Query peer | Histogram of total query response times, in seconds. |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of total query rows read. |
query_read_bytes |
Histogram | Head/Query peer | Histogram of total query read bytes. |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of total query rows written. |
peer_coord_active |
Gauge | Query head | Coordinators currently alive. |
peer_coord_pending_partitions |
Gauge | Query head | Partitions currently waiting for dispatch. |
peer_coord_inflight_partitions |
Gauge | Query head | Partitions currently in dispatched batches. |
peer_coord_connected_peers |
Gauge | Query head | Peers currently connected across all coordinators. |
peer_coord_free_peers |
Gauge | Query head | Peers currently idle and available for the next batch. |
peer_coord_connection_failures_total |
Counter | Query head | Count of peer connection failures. |
peer_coord_partitions_completed_total |
Counter | Query head | Count of partitions completed. |
peer_coord_partitions_errored_total |
Counter | Query head | Count of partition errors. |
peer_coord_batch_processing_seconds |
Histogram | Query head | Wall time from batch dispatch to completion, in seconds. |
Custom batch metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
processed_count |
Counter | Batch peer | Count of items processed. |
processed_failure |
Counter | Batch peer | Count of processing failures. |
processing_duration_histo |
Histogram | Batch peer | Histogram of Batch processing durations (milliseconds). |
processing_duration_summary |
Summary | Batch peer | Summary of Batch processing durations (milliseconds). |
rows_read |
Counter | Batch peer | Count of rows read. |
Custom batch controller metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
batch_peer_idle_duration_seconds |
Histogram | Batch controller | Time peers spend waiting for work after completing a job. High values indicate controller bottlenecks. |
batch_work_request_duration_seconds |
Histogram | Batch controller | Time to process a work request. |
batch_job_completion_duration_seconds |
Histogram | Batch controller | Time to process job completion. |
batch_connected_peers |
Gauge | Batch controller | Number of currently connected batch peers. |
batch_active_leases |
Gauge | Batch controller | Number of active job leases (jobs currently being processed). |
batch_jobs_created_total |
Counter | Batch controller | Total jobs created. |
batch_jobs_completed_total |
Counter | Batch controller | Jobs completed by status. |
batch_job_creation_duration_seconds |
Histogram | Batch controller | Time to create and enqueue a new job. |
batch_work_requests_total |
Counter | Batch controller | Work requests by outcome. |
batch_bytes_indexed |
Counter | Batch controller | Total bytes indexed by batch peers. |
batch_job_processing_duration_seconds |
Histogram | Batch controller | Time peers spend processing jobs (reported by peers on completion). |
batch_queue_depth |
Gauge | Batch controller | Current queue depth. |
batch_cache_size |
Gauge | Batch controller | Current cache size. |
batch_persistence_queue_depth |
Gauge | Batch controller | Async persistence queue depth. High values indicate storage write bottleneck. |
batch_lease_acquisitions_total |
Counter | Batch controller | Lease acquisition attempts. |
batch_lease_expirations_total |
Counter | Batch controller | Leases that expired (worker timeout). |
batch_state_transitions_total |
Counter | Batch controller | Job state machine transitions. |
batch_persistence_action_duration_seconds |
Histogram | Batch controller | Time for persistence operations. |
batch_storage_requests_total |
Counter | Batch controller | Storage API calls by operation. |
batch_storage_errors_total |
Counter | Batch controller | Storage errors by type. |
batch_storage_rate_limits_total |
Counter | Batch controller | Rate limit responses from object store when writing. |
batch_storage_inflight |
Gauge | Batch controller | Currently in-flight storage operations. |
batch_storage_duration_seconds |
Histogram | Batch controller | Storage operation latency. |
Custom merge metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
merge_duration_summary |
Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_duration_histo |
Histogram | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_summary |
Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_sdk_duration_histo |
Histogram | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_candidate_histo |
Histogram | Merge peer | Partitions per merge candidate. |
merge_candidate_inactive |
Counter | Merge peer | Merge candidates skipped due to an inactive partition within the candidate. |
merge_success |
Counter | Merge peer | Count of merge successes. |
merge_failure |
Counter | Merge peer | Count of merge failures. |
Custom streaming metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in the backlog (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds (requires intake_head_index_backlog_enabled = true). |
http_source_outstanding_reqs |
Gauge | Intake head | Number of outstanding ingest event requests. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of HTTP requests. |
http_source_request_duration_ns |
Histogram | Stream head | Histogram of HTTP request durations in nanoseconds. |
http_source_request_error_count |
Counter | Stream head | Count of HTTP request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Custom AWS Kinesis metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations, in nanoseconds. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of Kinesis checkpoint operation errors. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors reading from Kinesis. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis, in milliseconds. |
kinesis_source_operation_count |
Counter | Stream peer | Count of Kinesis operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of Kinesis operation durations, in nanoseconds. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
Custom Kafka metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration, in nanoseconds. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration, in nanoseconds. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed from Kafka. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed from Kafka. |
Custom DNS metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
dns_num_ips_in_cache |
Histogram | Any intake peer | The size of the IP pool used in the DNS system. |
dns_lookup_time |
Histogram | Any intake peer | Milliseconds per lookup. |
dns_ttl |
Histogram | Any intake peer | TTLs received per lookup. |
Custom PostgreSQL metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
pgx_pool_total_acquire_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool. |
pgx_pool_total_acquire_duration_ns_count |
Counter | CatalogDB | Total duration in nanoseconds of all successful acquires from the pool. |
pgx_pool_total_acquire_cancel_count |
Counter | CatalogDB | Cumulative count of acquires from the pool that were canceled by a context. |
pgx_pool_total_acquire_empty_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool that waited for a resource because the pool was empty. |
pgx_pool_total_conns_opened_count |
Counter | CatalogDB | Cumulative count of new connections opened. |
pgx_pool_total_destroyed_max_lifetime_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnLifetime. |
pgx_pool_total_destroyed_max_idle_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnIdleTime. |
pgx_pool_current_size |
Gauge | CatalogDB | Total number of resources currently in the pool. |
pgx_pool_current_constructing |
Gauge | CatalogDB | Number of connections currently being constructed. |
pgx_pool_current_acquired |
Gauge | CatalogDB | Number of connections currently acquired. |
pgx_pool_current_idle |
Gauge | CatalogDB | Number of currently idle connections in the pool. |
pgx_pool_max |
Gauge | CatalogDB | Maximum size of the pool. |
Custom partition cleaner metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
bulk_delete_duration |
Summary | Partition cleaner | Time spent performing bulk deletes. |
bulk_delete_success_count |
Counter | Partition cleaner | Files deleted. |
bulk_delete_failure_count |
Counter | Partition cleaner | Files unable to be deleted. |
bulk_delete_bytes |
Counter | Partition cleaner | Bytes deleted from storage. |
Custom scaler metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
hdxscaler_goroutines |
Gauge | hdx-scaler-go | Current number of goroutines. |
hdxscaler_kubelet_fetch_total |
Counter | hdx-scaler-go | Total kubelet stats/summary fetches by result (hit, miss, error). |
hdxscaler_kubelet_fetch_duration_seconds |
Histogram | hdx-scaler-go | Kubelet stats/summary API call latency (cache misses only). |
hdxscaler_kubelet_cache_entries |
Gauge | hdx-scaler-go | Number of entries in the kubelet stats cache. |
hdxscaler_sync_pods_total |
Counter | hdx-scaler-go | Total ListPods API calls per app. |
hdxscaler_event_cache_entries |
Gauge | hdx-scaler-go | Number of entries in the event watcher cache. |
hdxscaler_restart_count_entries |
Gauge | hdx-scaler-go | Number of entries in the OOM restart count tracker. |
hdxscaler_scrapes_total |
Counter | hdx-scaler-go | Total number of scrape requests. |
Age metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
reaper_stale_storage_total |
Counter | Reaper | Count of reap events where the storage ID was missing from the published configuration. |
reaper_storage_not_found_total |
Counter | Reaper | Count of reap events where the underlying storage returned a not-found error on a list or delete. |
query_count (deprecated) |
Counter | Reaper | Count of Reaper calls to the catalog. |
query_failure (deprecated) |
Counter | Reaper | Count of failed Reaper catalog calls. |
query_latency_summary (deprecated) |
Summary | Reaper | Latency of calls to the catalog. |
Alter metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_count (deprecated) |
Counter | Alter | Count of Alter calls to the catalog. |
query_failure (deprecated) |
Counter | Alter | Count of failed Alter catalog calls. |
query_latency_summary (deprecated) |
Summary | Alter | Latency of calls to the catalog. |
ClickProx metrics⚓︎
| Metric Name | Type | Components | Purpose | Labels |
|---|---|---|---|---|
clickprox_frontend_connections_total |
Gauge | ClickProx | Current number of client connections to ClickProx. | |
clickprox_backend_connections_total |
Gauge | ClickProx | Current number of back-end connections per query head pool. | pool |
clickprox_exception_packets_sent_total |
Counter | ClickProx | Count of ClickHouse exception packets sent to clients. | type pool |
clickprox_backend_connection_failures_total |
Counter | ClickProx | Count of failed back-end connection attempts. | pool |
clickprox_dropped_connections_total |
Counter | ClickProx | Count of connections dropped due to errors. | pool direction |
clickprox_routing_decisions_total |
Counter | ClickProx | Count of routing decisions by matching criterion. | criterion value pool |
Descheduler metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
descheduler_pods_evicted_total |
Counter | Descheduler | Total pods evicted since start. |
descheduler_pods_evicted |
Gauge | Descheduler | Pods evicted in last descheduler loop. |
descheduler_loop_duration_seconds |
Histogram | Descheduler | Descheduler loop duration. |
descheduler_strategy_duration_seconds |
Histogram | Descheduler | Strategy execution duration per loop. |
Intake spill metrics⚓︎
| Metric Name | Type | Components | Description |
|---|---|---|---|
hdx_sink_spill_catalog_accepted_count |
Counter | Intake head | Count of spilled catalog entries accepted into the indexing pipeline for processing. |
hdx_sink_spill_catalog_duration_ns |
Histogram | Intake head | Duration in nanoseconds taken to spill catalog adds. |
hdx_sink_spill_catalog_failure_count |
Counter | Intake head | Count of spill failures for catalog adds resulting in lost data. |
hdx_sink_spill_catalog_race_lost_count |
Counter | Intake head | Count of lost races attempting to add a spilled data entry to the catalog. |
hdx_sink_spill_raw_accepted_count |
Counter | Intake head | Count of spilled raw data entries accepted into the indexing pipeline for processing. |
hdx_sink_spill_raw_duration_ns |
Histogram | Intake head | Duration in nanoseconds taken to spill raw data buckets. |
hdx_sink_spill_raw_failure_count |
Counter | Intake head | Count of spill failures for raw data resulting in lost data. |
Merge metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
merge_success (deprecated) |
Counter | Merge peer | Count of merge successes. |
merge_failure (deprecated) |
Counter | Merge peer | Count of merge failures. |
query_count (deprecated) |
Counter | Merge peer | Count of Merge calls to the catalog. |
query_failure (deprecated) |
Counter | Merge peer | Count of failed Merge catalog calls. |
query_latency_summary (deprecated) |
Summary | Merge peer | Latency of calls to the catalog. |
merge_duration_summary (deprecated) |
Summary | Merge peer | Merge processing duration, in milliseconds. |
Merge controller metrics⚓︎
| Metric Name | Type | Components | Description | Labels |
|---|---|---|---|---|
partition_distribution |
Histogram | Merge Controller | Histogram of partition memory size. | project_id project_name table_id table_name target |
segments |
Gauge | Merge Controller | Current count of known segments. A segment is a group of buckets that share a merge target. | project_id project_name table_id table_name target |
active_merges |
Gauge | Merge Controller | Current count of active merge operations. | project_id project_name table_id table_name target |
partitions |
Gauge | Merge Controller | Current count of partitions awaiting segmentation. | project_id project_name table_id table_name target |
partitions_dispatched |
Counter | Merge Controller | Count of partitions dispatched for merging. Determine "partitions dispatched for merge per second" by combining with rate function. | project_id project_name table_id table_name target |
candidates |
Gauge | Merge Controller | Current count of constructed candidates waiting dispatch. | project_id project_name table_id table_name target |
candidates_dispatched |
Counter | Merge Controller | Count of candidates dispatched for merging. Determine "candidates dispatched for merge per second" by combining with rate function. | project_id project_name table_id table_name target |
partitions_per_candidate |
Histogram | Merge Controller | Distribution of partitions within a dispatched candidate. | project_id project_name table_id table_name target |
candidate_mem_size |
Histogram | Merge Controller | Distribution of calculated memory requirements for dispatched candidates. | project_id project_name table_id table_name target |
duplicate_partitions |
Counter | Merge Controller | Count of partitions sourced that are already being tracked. | project_id project_name table_id table_name target |
memory_coefficient |
Gauge | Merge Controller | Current value of the memory coefficient being applied to sourced partitions. | project_id project_name table_id table_name target |
expired_segments |
Counter | Merge Controller | Count of segments expired for moving out of the target range. | project_id project_name table_id table_name target |
bucket_duration |
Histogram | Merge Controller | Distribution of time in milliseconds of a bucket's age upon closing. The basis label gives the reason for closing: full = the bucket reached max size, idle_ttl = the bucket wasn't added to within the idle timeout, age_ttl = the bucket was open longer than the max age, segment_ttl = the bucket was part of a segment that fell out of the target range. |
project_id project_name table_id table_name target basis |
connected_clients |
Gauge | Merge Controller | Count of currently connected merge-peers. | pool_id |
query_latency |
Histogram | Merge Controller | Distribution of response times in milliseconds for catalog operations. | project_id table_id method |
admin_query_latency |
Histogram | Merge Controller | Distribution of response times in milliseconds for admin catalog operations. | project_id table_id method |
state_duration |
Histogram | Merge Controller | Distribution of time in milliseconds spent in each reactor state. | project_id table_id state |
merge_duty_cycle |
Gauge | Merge Controller | Overall work rate of the merge-controller, as a value between 0 and 1. | project_id project_name table_id table_name target |
RabbitMQ metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
rabbitmq_queue_messages |
Gauge | RabbitMQ | Sum of ready and unacknowledged messages (queue depth). |
rabbitmq_queues |
Gauge | RabbitMQ | Count of RabbitMQ queues. |
erlang_vm_statistics_bytes_received_total |
Counter | RabbitMQ | Total number of bytes received through RabbitMQ ports. |
erlang_vm_statistics_bytes_output_total |
Counter | RabbitMQ | Total number of bytes output through RabbitMQ ports. |
Prometheus⚓︎
Prometheus batch metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
processed_count |
Counter | Batch peer | Count of items processed. |
processed_failure |
Counter | Batch peer | Count of processing failures. |
processing_duration_histo |
Histogram | Batch peer | Histogram of Batch processing durations (milliseconds). |
processing_duration_summary |
Summary | Batch peer | Summary of Batch processing durations (milliseconds). |
rows_read |
Counter | Batch peer | Count of rows read. |
Prometheus DNS metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
dns_num_ips_in_cache |
Histogram | Any intake peer | The size of the IP pool used in the DNS system. |
dns_lookup_time |
Histogram | Any intake peer | Milliseconds per lookup. |
dns_ttl |
Histogram | Any intake peer | TTLs received per lookup. |
Prometheus general metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
bytes_written |
Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created |
Counter | Batch peer, Stream peer | Count of partitions created. |
process_cpu_seconds_total |
Counter | Batch peer, Stream head, Stream peer | Total user and system CPU time spent in seconds. |
promhttp_metric_handler_requests_total |
Counter | Batch peer, Stream head, Stream peer | Total number of scrapes by HTTP status code. |
process_max_fds |
Gauge | Batch peer, Stream head, Stream peer | Maximum number of open file descriptors. |
process_open_fds |
Gauge | Batch peer, Stream head, Stream peer | Number of open file descriptors. |
process_resident_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Resident memory size in bytes. |
process_start_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Virtual memory size in bytes. |
process_virtual_memory_max_bytes |
Gauge | Batch peer, Stream head, Stream peer | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight |
Gauge | Batch peer, Stream head, Stream peer | Current number of scrapes being served. |
upload_duration |
Summary | Any intake peer | Time spent uploading a file, in milliseconds. |
Prometheus Go metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
go_memstats_alloc_bytes_total |
Counter | Batch peer, Stream head, Stream peer | Total number of bytes allocated, even if freed. |
go_memstats_frees_total |
Counter | Batch peer, Stream head, Stream peer | Total number of frees. |
go_memstats_lookups_total |
Counter | Batch peer, Stream head, Stream peer | Total number of pointer lookups. |
go_memstats_mallocs_total |
Counter | Batch peer, Stream head, Stream peer | Total number of mallocs. |
go_goroutines |
Gauge | Batch peer, Stream head, Stream peer | Number of goroutines that currently exist. |
go_info |
Gauge | Batch peer, Stream head, Stream peer | Information about the Go environment. |
go_memstats_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes allocated and still in use. |
go_memstats_buck_hash_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used by the profiling bucket hash table. |
go_memstats_gc_cpu_fraction |
Gauge | Batch peer, Stream head, Stream peer | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes that are in use. |
go_memstats_heap_objects |
Gauge | Batch peer, Stream head, Stream peer | Number of allocated objects. |
go_memstats_heap_released_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Number of seconds since 1970 of last garbage collection. |
go_memstats_mcache_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system. |
go_threads |
Gauge | Batch peer, Stream head, Stream peer | Number of OS threads created. |
go_gc_duration_seconds |
Summary | Batch peer, Stream head, Stream peer | A summary of the pause duration of garbage collection cycles. |
Prometheus stream ingest metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog growing too big. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog growing too big. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing. Only produced when intake_head_index_backlog_enabled is true. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of errors in Kinesis checkpoint operations. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors in Kinesis source reads. |
kinesis_source_operation_count |
Counter | Stream peer | Count of operations on Kinesis. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in sink backlog waiting to be indexed. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in sink backlog waiting to be indexed. Only produced when intake_head_index_backlog_enabled is true. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis source. |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds. Only produced when intake_head_index_backlog_enabled is true. |
http_source_request_duration_ns |
Histogram | Stream head | A histogram of HTTP request durations in nanoseconds. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of duration of operations on Kinesis. |
Prometheus Kafka metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration. |
Prometheus merge metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
merge_candidate_inactive (deprecated) |
Counter | Merge peer | Merge candidates skipped due to an inactive partition within the candidate. |
merge_success (deprecated) |
Counter | Merge peer | Count of merge successes. |
merge_failure (deprecated) |
Counter | Merge peer | Count of merge failures. |
merge_duration_histo (deprecated) |
Histogram | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_histo (deprecated) |
Histogram | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_candidate_histo (deprecated) |
Histogram | Merge peer | Partitions per merge candidate. |
merge_duration_summary (deprecated) |
Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_summary (deprecated) |
Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
Prometheus operator metrics⚓︎
| Metric | Type | Components | Description |
|---|---|---|---|
o6r_up |
Gauge | Hydrolix Operator | Returns 1 if the Operator is running, otherwise 0. |
o6r_reconcile_seconds_created |
Gauge | Hydrolix Operator | Total seconds spent in the Operator reconcile loop. |
o6r_reconcile_seconds_total |
Counter | Hydrolix Operator | Cumulative total seconds spent in the Operator reconcile loop. |
o6r_hdx_ready |
Gauge | Hydrolix Operator | Reports the hydrolixcluster's current status through a status label. The series whose label matches the current status is 1 and all others are 0. Possible values are Ready, Not Ready, Upgrading, Scaled Off, and Invalid Spec. |
o6r_hdx_synced |
Gauge | Hydrolix Operator | Returns 1 if the Operator is successfully syncing resources, otherwise 0. |
o6r_hdx_warnings |
Gauge | Hydrolix Operator | Number of validation warnings for a hydrolixcluster. |
o6r_hdx_ignored_diffs |
Gauge | Hydrolix Operator | Number of resources with pending changes ignored by the Operator. |
o6r_hdx_htns |
Gauge | Hydrolix Operator | Number of HTNs in use in a hydrolixcluster. |
Prometheus PostgreSQL metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
pgx_pool_total_acquire_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool. |
pgx_pool_total_acquire_duration_ns_count |
Counter | CatalogDB | Total duration in nanoseconds of all successful acquires from the pool. |
pgx_pool_total_acquire_cancel_count |
Counter | CatalogDB | Cumulative count of acquires from the pool that were canceled by a context. |
pgx_pool_total_acquire_empty_count |
Counter | CatalogDB | Cumulative count of successful acquires from the pool that waited for a resource because the pool was empty. |
pgx_pool_total_conns_opened_count |
Counter | CatalogDB | Cumulative count of new connections opened. |
pgx_pool_total_destroyed_max_lifetime_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnLifetime. |
pgx_pool_total_destroyed_max_idle_count |
Counter | CatalogDB | Cumulative count of connections destroyed because they exceeded MaxConnIdleTime. |
pgx_pool_current_size |
Gauge | CatalogDB | Total number of resources currently in the pool. |
pgx_pool_current_constructing |
Gauge | CatalogDB | Number of connections currently being constructed. |
pgx_pool_current_acquired |
Gauge | CatalogDB | Number of connections currently acquired. |
pgx_pool_current_idle |
Gauge | CatalogDB | Number of currently idle connections in the pool. |
pgx_pool_max |
Gauge | CatalogDB | Maximum size of the pool. |
Prometheus query metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempts to the storage service. |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to the storage service, in seconds. |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time for the storage service, in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram of HTTP response times from the storage service, in seconds. |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service. |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempts to the storage service. |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status codes from the storage service. |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit). |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the cache. |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the network. |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram of file evictions from the cache. |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram of CPU usage in seconds. |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram of I/O times in seconds. |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram of wait times for file descriptors, in seconds. |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per-query catalog partition count. |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per-query partition read count. |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per-core partition usage count. |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total. |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of cores used total. |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time-range distribution. |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of columns read. |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding HDX blocks, in seconds. |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening HDX partitions, in seconds. |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading HDX partitions, in seconds. |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partitions skipped due to no matching columns. |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition-block read counts. |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available. |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match). |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup times, in seconds. |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocks, in percentage. |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of total skipped index blocks. |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error). |
hdx_partition_cache_data_read_bytes |
Histogram | Query peer | Histogram of bytes read from the local data cache. Tracks data.hdx bytes served from ephemeral storage |
hdx_partition_net_data_read_bytes |
Histogram | Query peer | Histogram of data.hdx bytes read from cloud object storage. Complementing hdx_partition_cache_data_read_bytes for cache hit ratio analysis |
hdx_partition_bloom_skip |
Histogram | Query peer | Histogram of bloom pre-filter outcomes per partition (bucket=0 not skipped, bucket=1 skipped), for measuring how many partitions the pre-filter removes from a scan (query acceleration). |
hdx_partition_bloom_false_positive |
Histogram | Query peer | Histogram of bloom false positives per partition (bucket=0 no, bucket=1 yes), where the filter reported a possible match but the index found no matching blocks; for tracking pre-filter accuracy, since a high rate erodes the benefit. |
hdx_partition_bloom_columns_checked |
Histogram | Query peer | Histogram of bloom filter columns probed per partition during query pre-filtering, for confirming the pre-filter is engaging on a query. |
hdx_partition_bloom_check_seconds |
Histogram | Query peer | Histogram of time spent probing bloom filters per partition, in seconds, for weighing pre-filter overhead against the partitions it skips. |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram of query I/O wait times, in seconds. |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram of query CPU wait times, in seconds. |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting HDX blocks to ClickHouse, in seconds. |
query_health |
Histogram | Head/Query peer | Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error). |
query_peer_availability |
Histogram | Head/Query peer | Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers). |
query_attempts_total |
Histogram | Head/Query peer | Histogram of total query attempts. |
query_response_seconds |
Histogram | Head/Query peer | Histogram of total query response times, in seconds. |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of total query rows read. |
query_read_bytes |
Histogram | Head/Query peer | Histogram of total query read bytes. |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of total query rows written. |
peer_coord_active |
Gauge | Query head | Coordinators currently alive. |
peer_coord_pending_partitions |
Gauge | Query head | Partitions currently waiting for dispatch. |
peer_coord_inflight_partitions |
Gauge | Query head | Partitions currently in dispatched batches. |
peer_coord_connected_peers |
Gauge | Query head | Peers currently connected across all coordinators. |
peer_coord_free_peers |
Gauge | Query head | Peers currently idle and available for the next batch. |
peer_coord_connection_failures_total |
Counter | Query head | Count of peer connection failures. |
peer_coord_partitions_completed_total |
Counter | Query head | Count of partitions completed. |
peer_coord_partitions_errored_total |
Counter | Query head | Count of partition errors. |
peer_coord_batch_processing_seconds |
Histogram | Query head | Wall time from batch dispatch to completion, in seconds. |
Scheduler metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
scheduler_schedule_attempts_total |
Counter | Scheduler | Scheduling outcomes (result=scheduled/unschedulable/error). |
scheduler_pending_pods |
Gauge | Scheduler | Pods in scheduler queues (activeQ/backoffQ/unschedulable). |
scheduler_scheduling_attempt_duration_seconds |
Histogram | Scheduler | End-to-end scheduling latency per attempt. |
scheduler_pod_scheduling_attempts |
Histogram | Scheduler | Successful scheduling attempts per pod. |
scheduler_queue_incoming_pods_total |
Counter | Scheduler | Pods entering the scheduling pipeline. |
scheduler_plugin_evaluation_total |
Counter | Scheduler | Plugin evaluation calls. |
scheduler_framework_extension_point_duration_seconds |
Histogram | Scheduler | Extension point latency (Filter/Score/Bind). |
scheduler_plugin_execution_duration_seconds |
Histogram | Scheduler | Per-plugin execution latency (NodeResourcesFit etc.). |
scheduler_inflight_events |
Gauge | Scheduler | Inflight scheduling events. |
scheduler_cache_size |
Gauge | Scheduler | Scheduler cache entries (nodes/pods/assumed). |
scheduler_goroutines |
Gauge | Scheduler | Goroutine count. |
scheduler_preemption_attempts_total |
Counter | Scheduler | Preemption attempts. |
scheduler_unschedulable_pods |
Gauge | Scheduler | Not emitted when 0 unschedulable pods exist. |
Ingest metrics⚓︎
General metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
process_cpu_seconds_total |
Counter | Batch peer, Stream head, Stream peer | Total user and system CPU time spent in seconds. |
promhttp_metric_handler_requests_total |
Counter | Batch peer, Stream head, Stream peer | Total number of scrapes by HTTP status code. |
process_max_fds |
Gauge | Batch peer, Stream head, Stream peer | Maximum number of open file descriptors. |
process_open_fds |
Gauge | Batch peer, Stream head, Stream peer | Number of open file descriptors. |
process_resident_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Resident memory size in bytes. |
process_start_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Virtual memory size in bytes. |
process_virtual_memory_max_bytes |
Gauge | Batch peer, Stream head, Stream peer | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight |
Gauge | Batch peer, Stream head, Stream peer | Current number of scrapes being served. |
Go metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
go_memstats_alloc_bytes_total |
Counter | Batch peer, Stream head, Stream peer | Total number of bytes allocated, even if freed. |
go_memstats_frees_total |
Counter | Batch peer, Stream head, Stream peer | Total number of frees. |
go_memstats_lookups_total |
Counter | Batch peer, Stream head, Stream peer | Total number of pointer lookups. |
go_memstats_mallocs_total |
Counter | Batch peer, Stream head, Stream peer | Total number of mallocs. |
go_goroutines |
Gauge | Batch peer, Stream head, Stream peer | Number of goroutines that currently exist. |
go_info |
Gauge | Batch peer, Stream head, Stream peer | Information about the Go environment. |
go_memstats_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes allocated and still in use. |
go_memstats_buck_hash_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used by the profiling bucket hash table. |
go_memstats_gc_cpu_fraction |
Gauge | Batch peer, Stream head, Stream peer | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes that are in use. |
go_memstats_heap_objects |
Gauge | Batch peer, Stream head, Stream peer | Number of allocated objects. |
go_memstats_heap_released_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Number of seconds since 1970 of last garbage collection. |
go_memstats_mcache_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system. |
go_threads |
Gauge | Batch peer, Stream head, Stream peer | Number of OS threads created. |
go_gc_duration_seconds |
Summary | Batch peer, Stream head, Stream peer | A summary of the pause duration of garbage collection cycles. |
Indexer metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_row_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of rows processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_byte_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of bytes processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_value_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of values processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_error_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of errors in indexing and uploading to storage. |
indexer_rows_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total rows indexed (written to partitions). |
indexer_bytes_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total bytes indexed (written to partitions). |
indexer_partitions_rejected_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Histogram of partitions not written. Values: 0 = raw data parsing failed, 1 = raw data or transform schema mismatch, 3 = error writing partition file, 4 = other error during indexing. |
indexer_partitions_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total partitions created. |
indexer_partition_write_seconds_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Time from receiving indexing query to writing partition file, in seconds. |
hdx_partition_bloom_columns_total_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Number of columns given a bloom filter per partition at write time, for monitoring bloom filter coverage. Only high-cardinality indexed string columns get a filter. |
hdx_partition_bloom_bytes_total_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total bloom filter bytes per partition at write time, for monitoring the storage overhead bloom filters add. |
hdx_partition_bloom_build_seconds_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Time to build bloom filters per partition at write time, in seconds, for monitoring bloom build cost during indexing. |
Redpanda metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
internal_event_queue_byte_count{mode="sink"} |
Counter | Stream head | Byte count sent to Redpanda. |
internal_event_queue_row_count{mode="sink"} |
Counter | Stream head | Row count sent to Redpanda. |
internal_event_queue_row_count{mode="source"} |
Counter | Stream Summary | Row count received from Redpanda. |
internal_event_queue_value_count{mode="source"} |
Counter | Stream Summary | Value count received from Redpanda. |
Storage metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
object_store_http_status_code_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful. |
object_store_http_error_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of HTTP request errors. For example, timeouts and connection errors. |
object_store_http_bytes_tx |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes transmitted to object storage (request body only). |
object_store_http_bytes_rx |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes received from object storage (response body only). |
net_http_status_code_bucket |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | HTTP status code histogram count from storage. |
object_store_http_histo |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A histogram of object storage interaction latencies. |
object_store_http_summary |
Summary | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A summary of object storage interaction latencies. |
Stream and intake head metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Stream summary metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_count |
Counter | Summary Peer | Count of calls to the catalog. |
query_failure |
Counter | Summary Peer | Count of failed catalog calls. |
query_latency_summary |
Counter | Summary Peer | Latency of calls to catalog. |
Traefik metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
traefik_service_requests_total |
Counter | Traefik | HTTP Traefik request information. |
traefik_service_request_duration_seconds_count/sum/bucket |
Counter | Traefik | Response time of Traefik to client. |
http_source_request_duration_ns_count/sum/bucket |
Counter | Traefik | Response time from Stream Head. |