All Metrics

This page contains most metrics used in Hydrolix for the following:

Use find on page to locate specific metrics.

Custom Hydrolix metrics

Custom general metrics

Metric	Type	Components	Purpose
`bytes_written`	Counter	Batch peer, Stream peer	Bytes written to the indexer.
`partitions_created`	Counter	Batch peer, Stream peer	Count of partitions created.
`upload_duration`	Summary	Any intake peer	Time spent uploading a file, in milliseconds.

Custom query metrics

Metric	Type	Components	Purpose
`net_connect_attempts_total`	Histogram	Head/Query peer	Histogram of TCP connection attempts to the storage service.
`net_connect_seconds`	Histogram	Head/Query peer	Histogram of time to connect over TCP to the storage service, in seconds.
`net_dns_resolve_seconds`	Histogram	Head/Query peer	Histogram of DNS resolution time for the storage service, in seconds.
`net_http_response_time`	Histogram	Head/Query peer	Histogram of HTTP response times from the storage service, in seconds.
`net_http_response_bytes`	Histogram	Head/Query peer	Histogram of HTTP bytes downloaded from the storage service.
`net_http_attempts_total`	Histogram	Head/Query peer	Histogram of HTTP connection attempts to the storage service.
`net_http_status_code`	Histogram	Head/Query peer	Histogram of HTTP status codes from the storage service.
`vfs_cache_hitmiss_total`	Histogram	Head/Query peer	Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit).
`vfs_cache_read_bytes`	Histogram	Head/Query peer	Histogram of bytes read from the cache.
`vfs_net_read_bytes`	Histogram	Head/Query peer	Histogram of bytes read from the network.
`vfs_cache_lru_file_eviction_total`	Histogram	Head/Query peer	Histogram of file evictions from the cache.
`epoll_cpu_seconds`	Histogram	Head/Query peer	Histogram of CPU usage in seconds.
`epoll_io_seconds`	Histogram	Head/Query peer	Histogram of I/O times in seconds.
`epoll_poll_seconds`	Histogram	Head/Query peer	Histogram of wait times for file descriptors, in seconds.
`hdx_storage_r_catalog_partitions_total`	Histogram	Head/Query peer	Histogram of per-query catalog partition count.
`hdx_storage_r_partitions_read_total`	Histogram	Head/Query peer	Histogram of per-query partition read count.
`hdx_storage_r_partitions_per_core_total`	Histogram	Head/Query peer	Histogram of per-core partition usage count.
`hdx_storage_r_peers_used_total`	Histogram	Query peer	Histogram of storage used total.
`hdx_storage_r_cores_used_total`	Histogram	Query peer	Histogram of cores used total.
`hdx_storage_r_catalog_timerange`	Histogram	Head/Query peer	Histogram of query time-range distribution.
`hdx_partition_columns_read_total`	Histogram	Head/Query peer	Histogram of columns read.
`hdx_partition_block_decode_seconds`	Histogram	Head/Query peer	Histogram of time spent decoding HDX blocks, in seconds.
`hdx_partition_open_seconds`	Histogram	Head/Query peer	Histogram of time spent opening HDX partitions, in seconds.
`hdx_partition_read_seconds`	Histogram	Head/Query peer	Histogram of time spent reading HDX partitions, in seconds.
`hdx_partition_skipped_total`	Histogram	Head/Query peer	Histogram of partitions skipped due to no matching columns.
`hdx_partition_blocks_read_total`	Histogram	Head/Query peer	Histogram of partition-block read counts.
`hdx_partition_blocks_avail_total`	Histogram	Head/Query peer	Histogram of partition blocks available.
`hdx_partition_index_decision`	Histogram	Head/Query peer	Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match).
`hdx_partition_index_lookup_seconds`	Histogram	Head/Query peer	Histogram of index lookup times, in seconds.
`hdx_partition_index_blocks_skipped_percent`	Histogram	Head/Query peer	Histogram of skipped index blocks, in percentage.
`hdx_partition_index_blocks_skipped_total`	Histogram	Head/Query peer	Histogram of total skipped index blocks.
`hdx_partition_rd_w_err_total`	Histogram	Head/Query peer	Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error).
`query_iowait_seconds`	Histogram	Head/Query peer	Histogram of query I/O wait times, in seconds.
`query_cpuwait_seconds`	Histogram	Head/Query peer	Histogram of query CPU wait times, in seconds.
`query_hdx_ch_conv_seconds`	Histogram	Head/Query peer	Histogram of time spent converting HDX blocks to ClickHouse, in seconds.
`query_health`	Histogram	Head/Query peer	Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error).
`query_peer_availability`	Histogram	Head/Query peer	Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers).
`query_attempts_total`	Histogram	Head/Query peer	Histogram of total query attempts.
`query_response_seconds`	Histogram	Head/Query peer	Histogram of total query response times, in seconds.
`query_rows_read_total`	Histogram	Head/Query peer	Histogram of total query rows read.
`query_read_bytes`	Histogram	Head/Query peer	Histogram of total query read bytes.
`query_rows_written_total`	Histogram	Head/Query peer	Histogram of total query rows written.

Custom batch metrics

Metric	Type	Components	Purpose
`processed_count`	Counter	Batch peer	Count of items processed.
`processed_failure`	Counter	Batch peer	Count of processing failures.
`processing_duration_histo`	Histogram	Batch peer	Histogram of Batch processing durations (milliseconds).
`processing_duration_summary`	Summary	Batch peer	Summary of Batch processing durations (milliseconds).
`rows_read`	Counter	Batch peer	Count of rows read.

Custom merge metrics

Metric	Type	Components	Purpose
`merge_duration_summary`	Summary	Merge peer	Merge processing duration, in milliseconds.
`merge_duration_histo`	Histogram	Merge peer	Merge processing duration, in milliseconds.
`merge_sdk_duration_summary`	Summary	Merge peer	Merge SDK processing duration, in milliseconds.
`merge_sdk_duration_histo`	Histogram	Merge peer	Merge SDK processing duration, in milliseconds.
`merge_candidate_histo`	Histogram	Merge peer	Partitions per merge candidate.
`merge_candidate_inactive`	Counter	Merge peer	Merge candidates skipped due to an inactive partition within the candidate.
`merge_candidate_construction_summary`	Summary	Merge head	Time spent building merge candidates, in milliseconds.
`merge_queue_full`	Counter	Merge head	Times candidate generation was skipped due to a full queue.
`merge_success`	Counter	Merge peer	Count of merge successes.
`merge_failure`	Counter	Merge peer	Count of merge failures.

Custom streaming metrics

Metric	Type	Components	Purpose
`hdx_sink_backlog_bytes_count`	Gauge	Intake head	Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_items_count`	Gauge	Intake head	Total count of partition buckets in the backlog (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_dropped_bytes_count`	Counter	Intake head	Total bytes of partition buckets dropped due to backlog overflow (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_dropped_items_count`	Counter	Intake head	Count of partition buckets dropped due to backlog overflow (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_delivery_count`	Counter	Intake head	Count of backlog buckets successfully handed off to indexing (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_trim_duration_ns`	Histogram	Intake head	Time to trim the backlog in nanoseconds (requires `intake_head_index_backlog_enabled = true`).
`http_source_byte_count`	Counter	Stream head	Count of bytes processed.
`http_source_request_count`	Counter	Stream head	Count of HTTP requests.
`http_source_request_duration_ns`	Histogram	Stream head	Histogram of HTTP request durations in nanoseconds.
`http_source_request_error_count`	Counter	Stream head	Count of HTTP request failures.
`http_source_row_count`	Counter	Stream head	Count of rows processed.
`http_source_value_count`	Counter	Stream head	Count of values processed.

Custom Kinesis metrics

Metric	Type	Components	Purpose
`kinesis_source_byte_count`	Counter	Stream peer	Count of bytes read from Kinesis.
`kinesis_source_checkpoint_count`	Counter	Stream peer	Count of Kinesis checkpoint operations.
`kinesis_source_checkpoint_duration_ns`	Histogram	Stream peer	Duration of Kinesis checkpoint operations, in nanoseconds.
`kinesis_source_checkpoint_error_count`	Counter	Stream peer	Count of Kinesis checkpoint operation errors.
`kinesis_source_error_count`	Counter	Stream peer	Count of errors reading from Kinesis.
`kinesis_source_lag_ms`	Gauge	Stream peer	Measure of lag in Kinesis, in milliseconds.
`kinesis_source_operation_count`	Counter	Stream peer	Count of Kinesis operations.
`kinesis_source_operation_duration_ns`	Histogram	Stream peer	Histogram of Kinesis operation durations, in nanoseconds.
`kinesis_source_record_count`	Counter	Stream peer	Count of records read from Kinesis.
`kinesis_source_row_count`	Counter	Stream peer	Count of rows read from Kinesis.
`kinesis_source_value_count`	Counter	Stream peer	Count of values read from Kinesis.

Custom Kafka metrics

Metric	Type	Components	Purpose
`kafka_source_byte_count`	Counter	Stream peer	Count of bytes read from Kafka.
`kafka_source_commit_duration_ns`	Histogram	Stream peer	Kafka commit duration, in nanoseconds.
`kafka_source_read_count`	Counter	Stream peer	Count of Kafka reads.
`kafka_source_read_duration_ns`	Histogram	Stream peer	Kafka read duration, in nanoseconds.
`kafka_source_read_error_count`	Counter	Stream peer	Count of Kafka errors.
`kafka_source_row_count`	Counter	Stream peer	Count of rows processed from Kafka.
`kafka_source_value_count`	Counter	Stream peer	Count of values processed from Kafka.

Custom DNS metrics

Metric	Type	Components	Purpose
`dns_num_ips_in_cache`	Histogram	(ingest)	The size of the IP pool used in the DNS system.
`dns_lookup_time`	Histogram	(ingest)	Milliseconds per lookup.
`dns_ttl`	Histogram	(ingest)	TTLs received per lookup.

Custom PostgreSQL metrics

Metric	Type	Purpose
`pgx_pool_total_acquire_count`	Count	The cumulative count of successful acquires from the pool.
`pgx_pool_total_acquire_duration_ns_count`	Count	The total duration (ns) of all successful acquires from the pool.
`pgx_pool_total_acquire_cancel_count`	Count	The cumulative count of acquires from the pool that were canceled by a context.
`pgx_pool_total_acquire_empty_count`	Count	The cumulative count of successful acquires from the pool that had to wait for a resource because the pool was empty.
`pgx_pool_total_conns_opened_count`	Count	The cumulative count of new connections opened.
`pgx_pool_total_destroyed_max_lifetime_count`	Count	The cumulative count of connections destroyed because they exceeded `MaxConnLifetime`.
`pgx_pool_total_destroyed_max_idle_count`	Count	The cumulative count of connections destroyed because they exceeded `MaxConnIdleTime`.
`pgx_pool_current_size`	Gauge	The total number of resources currently in the pool.
`pgx_pool_current_constructing`	Gauge	The number of connections currently being constructed.
`pgx_pool_current_acquired`	Gauge	The number of connections currently acquired.
`pgx_pool_current_idle`	Gauge	The number of currently idle connections in the pool.
`pgx_pool_max`	Gauge	The maximum size of the pool.

Age metrics

Metric Name	Type	Components
`query_count`	Counter	Decay/Reaper calls to the Catalog.
`query_failure`	Counter	Decay/Reaper failed Catalog calls.
`query_latency_summary`	Counter	Latency in calls to catalog.
`query_latency_summary_count/sum`	Counter	Latency in calls to catalog.

Alter metrics

Metric Name	Type	Components
`query_count`	Counter	Decay/Reaper calls to the Catalog.
`query_failure`	Counter	Decay/Reaper failed Catalog calls.
`query_latency_summary`	Counter	Latency in calls to catalog.
`query_latency_summary_count/sum`	Counter	Latency in calls to catalog.

Merge metrics

Metric Name	Type	Components
`merge_success`	Counter	Count of merge successes.
`merge_failure`	Counter	Count of merge successes.
`query_count`	Counter	Merge calls to the Catalog.
`query_failure`	Counter	Merge failed Catalog calls.
`query_latency_summary`	Counter	Latency in calls to catalog.
`query_latency_summary_count/sum`	Counter	Latency in calls to catalog.
`merge_duration_histo_bucket/count/sum`	Histogram	Merge processing duration, in milliseconds.
`merge_duration_summary`	Summary	Merge processing duration, in milliseconds.

RabbitMQ metrics

Metric Name	Type	Components
`rabbitmq_queue_messages`	Counter	Sum of ready and unacknowledged messages (queue depth).
`rabbitmq_queues`	Counter	RabbitMQ Queues count.
`erlang_vm_statistics_bytes_received_total`	Counter	The total number of bytes received through RabbitMQ ports.
`erlang_vm_statistics_bytes_output_total`	Counter	The total number of bytes output through RabbitMQ ports.

Prometheus

Prometheus batch metrics

Metric Name	Type	Components	Purpose
`processed_count`	Counter	Batch peer	Count of items processed.
`processed_failure`	Counter	Batch peer	Count of processing failures.
`rows_read`	Counter	Batch peer	Count of rows read.
`processing_duration_histo`	Histogram	Batch peer	Histogram of Batch processing durations in milliseconds.
`processing_duration_summary`	Summary	Batch peer	Summary of Batch processing durations in milliseconds.

Prometheus DNS metrics

Metric Name	Type	Components	Purpose
`dns_num_ips_in_cache`	Histogram	(ingest)	The size of the IP pool used in the DNS system.
`dns_lookup_time`	Histogram	(ingest)	Milliseconds per lookup.
`dns_ttl`	Histogram	(ingest)	TTLs received per lookup.

Prometheus general metrics

Metric Name	Type	Components	Purpose
`bytes_written`	Counter	Batch peer, Stream peer	Bytes written to the indexer.
`partitions_created`	Counter	Batch peer, Stream peer	Count of partitions created.
`process_cpu_seconds_total`	Counter	Batch peer, Stream head, Stream peer	Total user and system CPU time spent in seconds.
`promhttp_metric_handler_requests_total`	Counter	Batch peer, Stream head, Stream peer	Total number of scrapes by HTTP status code.
`process_max_fds`	Gauge	Batch peer, Stream head, Stream peer	Maximum number of open file descriptors.
`process_open_fds`	Gauge	Batch peer, Stream head, Stream peer	Number of open file descriptors.
`process_resident_memory_bytes`	Gauge	Batch peer, Stream head, Stream peer	Resident memory size in bytes.
`process_start_time_seconds`	Gauge	Batch peer, Stream head, Stream peer	Start time of the process since unix epoch in seconds.
`process_virtual_memory_bytes`	Gauge	Batch peer, Stream head, Stream peer	Virtual memory size in bytes.
`process_virtual_memory_max_bytes`	Gauge	Batch peer, Stream head, Stream peer	Maximum amount of virtual memory available in bytes.
`promhttp_metric_handler_requests_in_flight`	Gauge	Batch peer, Stream head, Stream peer	Current number of scrapes being served.
`upload_duration`	Summary	Any intake peer	Time spent uploading a file, in milliseconds

Prometheus Go metrics

Metric Name	Type	Components	Purpose
`go_memstats_alloc_bytes_total`	Counter	Batch peer, Stream head, Stream peer	Total number of bytes allocated, even if freed.
`go_memstats_frees_total`	Counter	Batch peer, Stream head, Stream peer	Total number of frees.
`go_memstats_lookups_total`	Counter	Batch peer, Stream head, Stream peer	Total number of pointer lookups.
`go_memstats_mallocs_total`	Counter	Batch peer, Stream head, Stream peer	Total number of mallocs.
`go_goroutines`	Gauge	Batch peer, Stream head, Stream peer	Number of goroutines that currently exist.
`go_info`	Gauge	Batch peer, Stream head, Stream peer	Information about the Go environment.
`go_memstats_alloc_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes allocated and still in use.
`go_memstats_buck_hash_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes used by the profiling bucket hash table.
`go_memstats_gc_cpu_fraction`	Gauge	Batch peer, Stream head, Stream peer	The fraction of this program's available CPU time used by the GC since the program started.
`go_memstats_gc_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes used for garbage collection system metadata.
`go_memstats_heap_alloc_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of heap bytes allocated and still in use.
`go_memstats_heap_idle_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of heap bytes waiting to be used.
`go_memstats_heap_inuse_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of heap bytes that are in use.
`go_memstats_heap_objects`	Gauge	Batch peer, Stream head, Stream peer	Number of allocated objects.
`go_memstats_heap_released_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of heap bytes released to OS.
`go_memstats_heap_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of heap bytes obtained from system.
`go_memstats_last_gc_time_seconds`	Gauge	Batch peer, Stream head, Stream peer	Number of seconds since 1970 of last garbage collection.
`go_memstats_mcache_inuse_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes in use by mcache structures.
`go_memstats_mcache_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes used for mcache structures obtained from system.
`go_memstats_mspan_inuse_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes in use by mspan structures.
`go_memstats_mspan_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes used for mspan structures obtained from system.
`go_memstats_next_gc_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of heap bytes when next garbage collection will take place.
`go_memstats_other_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes used for other system allocations.
`go_memstats_stack_inuse_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes in use by the stack allocator.
`go_memstats_stack_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes obtained from system for stack allocator.
`go_memstats_sys_bytes`	Gauge	Batch peer, Stream head, Stream peer	Number of bytes obtained from system.
`go_threads`	Gauge	Batch peer, Stream head, Stream peer	Number of OS threads created.
`go_gc_duration_seconds`	Summary	Batch peer, Stream head, Stream peer	A summary of the pause duration of garbage collection cycles.

Prometheus stream ingest metrics

Metric Name	Type	Components	Purpose
`hdx_sink_backlog_dropped_bytes_count`	Counter	Intake head	Total bytes of partition buckets dropped due to backlog growing too big. Only produced when `intake_head_index_backlog_enabled` is `true`.
`hdx_sink_backlog_dropped_items_count`	Counter	Intake head	Count of partition buckets dropped due to backlog growing too big. Only produced when `intake_head_index_backlog_enabled` is `true`.
`hdx_sink_backlog_delivery_count`	Counter	Intake head	Count of backlog buckets successfully handed off to indexing. Only produced when `intake_head_index_backlog_enabled` is `true`.
`http_source_byte_count`	Counter	Stream head	Count of bytes processed.
`http_source_request_count`	Counter	Stream head	Count of http requests.
`http_source_request_error_count`	Counter	Stream head	Count of http request failures.
`http_source_row_count`	Counter	Stream head	Count of rows processed.
`http_source_value_count`	Counter	Stream head	Count of values processed.
`kinesis_source_byte_count`	Counter	Stream peer	Count of bytes read from Kinesis.
`kinesis_source_checkpoint_count`	Counter	Stream peer	Count of Kinesis checkpoint operations.
`kinesis_source_checkpoint_error_count`	Counter	Stream peer	Count of errors in Kinesis checkpoint operations.
`kinesis_source_error_count`	Counter	Stream peer	Count of errors in Kinesis source reads.
`kinesis_source_operation_count`	Counter	Stream peer	Count of operations on Kinesis.
`kinesis_source_record_count`	Counter	Stream peer	Count of records read from Kinesis.
`kinesis_source_row_count`	Counter	Stream peer	Count of rows read from Kinesis.
`kinesis_source_value_count`	Counter	Stream peer	Count of values read from Kinesis.
`hdx_sink_backlog_bytes_count`	Gauge	Intake head	Total bytes of all partition buckets in sink backlog waiting to be indexed. Only produced when `intake_head_index_backlog_enabled` is `true`.
`hdx_sink_backlog_items_count`	Gauge	Intake head	Total count of partition buckets in sink backlog waiting to be indexed. Only produced when `intake_head_index_backlog_enabled` is `true`.
`kinesis_source_lag_ms`	Gauge	Stream peer	Measure of lag in Kinesis source.
`hdx_sink_backlog_trim_duration_ns`	Histogram	Intake head	Time to trim the backlog in nanoseconds. Only produced when `intake_head_index_backlog_enabled` is `true`.
`http_source_request_duration_ns`	Histogram	Stream head	A histogram of HTTP request durations in nanoseconds.
`kinesis_source_checkpoint_duration_ns`	Histogram	Stream peer	Duration of Kinesis checkpoint operations.
`kinesis_source_operation_duration_ns`	Histogram	Stream peer	Histogram of duration of operations on Kinesis.

Prometheus Kafka metrics

Metric Name	Type	Components	Purpose
`kafka_source_byte_count`	Counter	Stream peer	Count of bytes read from Kafka.
`kafka_source_read_count`	Counter	Stream peer	Count of Kafka reads.
`kafka_source_read_error_count`	Counter	Stream peer	Count of Kafka errors.
`kafka_source_row_count`	Counter	Stream peer	Count of rows processed.
`kafka_source_value_count`	Counter	Stream peer	Count of values processed.
`kafka_source_commit_duration_ns`	Histogram	Stream peer	Kafka commit duration.
`kafka_source_read_duration_ns`	Histogram	Stream peer	Kafka read duration.

Prometheus merge metrics

Metric Name	Type	Components	Purpose
`merge_candidate_inactive`	Counter	Merge peer	Merge candidates skipped due to an inactive partition within the candidate
`merge_queue_full`	Counter	Merge head	Times candidate generation was skipped due to a full queue
`merge_success`	Counter	Merge peer	Count of merge successes.
`merge_failure`	Counter	Merge peer	Count of merge successes.
`merge_duration_histo`	Histogram	Merge peer	Merge processing duration, in milliseconds.
`merge_sdk_duration_histo`	Histogram	Merge peer	Merge SDK processing duration, in milliseconds.
`merge_candidate_histo`	Histogram	Merge peer	Partitions per merge candidate.
`merge_duration_summary`	Summary	Merge peer	Merge processing duration, in milliseconds.
`merge_sdk_duration_summary`	Summary	Merge peer	Merge SDK processing duration, in milliseconds.
`merge_candidate_construction_summary`	Summary	Merge head	Time spent building merge candidates, in milliseconds.

Prometheus PostgreSQL metrics

Metric Name	Type	Components
`pgx_pool_total_acquire_count`	Count	The cumulative count of successful acquires from the pool.
`pgx_pool_total_acquire_duration_ns_count`	Count	The total duration of all successful acquires from the pool.
`pgx_pool_total_acquire_cancel_count`	Count	The cumulative count of acquires from the pool that were canceled by a context.
`pgx_pool_total_acquire_empty_count`	Count	The cumulative count of successful acquires from the pool that waited for a resource to be released or constructed because the pool was empty.
`pgx_pool_total_conns_opened_count`	Count	The cumulative count of new connections opened.
`pgx_pool_total_destroyed_max_lifetime_count`	Count	The cumulative count of connections destroyed because they exceeded MaxConnLifetime.
`pgx_pool_total_destroyed_max_idle_count`	Count	The cumulative count of connections destroyed because they exceeded MaxConnIdleTime.
`pgx_pool_current_size`	Gauge	The total number of resources currently in the pool.
`pgx_pool_current_constructing`	Gauge	The number of connections with construction in progress in the pool.
`pgx_pool_current_acquired`	Gauge	The number of currently acquired connections in the pool.
`pgx_pool_current_idle`	Gauge	The number of currently idle connections in the pool.
`pgx_pool_max`	Gauge	The maximum size of the pool.

Prometheus query metrics

Metric Name	Type	Components	Purpose
`net_connect_attempts_total`	Histogram	Head/Query peer	Histogram of TCP connection attempted to storage service
`net_connect_seconds`	Histogram	Head/Query peer	Histogram of time to connect over TCP to storage service in seconds
`net_dns_resolve_seconds`	Histogram	Head/Query peer	Histogram of DNS resolution time to storage service in seconds.
`net_http_response_time`	Histogram	Head/Query peer	Histogram HTTP response time to storage service in seconds
`net_http_response_bytes`	Histogram	Head/Query peer	Histogram of HTTP bytes downloaded from the storage service
`net_http_attempts_total`	Histogram	Head/Query peer	Histogram of HTTP connection attempted to storage service
`net_http_status_code`	Histogram	Head/Query peer	Histogram of HTTP status code result from storage service
`vfs_cache_hitmiss_total`	Histogram	Head/Query peer	Histogram of cache status if bucket = 0 cache miss, and 1 cache hit
`vfs_cache_read_bytes`	Histogram	Head/Query peer	Histogram bytes read from cache
`vfs_net_read_bytes`	Histogram	Head/Query peer	Histogram bytes read from network
`vfs_cache_lru_file_eviction_total`	Histogram	Head/Query peer	Histogram cache eviction of files
`epoll_cpu_seconds`	Histogram	Head/Query peer	Histogram CPU used in seconds
`epoll_io_seconds`	Histogram	Head/Query peer	Histogram I/O in seconds
`epoll_poll_seconds`	Histogram	Head/Query peer	Histogram wait for file descriptor in seconds
`hdx_storage_r_catalog_partitions_total`	Histogram	Head/Query peer	Histogram of per query catalog partition count
`hdx_storage_r_partitions_read_total`	Histogram	Head/Query peer	Histogram of per query partition read count
`hdx_storage_r_partitions_per_core_total`	Histogram	Head/Query peer	Histogram of per core partition used count
`hdx_storage_r_peers_used_total`	Histogram	Query peer	Histogram of storage used total
`hdx_storage_r_cores_used_total`	Histogram	Query peer	Histogram of Cores used total
`hdx_storage_r_catalog_timerange`	Histogram	Head/Query peer	Histogram of query time range distribution
`hdx_partition_columns_read_total`	Histogram	Head/Query peer	Histogram of column read
`hdx_partition_block_decode_seconds`	Histogram	Head/Query peer	Histogram of time spent decoding hdx blocks in seconds
`hdx_partition_open_seconds`	Histogram	Head/Query peer	Histogram of time spent opening hdx partition in seconds
`hdx_partition_read_seconds`	Histogram	Head/Query peer	Histogram of time spent reading hdx partition in seconds
`hdx_partition_skipped_total`	Histogram	Head/Query peer	Histogram of partition skip count due to no matching columns
`hdx_partition_blocks_read_total`	Histogram	Head/Query peer	Histogram of partition read count
`hdx_partition_blocks_avail_total`	Histogram	Head/Query peer	Histogram of partition blocks available
`hdx_partition_index_decision`	Histogram	Head/Query peer	Histogram of partition decision if bucket = 0 fullscan, 1 partial scan and 2 no match
`hdx_partition_index_lookup_seconds`	Histogram	Head/Query peer	Histogram of index lookup in seconds
`hdx_partition_index_blocks_skipped_percent`	Histogram	Head/Query peer	Histogram of skipped index blocked in percentage
`hdx_partition_index_blocks_skipped_total`	Histogram	Head/Query peer	Histogram of skipped index blocked in total
`hdx_partition_rd_w_err_total`	Histogram	Head/Query peer	Histogram of errors if bucket = 0 read error, 1 written error and 3 error
`query_iowait_seconds`	Histogram	Head/Query peer	Histogram query IO wait in seconds
`query_cpuwait_seconds`	Histogram	Head/Query peer	Histogram query cpu wait in seconds
`query_hdx_ch_conv_seconds`	Histogram	Head/Query peer	Histogram of time spent converting hdx blocks to clickhouse in seconds
`query_health`	Histogram	Head/Query peer	Histogram of query health if bucket = 0 initiated error, 1 succeeded and 2 error
`query_peer_availability`	Histogram	Head/Query peer	Histogram of query peer availability if bucket = 0 primary_peer_available, 1 secondary_peer_available and 2 no_reachable_peers
`query_attempts_total`	Histogram	Head/Query peer	Histogram of query attempts total
`query_response_seconds`	Histogram	Head/Query peer	Histogram of query response total in seconds
`query_rows_read_total`	Histogram	Head/Query peer	Histogram of query rows read total
`query_read_bytes`	Histogram	Head/Query peer	Histogram of query read bytes total
`query_rows_written_total`	Histogram	Head/Query peer	Histogram of query rows written total

Summary table

Summary table general metrics

Metric Name	Type	Components
`process_cpu_seconds_total`	Counter	Total user and system CPU time spent in seconds.
`promhttp_metric_handler_requests_total`	Counter	Total number of scrapes by HTTP status code.
`process_max_fds`	Gauge	Maximum number of open file descriptors.
`process_open_fds`	Gauge	Number of open file descriptors.
`process_resident_memory_bytes`	Gauge	Resident memory size in bytes.
`process_start_time_seconds`	Gauge	Start time of the process since unix epoch in seconds.
`process_virtual_memory_bytes`	Gauge	Virtual memory size in bytes.
`process_virtual_memory_max_bytes`	Gauge	Maximum amount of virtual memory available in bytes.
`promhttp_metric_handler_requests_in_flight`	Gauge	Current number of scrapes being served.

Summary table Go metrics

Metric Name	Type	Components
`go_memstats_alloc_bytes_total`	Counter	Total number of bytes allocated, even if freed.
`go_memstats_frees_total`	Counter	Total number of frees.
`go_memstats_lookups_total`	Counter	Total number of pointer lookups.
`go_memstats_mallocs_total`	Counter	Total number of mallocs.
`go_goroutines`	Gauge	Number of goroutines that currently exist.
`go_info`	Gauge	Information about the Go environment.
`go_memstats_alloc_bytes`	Gauge	Number of bytes allocated and still in use.
`go_memstats_buck_hash_sys_bytes`	Gauge	Number of bytes used by the profiling bucket hash table.
`go_memstats_gc_cpu_fraction`	Gauge	The fraction of this program's available CPU time used by the GC since the program started.
`go_memstats_gc_sys_bytes`	Gauge	Number of bytes used for garbage collection system metadata.
`go_memstats_heap_alloc_bytes`	Gauge	Number of heap bytes allocated and still in use.
`go_memstats_heap_idle_bytes`	Gauge	Number of heap bytes waiting to be used.
`go_memstats_heap_inuse_bytes`	Gauge	Number of heap bytes that are in use.
`go_memstats_heap_objects`	Gauge	Number of allocated objects.
`go_memstats_heap_released_bytes`	Gauge	Number of heap bytes released to OS.
`go_memstats_heap_sys_bytes`	Gauge	Number of heap bytes obtained from system.
`go_memstats_last_gc_time_seconds`	Gauge	Number of seconds since 1970 of last garbage collection.
`go_memstats_mcache_inuse_bytes`	Gauge	Number of bytes in use by mcache structures.
`go_memstats_mcache_sys_bytes`	Gauge	Number of bytes used for mcache structures obtained from system.
`go_memstats_mspan_inuse_bytes`	Gauge	Number of bytes in use by mspan structures.
`go_memstats_mspan_sys_bytes`	Gauge	Number of bytes used for mspan structures obtained from system.
`go_memstats_next_gc_bytes`	Gauge	Number of heap bytes when next garbage collection will take place.
`go_memstats_other_sys_bytes`	Gauge	Number of bytes used for other system allocations.
`go_memstats_stack_inuse_bytes`	Gauge	Number of bytes in use by the stack allocator.
`go_memstats_stack_sys_bytes`	Gauge	Number of bytes obtained from system for stack allocator.
`go_memstats_sys_bytes`	Gauge	Number of bytes obtained from system.
`go_threads`	Gauge	Number of OS threads created.
`go_gc_duration_seconds`	Summary	A summary of the pause duration of garbage collection cycles.

Summary table indexer metrics

Metric Name	Type	Components	Purpose
`hdx_sink_row_count`	Counter	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Count of rows processed by the indexer and uploaded to storage. Includes Hot and Cold reporting.
`hdx_sink_byte_count`	Counter	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Count of bytes processed by the indexer and uploaded to storage. Includes Hot and Cold reporting.
`hdx_sink_value_count`	Counter	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Count of values processed by the indexer and uploaded to storage. Includes Hot and Cold reporting.
`hdx_sink_error_count`	Counter	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Count of errors in indexing and uploading to storage.
`indexer_rows_written_count/bucket/sum`	Histogram	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Total rows indexed (written to partitions)
`indexer_bytes_written_count/bucket/sum`	Histogram	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Total bytes indexed (written to partitions)
`indexer_partitions_rejected_count/bucket/sum`	Histogram	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Histogram of partitions not able to written. If value is 0=raw data parsing failed, 1=raw data / transform schema mismatch, 3=Error writing partition file, 4= Other Error during indexing
`indexer_partitions_written_count/bucket/sum`	Histogram	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Total partitions created
`indexer_partition_write_seconds_count/bucket/sum`	Histogram	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	Time from receiving indexing query to writing partition file (seconds)

Summary table Redpanda metrics

Metric Name	Type	Components	Purpose
`internal_event_queue_byte_count{mode="sink"}`	Counter	Stream Head	Byte Count sent to Redpanda
`internal_event_queue_row_count{mode="sink""}`	Counter	Stream Head	Row Count sent to Redpanda
`internal_event_queue_byte_count{mode="sink""}`	Counter	Stream Head	Byte Count sent to Redpanda
`internal_event_queue_row_count{mode="source"}`	Counter	Stream Summary	Row Count received from Redpanda
`internal_event_queue_value_count{mode="source"}`	Counter	Stream Summary	Value Count recieved from Redpanda

Summary table storage metrics

Metric Name	Type	Components	Purpose
`object_store_http_status_code_count`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful.
`object_store_http_error_count`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of HTTP request errors (timeouts, connection errors, etc.)
`object_store_http_bytes_tx`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of bytes transmitted to object storage (request body only)
`object_store_http_bytes_rx`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of bytes received from object storage (response body only)
`net_http_status_code_bucket`	Counter	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	HTTP Status Code histogram count from Storage.
`object_store_http_histo`	Histogram	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A histogram of object storage interaction latencies
`object_store_http_summary`	Summary	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A summary of object storage interaction latencies

Summary table stream and intake head metrics

Metric Name	Type	Components	Purpose
`http_source_byte_count`	Counter	Stream head	Count of bytes processed.
`http_source_request_count`	Counter	Stream head	Count of http requests.
`http_source_request_error_count`	Counter	Stream head	Count of http request failures.
`http_source_row_count`	Counter	Stream head	Count of rows processed.
`http_source_value_count`	Counter	Stream head	Count of values processed.
`http_source_request_duration_ns_count/bucket/sum`	Histogram	Stream head	A histogram of HTTP request durations in nanoseconds.

Summary table stream summary metrics

Metric Name	Type	Components	Purpose
`query_latency_summary_count/sum`	Count/Sum	Summary Peer	Latency of calls to catalog.
`query_count`	Counter	Summary Peer	Count of calls to the Catalog.
`query_failure`	Counter	Summary Peer	Count of failed Catalog calls.
`query_latency_summary`	Counter	Summary Peer	Latency of calls to catalog.

Summary table Traefik metrics

Metric Name	Type	Components	Purpose
`traefik_service_requests_total`	Counter	Traefik	HTTP Traefik request information.
`traefik_service_request_duration_seconds_count/sum/bucket`	Counter	Traefik	Response time of traefik to client.
`http_source_request_duration_ns_count/sum/bucket`	Counter	Traefik	Response time from Stream-Head.