Hydrolix Metrics

This page lists the custom Hydrolix metrics available, and which components update them.

If more than one component uses a given metric, then querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query. For example, process_open_fds{service="stream-peer"}.

For more information about metric types, refer to Prometheus's documentation.

Custom general metrics

Metric	Type	Components	Purpose
`bytes_written`	Counter	Batch peer, Stream peer	Bytes written to the indexer.
`partitions_created`	Counter	Batch peer, Stream peer	Count of partitions created.
`upload_duration`	Summary	Any intake peer	Time spent uploading a file, in milliseconds.

Custom query metrics

Metric	Type	Components	Purpose
`net_connect_attempts_total`	Histogram	Head/Query peer	Histogram of TCP connection attempts to the storage service.
`net_connect_seconds`	Histogram	Head/Query peer	Histogram of time to connect over TCP to the storage service, in seconds.
`net_dns_resolve_seconds`	Histogram	Head/Query peer	Histogram of DNS resolution time for the storage service, in seconds.
`net_http_response_time`	Histogram	Head/Query peer	Histogram of HTTP response times from the storage service, in seconds.
`net_http_response_bytes`	Histogram	Head/Query peer	Histogram of HTTP bytes downloaded from the storage service.
`net_http_attempts_total`	Histogram	Head/Query peer	Histogram of HTTP connection attempts to the storage service.
`net_http_status_code`	Histogram	Head/Query peer	Histogram of HTTP status codes from the storage service.
`vfs_cache_hitmiss_total`	Histogram	Head/Query peer	Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit).
`vfs_cache_read_bytes`	Histogram	Head/Query peer	Histogram of bytes read from the cache.
`vfs_net_read_bytes`	Histogram	Head/Query peer	Histogram of bytes read from the network.
`vfs_cache_lru_file_eviction_total`	Histogram	Head/Query peer	Histogram of file evictions from the cache.
`epoll_cpu_seconds`	Histogram	Head/Query peer	Histogram of CPU usage in seconds.
`epoll_io_seconds`	Histogram	Head/Query peer	Histogram of I/O times in seconds.
`epoll_poll_seconds`	Histogram	Head/Query peer	Histogram of wait times for file descriptors, in seconds.
`hdx_storage_r_catalog_partitions_total`	Histogram	Head/Query peer	Histogram of per-query catalog partition count.
`hdx_storage_r_partitions_read_total`	Histogram	Head/Query peer	Histogram of per-query partition read count.
`hdx_storage_r_partitions_per_core_total`	Histogram	Head/Query peer	Histogram of per-core partition usage count.
`hdx_storage_r_peers_used_total`	Histogram	Query peer	Histogram of storage used total.
`hdx_storage_r_cores_used_total`	Histogram	Query peer	Histogram of cores used total.
`hdx_storage_r_catalog_timerange`	Histogram	Head/Query peer	Histogram of query time-range distribution.
`hdx_partition_columns_read_total`	Histogram	Head/Query peer	Histogram of columns read.
`hdx_partition_block_decode_seconds`	Histogram	Head/Query peer	Histogram of time spent decoding HDX blocks, in seconds.
`hdx_partition_open_seconds`	Histogram	Head/Query peer	Histogram of time spent opening HDX partitions, in seconds.
`hdx_partition_read_seconds`	Histogram	Head/Query peer	Histogram of time spent reading HDX partitions, in seconds.
`hdx_partition_skipped_total`	Histogram	Head/Query peer	Histogram of partitions skipped due to no matching columns.
`hdx_partition_blocks_read_total`	Histogram	Head/Query peer	Histogram of partition-block read counts.
`hdx_partition_blocks_avail_total`	Histogram	Head/Query peer	Histogram of partition blocks available.
`hdx_partition_index_decision`	Histogram	Head/Query peer	Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match).
`hdx_partition_index_lookup_seconds`	Histogram	Head/Query peer	Histogram of index lookup times, in seconds.
`hdx_partition_index_blocks_skipped_percent`	Histogram	Head/Query peer	Histogram of skipped index blocks, in percentage.
`hdx_partition_index_blocks_skipped_total`	Histogram	Head/Query peer	Histogram of total skipped index blocks.
`hdx_partition_rd_w_err_total`	Histogram	Head/Query peer	Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error).
`query_iowait_seconds`	Histogram	Head/Query peer	Histogram of query I/O wait times, in seconds.
`query_cpuwait_seconds`	Histogram	Head/Query peer	Histogram of query CPU wait times, in seconds.
`query_hdx_ch_conv_seconds`	Histogram	Head/Query peer	Histogram of time spent converting HDX blocks to ClickHouse, in seconds.
`query_health`	Histogram	Head/Query peer	Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error).
`query_peer_availability`	Histogram	Head/Query peer	Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers).
`query_attempts_total`	Histogram	Head/Query peer	Histogram of total query attempts.
`query_response_seconds`	Histogram	Head/Query peer	Histogram of total query response times, in seconds.
`query_rows_read_total`	Histogram	Head/Query peer	Histogram of total query rows read.
`query_read_bytes`	Histogram	Head/Query peer	Histogram of total query read bytes.
`query_rows_written_total`	Histogram	Head/Query peer	Histogram of total query rows written.

Custom batch metrics

Metric	Type	Components	Purpose
`processed_count`	Counter	Batch peer	Count of items processed.
`processed_failure`	Counter	Batch peer	Count of processing failures.
`processing_duration_histo`	Histogram	Batch peer	Histogram of Batch processing durations (milliseconds).
`processing_duration_summary`	Summary	Batch peer	Summary of Batch processing durations (milliseconds).
`rows_read`	Counter	Batch peer	Count of rows read.

Custom merge metrics

Metric	Type	Components	Purpose
`merge_duration_summary`	Summary	Merge peer	Merge processing duration, in milliseconds.
`merge_duration_histo`	Histogram	Merge peer	Merge processing duration, in milliseconds.
`merge_sdk_duration_summary`	Summary	Merge peer	Merge SDK processing duration, in milliseconds.
`merge_sdk_duration_histo`	Histogram	Merge peer	Merge SDK processing duration, in milliseconds.
`merge_candidate_histo`	Histogram	Merge peer	Partitions per merge candidate.
`merge_candidate_inactive`	Counter	Merge peer	Merge candidates skipped due to an inactive partition within the candidate.
`merge_candidate_construction_summary`	Summary	Merge head	Time spent building merge candidates, in milliseconds.
`merge_queue_full`	Counter	Merge head	Times candidate generation was skipped due to a full queue.
`merge_success`	Counter	Merge peer	Count of merge successes.
`merge_failure`	Counter	Merge peer	Count of merge failures.

Custom streaming metrics

Metric	Type	Components	Purpose
`hdx_sink_backlog_bytes_count`	Gauge	Intake head	Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_items_count`	Gauge	Intake head	Total count of partition buckets in the backlog (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_dropped_bytes_count`	Counter	Intake head	Total bytes of partition buckets dropped due to backlog overflow (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_dropped_items_count`	Counter	Intake head	Count of partition buckets dropped due to backlog overflow (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_delivery_count`	Counter	Intake head	Count of backlog buckets successfully handed off to indexing (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_trim_duration_ns`	Histogram	Intake head	Time to trim the backlog in nanoseconds (requires `intake_head_index_backlog_enabled = true`).
`http_source_byte_count`	Counter	Stream head	Count of bytes processed.
`http_source_request_count`	Counter	Stream head	Count of HTTP requests.
`http_source_request_duration_ns`	Histogram	Stream head	Histogram of HTTP request durations in nanoseconds.
`http_source_request_error_count`	Counter	Stream head	Count of HTTP request failures.
`http_source_row_count`	Counter	Stream head	Count of rows processed.
`http_source_value_count`	Counter	Stream head	Count of values processed.

Custom Kinesis metrics

Metric	Type	Components	Purpose
`kinesis_source_byte_count`	Counter	Stream peer	Count of bytes read from Kinesis.
`kinesis_source_checkpoint_count`	Counter	Stream peer	Count of Kinesis checkpoint operations.
`kinesis_source_checkpoint_duration_ns`	Histogram	Stream peer	Duration of Kinesis checkpoint operations, in nanoseconds.
`kinesis_source_checkpoint_error_count`	Counter	Stream peer	Count of Kinesis checkpoint operation errors.
`kinesis_source_error_count`	Counter	Stream peer	Count of errors reading from Kinesis.
`kinesis_source_lag_ms`	Gauge	Stream peer	Measure of lag in Kinesis, in milliseconds.
`kinesis_source_operation_count`	Counter	Stream peer	Count of Kinesis operations.
`kinesis_source_operation_duration_ns`	Histogram	Stream peer	Histogram of Kinesis operation durations, in nanoseconds.
`kinesis_source_record_count`	Counter	Stream peer	Count of records read from Kinesis.
`kinesis_source_row_count`	Counter	Stream peer	Count of rows read from Kinesis.
`kinesis_source_value_count`	Counter	Stream peer	Count of values read from Kinesis.

Custom Kafka metrics

Metric	Type	Components	Purpose
`kafka_source_byte_count`	Counter	Stream peer	Count of bytes read from Kafka.
`kafka_source_commit_duration_ns`	Histogram	Stream peer	Kafka commit duration, in nanoseconds.
`kafka_source_read_count`	Counter	Stream peer	Count of Kafka reads.
`kafka_source_read_duration_ns`	Histogram	Stream peer	Kafka read duration, in nanoseconds.
`kafka_source_read_error_count`	Counter	Stream peer	Count of Kafka errors.
`kafka_source_row_count`	Counter	Stream peer	Count of rows processed from Kafka.
`kafka_source_value_count`	Counter	Stream peer	Count of values processed from Kafka.

Custom DNS metrics

Metric	Type	Components	Purpose
`dns_num_ips_in_cache`	Histogram	(ingest)	The size of the IP pool used in the DNS system.
`dns_lookup_time`	Histogram	(ingest)	Milliseconds per lookup.
`dns_ttl`	Histogram	(ingest)	TTLs received per lookup.

Custom PostgreSQL metrics

Metric	Type	Purpose
`pgx_pool_total_acquire_count`	Count	The cumulative count of successful acquires from the pool.
`pgx_pool_total_acquire_duration_ns_count`	Count	The total duration (ns) of all successful acquires from the pool.
`pgx_pool_total_acquire_cancel_count`	Count	The cumulative count of acquires from the pool that were canceled by a context.
`pgx_pool_total_acquire_empty_count`	Count	The cumulative count of successful acquires from the pool that had to wait for a resource because the pool was empty.
`pgx_pool_total_conns_opened_count`	Count	The cumulative count of new connections opened.
`pgx_pool_total_destroyed_max_lifetime_count`	Count	The cumulative count of connections destroyed because they exceeded `MaxConnLifetime`.
`pgx_pool_total_destroyed_max_idle_count`	Count	The cumulative count of connections destroyed because they exceeded `MaxConnIdleTime`.
`pgx_pool_current_size`	Gauge	The total number of resources currently in the pool.
`pgx_pool_current_constructing`	Gauge	The number of connections currently being constructed.
`pgx_pool_current_acquired`	Gauge	The number of connections currently acquired.
`pgx_pool_current_idle`	Gauge	The number of currently idle connections in the pool.
`pgx_pool_max`	Gauge	The maximum size of the pool.