Custom Metrics
Hydrolix Metrics⚓︎
This page lists the custom Hydrolix metrics available, and which components update them.
If more than one component uses a given metric, then querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query. For example, process_open_fds{service="stream-peer"}.
For more information about metric types, refer to Prometheus's documentation.
Custom general metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
bytes_written |
Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created |
Counter | Batch peer, Stream peer | Count of partitions created. |
upload_duration |
Summary | Any intake peer | Time spent uploading a file, in milliseconds. |
Custom query metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempts to the storage service. |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to the storage service, in seconds. |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time for the storage service, in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram of HTTP response times from the storage service, in seconds. |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service. |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempts to the storage service. |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status codes from the storage service. |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit). |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the cache. |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the network. |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram of file evictions from the cache. |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram of CPU usage in seconds. |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram of I/O times in seconds. |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram of wait times for file descriptors, in seconds. |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per-query catalog partition count. |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per-query partition read count. |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per-core partition usage count. |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total. |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of cores used total. |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time-range distribution. |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of columns read. |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding HDX blocks, in seconds. |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening HDX partitions, in seconds. |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading HDX partitions, in seconds. |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partitions skipped due to no matching columns. |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition-block read counts. |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available. |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match). |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup times, in seconds. |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocks, in percentage. |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of total skipped index blocks. |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error). |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram of query I/O wait times, in seconds. |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram of query CPU wait times, in seconds. |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting HDX blocks to ClickHouse, in seconds. |
query_health |
Histogram | Head/Query peer | Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error). |
query_peer_availability |
Histogram | Head/Query peer | Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers). |
query_attempts_total |
Histogram | Head/Query peer | Histogram of total query attempts. |
query_response_seconds |
Histogram | Head/Query peer | Histogram of total query response times, in seconds. |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of total query rows read. |
query_read_bytes |
Histogram | Head/Query peer | Histogram of total query read bytes. |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of total query rows written. |
Custom batch metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
processed_count |
Counter | Batch peer | Count of items processed. |
processed_failure |
Counter | Batch peer | Count of processing failures. |
processing_duration_histo |
Histogram | Batch peer | Histogram of Batch processing durations (milliseconds). |
processing_duration_summary |
Summary | Batch peer | Summary of Batch processing durations (milliseconds). |
rows_read |
Counter | Batch peer | Count of rows read. |
Custom merge metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
merge_duration_summary |
Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_duration_histo |
Histogram | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_summary |
Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_sdk_duration_histo |
Histogram | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_candidate_histo |
Histogram | Merge peer | Partitions per merge candidate. |
merge_candidate_inactive |
Counter | Merge peer | Merge candidates skipped due to an inactive partition within the candidate. |
merge_candidate_construction_summary |
Summary | Merge head | Time spent building merge candidates, in milliseconds. |
merge_queue_full |
Counter | Merge head | Times candidate generation was skipped due to a full queue. |
merge_success |
Counter | Merge peer | Count of merge successes. |
merge_failure |
Counter | Merge peer | Count of merge failures. |
Custom streaming metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in the backlog (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds (requires intake_head_index_backlog_enabled = true). |
http_source_outstanding_reqs |
Gauge | Intake head | Number of outstanding ingest event requests. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of HTTP requests. |
http_source_request_duration_ns |
Histogram | Stream head | Histogram of HTTP request durations in nanoseconds. |
http_source_request_error_count |
Counter | Stream head | Count of HTTP request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Custom Kinesis metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations, in nanoseconds. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of Kinesis checkpoint operation errors. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors reading from Kinesis. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis, in milliseconds. |
kinesis_source_operation_count |
Counter | Stream peer | Count of Kinesis operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of Kinesis operation durations, in nanoseconds. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
Custom Kafka metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration, in nanoseconds. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration, in nanoseconds. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed from Kafka. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed from Kafka. |
Custom DNS metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
dns_num_ips_in_cache |
Histogram | (ingest) | The size of the IP pool used in the DNS system. |
dns_lookup_time |
Histogram | (ingest) | Milliseconds per lookup. |
dns_ttl |
Histogram | (ingest) | TTLs received per lookup. |
Custom PostgreSQL metrics⚓︎
| Metric | Type | Purpose |
|---|---|---|
pgx_pool_total_acquire_count |
Count | The cumulative count of successful acquires from the pool. |
pgx_pool_total_acquire_duration_ns_count |
Count | The total duration (ns) of all successful acquires from the pool. |
pgx_pool_total_acquire_cancel_count |
Count | The cumulative count of acquires from the pool that were canceled by a context. |
pgx_pool_total_acquire_empty_count |
Count | The cumulative count of successful acquires from the pool that had to wait for a resource because the pool was empty. |
pgx_pool_total_conns_opened_count |
Count | The cumulative count of new connections opened. |
pgx_pool_total_destroyed_max_lifetime_count |
Count | The cumulative count of connections destroyed because they exceeded MaxConnLifetime. |
pgx_pool_total_destroyed_max_idle_count |
Count | The cumulative count of connections destroyed because they exceeded MaxConnIdleTime. |
pgx_pool_current_size |
Gauge | The total number of resources currently in the pool. |
pgx_pool_current_constructing |
Gauge | The number of connections currently being constructed. |
pgx_pool_current_acquired |
Gauge | The number of connections currently acquired. |
pgx_pool_current_idle |
Gauge | The number of currently idle connections in the pool. |
pgx_pool_max |
Gauge | The maximum size of the pool. |
| Metric Name | Description |
|---|---|
hdx_sink_spill_catalog_accepted_count |
Count of spilled catalog entries accepted into the indexing pipeline for processing |
hdx_sink_spill_catalog_duration_ns |
A histogram of the duration in nanoseconds taken to spill catalog adds |
hdx_sink_spill_catalog_failure_count |
Count of spill failures for catalog adds resulting in lost data |
hdx_sink_spill_catalog_race_lost_count |
Count of lost races attempting to add spilled data entry to catalog |
hdx_sink_spill_raw_accepted_count |
Count of spilled raw data entries accepted into the indexing pipeline for processing |
hdx_sink_spill_raw_duration_ns |
A histogram of the duration in nanoseconds taken to spill raw data buckets |
hdx_sink_spill_raw_failure_count |
Count of spill failures for raw data resulting in lost data |
| Metric Name | Description |
|---|---|
hdx_sink_spill_catalog_accepted_count |
Count of spilled catalog entries accepted into the indexing pipeline for processing |
hdx_sink_spill_catalog_duration_ns |
A histogram of the duration in nanoseconds taken to spill catalog adds |
hdx_sink_spill_catalog_failure_count |
Count of spill failures for catalog adds resulting in lost data |
hdx_sink_spill_catalog_race_lost_count |
Count of lost races attempting to add spilled data entry to catalog |
hdx_sink_spill_raw_accepted_count |
Count of spilled raw data entries accepted into the indexing pipeline for processing |
hdx_sink_spill_raw_duration_ns |
A histogram of the duration in nanoseconds taken to spill raw data buckets |
hdx_sink_spill_raw_failure_count |
Count of spill failures for raw data resulting in lost data |