Overview
This page contains most metrics used in Hydrolix for the following:
Use find on page to locate specific metrics.
Custom Hydrolix metrics⚓︎
Custom general metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
bytes_written |
Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created |
Counter | Batch peer, Stream peer | Count of partitions created. |
upload_duration |
Summary | Any intake peer | Time spent uploading a file, in milliseconds. |
Custom query metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempts to the storage service. |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to the storage service, in seconds. |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time for the storage service, in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram of HTTP response times from the storage service, in seconds. |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service. |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempts to the storage service. |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status codes from the storage service. |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status (bucket=0 cache miss, bucket=1 cache hit). |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the cache. |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram of bytes read from the network. |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram of file evictions from the cache. |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram of CPU usage in seconds. |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram of I/O times in seconds. |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram of wait times for file descriptors, in seconds. |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per-query catalog partition count. |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per-query partition read count. |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per-core partition usage count. |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total. |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of cores used total. |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time-range distribution. |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of columns read. |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding HDX blocks, in seconds. |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening HDX partitions, in seconds. |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading HDX partitions, in seconds. |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partitions skipped due to no matching columns. |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition-block read counts. |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available. |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decisions (bucket=0 full scan, 1 partial scan, 2 no match). |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup times, in seconds. |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocks, in percentage. |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of total skipped index blocks. |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of read/write errors (bucket=0 read error, 1 write error, 3 error). |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram of query I/O wait times, in seconds. |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram of query CPU wait times, in seconds. |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting HDX blocks to ClickHouse, in seconds. |
query_health |
Histogram | Head/Query peer | Histogram of query health (bucket=0 initiated error, 1 succeeded, 2 error). |
query_peer_availability |
Histogram | Head/Query peer | Histogram of peer availability (bucket=0 primary_peer_available, 1 secondary_peer_available, 2 no_reachable_peers). |
query_attempts_total |
Histogram | Head/Query peer | Histogram of total query attempts. |
query_response_seconds |
Histogram | Head/Query peer | Histogram of total query response times, in seconds. |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of total query rows read. |
query_read_bytes |
Histogram | Head/Query peer | Histogram of total query read bytes. |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of total query rows written. |
Custom batch metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
processed_count |
Counter | Batch peer | Count of items processed. |
processed_failure |
Counter | Batch peer | Count of processing failures. |
processing_duration_histo |
Histogram | Batch peer | Histogram of Batch processing durations (milliseconds). |
processing_duration_summary |
Summary | Batch peer | Summary of Batch processing durations (milliseconds). |
rows_read |
Counter | Batch peer | Count of rows read. |
Custom merge metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
merge_duration_summary |
Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_duration_histo |
Histogram | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_summary |
Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_sdk_duration_histo |
Histogram | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_candidate_histo |
Histogram | Merge peer | Partitions per merge candidate. |
merge_candidate_inactive |
Counter | Merge peer | Merge candidates skipped due to an inactive partition within the candidate. |
merge_candidate_construction_summary |
Summary | Merge head | Time spent building merge candidates, in milliseconds. |
merge_queue_full |
Counter | Merge head | Times candidate generation was skipped due to a full queue. |
merge_success |
Counter | Merge peer | Count of merge successes. |
merge_failure |
Counter | Merge peer | Count of merge failures. |
Custom streaming metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in the backlog (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds (requires intake_head_index_backlog_enabled = true). |
http_source_outstanding_reqs |
Gauge | Intake head | Number of outstanding ingest event requests. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of HTTP requests. |
http_source_request_duration_ns |
Histogram | Stream head | Histogram of HTTP request durations in nanoseconds. |
http_source_request_error_count |
Counter | Stream head | Count of HTTP request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Custom AWS Kinesis metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations, in nanoseconds. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of Kinesis checkpoint operation errors. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors reading from Kinesis. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis, in milliseconds. |
kinesis_source_operation_count |
Counter | Stream peer | Count of Kinesis operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of Kinesis operation durations, in nanoseconds. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
Custom Kafka metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration, in nanoseconds. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration, in nanoseconds. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed from Kafka. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed from Kafka. |
Custom DNS metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
dns_num_ips_in_cache |
Histogram | (ingest) | The size of the IP pool used in the DNS system. |
dns_lookup_time |
Histogram | (ingest) | Milliseconds per lookup. |
dns_ttl |
Histogram | (ingest) | TTLs received per lookup. |
Custom PostgreSQL metrics⚓︎
| Metric | Type | Purpose |
|---|---|---|
pgx_pool_total_acquire_count |
Count | The cumulative count of successful acquires from the pool. |
pgx_pool_total_acquire_duration_ns_count |
Count | The total duration (ns) of all successful acquires from the pool. |
pgx_pool_total_acquire_cancel_count |
Count | The cumulative count of acquires from the pool that were canceled by a context. |
pgx_pool_total_acquire_empty_count |
Count | The cumulative count of successful acquires from the pool that had to wait for a resource because the pool was empty. |
pgx_pool_total_conns_opened_count |
Count | The cumulative count of new connections opened. |
pgx_pool_total_destroyed_max_lifetime_count |
Count | The cumulative count of connections destroyed because they exceeded MaxConnLifetime. |
pgx_pool_total_destroyed_max_idle_count |
Count | The cumulative count of connections destroyed because they exceeded MaxConnIdleTime. |
pgx_pool_current_size |
Gauge | The total number of resources currently in the pool. |
pgx_pool_current_constructing |
Gauge | The number of connections currently being constructed. |
pgx_pool_current_acquired |
Gauge | The number of connections currently acquired. |
pgx_pool_current_idle |
Gauge | The number of currently idle connections in the pool. |
pgx_pool_max |
Gauge | The maximum size of the pool. |
| Metric Name | Description |
|---|---|
hdx_sink_spill_catalog_accepted_count |
Count of spilled catalog entries accepted into the indexing pipeline for processing |
hdx_sink_spill_catalog_duration_ns |
A histogram of the duration in nanoseconds taken to spill catalog adds |
hdx_sink_spill_catalog_failure_count |
Count of spill failures for catalog adds resulting in lost data |
hdx_sink_spill_catalog_race_lost_count |
Count of lost races attempting to add spilled data entry to catalog |
hdx_sink_spill_raw_accepted_count |
Count of spilled raw data entries accepted into the indexing pipeline for processing |
hdx_sink_spill_raw_duration_ns |
A histogram of the duration in nanoseconds taken to spill raw data buckets |
hdx_sink_spill_raw_failure_count |
Count of spill failures for raw data resulting in lost data |
Age metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_count |
Counter | Decay/Reaper calls to the catalog. | |
query_failure |
Counter | Decay/Reaper failed catalog calls. | |
query_latency_summary |
Counter | Latency in calls to catalog. | |
query_latency_summary_count/sum |
Counter | Latency in calls to catalog. |
Alter metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_count |
Counter | Decay/Reaper calls to the catalog. | |
query_failure |
Counter | Decay/Reaper failed catalog calls. | |
query_latency_summary |
Counter | Latency in calls to catalog. | |
query_latency_summary_count/sum |
Counter | Latency in calls to catalog. |
Intake spill metrics⚓︎
| Metric Name | Description |
|---|---|
hdx_sink_spill_catalog_accepted_count |
Count of spilled catalog entries accepted into the indexing pipeline for processing |
hdx_sink_spill_catalog_duration_ns |
A histogram of the duration in nanoseconds taken to spill catalog adds |
hdx_sink_spill_catalog_failure_count |
Count of spill failures for catalog adds resulting in lost data |
hdx_sink_spill_catalog_race_lost_count |
Count of lost races attempting to add spilled data entry to catalog |
hdx_sink_spill_raw_accepted_count |
Count of spilled raw data entries accepted into the indexing pipeline for processing |
hdx_sink_spill_raw_duration_ns |
A histogram of the duration in nanoseconds taken to spill raw data buckets |
hdx_sink_spill_raw_failure_count |
Count of spill failures for raw data resulting in lost data |
Merge metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
merge_success |
Counter | Count of merge successes. | |
merge_failure |
Counter | Count of merge successes. | |
query_count |
Counter | Merge calls to the catalog. | |
query_failure |
Counter | Merge failed catalog calls. | |
query_latency_summary |
Counter | Latency in calls to catalog. | |
query_latency_summary_count/sum |
Counter | Latency in calls to catalog. | |
merge_duration_histo_bucket/count/sum |
Histogram | Merge processing duration, in milliseconds. | |
merge_duration_summary |
Summary | Merge processing duration, in milliseconds. |
Merge controller metrics⚓︎
| Metric Name | Type | Components | Description | Labels |
|---|---|---|---|---|
partition_distribution |
Histogram | Merge Controller | Histogram of partition memory size | project_id project_name table_id table_name target |
segments |
Gauge | Merge Controller | Current count of known segments. A segment is a group of buckets that share a merge target. | project_id project_name table_id table_name target |
active_merges |
Gauge | Merge Controller | Current count of active merge operations | project_id project_name table_id table_name target |
partitions |
Gauge | Merge Controller | Current count of partitions awaiting segmentation | project_id project_name table_id table_name target |
partitions_dispatched |
Counter | Merge Controller | Count of partitions dispatched for merging. Can be used in combination with the rate function to determine “partitions dispatched for merge per second” | project_id project_name table_id table_name target |
candidates |
Gauge | Merge Controller | Current count of constructed candidates waiting dispatch. | project_id project_name table_id table_name target |
candidates_dispatched |
Counter | Merge Controller | Count of candidates dispatched for merging. Can be used in combination with the rate function to determine “candidates dispatched for merge per second” | project_id project_name table_id table_name target |
partitions_per_candidate |
Histogram | Merge Controller | Distribution of partitions within a dispatched candidate. | project_id project_name table_id table_name target |
candidate_mem_size |
Histogram | Merge Controller | Distribution of calculated memory requirements for dispatched candidates | project_id project_name table_id table_name target |
duplicate_partitions |
Counter | Merge Controller | Count of partitions sourced that are already being tracked. | project_id project_name table_id table_name target |
memory_coefficient |
Gauge | Merge Controller | Current value of the memory coefficient being applied to sourced partitions. | project_id project_name table_id table_name target |
expired_segments |
Counter | Merge Controller | Count of segments expired for moving out of the target range | project_id project_name table_id table_name target |
bucket_duration |
Histogram | Merge Controller | Distribution of time in milliseconds of a bucket’s age upon closing. The basis label captures the reason for closing. Options are: full: The bucket reached max size idle_ttl: The bucket wasn't added to within the idle timeout age_ttl: The bucket was open longer than the max age segment_ttl: The bucket was part of a segment that fell out of the target range |
project_id project_name table_id table_name target basis |
connected_clients |
Gauge | Merge Controller | Count of currently connected merge-peers | pool_id |
query_latency |
Histogram | Merge Controller | Distribution of response times in milliseconds for catalog operations | project_id table_id method |
RabbitMQ metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
rabbitmq_queue_messages |
Counter | Sum of ready and unacknowledged messages (queue depth). | |
rabbitmq_queues |
Counter | RabbitMQ Queues count. | |
erlang_vm_statistics_bytes_received_total |
Counter | The total number of bytes received through RabbitMQ ports. | |
erlang_vm_statistics_bytes_output_total |
Counter | The total number of bytes output through RabbitMQ ports. |
Prometheus⚓︎
Prometheus batch metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
processed_count |
Counter | Batch peer | Count of items processed. |
processed_failure |
Counter | Batch peer | Count of processing failures. |
rows_read |
Counter | Batch peer | Count of rows read. |
processing_duration_histo |
Histogram | Batch peer | Histogram of batch processing durations in milliseconds. |
processing_duration_summary |
Summary | Batch peer | Summary of batch processing durations in milliseconds. |
Prometheus DNS metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
dns_num_ips_in_cache |
Histogram | (ingest) | The size of the IP pool used in the DNS system. |
dns_lookup_time |
Histogram | (ingest) | Milliseconds per lookup. |
dns_ttl |
Histogram | (ingest) | TTLs received per lookup. |
Prometheus general metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
bytes_written |
Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created |
Counter | Batch peer, Stream peer | Count of partitions created. |
process_cpu_seconds_total |
Counter | Batch peer, Stream head, Stream peer | Total user and system CPU time spent in seconds. |
promhttp_metric_handler_requests_total |
Counter | Batch peer, Stream head, Stream peer | Total number of scrapes by HTTP status code. |
process_max_fds |
Gauge | Batch peer, Stream head, Stream peer | Maximum number of open file descriptors. |
process_open_fds |
Gauge | Batch peer, Stream head, Stream peer | Number of open file descriptors. |
process_resident_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Resident memory size in bytes. |
process_start_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Virtual memory size in bytes. |
process_virtual_memory_max_bytes |
Gauge | Batch peer, Stream head, Stream peer | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight |
Gauge | Batch peer, Stream head, Stream peer | Current number of scrapes being served. |
upload_duration |
Summary | Any intake peer | Time spent uploading a file, in milliseconds |
Prometheus Go metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
go_memstats_alloc_bytes_total |
Counter | Batch peer, Stream head, Stream peer | Total number of bytes allocated, even if freed. |
go_memstats_frees_total |
Counter | Batch peer, Stream head, Stream peer | Total number of frees. |
go_memstats_lookups_total |
Counter | Batch peer, Stream head, Stream peer | Total number of pointer lookups. |
go_memstats_mallocs_total |
Counter | Batch peer, Stream head, Stream peer | Total number of mallocs. |
go_goroutines |
Gauge | Batch peer, Stream head, Stream peer | Number of goroutines that currently exist. |
go_info |
Gauge | Batch peer, Stream head, Stream peer | Information about the Go environment. |
go_memstats_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes allocated and still in use. |
go_memstats_buck_hash_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used by the profiling bucket hash table. |
go_memstats_gc_cpu_fraction |
Gauge | Batch peer, Stream head, Stream peer | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes that are in use. |
go_memstats_heap_objects |
Gauge | Batch peer, Stream head, Stream peer | Number of allocated objects. |
go_memstats_heap_released_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Number of seconds since 1970 of last garbage collection. |
go_memstats_mcache_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system. |
go_threads |
Gauge | Batch peer, Stream head, Stream peer | Number of OS threads created. |
go_gc_duration_seconds |
Summary | Batch peer, Stream head, Stream peer | A summary of the pause duration of garbage collection cycles. |
Prometheus stream ingest metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog growing too big. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog growing too big. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing. Only produced when intake_head_index_backlog_enabled is true. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of errors in Kinesis checkpoint operations. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors in Kinesis source reads. |
kinesis_source_operation_count |
Counter | Stream peer | Count of operations on Kinesis. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in sink backlog waiting to be indexed. Only produced when intake_head_index_backlog_enabled is true. |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in sink backlog waiting to be indexed. Only produced when intake_head_index_backlog_enabled is true. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis source. |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds. Only produced when intake_head_index_backlog_enabled is true. |
http_source_request_duration_ns |
Histogram | Stream head | A histogram of HTTP request durations in nanoseconds. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of duration of operations on Kinesis. |
Prometheus Kafka metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration. |
Prometheus merge metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
merge_candidate_inactive |
Counter | Merge peer | Merge candidates skipped due to an inactive partition within the candidate |
merge_queue_full |
Counter | Merge head | Times candidate generation was skipped due to a full queue |
merge_success |
Counter | Merge peer | Count of merge successes. |
merge_failure |
Counter | Merge peer | Count of merge successes. |
merge_duration_histo |
Histogram | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_histo |
Histogram | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_candidate_histo |
Histogram | Merge peer | Partitions per merge candidate. |
merge_duration_summary |
Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_summary |
Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_candidate_construction_summary |
Summary | Merge head | Time spent building merge candidates, in milliseconds. |
Prometheus operator metrics⚓︎
| Metric | Type | Description |
|---|---|---|
o6r_up |
Gauge | Returns 1 if Operator is running, otherwise 0 |
o6r_reconcile_seconds_created |
Gauge | Total seconds spent in the Operator reconcile loop |
o6r_reconcile_seconds_total |
Counter | Cumulative total seconds spent in the Operator reconcile loop |
o6r_hdx_ready |
Gauge | Returns 1 if the hydrolixcluster is ready, otherwise 0 |
o6r_hdx_synced |
Gauge | 1 if the Operator is successfully syncing resources, otherwise 0 |
o6r_hdx_warnings |
Gauge | Number of validation warnings for a hydrolixcluster |
o6r_hdx_ignored_diffs |
Gauge | Number of resources wirh pending changes ignored by the Operator |
o6r_hdx_htns |
Gauge | Number of HTNs being used in a hydrolixcluster. See Hydrolix Tunables |
Prometheus PostgreSQL metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
pgx_pool_total_acquire_count |
Count | The cumulative count of successful acquires from the pool. | |
pgx_pool_total_acquire_duration_ns_count |
Count | The total duration of all successful acquires from the pool. | |
pgx_pool_total_acquire_cancel_count |
Count | The cumulative count of acquires from the pool that were canceled by a context. | |
pgx_pool_total_acquire_empty_count |
Count | The cumulative count of successful acquires from the pool that waited for a resource to be released or constructed because the pool was empty. | |
pgx_pool_total_conns_opened_count |
Count | The cumulative count of new connections opened. | |
pgx_pool_total_destroyed_max_lifetime_count |
Count | The cumulative count of connections destroyed because they exceeded MaxConnLifetime. | |
pgx_pool_total_destroyed_max_idle_count |
Count | The cumulative count of connections destroyed because they exceeded MaxConnIdleTime. | |
pgx_pool_current_size |
Gauge | The total number of resources currently in the pool. | |
pgx_pool_current_constructing |
Gauge | The number of connections with construction in progress in the pool. | |
pgx_pool_current_acquired |
Gauge | The number of currently acquired connections in the pool. | |
pgx_pool_current_idle |
Gauge | The number of currently idle connections in the pool. | |
pgx_pool_max |
Gauge | The maximum size of the pool. |
Prometheus query metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
net_connect_attempts_total |
Histogram | Head/Query peer | Histogram of TCP connection attempted to storage service |
net_connect_seconds |
Histogram | Head/Query peer | Histogram of time to connect over TCP to storage service in seconds |
net_dns_resolve_seconds |
Histogram | Head/Query peer | Histogram of DNS resolution time to storage service in seconds. |
net_http_response_time |
Histogram | Head/Query peer | Histogram HTTP response time to storage service in seconds |
net_http_response_bytes |
Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service |
net_http_attempts_total |
Histogram | Head/Query peer | Histogram of HTTP connection attempted to storage service |
net_http_status_code |
Histogram | Head/Query peer | Histogram of HTTP status code result from storage service |
vfs_cache_hitmiss_total |
Histogram | Head/Query peer | Histogram of cache status if bucket = 0 cache miss, and 1 cache hit |
vfs_cache_read_bytes |
Histogram | Head/Query peer | Histogram bytes read from cache |
vfs_net_read_bytes |
Histogram | Head/Query peer | Histogram bytes read from network |
vfs_cache_lru_file_eviction_total |
Histogram | Head/Query peer | Histogram cache eviction of files |
epoll_cpu_seconds |
Histogram | Head/Query peer | Histogram CPU used in seconds |
epoll_io_seconds |
Histogram | Head/Query peer | Histogram I/O in seconds |
epoll_poll_seconds |
Histogram | Head/Query peer | Histogram wait for file descriptor in seconds |
hdx_storage_r_catalog_partitions_total |
Histogram | Head/Query peer | Histogram of per query catalog partition count |
hdx_storage_r_partitions_read_total |
Histogram | Head/Query peer | Histogram of per query partition read count |
hdx_storage_r_partitions_per_core_total |
Histogram | Head/Query peer | Histogram of per core partition used count |
hdx_storage_r_peers_used_total |
Histogram | Query peer | Histogram of storage used total |
hdx_storage_r_cores_used_total |
Histogram | Query peer | Histogram of Cores used total |
hdx_storage_r_catalog_timerange |
Histogram | Head/Query peer | Histogram of query time range distribution |
hdx_partition_columns_read_total |
Histogram | Head/Query peer | Histogram of column read |
hdx_partition_block_decode_seconds |
Histogram | Head/Query peer | Histogram of time spent decoding hdx blocks in seconds |
hdx_partition_open_seconds |
Histogram | Head/Query peer | Histogram of time spent opening hdx partition in seconds |
hdx_partition_read_seconds |
Histogram | Head/Query peer | Histogram of time spent reading hdx partition in seconds |
hdx_partition_skipped_total |
Histogram | Head/Query peer | Histogram of partition skip count due to no matching columns |
hdx_partition_blocks_read_total |
Histogram | Head/Query peer | Histogram of partition read count |
hdx_partition_blocks_avail_total |
Histogram | Head/Query peer | Histogram of partition blocks available |
hdx_partition_index_decision |
Histogram | Head/Query peer | Histogram of partition decision if bucket = 0 fullscan, 1 partial scan and 2 no match |
hdx_partition_index_lookup_seconds |
Histogram | Head/Query peer | Histogram of index lookup in seconds |
hdx_partition_index_blocks_skipped_percent |
Histogram | Head/Query peer | Histogram of skipped index blocked in percentage |
hdx_partition_index_blocks_skipped_total |
Histogram | Head/Query peer | Histogram of skipped index blocked in total |
hdx_partition_rd_w_err_total |
Histogram | Head/Query peer | Histogram of errors if bucket = 0 read error, 1 written error and 3 error |
query_iowait_seconds |
Histogram | Head/Query peer | Histogram query IO wait in seconds |
query_cpuwait_seconds |
Histogram | Head/Query peer | Histogram query cpu wait in seconds |
query_hdx_ch_conv_seconds |
Histogram | Head/Query peer | Histogram of time spent converting hdx blocks to clickhouse in seconds |
query_health |
Histogram | Head/Query peer | Histogram of query health if bucket = 0 initiated error, 1 succeeded and 2 error |
query_peer_availability |
Histogram | Head/Query peer | Histogram of query peer availability if bucket = 0 primary_peer_available, 1 secondary_peer_available and 2 no_reachable_peers |
query_attempts_total |
Histogram | Head/Query peer | Histogram of query attempts total |
query_response_seconds |
Histogram | Head/Query peer | Histogram of query response total in seconds |
query_rows_read_total |
Histogram | Head/Query peer | Histogram of query rows read total |
query_read_bytes |
Histogram | Head/Query peer | Histogram of query read bytes total |
query_rows_written_total |
Histogram | Head/Query peer | Histogram of query rows written total |
Summary table⚓︎
Summary table general metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
process_cpu_seconds_total |
Counter | Total user and system CPU time spent in seconds. | |
promhttp_metric_handler_requests_total |
Counter | Total number of scrapes by HTTP status code. | |
process_max_fds |
Gauge | Maximum number of open file descriptors. | |
process_open_fds |
Gauge | Number of open file descriptors. | |
process_resident_memory_bytes |
Gauge | Resident memory size in bytes. | |
process_start_time_seconds |
Gauge | Start time of the process since unix epoch in seconds. | |
process_virtual_memory_bytes |
Gauge | Virtual memory size in bytes. | |
process_virtual_memory_max_bytes |
Gauge | Maximum amount of virtual memory available in bytes. | |
promhttp_metric_handler_requests_in_flight |
Gauge | Current number of scrapes being served. |
Summary table Go metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
go_memstats_alloc_bytes_total |
Counter | Total number of bytes allocated, even if freed. | |
go_memstats_frees_total |
Counter | Total number of frees. | |
go_memstats_lookups_total |
Counter | Total number of pointer lookups. | |
go_memstats_mallocs_total |
Counter | Total number of mallocs. | |
go_goroutines |
Gauge | Number of goroutines that currently exist. | |
go_info |
Gauge | Information about the Go environment. | |
go_memstats_alloc_bytes |
Gauge | Number of bytes allocated and still in use. | |
go_memstats_buck_hash_sys_bytes |
Gauge | Number of bytes used by the profiling bucket hash table. | |
go_memstats_gc_cpu_fraction |
Gauge | The fraction of this program's available CPU time used by the GC since the program started. | |
go_memstats_gc_sys_bytes |
Gauge | Number of bytes used for garbage collection system metadata. | |
go_memstats_heap_alloc_bytes |
Gauge | Number of heap bytes allocated and still in use. | |
go_memstats_heap_idle_bytes |
Gauge | Number of heap bytes waiting to be used. | |
go_memstats_heap_inuse_bytes |
Gauge | Number of heap bytes that are in use. | |
go_memstats_heap_objects |
Gauge | Number of allocated objects. | |
go_memstats_heap_released_bytes |
Gauge | Number of heap bytes released to OS. | |
go_memstats_heap_sys_bytes |
Gauge | Number of heap bytes obtained from system. | |
go_memstats_last_gc_time_seconds |
Gauge | Number of seconds since 1970 of last garbage collection. | |
go_memstats_mcache_inuse_bytes |
Gauge | Number of bytes in use by mcache structures. | |
go_memstats_mcache_sys_bytes |
Gauge | Number of bytes used for mcache structures obtained from system. | |
go_memstats_mspan_inuse_bytes |
Gauge | Number of bytes in use by mspan structures. | |
go_memstats_mspan_sys_bytes |
Gauge | Number of bytes used for mspan structures obtained from system. | |
go_memstats_next_gc_bytes |
Gauge | Number of heap bytes when next garbage collection will take place. | |
go_memstats_other_sys_bytes |
Gauge | Number of bytes used for other system allocations. | |
go_memstats_stack_inuse_bytes |
Gauge | Number of bytes in use by the stack allocator. | |
go_memstats_stack_sys_bytes |
Gauge | Number of bytes obtained from system for stack allocator. | |
go_memstats_sys_bytes |
Gauge | Number of bytes obtained from system. | |
go_threads |
Gauge | Number of OS threads created. | |
go_gc_duration_seconds |
Summary | A summary of the pause duration of garbage collection cycles. |
Summary table indexer metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_row_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of rows processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_byte_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of bytes processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_value_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of values processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_error_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of errors in indexing and uploading to storage. |
indexer_rows_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total rows indexed (written to partitions) |
indexer_bytes_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total bytes indexed (written to partitions) |
indexer_partitions_rejected_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Histogram of partitions not able to written. If value is 0=raw data parsing failed, 1=raw data / transform schema mismatch, 3=Error writing partition file, 4= Other Error during indexing |
indexer_partitions_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total partitions created |
indexer_partition_write_seconds_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Time from receiving indexing query to writing partition file (seconds) |
Summary table Redpanda metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
internal_event_queue_byte_count{mode="sink"} |
Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="sink""} |
Counter | Stream Head | Row count sent to Redpanda |
internal_event_queue_byte_count{mode="sink""} |
Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="source"} |
Counter | Stream Summary | Row count received from Redpanda |
internal_event_queue_value_count{mode="source"} |
Counter | Stream Summary | Value count recieved from Redpanda |
Summary table storage metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
object_store_http_status_code_count |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful. |
object_store_http_error_count |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of HTTP request errors (timeouts, connection errors, etc.) |
object_store_http_bytes_tx |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes transmitted to object storage (request body only) |
object_store_http_bytes_rx |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes received from object storage (response body only) |
net_http_status_code_bucket |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | HTTP Status Code histogram count from Storage. |
object_store_http_histo |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A histogram of object storage interaction latencies |
object_store_http_summary |
Summary | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A summary of object storage interaction latencies |
Summary table stream and intake head metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
http_source_request_duration_ns_count/bucket/sum |
Histogram | Stream head | A histogram of HTTP request durations in nanoseconds. |
Summary table stream summary metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_latency_summary_count/sum |
Count/Sum | Summary Peer | Latency of calls to catalog. |
query_count |
Counter | Summary Peer | Count of calls to the catalog. |
query_failure |
Counter | Summary Peer | Count of failed catalog calls. |
query_latency_summary |
Counter | Summary Peer | Latency of calls to catalog. |
Summary table Traefik metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
traefik_service_requests_total |
Counter | Traefik | HTTP Traefik request information. |
traefik_service_request_duration_seconds_count/sum/bucket |
Counter | Traefik | Response time of traefik to client. |
http_source_request_duration_ns_count/sum/bucket |
Counter | Traefik | Response time from Stream-Head. |