Prometheus Integration
The Hydrolix stack includes Prometheus, an open-source metrics database. While the Stack runs, Hydrolix continuously updates its Prometheus instance with metrics information. You can query, view, and actively monitor this information through the use of a stack's Grafana instance or you can access it via your own monitoring platform.
Using Prometheus directly
Prometheus has its own web-based UI, available by visiting https://YOUR-HYDROLIX-HOSTNAME.hydrolix.live/prometheus
in your web browser.
This view is far more basic than Grafana's, suitable for quickly entering queries and seeing simple, graphed results. Hydrolix does make this feature available immediately, without any additional setup.
Hydrolix's metrics
This table lists the metrics available, and which components update them.
If more than one component uses a given metric, then querying it will return results from all relevant components. You can restrict results to a specific component by adding a service
keyword to your query, e.g. "process_open_fds{service="stream-peer"}
".
For more information about metric types, refer to Prometheus's documentation.
General metrics
These metrics track various counters and statistics regarding data ingestion.
Metric | Type | Components | Purpose |
---|---|---|---|
bytes_written | Counter | Batch peer, Stream peer | Bytes written to the indexer. |
partitions_created | Counter | Batch peer, Stream peer | Count of partitions created. |
process_cpu_seconds_total | Counter | Batch peer, Stream head, Stream peer | Total user and system CPU time spent in seconds. |
process_max_fds | Gauge | Batch peer, Stream head, Stream peer | Maximum number of open file descriptors. |
process_open_fds | Gauge | Batch peer, Stream head, Stream peer | Number of open file descriptors. |
process_resident_memory_bytes | Gauge | Batch peer, Stream head, Stream peer | Resident memory size in bytes. |
process_start_time_seconds | Gauge | Batch peer, Stream head, Stream peer | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes | Gauge | Batch peer, Stream head, Stream peer | Virtual memory size in bytes. |
process_virtual_memory_max_bytes | Gauge | Batch peer, Stream head, Stream peer | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight | Gauge | Batch peer, Stream head, Stream peer | Current number of scrapes being served. |
promhttp_metric_handler_requests_total | Counter | Batch peer, Stream head, Stream peer | Total number of scrapes by HTTP status code. |
Query metrics
These metrics track activity specific to batch ingestions.
Metric | Type | Components | Purpose |
---|---|---|---|
net_connect_attempts_total | Histogram | Head/Query peer | Histogram of TCP connection attempted to storage service |
net_connect_seconds | Histogram | Head/Query peer | Histogram of time to connect over TCP to storage service in seconds |
net_dns_resolve_seconds | Histogram | Head/Query peer | Histogram of DNS resolution time to storage service in seconds. |
net_http_response_time | Histogram | Head/Query peer | Histogram HTTP response time to storage service in seconds |
net_http_response_bytes | Histogram | Head/Query peer | Histogram of HTTP bytes downloaded from the storage service |
net_http_attempts_total | Histogram | Head/Query peer | Histogram of HTTP connection attempted to storage service |
net_http_status_code | Histogram | Head/Query peer | Histogram of HTTP status code result from storage service |
vfs_cache_hitmiss_total | Histogram | Head/Query peer | Histogram of cache status if bucket = 0 cache miss, and 1 cache hit |
vfs_cache_read_bytes | Histogram | Head/Query peer | Histogram bytes read from cache |
vfs_net_read_bytes | Histogram | Head/Query peer | Histogram bytes read from network |
vfs_cache_lru_file_eviction_total | Histogram | Head/Query peer | Histogram cache eviction of files |
epoll_cpu_seconds | Histogram | Head/Query peer | Histogram CPU used in seconds |
epoll_io_seconds | Histogram | Head/Query peer | Histogram I/O in seconds |
epoll_poll_seconds | Histogram | Head/Query peer | Histogram wait for file descriptor in seconds |
hdx_storage_r_catalog_partitions_total | Histogram | Head/Query peer | Histogram of per query catalog partition count |
hdx_storage_r_partitions_read_total | Histogram | Head/Query peer | Histogram of per query partition read count |
hdx_storage_r_partitions_per_core_total | Histogram | Head/Query peer | Histogram of per core partition used count |
hdx_storage_r_peers_used_total | Histogram | Query peer | Histogram of storage used total |
hdx_storage_r_cores_used_total | Histogram | Query peer | Histogram of Cores used total |
hdx_storage_r_catalog_timerange | Histogram | Head/Query peer | Histogram of query time range distribution |
hdx_partition_columns_read_total | Histogram | Head/Query peer | Histogram of column read |
hdx_partition_block_decode_seconds | Histogram | Head/Query peer | Histogram of time spent decoding hdx blocks in seconds |
hdx_partition_open_seconds | Histogram | Head/Query peer | Histogram of time spent opening hdx partition in seconds |
hdx_partition_read_seconds | Histogram | Head/Query peer | Histogram of time spent reading hdx partition in seconds |
hdx_partition_skipped_total | Histogram | Head/Query peer | Histogram of partition skip count due to no matching columns |
hdx_partition_blocks_read_total | Histogram | Head/Query peer | Histogram of partition read count |
hdx_partition_blocks_avail_total | Histogram | Head/Query peer | Histogram of partition blocks available |
hdx_partition_index_decision | Histogram | Head/Query peer | Histogram of partition decision if bucket = 0 fullscan, 1 partial scan and 2 no match |
hdx_partition_index_lookup_seconds | Histogram | Head/Query peer | Histogram of index lookup in seconds |
hdx_partition_index_blocks_skipped_percent | Histogram | Head/Query peer | Histogram of skipped index blocked in percentage |
hdx_partition_index_blocks_skipped_total | Histogram | Head/Query peer | Histogram of skipped index blocked in total |
hdx_partition_rd_w_err_total | Histogram | Head/Query peer | Histogram of errors if bucket = 0 read error, 1 written error and 3 error |
query_iowait_seconds | Histogram | Head/Query peer | Histogram query IO wait in seconds |
query_cpuwait_seconds | Histogram | Head/Query peer | Histogram query cpu wait in seconds |
query_hdx_ch_conv_seconds | Histogram | Head/Query peer | Histogram of time spent converting hdx blocks to clickhouse in seconds |
query_health | Histogram | Head/Query peer | Histogram of query health if bucket = 0 initiated error, 1 succeeded and 2 error |
query_peer_availability | Histogram | Head/Query peer | Histogram of query peer availability if bucket = 0 primary_peer_available, 1 secondary_peer_available and 2 no_reachable_peers |
query_attempts_total | Histogram | Head/Query peer | Histogram of query attempts total |
query_response_seconds | Histogram | Head/Query peer | Histogram of query response total in seconds |
query_rows_read_total | Histogram | Head/Query peer | Histogram of query rows read total |
query_read_bytes | Histogram | Head/Query peer | Histogram of query read bytes total |
query_rows_written_total | Histogram | Head/Query peer | Histogram of query rows written total |
Batch metrics
These metrics track activity specific to batch ingestions.
Metric | Type | Components | Purpose |
---|---|---|---|
processed_count | Counter | Batch peer | Count of items processed. |
processed_failure | Counter | Batch peer | Count of processing failures. |
processing_duration_histo | Histogram | Batch peer | Histogram of Batch processing durations in milliseconds. |
processing_duration_summary | Summary | Batch peer | Summary of Batch processing durations in milliseconds. |
rows_read | Counter | Batch peer | Count of rows read. |
Merge metrics
These metrics correspond to Hydrolix's merge service.
Metric | Type | Components | Purpose |
---|---|---|---|
merge_duration_summary | Summary | Merge peer | Merge processing duration, in milliseconds. |
merge_duration_histo | Histogram | Merge peer | Merge processing duration, in milliseconds. |
merge_sdk_duration_summary | Summary | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_sdk_duration_histo | Histogram | Merge peer | Merge SDK processing duration, in milliseconds. |
merge_candidate_histo | Histogram | Merge peer | Partitions per merge candidate. |
merge_success | Counter | Merge peer | Count of merge successes. |
merge_failure | Counter | Merge peer | Count of merge successes. |
Rabbit MQ
https://www.rabbitmq.com/prometheus.html
Streaming metrics
HTTP Stream Ingest
These metrics are specific to the use of streaming data sources.
Metric | Type | Components | Purpose |
---|---|---|---|
http_source_byte_count | Counter | Stream head | Count of bytes processed. |
http_source_request_count | Counter | Stream head | Count of http requests. |
http_source_request_duration_ns | Histogram | Stream head | A histogram of HTTP request durations in nanoseconds. |
http_source_request_error_count | Counter | Stream head | Count of http request failures. |
http_source_row_count | Counter | Stream head | Count of rows processed. |
http_source_value_count | Counter | Stream head | Count of values processed. |
kinesis_source_byte_count | Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count | Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_duration_ns | Histogram | Stream peer | Duration of Kinesis checkpoint operations. |
kinesis_source_checkpoint_error_count | Counter | Stream peer | Count of errors in Kinesis checkpoint operations. |
kinesis_source_error_count | Counter | Stream peer | Count of errors in Kinesis source reads. |
kinesis_source_lag_ms | Gauge | Stream peer | Measure of lag in Kinesis source. |
kinesis_source_operation_count | Counter | Stream peer | Count of operations on Kinesis. |
kinesis_source_operation_duration_ns | Histogram | Stream peer | Histogram of duration of operations on Kinesis. |
kinesis_source_record_count | Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count | Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count | Counter | Stream peer | Count of values read from Kinesis. |
Redpanda
https://docs.redpanda.com/docs/cluster-administration/monitoring/
Kafka Ingest
These metrics are specific to the use of Kafka data sources.
Metric | Type | Components | Purpose |
---|---|---|---|
kafka_source_byte_count | Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_commit_duration_ns | Histogram | Stream peer | Kafka commit duration. |
kafka_source_read_count | Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_duration_ns | Histogram | Stream peer | Kafka read duration. |
kafka_source_read_error_count | Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count | Counter | Stream peer | Count of rows processed. |
kafka_source_value_count | Counter | Stream peer | Count of values processed. |
Go environment metrics
These metrics track resources used by Hydrolix's Go environments.
Metric | Type | Components | Purpose |
---|---|---|---|
go_gc_duration_seconds | Summary | Batch peer, Stream head, Stream peer | A summary of the pause duration of garbage collection cycles. |
go_goroutines | Gauge | Batch peer, Stream head, Stream peer | Number of goroutines that currently exist. |
go_info | Gauge | Batch peer, Stream head, Stream peer | Information about the Go environment. |
go_memstats_alloc_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes allocated and still in use. |
go_memstats_alloc_bytes_total | Counter | Batch peer, Stream head, Stream peer | Total number of bytes allocated, even if freed. |
go_memstats_buck_hash_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes used by the profiling bucket hash table. |
go_memstats_frees_total | Counter | Batch peer, Stream head, Stream peer | Total number of frees. |
go_memstats_gc_cpu_fraction | Gauge | Batch peer, Stream head, Stream peer | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes that are in use. |
go_memstats_heap_objects | Gauge | Batch peer, Stream head, Stream peer | Number of allocated objects. |
go_memstats_heap_released_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds | Gauge | Batch peer, Stream head, Stream peer | Number of seconds since 1970 of last garbage collection. |
go_memstats_lookups_total | Counter | Batch peer, Stream head, Stream peer | Total number of pointer lookups. |
go_memstats_mallocs_total | Counter | Batch peer, Stream head, Stream peer | Total number of mallocs. |
go_memstats_mcache_inuse_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes | Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system. |
go_threads | Gauge | Batch peer, Stream head, Stream peer | Number of OS threads created. |
Updated 11 months ago