Summary Table Metrics
Summary table metrics overview⚓︎
Prometheus in Hydrolix⚓︎
The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information.
You can query, view, and actively monitor this information using a stack's Grafana instance, or you can access it with your own monitoring platform. See Prometheus Integration for more information about setting up Prometheus with an external server.
Use the Prometheus UI⚓︎
Prometheus has its own web-based UI.
This view is a basic metric view, suitable for quickly entering queries and seeing simple, graphed results. This feature is available in Hydrolix without any additional setup.
Navigate to https://<YourHostname>/prometheus to view dashboards.
For more information about Prometheus metric types, refer to the Prometheus documentation.
Hydrolix metrics⚓︎
Use multiple components with a metric⚓︎
If more than one component uses a given metric, querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query.
For example, process_open_fds{service="stream-peer"}.
Ingest peers⚓︎
For a list of all Hydrolix metrics, see All Metrics.
Each ingest method's peers has multiple containers. One container runs message acquisition, and the other is the indexer which indexes and completes enrichment jobs.
HTTP stream summary ingest⚓︎
These metrics are specific to the use of streaming data sources.
Traefik⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
traefik_service_requests_total |
Counter | Traefik | HTTP Traefik request information. |
traefik_service_request_duration_seconds_count/sum/bucket |
Counter | Traefik | Response time of Traefik to client. |
http_source_request_duration_ns_count/sum/bucket |
Counter | Traefik | Response time from Stream Head. |
Stream head and intake head⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Redpanda metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
internal_event_queue_byte_count{mode="sink"} |
Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="sink""} |
Counter | Stream Head | Row count sent to Redpanda |
internal_event_queue_byte_count{mode="sink""} |
Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="source"} |
Counter | Stream Summary | Row count received from Redpanda |
internal_event_queue_value_count{mode="source"} |
Counter | Stream Summary | Value count recieved from Redpanda |
Stream summary metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_count |
Counter | Summary Peer | Count of calls to the catalog. |
query_failure |
Counter | Summary Peer | Count of failed catalog calls. |
query_latency_summary |
Counter | Summary Peer | Latency of calls to catalog. |
Indexer Metrics⚓︎
Indexer metrics are metrics that cover indexing of ingested data.
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_row_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of rows processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_byte_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of bytes processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_value_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of values processed by the indexer and uploaded to storage. Includes Hot and Cold reporting. |
hdx_sink_error_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Count of errors in indexing and uploading to storage. |
indexer_rows_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total rows indexed (written to partitions). |
indexer_bytes_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total bytes indexed (written to partitions). |
indexer_partitions_rejected_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Histogram of partitions not written. Values: 0 = raw data parsing failed, 1 = raw data or transform schema mismatch, 3 = error writing partition file, 4 = other error during indexing. |
indexer_partitions_written_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Total partitions created. |
indexer_partition_write_seconds_count/bucket/sum |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | Time from receiving indexing query to writing partition file, in seconds. |
Storage⚓︎
Each of the object_store* metrics has these labels:
provider- Object storage provider. For example, AWS, Azure, GCS.code- HTTP response code.method- HTTP method used. For examplePOSTandGET.host- (optional) HTTP host used to target object storage.
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
object_store_http_status_code_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful. |
object_store_http_error_count |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of HTTP request errors (timeouts, connection errors, etc.) |
object_store_http_bytes_tx |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes transmitted to object storage (request body only) |
object_store_http_bytes_rx |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes received from object storage (response body only) |
net_http_status_code_bucket |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | HTTP Status Code histogram count from Storage. |
object_store_http_histo |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A histogram of object storage interaction latencies |
object_store_http_summary |
Summary | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A summary of object storage interaction latencies |
General ingestion metrics⚓︎
These metrics track various counters and statistics regarding data ingestion.
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
process_cpu_seconds_total |
Counter | Batch peer, Stream head, Stream peer | Total user and system CPU time spent in seconds. |
promhttp_metric_handler_requests_total |
Counter | Batch peer, Stream head, Stream peer | Total number of scrapes by HTTP status code. |
process_max_fds |
Gauge | Batch peer, Stream head, Stream peer | Maximum number of open file descriptors. |
process_open_fds |
Gauge | Batch peer, Stream head, Stream peer | Number of open file descriptors. |
process_resident_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Resident memory size in bytes. |
process_start_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes |
Gauge | Batch peer, Stream head, Stream peer | Virtual memory size in bytes. |
process_virtual_memory_max_bytes |
Gauge | Batch peer, Stream head, Stream peer | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight |
Gauge | Batch peer, Stream head, Stream peer | Current number of scrapes being served. |
Go environment metrics⚓︎
These metrics track resources used by Hydrolix in Go environments.
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
go_memstats_alloc_bytes_total |
Counter | Batch peer, Stream head, Stream peer | Total number of bytes allocated, even if freed. |
go_memstats_frees_total |
Counter | Batch peer, Stream head, Stream peer | Total number of frees. |
go_memstats_lookups_total |
Counter | Batch peer, Stream head, Stream peer | Total number of pointer lookups. |
go_memstats_mallocs_total |
Counter | Batch peer, Stream head, Stream peer | Total number of mallocs. |
go_goroutines |
Gauge | Batch peer, Stream head, Stream peer | Number of goroutines that currently exist. |
go_info |
Gauge | Batch peer, Stream head, Stream peer | Information about the Go environment. |
go_memstats_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes allocated and still in use. |
go_memstats_buck_hash_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used by the profiling bucket hash table. |
go_memstats_gc_cpu_fraction |
Gauge | Batch peer, Stream head, Stream peer | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes that are in use. |
go_memstats_heap_objects |
Gauge | Batch peer, Stream head, Stream peer | Number of allocated objects. |
go_memstats_heap_released_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds |
Gauge | Batch peer, Stream head, Stream peer | Number of seconds since 1970 of last garbage collection. |
go_memstats_mcache_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes |
Gauge | Batch peer, Stream head, Stream peer | Number of bytes obtained from system. |
go_threads |
Gauge | Batch peer, Stream head, Stream peer | Number of OS threads created. |
go_gc_duration_seconds |
Summary | Batch peer, Stream head, Stream peer | A summary of the pause duration of garbage collection cycles. |
For a complete list of the metrics used by Hydrolix, including Prometheus, RabbitMQ, and others, see All Metrics.