Stream Metrics
Stream metrics overview⚓︎
Prometheus in Hydrolix⚓︎
The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information.
You can query, view, and actively monitor this information using a stack's Grafana instance, or you can access it with your own monitoring platform. See Prometheus Integration for more information about setting up Prometheus with an external server.
Use the Prometheus UI⚓︎
Prometheus has its own web-based UI.
This view is a basic metric view, suitable for quickly entering queries and seeing simple, graphed results. This feature is available in Hydrolix without any additional setup.
Navigate to https://<YourHostname>/prometheus to view dashboards.
For more information about Prometheus metric types, refer to the Prometheus documentation.
Use multiple components with a metric⚓︎
If more than one component uses a given metric, querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query.
For example, "process_open_fds{service="stream-peer"}".
For a list of all Hydrolix metrics, see All Metrics.
Each ingest method's peers has multiple containers. One container runs message acquisition, and the other is the indexer which indexes and completes enrichment jobs.
HTTP stream ingest metrics⚓︎
These metrics are specific to the use of streaming data sources.
Traefik metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
traefik_service_requests_total |
Counter | Traefik | HTTP Traefik request information. |
traefik_service_request_duration_seconds_count/sum/bucket |
Counter | Traefik | Response time of traefik to client. |
http_source_request_duration_ns_count/sum/bucket |
Counter | Traefik | Response time from Stream-Head. |
Intake-head metrics⚓︎
Intake-head replaces the older Stream-head/Stream-peer architecture.
| Metric | Type | Components | Purpose |
|---|---|---|---|
hdx_sink_backlog_bytes_count |
Gauge | Intake head | Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_items_count |
Gauge | Intake head | Total count of partition buckets in the backlog (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_bytes_count |
Counter | Intake head | Total bytes of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_dropped_items_count |
Counter | Intake head | Count of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_delivery_count |
Counter | Intake head | Count of backlog buckets successfully handed off to indexing (requires intake_head_index_backlog_enabled = true). |
hdx_sink_backlog_trim_duration_ns |
Histogram | Intake head | Time to trim the backlog in nanoseconds (requires intake_head_index_backlog_enabled = true). |
http_source_outstanding_reqs |
Gauge | Intake head | Number of outstanding ingest event requests. |
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of HTTP requests. |
http_source_request_duration_ns |
Histogram | Stream head | Histogram of HTTP request durations in nanoseconds. |
http_source_request_error_count |
Counter | Stream head | Count of HTTP request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
Stream-head metrics⚓︎
Stream-heads coordinate streaming jobs.
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
http_source_byte_count |
Counter | Stream head | Count of bytes processed. |
http_source_request_count |
Counter | Stream head | Count of http requests. |
http_source_request_error_count |
Counter | Stream head | Count of http request failures. |
http_source_row_count |
Counter | Stream head | Count of rows processed. |
http_source_value_count |
Counter | Stream head | Count of values processed. |
http_source_request_duration_ns_count/bucket/sum |
Histogram | Stream head | A histogram of HTTP request durations in nanoseconds. |
Redpanda metrics⚓︎
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
internal_event_queue_byte_count{mode="sink"} |
Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="sink""} |
Counter | Stream Head | Row count sent to Redpanda |
internal_event_queue_byte_count{mode="sink""} |
Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="source"} |
Counter | Stream Summary | Row count received from Redpanda |
internal_event_queue_value_count{mode="source"} |
Counter | Stream Summary | Value count recieved from Redpanda |
Stream-peer metrics⚓︎
Stream-peers carry out streaming jobs.
| Metric Name | Type | Components | Purpose |
|---|---|---|---|
query_latency_summary_count/sum |
Count/Sum | Summary Peer | Latency of calls to catalog. |
query_count |
Counter | Summary Peer | Count of calls to the catalog. |
query_failure |
Counter | Summary Peer | Count of failed catalog calls. |
query_latency_summary |
Counter | Summary Peer | Latency of calls to catalog. |
Stream ingest also produces Indexer metrics.
Kafka ingest metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kafka_source_byte_count |
Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_commit_duration_ns |
Histogram | Stream peer | Kafka commit duration, in nanoseconds. |
kafka_source_read_count |
Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_duration_ns |
Histogram | Stream peer | Kafka read duration, in nanoseconds. |
kafka_source_read_error_count |
Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count |
Counter | Stream peer | Count of rows processed from Kafka. |
kafka_source_value_count |
Counter | Stream peer | Count of values processed from Kafka. |
Kafka ingest also produces Indexer metrics.
AWS Kinesis ingest metrics⚓︎
| Metric | Type | Components | Purpose |
|---|---|---|---|
kinesis_source_byte_count |
Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count |
Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_duration_ns |
Histogram | Stream peer | Duration of Kinesis checkpoint operations, in nanoseconds. |
kinesis_source_checkpoint_error_count |
Counter | Stream peer | Count of Kinesis checkpoint operation errors. |
kinesis_source_error_count |
Counter | Stream peer | Count of errors reading from Kinesis. |
kinesis_source_lag_ms |
Gauge | Stream peer | Measure of lag in Kinesis, in milliseconds. |
kinesis_source_operation_count |
Counter | Stream peer | Count of Kinesis operations. |
kinesis_source_operation_duration_ns |
Histogram | Stream peer | Histogram of Kinesis operation durations, in nanoseconds. |
kinesis_source_record_count |
Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count |
Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count |
Counter | Stream peer | Count of values read from Kinesis. |
Kinesis ingest also produces Indexer metrics.
Storage⚓︎
Each of the object_store* metrics has these labels:
provider- Object storage provider. For example, AWS, Azure, GCS.code- HTTP response code.method- HTTP method used. For examplePOSTandGET.host- (optional) HTTP host used to target object storage.
| Metric | Type | Components | Purpose |
|---|---|---|---|
net_http_status_code_bucket |
Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | HTTP status code histogram count from storage. |
object_store_http_histo |
Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A histogram of object storage interaction latencies |
object_store_http_summary |
Summary | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A summary of object storage interaction latencies |
object_store_http_status_code_count |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful. |
object_store_http_error_count |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of HTTP request errors. For example, timeouts and connection errors. |
object_store_http_bytes_tx |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes transmitted to object storage (request body only) |
object_store_http_bytes_rx |
Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes received from object storage (response body only) |
For a complete list of the metrics used by Hydrolix, including Prometheus, RabbitMQ, and others, see All Metrics.