Stream Metrics
Information about stream metrics used by Hydrolix
Stream metrics overview
Prometheus in Hydrolix
The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information.
You can query, view, and actively monitor this information using a stack's Grafana instance, or you can access it with your own monitoring platform. See Prometheus Integration for more information about setting up Prometheus with an external server.
Use the Prometheus UI
Prometheus has its own web-based UI.
This view is a basic metric view, suitable for quickly entering queries and seeing simple, graphed results. This feature is available in Hydrolix without any additional setup.
Navigate to https://<YourHostname>/prometheus
to view dashboards.
For more information about Prometheus metric types, refer to the Prometheus documentation.
Use multiple components with a metric
If more than one component uses a given metric, querying it will return results from all relevant components. You can restrict results to a specific component by adding a service
keyword to your query.
For example, "process_open_fds{service="stream-peer"}
".
For a list of all Hydrolix metrics, see All Metrics.
Each ingest method's peers
has multiple containers. One container runs message acquisition, and the other is the indexer
which indexes and completes enrichment jobs.
HTTP stream ingest metrics
These metrics are specific to the use of streaming data sources.
Traefik metrics
Metric Name | Type | Components | Purpose |
---|---|---|---|
traefik_service_requests_total | Counter | Traefik | HTTP Traefik request information. |
traefik_service_request_duration_seconds_count/sum/bucket | Counter | Traefik | Response time of traefik to client. |
http_source_request_duration_ns_count/sum/bucket | Counter | Traefik | Response time from Stream-Head. |
Intake-head metrics
Intake-head replaces the older Stream-head/Stream-peer architecture.
Metric | Type | Components | Purpose |
---|---|---|---|
hdx_sink_backlog_bytes_count | Gauge | Intake head | Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires intake_head_index_backlog_enabled = true ). |
hdx_sink_backlog_items_count | Gauge | Intake head | Total count of partition buckets in the backlog (requires intake_head_index_backlog_enabled = true ). |
hdx_sink_backlog_dropped_bytes_count | Counter | Intake head | Total bytes of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true ). |
hdx_sink_backlog_dropped_items_count | Counter | Intake head | Count of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true ). |
hdx_sink_backlog_delivery_count | Counter | Intake head | Count of backlog buckets successfully handed off to indexing (requires intake_head_index_backlog_enabled = true ). |
hdx_sink_backlog_trim_duration_ns | Histogram | Intake head | Time to trim the backlog in nanoseconds (requires intake_head_index_backlog_enabled = true ). |
http_source_outstanding_reqs | Gauge | Intake head | Number of outstanding ingest event requests. |
http_source_byte_count | Counter | Stream head | Count of bytes processed. |
http_source_request_count | Counter | Stream head | Count of HTTP requests. |
http_source_request_duration_ns | Histogram | Stream head | Histogram of HTTP request durations in nanoseconds. |
http_source_request_error_count | Counter | Stream head | Count of HTTP request failures. |
http_source_row_count | Counter | Stream head | Count of rows processed. |
http_source_value_count | Counter | Stream head | Count of values processed. |
Stream-head metrics
Stream-heads coordinate streaming jobs.
Metric Name | Type | Components | Purpose |
---|---|---|---|
http_source_byte_count | Counter | Stream head | Count of bytes processed. |
http_source_request_count | Counter | Stream head | Count of http requests. |
http_source_request_error_count | Counter | Stream head | Count of http request failures. |
http_source_row_count | Counter | Stream head | Count of rows processed. |
http_source_value_count | Counter | Stream head | Count of values processed. |
http_source_request_duration_ns_count/bucket/sum | Histogram | Stream head | A histogram of HTTP request durations in nanoseconds. |
Redpanda metrics
Metric Name | Type | Components | Purpose |
---|---|---|---|
internal_event_queue_byte_count{mode="sink"} | Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="sink""} | Counter | Stream Head | Row count sent to Redpanda |
internal_event_queue_byte_count{mode="sink""} | Counter | Stream Head | Byte count sent to Redpanda |
internal_event_queue_row_count{mode="source"} | Counter | Stream Summary | Row count received from Redpanda |
internal_event_queue_value_count{mode="source"} | Counter | Stream Summary | Value count recieved from Redpanda |
Stream-peer metrics
Stream-peers carry out streaming jobs.
Metric Name | Type | Components | Purpose |
---|---|---|---|
query_latency_summary_count/sum | Count/Sum | Summary Peer | Latency of calls to catalog. |
query_count | Counter | Summary Peer | Count of calls to the catalog. |
query_failure | Counter | Summary Peer | Count of failed catalog calls. |
query_latency_summary | Counter | Summary Peer | Latency of calls to catalog. |
Stream ingest also produces Indexer metrics.
Kafka ingest metrics
Metric | Type | Components | Purpose |
---|---|---|---|
kafka_source_byte_count | Counter | Stream peer | Count of bytes read from Kafka. |
kafka_source_commit_duration_ns | Histogram | Stream peer | Kafka commit duration, in nanoseconds. |
kafka_source_read_count | Counter | Stream peer | Count of Kafka reads. |
kafka_source_read_duration_ns | Histogram | Stream peer | Kafka read duration, in nanoseconds. |
kafka_source_read_error_count | Counter | Stream peer | Count of Kafka errors. |
kafka_source_row_count | Counter | Stream peer | Count of rows processed from Kafka. |
kafka_source_value_count | Counter | Stream peer | Count of values processed from Kafka. |
Kafka ingest also produces Indexer metrics.
AWS Kinesis ingest metrics
Metric | Type | Components | Purpose |
---|---|---|---|
kinesis_source_byte_count | Counter | Stream peer | Count of bytes read from Kinesis. |
kinesis_source_checkpoint_count | Counter | Stream peer | Count of Kinesis checkpoint operations. |
kinesis_source_checkpoint_duration_ns | Histogram | Stream peer | Duration of Kinesis checkpoint operations, in nanoseconds. |
kinesis_source_checkpoint_error_count | Counter | Stream peer | Count of Kinesis checkpoint operation errors. |
kinesis_source_error_count | Counter | Stream peer | Count of errors reading from Kinesis. |
kinesis_source_lag_ms | Gauge | Stream peer | Measure of lag in Kinesis, in milliseconds. |
kinesis_source_operation_count | Counter | Stream peer | Count of Kinesis operations. |
kinesis_source_operation_duration_ns | Histogram | Stream peer | Histogram of Kinesis operation durations, in nanoseconds. |
kinesis_source_record_count | Counter | Stream peer | Count of records read from Kinesis. |
kinesis_source_row_count | Counter | Stream peer | Count of rows read from Kinesis. |
kinesis_source_value_count | Counter | Stream peer | Count of values read from Kinesis. |
Kinesis ingest also produces Indexer metrics.
Storage
Each of the object_store*
metrics has these labels:
- Provider - Object storage provider. For example, AWS, Azure, GCS.
- Code - HTTP response code.
- Method - HTTP method used. For example
POST
andGET
. - Host - HTTP host used to target object storage.
Metric | Type | Components | Purpose |
---|---|---|---|
net_http_status_code_bucket | Counter | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | HTTP status code histogram count from storage. |
object_store_http_histo | Histogram | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A histogram of object storage interaction latencies |
object_store_http_summary | Summary | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A summary of object storage interaction latencies |
object_store_http_status_code_count | Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of successful HTTP requests against object storage (replaces net_http_status_code_count ). Requests resulting in 500 are still considered successful. |
object_store_http_error_count | Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of HTTP request errors. For example, timeouts and connection errors. |
object_store_http_bytes_tx | Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes transmitted to object storage (request body only) |
object_store_http_bytes_rx | Count | Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head | A count of bytes received from object storage (response body only) |
For a complete list of the metrics used by Hydrolix, including Prometheus, RabbitMQ, and others, see All Metrics.
Updated 27 days ago