Stream metrics overview

Prometheus in Hydrolix

The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information.

You can query, view, and actively monitor this information using a stack's Grafana instance, or you can access it with your own monitoring platform. See Prometheus Integration for more information about setting up Prometheus with an external server.

Use the Prometheus UI

Prometheus has its own web-based UI.

This view is a basic metric view, suitable for quickly entering queries and seeing simple, graphed results. This feature is available in Hydrolix without any additional setup.

Navigate to https://<YourHostname>/prometheus to view dashboards.

For more information about Prometheus metric types, refer to the Prometheus documentation.

Use multiple components with a metric

If more than one component uses a given metric, querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query.

For example, "process_open_fds{service="stream-peer"}".

For a list of all Hydrolix metrics, see All Metrics.

Each ingest method's peers has multiple containers. One container runs message acquisition, and the other is the indexer which indexes and completes enrichment jobs.

HTTP stream ingest metrics

These metrics are specific to the use of streaming data sources.

Traefik metrics

Metric Name	Type	Components	Purpose
`traefik_service_requests_total`	Counter	Traefik	HTTP Traefik request information.
`traefik_service_request_duration_seconds_count/sum/bucket`	Counter	Traefik	Response time of traefik to client.
`http_source_request_duration_ns_count/sum/bucket`	Counter	Traefik	Response time from Stream-Head.

Intake-head metrics

Intake-head replaces the older Stream-head/Stream-peer architecture.

Metric	Type	Components	Purpose
`hdx_sink_backlog_bytes_count`	Gauge	Intake head	Total bytes of all partition buckets in the sink backlog waiting to be indexed (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_items_count`	Gauge	Intake head	Total count of partition buckets in the backlog (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_dropped_bytes_count`	Counter	Intake head	Total bytes of partition buckets dropped due to backlog overflow (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_dropped_items_count`	Counter	Intake head	Count of partition buckets dropped due to backlog overflow (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_delivery_count`	Counter	Intake head	Count of backlog buckets successfully handed off to indexing (requires `intake_head_index_backlog_enabled = true`).
`hdx_sink_backlog_trim_duration_ns`	Histogram	Intake head	Time to trim the backlog in nanoseconds (requires `intake_head_index_backlog_enabled = true`).
`http_source_outstanding_reqs`	Gauge	Intake head	Number of outstanding ingest event requests.
`http_source_byte_count`	Counter	Stream head	Count of bytes processed.
`http_source_request_count`	Counter	Stream head	Count of HTTP requests.
`http_source_request_duration_ns`	Histogram	Stream head	Histogram of HTTP request durations in nanoseconds.
`http_source_request_error_count`	Counter	Stream head	Count of HTTP request failures.
`http_source_row_count`	Counter	Stream head	Count of rows processed.
`http_source_value_count`	Counter	Stream head	Count of values processed.

Stream-head metrics

Stream-heads coordinate streaming jobs.

Metric Name	Type	Components	Purpose
`http_source_byte_count`	Counter	Stream head	Count of bytes processed.
`http_source_request_count`	Counter	Stream head	Count of http requests.
`http_source_request_error_count`	Counter	Stream head	Count of http request failures.
`http_source_row_count`	Counter	Stream head	Count of rows processed.
`http_source_value_count`	Counter	Stream head	Count of values processed.
`http_source_request_duration_ns_count/bucket/sum`	Histogram	Stream head	A histogram of HTTP request durations in nanoseconds.

Redpanda metrics

Metric Name	Type	Components	Purpose
`internal_event_queue_byte_count{mode="sink"}`	Counter	Stream Head	Byte count sent to Redpanda
`internal_event_queue_row_count{mode="sink""}`	Counter	Stream Head	Row count sent to Redpanda
`internal_event_queue_byte_count{mode="sink""}`	Counter	Stream Head	Byte count sent to Redpanda
`internal_event_queue_row_count{mode="source"}`	Counter	Stream Summary	Row count received from Redpanda
`internal_event_queue_value_count{mode="source"}`	Counter	Stream Summary	Value count recieved from Redpanda

Stream-peer metrics

Stream-peers carry out streaming jobs.

Metric Name	Type	Components	Purpose
`query_latency_summary_count/sum`	Count/Sum	Summary Peer	Latency of calls to catalog.
`query_count`	Counter	Summary Peer	Count of calls to the catalog.
`query_failure`	Counter	Summary Peer	Count of failed catalog calls.
`query_latency_summary`	Counter	Summary Peer	Latency of calls to catalog.

Stream ingest also produces Indexer metrics.

Kafka ingest metrics

Metric	Type	Components	Purpose
`kafka_source_byte_count`	Counter	Stream peer	Count of bytes read from Kafka.
`kafka_source_commit_duration_ns`	Histogram	Stream peer	Kafka commit duration, in nanoseconds.
`kafka_source_read_count`	Counter	Stream peer	Count of Kafka reads.
`kafka_source_read_duration_ns`	Histogram	Stream peer	Kafka read duration, in nanoseconds.
`kafka_source_read_error_count`	Counter	Stream peer	Count of Kafka errors.
`kafka_source_row_count`	Counter	Stream peer	Count of rows processed from Kafka.
`kafka_source_value_count`	Counter	Stream peer	Count of values processed from Kafka.

Kafka ingest also produces Indexer metrics.

AWS Kinesis ingest metrics

Metric	Type	Components	Purpose
`kinesis_source_byte_count`	Counter	Stream peer	Count of bytes read from Kinesis.
`kinesis_source_checkpoint_count`	Counter	Stream peer	Count of Kinesis checkpoint operations.
`kinesis_source_checkpoint_duration_ns`	Histogram	Stream peer	Duration of Kinesis checkpoint operations, in nanoseconds.
`kinesis_source_checkpoint_error_count`	Counter	Stream peer	Count of Kinesis checkpoint operation errors.
`kinesis_source_error_count`	Counter	Stream peer	Count of errors reading from Kinesis.
`kinesis_source_lag_ms`	Gauge	Stream peer	Measure of lag in Kinesis, in milliseconds.
`kinesis_source_operation_count`	Counter	Stream peer	Count of Kinesis operations.
`kinesis_source_operation_duration_ns`	Histogram	Stream peer	Histogram of Kinesis operation durations, in nanoseconds.
`kinesis_source_record_count`	Counter	Stream peer	Count of records read from Kinesis.
`kinesis_source_row_count`	Counter	Stream peer	Count of rows read from Kinesis.
`kinesis_source_value_count`	Counter	Stream peer	Count of values read from Kinesis.

Kinesis ingest also produces Indexer metrics.

Storage

Each of the object_store* metrics has these labels:

provider - Object storage provider. For example, AWS, Azure, GCS.
code - HTTP response code.
method - HTTP method used. For example POST and GET.
host - (optional) HTTP host used to target object storage.

Metric	Type	Components	Purpose
`net_http_status_code_bucket`	Counter	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	HTTP status code histogram count from storage.
`object_store_http_histo`	Histogram	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A histogram of object storage interaction latencies
`object_store_http_summary`	Summary	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A summary of object storage interaction latencies
`object_store_http_status_code_count`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of successful HTTP requests against object storage (replaces `net_http_status_code_count`). Requests resulting in `500` are still considered successful.
`object_store_http_error_count`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of HTTP request errors. For example, timeouts and connection errors.
`object_store_http_bytes_tx`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of bytes transmitted to object storage (request body only)
`object_store_http_bytes_rx`	Count	Batch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake Head	A count of bytes received from object storage (response body only)

For a complete list of the metrics used by Hydrolix, including Prometheus, RabbitMQ, and others, see All Metrics.