Stream Metrics

Information about stream metrics used by Hydrolix

Stream metrics overview

Prometheus in Hydrolix

The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information.

You can query, view, and actively monitor this information using a stack's Grafana instance, or you can access it with your own monitoring platform. See Prometheus Integration for more information about setting up Prometheus with an external server.


Use the Prometheus UI

Prometheus has its own web-based UI.

This view is a basic metric view, suitable for quickly entering queries and seeing simple, graphed results. This feature is available in Hydrolix without any additional setup.

Navigate to https://<YourHostname>/prometheus to view dashboards.

For more information about Prometheus metric types, refer to the Prometheus documentation.


Use multiple components with a metric

If more than one component uses a given metric, querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query.

For example, "process_open_fds{service="stream-peer"}".

For a list of all Hydrolix metrics, see All Metrics.

Each ingest method's peers has multiple containers. One container runs message acquisition, and the other is the indexer which indexes and completes enrichment jobs.

HTTP stream ingest metrics

These metrics are specific to the use of streaming data sources.

Traefik metrics

Metric NameTypeComponentsPurpose
traefik_service_requests_totalCounterTraefikHTTP Traefik request information.
traefik_service_request_duration_seconds_count/sum/bucketCounterTraefikResponse time of traefik to client.
http_source_request_duration_ns_count/sum/bucketCounterTraefikResponse time from Stream-Head.

Intake-head metrics

Intake-head replaces the older Stream-head/Stream-peer architecture.

MetricTypeComponentsPurpose
hdx_sink_backlog_bytes_countGaugeIntake headTotal bytes of all partition buckets in the sink backlog waiting to be indexed (requires intake_head_index_backlog_enabled = true).
hdx_sink_backlog_items_countGaugeIntake headTotal count of partition buckets in the backlog (requires intake_head_index_backlog_enabled = true).
hdx_sink_backlog_dropped_bytes_countCounterIntake headTotal bytes of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true).
hdx_sink_backlog_dropped_items_countCounterIntake headCount of partition buckets dropped due to backlog overflow (requires intake_head_index_backlog_enabled = true).
hdx_sink_backlog_delivery_countCounterIntake headCount of backlog buckets successfully handed off to indexing (requires intake_head_index_backlog_enabled = true).
hdx_sink_backlog_trim_duration_nsHistogramIntake headTime to trim the backlog in nanoseconds (requires intake_head_index_backlog_enabled = true).
http_source_outstanding_reqsGaugeIntake headNumber of outstanding ingest event requests.
http_source_byte_countCounterStream headCount of bytes processed.
http_source_request_countCounterStream headCount of HTTP requests.
http_source_request_duration_nsHistogramStream headHistogram of HTTP request durations in nanoseconds.
http_source_request_error_countCounterStream headCount of HTTP request failures.
http_source_row_countCounterStream headCount of rows processed.
http_source_value_countCounterStream headCount of values processed.

Stream-head metrics

Stream-heads coordinate streaming jobs.

Metric NameTypeComponentsPurpose
http_source_byte_countCounterStream headCount of bytes processed.
http_source_request_countCounterStream headCount of http requests.
http_source_request_error_countCounterStream headCount of http request failures.
http_source_row_countCounterStream headCount of rows processed.
http_source_value_countCounterStream headCount of values processed.
http_source_request_duration_ns_count/bucket/sumHistogramStream headA histogram of HTTP request durations in nanoseconds.

Redpanda metrics

Metric NameTypeComponentsPurpose
internal_event_queue_byte_count{mode="sink"}CounterStream HeadByte count sent to Redpanda
internal_event_queue_row_count{mode="sink""}CounterStream HeadRow count sent to Redpanda
internal_event_queue_byte_count{mode="sink""}CounterStream HeadByte count sent to Redpanda
internal_event_queue_row_count{mode="source"}CounterStream SummaryRow count received from Redpanda
internal_event_queue_value_count{mode="source"}CounterStream SummaryValue count recieved from Redpanda

Stream-peer metrics

Stream-peers carry out streaming jobs.

Metric NameTypeComponentsPurpose
query_latency_summary_count/sumCount/SumSummary PeerLatency of calls to catalog.
query_countCounterSummary PeerCount of calls to the catalog.
query_failureCounterSummary PeerCount of failed catalog calls.
query_latency_summaryCounterSummary PeerLatency of calls to catalog.

Stream ingest also produces Indexer metrics.

Kafka ingest metrics

MetricTypeComponentsPurpose
kafka_source_byte_countCounterStream peerCount of bytes read from Kafka.
kafka_source_commit_duration_nsHistogramStream peerKafka commit duration, in nanoseconds.
kafka_source_read_countCounterStream peerCount of Kafka reads.
kafka_source_read_duration_nsHistogramStream peerKafka read duration, in nanoseconds.
kafka_source_read_error_countCounterStream peerCount of Kafka errors.
kafka_source_row_countCounterStream peerCount of rows processed from Kafka.
kafka_source_value_countCounterStream peerCount of values processed from Kafka.

Kafka ingest also produces Indexer metrics.

AWS Kinesis ingest metrics

MetricTypeComponentsPurpose
kinesis_source_byte_countCounterStream peerCount of bytes read from Kinesis.
kinesis_source_checkpoint_countCounterStream peerCount of Kinesis checkpoint operations.
kinesis_source_checkpoint_duration_nsHistogramStream peerDuration of Kinesis checkpoint operations, in nanoseconds.
kinesis_source_checkpoint_error_countCounterStream peerCount of Kinesis checkpoint operation errors.
kinesis_source_error_countCounterStream peerCount of errors reading from Kinesis.
kinesis_source_lag_msGaugeStream peerMeasure of lag in Kinesis, in milliseconds.
kinesis_source_operation_countCounterStream peerCount of Kinesis operations.
kinesis_source_operation_duration_nsHistogramStream peerHistogram of Kinesis operation durations, in nanoseconds.
kinesis_source_record_countCounterStream peerCount of records read from Kinesis.
kinesis_source_row_countCounterStream peerCount of rows read from Kinesis.
kinesis_source_value_countCounterStream peerCount of values read from Kinesis.

Kinesis ingest also produces Indexer metrics.

Storage

Each of the object_store* metrics has these labels:

  • Provider - Object storage provider. For example, AWS, Azure, GCS.
  • Code - HTTP response code.
  • Method - HTTP method used. For example POST and GET.
  • Host - HTTP host used to target object storage.
MetricTypeComponentsPurpose
net_http_status_code_bucketCounterBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadHTTP status code histogram count from storage.
object_store_http_histoHistogramBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA histogram of object storage interaction latencies
object_store_http_summarySummaryBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA summary of object storage interaction latencies
object_store_http_status_code_countCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful.
object_store_http_error_countCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of HTTP request errors. For example, timeouts and connection errors.
object_store_http_bytes_txCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of bytes transmitted to object storage (request body only)
object_store_http_bytes_rxCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of bytes received from object storage (response body only)

For a complete list of the metrics used by Hydrolix, including Prometheus, RabbitMQ, and others, see All Metrics.