Summary Metrics

The Hydrolix stack includes Prometheus, an open-source metrics database. While the Stack runs, Hydrolix continuously updates its Prometheus instance with metrics information.

Use Prometheus directly

Prometheus has its own web-based UI, available by visiting https://<YourHostname>/prometheus in your web browser.

This view is a basic metric view, suitable for quickly entering queries and seeing simple, graphed results. Hydrolix does make this feature available immediately, without any additional setup.

For more information about metric types, refer to Prometheus's documentation.

Hydrolix's metrics

If more than one component uses a given metric, then querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query, e.g. "process_open_fds{service="stream-peer"}".

For more information about metric types, refer to Prometheus's documentation.

Each of the Ingest method peers has multiple containers. One container that will do message acquisition and the other is the indexer which will index and complete enrichment jobs.

HTTP Stream Summary Ingest

These metrics are specific to the use of streaming data sources.

Traefik

MetricTypeComponentsPurpose
traefik_service_requests_totalCounterTraefikHTTP Traefik request information.
traefik_service_request_duration_seconds_count/sum/bucketCounterTraefikResponse time of traefik to client.
http_source_request_duration_ns_count/sum/bucketCounterTraefikResponse time from Stream-Head.

Stream Head and Intake Head

MetricTypeComponentsPurpose
http_source_byte_countCounterStream headCount of bytes processed.
http_source_request_countCounterStream headCount of http requests.
http_source_request_duration_ns_count/bucket/sumHistogramStream headA histogram of HTTP request durations in nanoseconds.
http_source_request_error_countCounterStream headCount of http request failures.
http_source_row_countCounterStream headCount of rows processed.
http_source_value_countCounterStream headCount of values processed.

Redpanda

MetricTypeComponentsPurpose
internal_event_queue_byte_count{mode="sink"}CounterStream HeadByte Count sent to Redpanda
internal_event_queue_row_count{mode="sink""}CounterStream HeadRow Count sent to Redpanda
internal_event_queue_byte_count{mode="sink""}CounterStream HeadByte Count sent to Redpanda
internal_event_queue_row_count{mode="source"}CounterStream SummaryRow Count received from Redpanda
internal_event_queue_value_count{mode="source"}CounterStream SummaryValue Count recieved from Redpanda

Stream Summary Metrics

MetricTypeComponentsPurpose
query_countCounterSummary PeerCount of calls to the Catalog.
query_failureCounterSummary PeerCount of failed Catalog calls.
query_latency_summaryCounterSummary PeerLatency of calls to catalog.
query_latency_summary_count/sumCount/SumSummary PeerLatency of calls to catalog.

Indexer Metrics

Indexer metrics are metrics that cover the indexing and enrichment of data being ingested

MetricTypeComponentsPurpose
indexer_rows_written_count/bucket/sumHistogramBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadTotal rows indexed (written to partitions)
indexer_bytes_written_count/bucket/sumHistogramBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadTotal bytes indexed (written to partitions)
indexer_partitions_rejected_count/bucket/sumHistogramBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadHistogram of partitions not able to written. If value is 0=raw data parsing failed, 1=raw data / transform schema mismatch, 3=Error writing partition file, 4= Other Error during indexing
indexer_partitions_written_count/bucket/sumHistogramBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadTotal partitions created
indexer_partition_write_seconds_count/bucket/sumHistogramBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadTime from receiving indexing query to writing partition file (seconds)
hdx_sink_row_countCounterBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadCount of rows processed by the indexer and uploaded to storage. Includes Hot and Cold reporting.
hdx_sink_byte_countCounterBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadCount of bytes processed by the indexer and uploaded to storage. Includes Hot and Cold reporting.
hdx_sink_value_countCounterBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadCount of values processed by the indexer and uploaded to storage. Includes Hot and Cold reporting.
hdx_sink_error_countCounterBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadCount of errors in indexing and uploading to storage.

Storage

Cloud/Object Storage metrics.

Each of the below object_store* metrics has these labels:

  • provider - Object storage provider (AWS, Azure, GCS)
  • code - HTTP Response code
  • method - HTTP Method used (POST, GET, etc)
  • host - HTTP Host used to target object storage.
MetricTypeComponentsPurpose
net_http_status_code_bucketCounterBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadHTTP Status Code histogram count from Storage.
object_store_http_histoHistogramBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA histogram of object storage interaction latencies
object_store_http_summarySummaryBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA summary of object storage interaction latencies
object_store_http_status_code_countCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of successful HTTP requests against object storage (replaces net_http_status_code_count). Requests resulting in 500 are still considered successful.
object_store_http_error_countCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of HTTP request errors (timeouts, connection errors, etc.)
object_store_http_bytes_txCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of bytes transmitted to object storage (request body only)
object_store_http_bytes_rxCountBatch (inc. Autoingest), Kafka, Kinesis, Stream HTTP, Intake HeadA count of bytes received from object storage (response body only)

Additional Metrics

Additional Metrics are provided for the management and control of different components.

General Metrics.

These metrics track various counters and statistics regarding data ingestion.

MetricTypePurpose
process_cpu_seconds_totalCounterTotal user and system CPU time spent in seconds.
process_max_fdsGaugeMaximum number of open file descriptors.
process_open_fdsGaugeNumber of open file descriptors.
process_resident_memory_bytesGaugeResident memory size in bytes.
process_start_time_secondsGaugeStart time of the process since unix epoch in seconds.
process_virtual_memory_bytesGaugeVirtual memory size in bytes.
process_virtual_memory_max_bytesGaugeMaximum amount of virtual memory available in bytes.
promhttp_metric_handler_requests_in_flightGaugeCurrent number of scrapes being served.
promhttp_metric_handler_requests_totalCounterTotal number of scrapes by HTTP status code.

Go environment metrics

These metrics track resources used by Hydrolix's Go environments.

MetricTypePurpose
go_gc_duration_secondsSummaryA summary of the pause duration of garbage collection cycles.
go_goroutinesGaugeNumber of goroutines that currently exist.
go_infoGaugeInformation about the Go environment.
go_memstats_alloc_bytesGaugeNumber of bytes allocated and still in use.
go_memstats_alloc_bytes_totalCounterTotal number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytesGaugeNumber of bytes used by the profiling bucket hash table.
go_memstats_frees_totalCounterTotal number of frees.
go_memstats_gc_cpu_fractionGaugeThe fraction of this program's available CPU time used by the GC since the program started.
go_memstats_gc_sys_bytesGaugeNumber of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytesGaugeNumber of heap bytes allocated and still in use.
go_memstats_heap_idle_bytesGaugeNumber of heap bytes waiting to be used.
go_memstats_heap_inuse_bytesGaugeNumber of heap bytes that are in use.
go_memstats_heap_objectsGaugeNumber of allocated objects.
go_memstats_heap_released_bytesGaugeNumber of heap bytes released to OS.
go_memstats_heap_sys_bytesGaugeNumber of heap bytes obtained from system.
go_memstats_last_gc_time_secondsGaugeNumber of seconds since 1970 of last garbage collection.
go_memstats_lookups_totalCounterTotal number of pointer lookups.
go_memstats_mallocs_totalCounterTotal number of mallocs.
go_memstats_mcache_inuse_bytesGaugeNumber of bytes in use by mcache structures.
go_memstats_mcache_sys_bytesGaugeNumber of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytesGaugeNumber of bytes in use by mspan structures.
go_memstats_mspan_sys_bytesGaugeNumber of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytesGaugeNumber of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytesGaugeNumber of bytes used for other system allocations.
go_memstats_stack_inuse_bytesGaugeNumber of bytes in use by the stack allocator.
go_memstats_stack_sys_bytesGaugeNumber of bytes obtained from system for stack allocator.
go_memstats_sys_bytesGaugeNumber of bytes obtained from system.
go_threadsGaugeNumber of OS threads created.