Merge Metrics

Merge metrics⚓︎

Metric Name	Type	Components	Purpose
`merge_success` (deprecated)	Counter	Merge peer	Count of merge successes.
`merge_failure` (deprecated)	Counter	Merge peer	Count of merge failures.
`query_count` (deprecated)	Counter	Merge peer	Count of Merge calls to the catalog.
`query_failure` (deprecated)	Counter	Merge peer	Count of failed Merge catalog calls.
`query_latency_summary` (deprecated)	Summary	Merge peer	Latency of calls to the catalog.
`merge_duration_summary` (deprecated)	Summary	Merge peer	Merge processing duration, in milliseconds.

Merge controller⚓︎

Metric Name	Type	Components	Description	Labels
`partition_distribution`	Histogram	Merge Controller	Histogram of partition memory size.	`project_id` `project_name` `table_id` `table_name` `target`
`segments`	Gauge	Merge Controller	Current count of known segments. A segment is a group of buckets that share a merge target.	`project_id` `project_name` `table_id` `table_name` `target`
`active_merges`	Gauge	Merge Controller	Current count of active merge operations.	`project_id` `project_name` `table_id` `table_name` `target`
`partitions`	Gauge	Merge Controller	Current count of partitions awaiting segmentation.	`project_id` `project_name` `table_id` `table_name` `target`
`partitions_dispatched`	Counter	Merge Controller	Count of partitions dispatched for merging. Determine "partitions dispatched for merge per second" by combining with rate function.	`project_id` `project_name` `table_id` `table_name` `target`
`candidates`	Gauge	Merge Controller	Current count of constructed candidates waiting dispatch.	`project_id` `project_name` `table_id` `table_name` `target`
`candidates_dispatched`	Counter	Merge Controller	Count of candidates dispatched for merging. Determine "candidates dispatched for merge per second" by combining with rate function.	`project_id` `project_name` `table_id` `table_name` `target`
`partitions_per_candidate`	Histogram	Merge Controller	Distribution of partitions within a dispatched candidate.	`project_id` `project_name` `table_id` `table_name` `target`
`candidate_mem_size`	Histogram	Merge Controller	Distribution of calculated memory requirements for dispatched candidates.	`project_id` `project_name` `table_id` `table_name` `target`
`duplicate_partitions`	Counter	Merge Controller	Count of partitions sourced that are already being tracked.	`project_id` `project_name` `table_id` `table_name` `target`
`memory_coefficient`	Gauge	Merge Controller	Current value of the memory coefficient being applied to sourced partitions.	`project_id` `project_name` `table_id` `table_name` `target`
`expired_segments`	Counter	Merge Controller	Count of segments expired for moving out of the target range.	`project_id` `project_name` `table_id` `table_name` `target`
`bucket_duration`	Histogram	Merge Controller	Distribution of time in milliseconds of a bucket's age upon closing. The `basis` label gives the reason for closing: `full` = the bucket reached max size, `idle_ttl` = the bucket wasn't added to within the idle timeout, `age_ttl` = the bucket was open longer than the max age, `segment_ttl` = the bucket was part of a segment that fell out of the target range.	`project_id` `project_name` `table_id` `table_name` `target` `basis`
`connected_clients`	Gauge	Merge Controller	Count of currently connected merge-peers.	`pool_id`
`query_latency`	Histogram	Merge Controller	Distribution of response times in milliseconds for catalog operations.	`project_id` `table_id` `method`
`admin_query_latency`	Histogram	Merge Controller	Distribution of response times in milliseconds for admin catalog operations.	`project_id` `table_id` `method`
`state_duration`	Histogram	Merge Controller	Distribution of time in milliseconds spent in each reactor state.	`project_id` `table_id` `state`
`merge_duty_cycle`	Gauge	Merge Controller	Overall work rate of the merge-controller, as a value between 0 and 1.	`project_id` `project_name` `table_id` `table_name` `target`
`row_weighted_efficiency`	Gauge	Merge Controller	Merge target efficiency as a value between `0.0` and `1.0`, where a value near `1.0` means the target's data sits in close to the fewest partitions its size allows. Efficiency is calculated for each hour, storage location, and shard key group, then combined as the sum of ideal partition counts divided by the sum of actual partition counts, so the groups holding the most partitions have the most influence. Despite the name, the weighting is by partition count rather than by row count.	`project_id` `project_name` `table_id` `table_name` `target`
`mem_weighted_efficiency`	Gauge	Merge Controller	Merge target efficiency as a value between `0.0` and `1.0`, weighted by memory size. Each hour, storage location, and shard key group contributes in proportion to the bytes it holds, so a small badly packed group has little influence on the result.	`project_id` `project_name` `table_id` `table_name` `target`
`harmonic_efficiency`	Gauge	Merge Controller	Merge target efficiency as a value between `0.0` and `1.0`, calculated as the harmonic mean of the per group efficiencies. Every group counts equally and low values weigh heavily, so a single badly merged group visibly lowers the result.	`project_id` `project_name` `table_id` `table_name` `target`
`ideal_partition_count`	Gauge	Merge Controller	Given the current required memory for all partitions in the target, what's the ideal number of partitions if each partition was at the target size.	`project_id` `project_name` `table_id` `table_name` `target`
`actual_partition_count`	Gauge	Merge Controller	Current count of partitions in the target.	`project_id` `project_name` `table_id` `table_name` `target`

Catalog⚓︎

For more information about merge operations in your tables, see Catalog Metadata.

For a complete list of the metrics used by Hydrolix, including Prometheus, RabbitMQ, and others, see All Metrics.

Prometheus in Hydrolix⚓︎

The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information.

You can query, view, and actively monitor this information using a stack's Grafana instance, or you can access it with your own monitoring platform. See Prometheus Integration for more information about setting up Prometheus with an external server.

Use the Prometheus UI⚓︎

Prometheus has its own web-based UI.

This view is a basic metric view, suitable for quickly entering queries and seeing simple, graphed results. This feature is available in Hydrolix without any additional setup.

Navigate to https://hostname.hydrolix.live/prometheus to view dashboards.

For more information about Prometheus metric types, refer to the Prometheus documentation.

Pass additional options to Prometheus⚓︎

Use a list of strings in the prometheus_extra_args tunable to append options after default flags set by Hydrolix.

Set a Custom TSDB Block Duration

spec:
  prometheus_extra_args:
  - --storage.tsdb.max-block-duration=2h

For a list of available flags, see the Prometheus CLI reference, or see Prometheus storage.

Prometheus option resolution⚓︎

Prometheus settings come from and are resolved in the following order from lowest to highest priority:

Prometheus: Built-in defaults
Hydrolix: Default settings for Prometheus
User-specified: Using prometheus_extra_args (highest precedence)

See https://hostname.hydrolix.live/prometheus/config for current Prometheus settings.

If a user-configured flag conflicts with a built-in Prometheus default, the value in prometheus_extra_args overrides the default one. This results in the following resolution by flag type:

Scalar flags accept a single value. If a scalar flag in prometheus_extra_args conflicts with a Prometheus built in or Hydrolix-specified default, the value in prometheus_extra_args overrides it.
Cumulative flags accept multiple values. If a flag in prometheus_extra_args matches a built-in cumulative flag (for example, --enable-feature), the values are appended rather than replaced.