Prometheus Remote Read and Write

Overview

The Hydrolix stack includes Prometheus, and continuously updates that instance with metrics information.

By default, Prometheus keeps about 15 days of metrics, or hot samples, to keep its storage small, fast, and cost-effective.

Use Prometheus remote read/write with Hydrolix as a receiver to store metrics without touching Prometheus disk or RAM resources. This is an efficient way to store metrics for long periods without storage overhead through Prometheus.

Remote write

Remote write sends compressed data to a receiver using HTTP/1.1. Hydrolix acts as the receiver for the streamed data, and can hold months or years of metrics in long-term storage.

Remote read

Remote read requests metrics from Hydrolix for data older than the Prometheus retention window. The Prometheus server stitches together local data and and remote results fetched from Hydrolix before returning the query data.

Set Hydrolix as the Prometheus ingestion point

Configure Prometheus to use Hydrolix as a custom metrics endpoint.

Edit the Prometheus config YAML file to enable remote read and write to Hydrolix.

   remote_write:
- url: 'http://stream-head:8089/ingest/event'       # Hydrolix’s ingest endpoint
  headers:
    x-hdx-table: 'prom.prom'                        # Full table name: <project>.<table>

remote_read:
- url: 'http://query-head:8088/prom_read'           # Hydrolix’s Prometheus query endpoint
  read_recent: true                                 # Include recent data from Prometheus memory (not just long-term)
  headers:
    x-hdx-project-name: 'prom'                      # Hydrolix project name
    x-hdx-table-name: 'prom'                        # Hydrolix table name within that project

Test the remote write connection

Testing that the remote read/write is working can be tricky, as the process is usually asynchronous.

Send a test metric from Prometheus

1. Open the Prometheus UI - https://{myhost}.hydrolix.live/prometheus/graph.
2. Run this query to confirm an active time series is returned: prometheus_engine_query_duration_seconds.

Test the remote read connection

1. Edit the Prometheus config file.

2. Add the following code to the config file:

   YAML

   remote_read:
     - url: 'http://{myhost}.hydrolix.live:8088/prom_read'
       read_recent: true
       headers:
         x-hdx-project-name: 'prom'
         x-hdx-table-name: 'prom'
   
   

3. Restart Prometheus to apply the change.

4. Query the metric.

   bash

   curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=test_metric'
   

   The query should return results from either memory or Hydrolix. The value read_recent: true includes both.