The Hydrolix stack includes Prometheus, an open-source metrics database. Hydrolix continuously updates its Prometheus instance with metrics information. You can query, view, and actively monitor this information through the use of a stack's Grafana instance, or you can access it via your own monitoring platform.

Using Prometheus Directly

Prometheus has its own web-based UI, available by visiting https://YOUR-HYDROLIX-HOSTNAME.hydrolix.live/prometheus in your web browser.

This view is far more basic than Grafana's, suitable for quickly entering queries and seeing simple, graphed results. Hydrolix does make this feature available immediately, without any additional setup.

Using a Separate Prometheus Server

How It Works

Rather than using the built-in Prometheus server to display and report metrics, you can use an external Prometheus server. To do this, configure the internal server to forward metrics to your external server and enable remote writing on the external server.

This solution uses Prometheus' Remote Write Server functionality to link the two servers.

📘

Both servers are doing work

Even though this uses a separate external Prometheus server, Hydrolix's internal Prometheus server still uses memory and CPU. It collects and aggregates metrics, then forwards the data to your external Prometheus server for query.

Steps

1. Tell Hydrolix to send data to the external Prometheus server. Include this line in the spec section of your hydrolixcluster.yaml file. This example assumes your external Prometheus server is running at the default port 9090, and that firewalls allow traffic on that port:

   YAML

   spec:
   ...
     prometheus_remote_write_url: http://<prometheus server hostname>:9090/api/v1/write
   

2. Run the external Prometheus sever with the --web.enable-remote-write-receiver switch.

3. If the external Prometheus installation uses basic auth, set the username in your hydrolixcluster.yaml file and set the password in a curated secret.

   Edit the hydrolixcluster.yaml file to add one line:

   YAML

   spec:
   ...
     prometheus_remote_write_username: <username>
   

   Apply this change to the Hydrolix cluster:

   Shell

   kubectl -f hydrolixcluster.yaml apply
   

   Create a file named prom-secret.yaml with these contents:

   YAML

   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: curated
     namespace: $HDX_KUBERNETES_NAMESPACE
   stringData:
     PROMETHEUS_REMOTE_WRITE_PASSWORD: <password>
   type: Opaque
   

   Finally, use the Kubernetes command line tool (kubectl) to interpolate the $HDX_KUBERNETES_NAMESPACE variable and apply the generated secret to your Kubernetes cluster:

   Shell

   eval "echo \"$(cat prom-secret.yaml)\"" > secrets.yaml
   kubectl apply -f prom-secret.yaml
   

   
   

   📘

   Not the same as Prometheus Remote Read/Write

   Hydrolix can also serve as the database for a Prometheus installation, providing longer retention and cost savings for large volumes of data. The settings for that feature are very similarly named to these settings, and can be easily confused.

Hydrolix's Metrics

This table lists the metrics available, and which components update them.

If more than one component uses a given metric, then querying it will return results from all relevant components. You can restrict results to a specific component by adding a service keyword to your query, e.g. "process_open_fds{service="stream-peer"}".

For more information about metric types, refer to Prometheus's documentation.

General Metrics

These metrics track various counters and statistics regarding data ingestion.

Query Metrics

These metrics track activity specific to batch ingestions.

Batch Metrics

These metrics track activity specific to batch ingestion.

Merge Metrics

These metrics correspond to Hydrolix's merge service.

Rabbit MQ

https://www.rabbitmq.com/prometheus.html

Streaming Metrics

HTTP Stream Ingest

These metrics are specific to the use of streaming data sources.

Redpanda

https://docs.redpanda.com/docs/cluster-administration/monitoring/

Kafka Ingest

These metrics are specific to the use of Kafka data sources.

DNS Metrics

These metrics are specific to the use of Kafka data sources.

Go Environment Metrics

These metrics track resources used by Hydrolix's Go environments.

PostgreSQL Pool Metrics