Grafana Automatic

Goal

Enable a working Grafana instance in your Hydrolix cluster using the automatically configured Grafana service.

Overview of steps

1. Modify the Hydrolix configuration
2. Obtain your Grafana administrative password
3. View Grafana interface
4. Use the Hydrolix data source for queries

Before you begin

• Make sure you're running Hydrolix version 5.1 or greater.
• Have kubectl and/or k9s configured for your Hydrolix instance.
• Ensure you have at least 6 GB memory and 6 CPUs available in your Kubernetes cluster.

Installation

Modify the Hydrolix configuration

Edit your Hydrolix cluster's configuration to include the Grafana in-cluster integration. Add the following lines to the spec: section of your Hydrolix configuration:

data_visualization_tools:
  - grafana

Example hydrolixcluster.yaml configuration:

spec:
  acme_enabled: true
  admin_email: admin@mycompany.com
  data_visualization_tools:
    - grafana
  db_bucket_endpoint: https://us-sea-1.linodeobjects.com
  db_bucket_region: us-sea-1

Load the configuration changes into your Hydrolix cluster with kubectl apply -f hydrolixcluster.yaml or by saving your changes in k9s.

Wait for the Grafana pod to start

Wait for the Grafana Kubernetes pod to start. This can take several minutes. You can tell when it's ready for use by spotting the Running indicator in the output of the kubectl command.

kubectl get pods | grep -E 'grafana'

grafana-864958596c-wnv6x  2/2  Running  0  8m37s

Verify the web interface

Make sure you're logged into your Hydrolix cluster's UI by visiting https://{hostname}.hydrolix.live in a web browser.

Visit https://{hostname}.hydrolix.live/grafana in your web browser to see Grafana's UI. You should be directed to a login page.

If the UI isn't available after a few minutes, see the troubleshooting section below.

Obtain and use your Grafana administrative password

The built-in Grafana instance comes pre-configured with an automatically generated administrative username and password.

• Using k9s

• Open :secrets in k9s.

• Select the general secret and press x to decode its values.

• Locate the value of GRAFANA_ADMIN_PASSWORD.

• Using kubectl

  kubectl get secret general -o jsonpath="{.data.GRAFANA_ADMIN_PASSWORD}" | base64 --decode
  

View the Grafana interface

1. Visit https://{hostname}.hydrolix.live/grafana in your web browser.

2. Use the admin username and the password you decoded in the last step.

Use the Hydrolix data source for queries

As of v5.5.0, in-cluster Grafana instances automatically deploy with the Hydrolix data source plugin installed. This data source is viewable at https://{hostname}.hydrolix.live/grafana/connections/datasources with the name Hydrolix - Local.

Use the Hydrolix data source when querying and configuring dashboards to take advantage of the features in the Hydrolix data source plugin.

To verify whether the installed plugin is the latest version:

1. Navigate to Administration > Plugins and data > Plugins and select the Hydrolix data source plugin.
2. Click Upgrade in the top-right corner if the option is available.

Verification

Verify metrics are queryable from Grafana by running a test query and optionally creating a dashboard.

1. Visit http://{hostname}.hydrolix.live/grafana/dashboard in your web browser.
2. Select New > New dashboard.
3. Select + Add visualization.
4. Select Hydrolix - Local from the data source dropdown.

5. Paste the following query into the query input box:

   SELECT
     count(*) as error_count,
     toStartOfMinute(timestamp) AS minute,
     container
   FROM
     hydro.logs
   WHERE
     timestamp > NOW() - INTERVAL 1 DAY
     AND level = 'error'
   GROUP BY
     minute, container
   ORDER BY
     minute ASC
   

6. Change Query Type to TimeSeries.

7. In the right-hand panel, change No value to 0.

   You should now have a timeseries visualization showing the count of errors by container for your Hydrolix cluster.

   

8. Select Save dashboard if you want to save this visualization in a new dashboard.

Troubleshoot

Not enough resources

If the grafana-* pod isn't starting and the kubectl or k9s utility is showing the pod in a pending state, check the resources on the nodes backing your Hydrolix installation. It's possible to lower the amount of memory so Grafana (and its associated renderer service) doesn't request as much from Kubernetes. More information about scaling pods and services can be found in Scale your Cluster. Here's a sample of a low-overhead configuration in the Hydrolix spec file:

spec:
  scale:
    grafana:
      cpu: 2
      memory: 2Gi
    renderer:
      cpu: 1
      memory: 1Gi

Postgres not available

Grafana uses the same Postgres instance as the Hydrolix catalog database, storing Grafana configuration data in a separate database called grafana. This is where Grafana dashboard definitions, users, alerts, and query history are stored.

The configuration provided by Hydrolix assumes that this Postgres instance is only available through SSL connections. If you're using an insecure configuration of Postgres, the built-in Grafana instance will refuse to connect.

If you're unable to use an SSL-enabled Postgres, perhaps for development or demo purposes, configure Grafana to make non-SSL connections with the kubectl utility.

Set pg_ssl_mode to disable:

kubectl edit hdx

Add the following line to the Hydrolix spec:

pg_ssl_mode: disable

Save the file, then disable Grafana and re-enable it by removing and then re-adding these lines to the Hydrolix spec:

data_visualization_tools:
  - grafana

Next steps

See the Hydrolix Data Source Plugin for Grafana for more details on configuring the data source plugin and a list of plugin features.