Grafana Automatic Installation

Goal

Enable a working Grafana instance in your Hydrolix cluster using the automatically configured Grafana service.

Before you begin

Make sure you're running Hydrolix version 5.1 or greater
Have kubectl or k9s configured for your Hydrolix instance
Ensure you have at least 6 GB memory and 6 CPUs available in your Kubernetes cluster

Overview of steps

Installation

Modify the Hydrolix cluster configuration

Edit your Hydrolix cluster's configuration to include the Grafana in-cluster integration.

Add these two lines to the spec: section of your Hydrolix configuration
```
  data_visualization_tools:
    - grafana
```
For example, here's what it could look like in your hydrolixcluster.yaml file.
```
spec:
  acme_enabled: true
  admin_email: [email protected]
  data_visualization_tools:
    - grafana
  db_bucket_endpoint: https://us-sea-1.linodeobjects.com
  db_bucket_region: us-sea-1
```
Load the configuration changes into your Hydrolix cluster with kubectl apply -f hydrolixcluster.yaml or by saving your changes in k9s.
Wait for the Grafana Kubernetes pod to start. This can take several minutes. You can tell when it's ready for use by spotting the Running indicator in the output of the kubectl command.
```
% kubectl get pods | grep -E 'grafana'       

grafana-864958596c-wnv6x                 2/2     Running     0               8m37s  
```
Verify the web interface is working.
1. Make sure you're logged into your Hydrolix cluster's UI by visiting https://{myhost}.hydrolix.live in a web browser.
2. Visit https://{myhost}.hydrolix.live/grafana in your web browser to see Kibana's Web UI. You should be directed to a login page.

If the UI isn't available after a few minutes, see the troubleshooting section below.

Obtain and use your Grafana administrative password

The built-in Grafana instance comes pre-configured with an automatically generated administrative username and password.

Using k9s
1. K9s, in :secrets
2. General, decode with x
3. Find the value of GRAFANA_ADMIN_PASSWORD

Using kubectl

% kubectl get secret general -o jsonpath="{.data.GRAFANA_ADMIN_PASSWORD}" | base64 --decode

View the Grafana interface

Visit https://{myhost}.hydrolix.live/grafana in your web browser.
Use the admin username and the password you decoded in the last step.

Querying: Use the Hydrolix data source

As of v5.5.0, in-cluster Grafana instances automatically deploy with the Hydrolix data source plugin installed. This data source is viewable at https://{myhost}.hydrolix.live/grafana/connections/datasources with the name Hydrolix - Local.

Use the Hydrolix data source when querying and configuring dashboards to take advantage of the features in the Hydrolix data source plugin.

To verify whether the installed plugin is the latest version:

Navigate to Administration > Plugins and data > Plugins and select the Hydrolix data source plugin.
Click Upgrade in the top-right corner if the option is available.

Verification

Verify metrics are queryable from Grafana by running a test query and optionally creating a dashboard.

Visit http://{myhost}.hydrolix.live/grafana/dashboard in your web browser.
Select New > New dashboard.
Select + Add visualization.
Select Hydrolix - Local from the data source dropdown.
Paste the following query into the query input box:

SELECT
  count(*) as error_count,
  toStartOfMinute(timestamp) AS minute,
  container
FROM
  hydro.logs
WHERE
  timestamp > NOW() - INTERVAL 1 DAY
  AND level = 'error'
GROUP BY
  minute, container
ORDER BY 
  minute ASC

Change Query Type to TimeSeries.
In the right-hand panel, change No value to 0.

You should now have a timeseries visualization showing the count of errors by container for your Hydrolix cluster.

Hydrolix cluster error count by container

Select Save dashboard if you want to save this visualization in a new dashboard.

Go further

See the Hydrolix Data Source Plugin for Grafana for more details on configuring the data source plugin and a list of plugin features.

Troubleshooting

Not enough resources

If the grafana-* pod isn't starting and the kubectl or k9s utility is showing the pod in a pending state, check the resources on the nodes backing your Hydrolix installation. It's possible to lower the amount of memory so Grafana (and its associated renderer service) doesn't request as much from Kubernetes. More information about scaling pods and services can be found in Scale your Cluster. Here's a sample of a low-overhead configuration in the Hydrolix spec file:

spec:
  scale:
    grafana:
      cpu: 2
      memory: 2Gi
    renderer:
      cpu: 1
      memory: 1Gi

Postgres not available

Grafana uses the same Postgres instance as the Hydrolix catalog database, storing Grafana configuration data in a separate database called grafana. This is where Grafana dashboard definitions, users, alerts, and query history are stored.

The configuration provided by Hydrolix assumes that this Postgres instance is only available through SSL connections. If you're using an insecure configuration of Postgres, the built-in Grafana instance will refuse to connect.

If you're unable to use an SSL-enabled Postgres, perhaps for development or demo purposes, configure Grafana to make non-SSL connections with the kubectl utility.

Set pg_ssl_mode to disable
```
kubectl edit hdx
```
Use the file editor to add this line to the Hydrolix spec:
```
pg_ssl_mode: disable
```
Save the file.
Disable Grafana, then re-enable Grafana by removing, then adding lines to the Hydrolix spec:

data_visualization_tools:
- grafana