Kibana Automatic Installation

Overview

Kibana visualizes your Hydrolix data using Quesma, a database proxy. All the components required for a working Kibana instance are packaged with Hydrolix as an in-cluster integration. Kibana provides dashboards, while Quesma and Elasticsearch are also automatically deployed within your cluster to convert data and maintain the Kibana installation.

This document covers the activation, configuration, and deactivation of Kibana in your Hydrolix cluster. Instructions on how to use Kibana are in the Kibana documentation.

Quesma is a lightweight database proxy that enables interoperability between a front end such as Kibana and multiple back end datastores such as Hydrolix and Elasticsearch. It provides this interoperability by acting as a SQL translation layer that decouples your dashboard and ingestion layers from your datastore.

Before you begin

Here's what you'll need to run Kibana.

• Kuberentes cluster-admin privileges as described in the Kubernetes documentation
• the name of the Hydrolix table you'd like Kibana to use
• sufficient resources in your Kubernetes cluster to accommodate new Kibana, Quesma, and Elasticsearch pods

Installation

1. Install the Elasticsearch Custom Resource Definitions (CRD) and operator into your cluster. With kubectl configured to work with your Hydrolix cluster, issue these two commands.

   Shell

   kubectl create -f https://download.elastic.co/downloads/eck/2.15.0/crds.yaml
   kubectl apply -f https://download.elastic.co/downloads/eck/2.15.0/operator.yaml
   

   More information about this step can be found in the Elasticsearch documentation.

2. Edit your Hydrolix cluster's configuration to include the Kibana in-cluster integration.

   Add these two lines to the spec: section of your hydrolixcluster.yaml file.

   YAML

   data_visualization_tools:
     - Kibana
   

   For example, here's what it could look like in your hydrolixcluster.yaml file.

   YAML

   spec:
     acme_enabled: true
     admin_email: [email protected]
     data_visualization_tools:
       - Kibana
     db_bucket_endpoint: https://us-sea-1.linodeobjects.com
     db_bucket_region: us-sea-1
   

3. Load the configuration changes into your Hydrolix cluster with kubectl apply -f hydrolixcluster.yaml.

4. Wait for Elasticsearch, Quesma, and Kibana pods to start. This can take several minutes. You can tell when they're ready for use by looking for the Running indicator in the output of the kubectl command.

   Shell

   % kubectl get pods | grep -E 'elastic|quesma|kibana'       
   
   elasticsearch-es-default-0            1/1     Running            3 (74m ago)       7h25m
   kibana-kb-85bdfdffd6-6x9n7            1/1     Running            0                 7h25m
   quesma-596b76b665-fhbpz               1/1     Running            0                 7h25m
   

5. Verify the web interface is working.

   1. Make sure you're logged into your Hydrolix cluster's UI by visiting https://{hostname}.hydrolix.live in a web browser.
   2. Visit https://{hostname}.hydrolix.live/kibana in your web browser to see Kibana's Web UI.

Using Kibana and Quesma

The Hydrolix-provided in-cluster Kibana integration is initially configured to use the Hydrolix hydro.logs table as a source. To see this data, use the Kibana interface.

1. Visit your Kibana installation in a web browser by visiting https://{hostname}.hydrolix.live/kibana.

2. Select Stack Management from the main menu on the left.

3. In the resulting menu, select Kibana > Data Views.

4. If this is your first data source, click the link at the bottom of the introduction page. The link is part of the "You can also create a data view against hidden, system, or default indices" text.

   ...or...

   If this isn't your first data source, select the Create data view button in the upper right-hand corner of the UI.

5. Fill in the next form with a name, index pattern, and timestamp.

6. Click the Save data view to Kibana button near the bottom of the page.

If you need assistance, more information can be found in the Quesma documentation.

Configuration

By default, Quesma is set up to use data from your Hydrolix clusters' hydro.logs table. To change this to another table, add three more lines to your hydrolixconfig.yaml file:

  quesma_config:
    project: {project_name}
    table: {table_name}

Note the spaces at the beginning of the lines in the example above.

Once you save and apply the changes with kubectl apply -f hydrolixcluster.yaml, the Quesma pod will restart and point to the specified table.
For example, to use a table called sample_project.sample_table, your configuration could look similar to the snippet below.

    replicas: 1
    service: kinesis-peer
  quesma_config:
    project: sample_project
    table: sample_table
  scale:
    intake-head:
      replicas: 10
    kinesis-peer:
      replicas: 1

Deactivation

Removing this integration is straightforward, reversing the installation steps.

1. Edit your hydrolixcluster.yaml file, removing the data_visualization_tools and quesma_config sections.

2. Load the configuration changes into your Hydrolix cluster with kubectl apply -f hydrolixcluster.yaml.

3. Remove the Elasticsearch CRDs and operator from your cluster.

   Shell

   kubectl delete -f https://download.elastic.co/downloads/eck/2.15.0/crds.yaml
   kubectl delete -f https://download.elastic.co/downloads/eck/2.15.0/operator.yaml