Kibana Automatic Installation

Overview

Visualize your Hydrolix data in Kibana using the Kibana Gateway database proxy.

All the components required for a working Kibana instance are packaged with Hydrolix as an in-cluster integration. Kibana provides dashboards, while the Kibana Gateway and Elasticsearch are also automatically deployed within your cluster to convert data and maintain the Kibana installation.

This document covers the activation, configuration, and deactivation of Kibana in your Hydrolix cluster. Instructions on how to use Kibana are in the Kibana documentation.

Kibana Gateway is a lightweight database proxy that enables interoperability between a front end such as Kibana and multiple back end datastores such as Hydrolix and Elasticsearch. It provides this interoperability by acting as a SQL translation layer that decouples your dashboard and ingestion layers from your datastore.

ℹ️

Container and pod names

Hydrolix acquired the database proxy software from Quesma. The Kubernetes pod, container, and tunable are named quesma since the cluster integration predated the software acquisition in 2025 September. The product is called Kibana Gateway.

Before you begin

Here's what you'll need to run Kibana.

• Kubernetes cluster-admin privileges as described in the Kubernetes documentation
• The name of the Hydrolix table you'd like Kibana to use
• Sufficient resources in your Kubernetes cluster to accommodate new Kibana, Kibana Gateway, and Elasticsearch pods

Installation

1. Install the Elasticsearch Custom Resource Definitions (CRD) and operator into your cluster. With kubectl configured to work with your Hydrolix cluster, issue these two commands.

   Shell

   kubectl create -f https://download.elastic.co/downloads/eck/2.15.0/crds.yaml
   kubectl apply -f https://download.elastic.co/downloads/eck/2.15.0/operator.yaml
   

   More information about this step can be found in the Elasticsearch documentation.

2. Edit your Hydrolix cluster's configuration to include the Kibana in-cluster integration.

   Add these two lines to the spec: section of your hydrolixcluster.yaml file.

   YAML

   data_visualization_tools:
     - Kibana
   

   For example, here's what it could look like in your hydrolixcluster.yaml file.

   YAML

   spec:
     acme_enabled: true
     admin_email: [email protected]
     data_visualization_tools:
       - Kibana
     db_bucket_endpoint: https://us-sea-1.linodeobjects.com
     db_bucket_region: us-sea-1
   

3. Load the configuration changes into your Hydrolix cluster with kubectl apply -f hydrolixcluster.yaml.

4. Wait for the elasticsearch, quesma (Kibana Gateway), and kibana-kb (Kibana) pods to start. This can take several minutes. You can tell when they're ready for use by looking for the Running indicator in the output of the kubectl command.

   Shell

   % kubectl get pods | grep -E 'elastic|quesma|kibana'       
   
   elasticsearch-es-default-0            1/1     Running            3 (74m ago)       7h25m
   kibana-kb-85bdfdffd6-6x9n7            1/1     Running            0                 7h25m
   quesma-596b76b665-fhbpz               1/1     Running            0                 7h25m
   

5. Verify the web interface is working.

   1. Make sure you're logged into your Hydrolix cluster's UI by visiting https://{hostname}.hydrolix.live in a web browser.
   2. Visit https://{hostname}.hydrolix.live/kibana in your web browser to see Kibana's Web UI.

Using Kibana and Kibana Gateway

The Hydrolix-provided in-cluster Kibana integration is initially configured to use the Hydrolix hydro.logs table as a source. To see this data, use the Kibana interface.

1. Visit your Kibana installation in a web browser by visiting https://{hostname}.hydrolix.live/kibana.

2. Select Stack Management from the main menu on the left.

3. In the resulting menu, select Kibana > Data Views.

4. If this is your first data source, click the link at the bottom of the introduction page. The link is part of the "You can also create a data view against hidden, system, or default indices" text.

   ...or...

   If this isn't your first data source, select the Create data view button in the upper right-hand corner of the UI.

5. Fill in the next form with a name, index pattern, and timestamp.

6. Click the Save data view to Kibana button near the bottom of the page.

If you need assistance, more information can be found in the Quesma documentation.

Configuration

The in-cluster Kibana Gateway can be configured to have access to additional tables within the Hydrolix cluster and to be interoperable with a Kibana instance running outside the Hydrolix cluster.

Query additional projects and tables

By default, Kibana Gateway is set up to use data from the hydro.logs table in your Hydrolix cluster. To change this to another table, add three more lines to your hydrolixconfig.yaml file:

  quesma_config:
    project: {project_name}
    table: {table_name}

Note the spaces at the beginning of the lines in the example above.

Once you save and apply the changes with kubectl apply -f hydrolixcluster.yaml, the quesma pod will restart and point to the specified table.
For example, to use a table called sample_project.sample_table, your configuration could look similar to the snippet below.

    replicas: 1
    service: kinesis-peer
  quesma_config:
    project: sample_project
    table: sample_table
  scale:
    intake-head:
      replicas: 10
    kinesis-peer:
      replicas: 1

Enable public access to Kibana Gateway

If Kibana or another client is running outside the cluster and needs to communicate with the Kibana Gateway, use the tunables quesma_config.enable_public_access: true and issue_wildcard_cert: true. This will expose the Kibana Gateway instance under a separate subdomain for external access by Kibana or a similar tool that can query the Hydrolix cluster. The additional configuration looks like the following:

spec:
  quesma_config:
    enable_public_access: true
  issue_wildcard_cert: true

This exposes the Kibana Gateway at quesma.{myhost}.hydrolix.live. The username and password for can be found in the hdx-elastic-user K8s secret. You can read more about interacting with Kubernetes secrets in Managing Secrets using kubectl.

Deactivation

Removing this integration is straightforward, reversing the installation steps.

1. Edit your hydrolixcluster.yaml file, removing the data_visualization_tools and quesma_config sections.

2. Load the configuration changes into your Hydrolix cluster with kubectl apply -f hydrolixcluster.yaml.

3. Remove the Elasticsearch CRDs and operator from your cluster.

   Shell

   kubectl delete -f https://download.elastic.co/downloads/eck/2.15.0/crds.yaml
   kubectl delete -f https://download.elastic.co/downloads/eck/2.15.0/operator.yaml