Kibana Automatic

Overview

Visualize your Hydrolix data in Kibana using the Kibana Gateway database proxy.

All the components required for a working Kibana instance are packaged with Hydrolix as an in-cluster integration. Kibana provides dashboards, while the Kibana Gateway and Elasticsearch are also automatically deployed within your cluster to convert data and maintain the Kibana installation.

This document covers the activation, configuration, and deactivation of Kibana in your Hydrolix cluster. Instructions on how to use Kibana are in the Kibana documentation.

Kibana Gateway is a lightweight database proxy that enables interoperability between a front end such as Kibana and multiple back end datastores such as Hydrolix and Elasticsearch. It provides this interoperability by acting as a SQL translation layer that decouples your dashboard and ingestion layers from your datastore.

Container and pod names

Hydrolix acquired the database proxy software from Quesma. The Kubernetes pod, container, and tunable are named quesma since the cluster integration predated the software acquisition in 2025 September. The product is called Kibana Gateway.

Before you begin

Ensure you have the following:

• Kubernetes cluster-admin privileges as described in the Kubernetes documentation
• The name of the Hydrolix table to use with Kibana
• Sufficient resources in your Kubernetes cluster to accommodate new Kibana, Kibana Gateway, and Elasticsearch pods

Installation

1. Install the Elasticsearch Custom Resource Definitions (CRD) and operator. With kubectl configured for your Hydrolix cluster, run the following commands:

   kubectl create -f https://download.elastic.co/downloads/eck/2.15.0/crds.yaml
   kubectl apply -f https://download.elastic.co/downloads/eck/2.15.0/operator.yaml
   

   For more information, see the Elasticsearch documentation.

2. Edit the Hydrolix cluster configuration to include the Kibana in-cluster integration. Add the following lines to the spec: section of your hydrolixcluster.yaml file:

   data_visualization_tools:
     - Kibana
   

   Example hydrolixcluster.yaml configuration:

   spec:
     acme_enabled: true
     admin_email: admin@mycompany.com
     data_visualization_tools:
       - Kibana
     db_bucket_endpoint: https://us-sea-1.linodeobjects.com
     db_bucket_region: us-sea-1
   

3. Apply the configuration changes:

   kubectl apply -f hydrolixcluster.yaml
   

4. Wait for the pods to start. The elasticsearch, quesma (Kibana Gateway), and kibana-kb (Kibana) pods can take several minutes to start. Verify the pods are running:

   kubectl get pods | grep -E 'elastic|quesma|kibana'
   

   Expected output showing Running status:

   elasticsearch-es-default-0            1/1     Running            3 (74m ago)       7h25m
   kibana-kb-85bdfdffd6-6x9n7            1/1     Running            0                 7h25m
   quesma-596b76b665-fhbpz               1/1     Running            0                 7h25m
   

5. Verify the Kibana web interface:

   1. Sign in to your Hydrolix cluster UI at https://{hostname}.hydrolix.live.
   2. Navigate to https://{hostname}.hydrolix.live/kibana to access the Kibana UI.

Use Kibana and Kibana Gateway

The Hydrolix in-cluster Kibana integration is configured to use the Hydrolix hydro.logs table as a data source.

1. Visit your Kibana installation at https://{hostname}.hydrolix.live/kibana.
2. Select Stack Management from the main menu.
3. Select Kibana > Data Views.
4. Create a data view:
   • If this is your first data source, select the link at the bottom of the introduction page. The link is part of the text "You can also create a data view against hidden, system, or default indices."
   • If this isn't your first data source, select the Create data view button.
5. Enter a name, index pattern, and timestamp in the form.
6. Select Save data view to Kibana.

Configuration

The in-cluster Kibana Gateway can be configured to have access to additional tables within the Hydrolix cluster and to be interoperable with a Kibana instance running outside the Hydrolix cluster.

Query additional projects and tables

By default, Kibana Gateway uses data from the hydro.logs table. To configure Kibana Gateway to use a different table, add the following configuration to your hydrolixcluster.yaml file:

quesma_config:
  project: {project_name}
  table: {table_name}

Apply the changes to restart the quesma pod and point it to the specified table:

kubectl apply -f hydrolixcluster.yaml

Example configuration using the sample_project.sample_table table:

    replicas: 1
    service: kinesis-peer
  quesma_config:
    project: sample_project
    table: sample_table
  scale:
    intake-head:
      replicas: 10
    kinesis-peer:
      replicas: 1

Enable public access to Kibana Gateway

To allow external clients (such as Kibana running outside the cluster) to communicate with Kibana Gateway, enable public access using the following configuration:

spec:
  quesma_config:
    enable_public_access: true
  issue_wildcard_cert: true

This configuration exposes Kibana Gateway at quesma.{myhost}.hydrolix.live for external access. The username and password are stored in the hdx-elastic-user Kubernetes secret. For more information about managing secrets, see Managing Secrets using kubectl.

Deactivation

To remove the Kibana integration:

1. Edit your hydrolixcluster.yaml file and remove the data_visualization_tools and quesma_config sections.

2. Apply the configuration changes:

   kubectl apply -f hydrolixcluster.yaml
   

3. Remove the Elasticsearch CRDs and operator:

   kubectl delete -f https://download.elastic.co/downloads/eck/2.15.0/crds.yaml
   kubectl delete -f https://download.elastic.co/downloads/eck/2.15.0/operator.yaml