Documentation
ProductPricingDocsBlog

Visualize Fastly Data with Grafana

Suggest Edits

Grafana is a popular open source visualization and analytics application that has gained significant traction with the advent of always-on/connected-devices and time-series oriented datasets. It allows you to query, visualize, alert on, and explore metrics independent of where they are stored.

In this final segment of the Working with Fastly CDN Logs tutorial, we'll focus on visualizing our CDN log data leveraging a new dashboard we have developed for Grafana. You can find the full dashboard configuration file for use with this walkthrough below:

Download: Fastly Composite Dashboard for Hydrolix

Grafana Installation

The easiest way to use Grafana is to sign up for an account with [Grafana Cloud] (https://grafana.com/products/cloud/). Alternatively, Grafana can be installed and configured locally using the standard Grafana installation process.

Dashboard Dependencies

Grafana Pie Chart Plugin

The Fastly dashboard provides a variety of panels and visualizations for use with the Fastly CDN log data. A few of the panels make use of the Grafana Pie Chart plugin which is not included in the core release.

To install the Pie Chart plugin on a local Grafana instance, use the grafana-cli tool from the commandline:

grafana-cli plugins install grafana-piechart-panel

Update Plugins

After installing any new plugins (or updating Grafana), it's always good to check for plugin updates. While plugins are not updated automatically, you will be notified when updates are available in the Configuration/Plugins section of the Grafana UI.

If you are on the CLI, you can update all plugins using the following command:

grafana-cli plugins update-all

Restart Grafana

Full documentation on how to restart the Grafana service for your specific operating system can be found here. For Ubuntu (and other systems systems that leverage init.d), restart Grafana using the following command from the CLI:

$ sudo service grafana-server restart

$ sudo /etc/init.d/grafana-server restart

Configuration

With the dependencies taken care of, we're ready to login to the Grafana user interface and begin the configuration process.

Login

To login to the Grafana instance included within the Hydrolix client stack:

https://<hydrolix_hostname>.hydrolix.live/grafana

Username: admin
Password: admin

Please change your credentials after the intial login.

Add the Clickhouse Data Source

In order to connect to your Hydrolix database instance, you'll need to use the Clickhouse Datasource for Grafana.

Navigate to Configuration/Data Sources and add the Clickhouse data source:

Enter the Required Configuration Settings for your Hydrolix Database:

Name: Enter a name for your Hydrolix data source

URL: The Query API Endpoint for your Hydrolix instance using the format https://<hydrolix_hostname>.hydrolix.live/query

Explore the Data

Once your Grafana instance is connected to your Hydrolix cluster, navigate to the Explore menu and perform some basic data exploration prior to importing the dashboard configuration file.

Populate Query Builder with the Hydrolix Project, Table, and associated DateTime for the Fastly CDN log data:

Using the Query Builder menu, select the following for the FROM field in the Query Builder menu:

  1. The Hydrolix Project that contains the desired Fastly tables. Once the appropriate project is selected, any associated tables will automatically populate into the query builder table pulldown menu. The project that contains our tables is called Fastly.

  2. The Table that contains the Fastly CDN log data. Our data is contained within the xlogs table.

  3. The DateTime field for supporting time-series functions and macros. The Clickhouse data source plugin will automatically attempt to detect the appropriate Column:DateTime or Column:Timestamp field from the selected table. It's time_start in this case.

This process essentially binds the Project, Table and Datetime fields to the data source connection for use in subsequent queries and integrated variables.

Basic data exploration and validation

If the Clickhouse data source successfully connects to your Hydrolix instance, you will be presented with a basic time-series graph that will allow you to experiment with your dataset. It's worth spending some time switching between time-series and table-based formats to get familiar with how the results of a query are returned. Ironically, one of the trickiest parts to creating a compelling panel has more to do with time-bucketing and time-windowing versus returning the right data. Fortunately, Grafana (and the Clickhouse datasource) provide some extremely useful macros and functions that assist in the process.

Don't forget to display the raw generated SQL as you work through your queries. It's critical to understanding how something like the $timeSeries macro is interpreted.

Preparing the Dashboard Configuration File for Import

Prior to importing the Fastly dashboard for Hydrolix, there are a few suggested modifications that need to be made to the configuration file. The dashboard configuration file, also known as the JSON Model for the dashboard, is a data structure that defines the entire dashboard. This includes all of the general and panel-related settings, layout, and supporting queries.

Since it's obviously in JSON format, a source-code editor like Visual Studio Code Visual Studio Code can help with the editing process.

Recommended edits:

  1. Replace all instances of "datasource": "Hydrolix" with that of your Clickhouse datasource name. If you chose Hydrolix for your datasource name, you are good to go!
"datasource": "Hydrolix",
  1. Modify the "url": value to reflect that of your Hydrolix instance hostname. You will need to replace the hostname portion of https://<hydrolix_hostname>.hydrolix.live/version
{
        "icon": "external link",
        "tags": [],
        "targetBlank": true,
        "title": "Hydrolix Version Information",
        "type": "link",
        "url": "https://<hydrolix_hostname>.hydrolix.live/version"
      }
  1. The majority of the panels leverage a dashboard query variable ($servers) that allows for filtering by origin server. The relevant section is highlighted below. You will need to potentially replace "datasource": "Hydrolix" (if you haven't done so already) and change fastly.xlogs to the Hydrolix project.table that houses your Fastly CDN logs. Updating Query Builder with your specific Hydrolix project, table, and datetime settings at the data source level will not automatically remap the dashboard variable settings.
{
          "allValue": null,
          "current": {
            "selected": false,
            "text": "All",
            "value": "$__all"
          },
          "datasource": "Hydrolix",
          "definition": "SELECT host FROM fastly.xlogs",
          "description": "Origin host list to be used in filtering various panels",
          "error": null,
          "hide": 0,
          "includeAll": true,
          "label": "Origin Host",
          "multi": false,
          "name": "servers",
          "options": [],
          "query": "SELECT host FROM fastly.xlogs",
          "refresh": 2,
          "regex": "",
          "skipUrlSync": false,
          "sort": 0,
          "tagValuesQuery": "",
          "tags": [],
          "tagsQuery": "",
          "type": "query",
          "useTags": false
        }

Importing the Grafana Fastly Dashboard for Hydrolix

Once the above modifications have been made, it's as easy as uploading the JSON Model file.

If the panels are not updating after import, please go into Edit on one of the panels and check to ensure that the Hydrolix project, table, and datetime settings are properly set in Query Builder configuration. Remedy if required.

Visualization

Congratulations! You've made it.

At this point, your new Grafana dashboard for visualizing Fastly CDN logs on Hydrolix should be filled with a plethora of colours and metrics. Keep in mind, this is a composite dashboard that has been designed for flexibility and inspiration. Please customize and adapt the layout, panels, and supporting queries to make it your own.

If you have any additional suggestions on how we can make future iterations better, of if there's another tutorial you would like to see, please don't hesitate to reach out. We welcome input and feedback from the community.

Updated 2 months ago