Overview

work with Metrics

Timeseries data are measurements or events that are monitored, where time is a critical dimension to the data. This could be server metrics, application performance monitoring metrics, network data, logs, sensor data, events, clicks, IOT, and many other types of analytics data.

Timeseries analytics is the measurement of change over time.

All fields in Hydrolix are indexed, however, time is the primary index. Time as the primary key is one of the reasons why Hydrolix can get query results from object storage quickly. Internally, a table consists of many time partitioned data files in a columnar format. Each file represents sequentially time ordered data and associated metadata. This is all handled behind the scenes and invisible to the user or administrator.

Work with time

Since Hydrolix uses time as a primary key, queries that reference tables should contain a time range in the WHERE clause. This helps maximize performance be reducing the scope of which portions of a table need to be included in a particular search.

The below transform can be used to load data.

When a transform is created, a corresponding automatic view is created or updated which allows read access to the data ingested into the table.

{
    "name": "metrics",
    "type": "csv",
    "table": "{{table_uuid}}",
    "settings": {
        "compression": "gzip",
        "format-detail": {
            "delimiter": 9
        },
        "output_columns": [
          {
                "name": "timestamp",
                "datatype": {
                    "primary": true,
                    "type": "datetime",
                    "format": "20060102150405",
                    "resolution": "seconds",
                    "source": { "from_input_index": 0 }
                }
            },
            {
                "name": "hostname",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 1 }
                }
            },
            {
                "name": "region",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 2 }
                }
            },
            {
                "name": "datacenter",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 3 }
                }
            },
            {
                "name": "rack",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 4 }
                }
            },
            {
                "name": "os",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 5 }
                }
            },
            {
                "name": "arch",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 6 }
                }
            },
            {
                "name": "team",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 7 }
                }
            },
            {
                "name": "service",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 8 }
                }
            },
            {
                "name": "service_version",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 9 }
                }
            },
            {
                "name": "service_environment",
                "datatype": {
                    "type": "string",
                    "source": { "from_input_index": 10 }
                }
            },
            {
                "name": "usage_user",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 11 }
                }
            },
            {
                "name": "usage_system",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 12 }
                }
            },
            {
                "name": "usage_idle",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 13 }
                }
            },
            {
                "name": "usage_nice",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 14 }
                }
            },
            {
                "name": "usage_iowait",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 15 }
                }
            },
            {
                "name": "usage_irq",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 16 }
                }
            },
            {
                "name": "usage_softirq",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 17 }
                }
            },
            {
                "name": "usage_steal",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 18 }
                }
            },
            {
                "name": "usage_guest",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 19 }
                }
            },
            {
                "name": "usage_guest_nice",
                "datatype": {
                    "type": "uint64",
                    "index": false,
                    "source": { "from_input_index": 20 }
                }
            }
        ]
    }
}