Work with Metrics
Working with Metrics
Timeseries data are measurements or events that are monitored, where time is a critical dimension to the data. This could be server metrics, application performance monitoring metrics, network data, logs, sensor data, events, clicks, IOT, and many other types of analytics data.
Timeseries analytics is the measurement of change over time.
All fields in Hydrolix are indexed, however, time is the primary index. Time as the primary key is one of the reasons why Hydrolix can get query results from object storage quickly. Internally, a table consists of many time partitioned data files in a columnar format. Each file represents sequentially time ordered data and associated metadata. This is all handled behind the scenes and invisible to the user or administrator.
Working with time
Since Hydrolix uses time as a primary key, queries that reference tables should contain a time range in the WHERE clause. This helps maximize performance be reducing the scope of which portions of a table need to be included in a particular search.
The below transform can be used to load data. An Auto-view will be created when the below is used, that will have all the columns needed to view the data.
{
"name": "metrics",
"type": "csv",
"table": "{{table_uuid}}",
"settings": {
"compression": "gzip",
"format-detail": {
"delimiter": 9
},
"output_columns": [
{
"name": "timestamp",
"datatype": {
"primary": true,
"type": "datetime",
"format": "20060102150405",
"resolution": "seconds",
"source": { "from_input_index": 0 }
}
},
{
"name": "hostname",
"datatype": {
"type": "string",
"source": { "from_input_index": 1 }
}
},
{
"name": "region",
"datatype": {
"type": "string",
"source": { "from_input_index": 2 }
}
},
{
"name": "datacenter",
"datatype": {
"type": "string",
"source": { "from_input_index": 3 }
}
},
{
"name": "rack",
"datatype": {
"type": "string",
"source": { "from_input_index": 4 }
}
},
{
"name": "os",
"datatype": {
"type": "string",
"source": { "from_input_index": 5 }
}
},
{
"name": "arch",
"datatype": {
"type": "string",
"source": { "from_input_index": 6 }
}
},
{
"name": "team",
"datatype": {
"type": "string",
"source": { "from_input_index": 7 }
}
},
{
"name": "service",
"datatype": {
"type": "string",
"source": { "from_input_index": 8 }
}
},
{
"name": "service_version",
"datatype": {
"type": "string",
"source": { "from_input_index": 9 }
}
},
{
"name": "service_environment",
"datatype": {
"type": "string",
"source": { "from_input_index": 10 }
}
},
{
"name": "usage_user",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 11 }
}
},
{
"name": "usage_system",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 12 }
}
},
{
"name": "usage_idle",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 13 }
}
},
{
"name": "usage_nice",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 14 }
}
},
{
"name": "usage_iowait",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 15 }
}
},
{
"name": "usage_irq",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 16 }
}
},
{
"name": "usage_softirq",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 17 }
}
},
{
"name": "usage_steal",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 18 }
}
},
{
"name": "usage_guest",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 19 }
}
},
{
"name": "usage_guest_nice",
"datatype": {
"type": "uint64",
"index": false,
"source": { "from_input_index": 20 }
}
}
]
}
}
Updated about 2 months ago