Timestamp formatting

Transform column definitions that use either the datetime or epoch datatypes require the additional definition of a format property. This property lets Hydrolix know how to parse and interpret the column’s incoming time data.

You can optionally specify a resolution property as well, in order to tune the granularity of the stored time data.

Formatting datetime values

The datetime datatype allows most any format of time representation in incoming data, provided that you can describe that format as a Go-style time format string. In that column definition’s format property, you must indicate how the data source would represent Go’s “reference time” of 03:04:05 PM on January 2, 2006, in the Mountain Standard Time zone (UTC-07:00).

A handful of examples:

Possible data value The literal format to specify
2020-12-01 2006-01-02
12-01-20 13:05:00 01-02-06 15:04:05
2020-12-01T08:05:00-0500 2006-01-02T15:04:05-0700

Formatting epoch values

Use the epoch datatype for Unix-style “epoch time”, or other representations of a timestamp a number of time-units that have passed since 1970-01-01 00:00:00 UTC, a.k.a. the epoch.

With epoch, acceptable values of format include the following:

Format Meaning
ns nanoseconds since epoch
us microseconds since epoch
ms milliseconds since epoch
cs centiseconds since epoch
s seconds since epoch (a.k.a. Unix time)

In all cases, the Hydrolix ingester accepts either integers or numerical strings as legal epoch data.

Hydrolix treats all epoch-based timestamps as UTC time.

Specifying resolution

Hydrolix supports storing time values with either second or millisecond resolution. You can set this granularity level for a given field by specifying a resolution property within the appropriate output_columns definition, as discussed in Column definitions.

Hydrolix uses seconds as its default stored-time resolution.

Resolution and truncation

If an incoming time value contains a finer resolution than you’ve set its target column definition to receive, Hydrolix will truncate the fractional-second part as necessary prior to storage.

For example, if you’ve set a given column to have millisecond resolution, and incoming data for that field includes a time down to the microsecond, Hydrolix will store the value after rounding down to the nearest millisecond.