Settings: Output Columns

The core of a transform is the Output Columns section under settings. This section is where you define what the data looks like and how it should be loaded into the Hydrolix platform. These definitions are used to specify Data-Types, indexing and other characteristics your data should have when it is being written into the HDX format.

"settings": {
   "output_columns": [
        {
            "name": "timestamp",
            "position": 0,
            "datatype": {
                "type": "datetime",
                "primary": true,
                "format": "2006-01-02 15:04:05 MST"
            }
        },
        {
            "name": "data",
            "position": 1,
            "datatype": {
                "type": "uint64"
            }
        }
    ],
    
    // Other "settings" properties go here

}

A transform must contain one object in its output_columns list for every column in the target table that its ingests will affect. This includes the table's primary timestamp column, as well as any column that will copy a new value based directly on incoming data. The transform enables Hydrolix to copy these values into the target table by specifying a positional or name-based mapping between the incoming data fields and the table's columns.

The transform can also set columns to derive a new value based on a JavaScript expression, or a simple default.

The properties supported by column definition objects include the following:

AttributePurposeRequired
nameA name for the Column, for use by subsequent SQL queries.Yes
positionColumn position in the source CSV data, starting with 0.Yes, if the data is CSV.
datatypeAn object containing detailed column definition.Yes

The datatype property requires an object as its value in order to allow recursive definitions, as described in arrays. The properties supported by datatype include the following:

AttributePurposeRequired
defaultA default value to apply when the source data's value for this column is empty. No
elementsA description of the constituent elements of array-type data. See arrays.No, except for array.
formatParsing information for datetime and epoch data types. See Timestamp formatting.No, except for datetime and epoch.
indexIf true, then Hydrolix will index this column's data as well as storing it as usual.No
primaryIf true, then this marks the target table's single, primary field. It must have either datetime or epoch type.Exactly one per transform
resolutionSet to "ms" (milliseconds) or "s" (seconds) to define the time-granularity that Hydrolix will apply when storing this column's datetime or epoch-type data. Defaults to "s".No
scriptA JavaScript expression whose output becomes the stored value of this column. See Scripted column values, below.No
sourceUsed with JSON encoded files, if specified this is used as the source column name. When data is then written to the database the column will be called by the "name" value aboveNo
typeThe data type of the field.Yes
virtualIf true, then Hydrolix will not map this column to any incoming data. Useful for constant or derived data when paired with the default or script settings, respectively.No

Did this page help you?