Basic Transform Structure

A transform is a JSON document that configures how Hydrolix will index and store incoming data onto a given table. A transform's content includes a general description of the incoming data's format, as well as a description of every column that data will produce and populate in the target table.

The following example transform would let a Hydrolix table ingest a GZIP-compressed, CSV-formatted data stream with two columns:

{
    "name": "my_special_transform",
    "description": "description of my data",
    "type": "csv",
    "settings": {
        "output_columns": [
            {
                "name": "timestamp",
                "position": 0,
                "datatype": {
                    "type": "datetime",
                    "primary": true,
                    "format": "2006-01-02 15:04:05 MST"
                }
            },
            {
                "name": "the_data",
                "position": 1,
                "datatype": {
                    "type": "uint64"
                }
            }
        ],
        "compression": "gzip",
        "format_details": {
            "skip_head": 2,
            "delimiter": "\t"
        }
    }
}

Basic properties

Transform documents have the following top-level properties:

PropertyPurpose
nameA name for this transform; must be unique within its table.
descriptionAn optional description of this transform.
typeThe file type of the input data this transform handles. One of csv, or json, or parquet.
settingsA JSON object describing the data to ingest.

The settings object

"settings": {
    "is_default": 1,
    "compression": "gzip",
    "format_details": {
        // Configuration specific to the data's overall 
        // format goes here
    },
    "output_columns": [
        // A list of column-definition objects goes here
    ]
}

The settings object contains the details of the content and structure of the data. This is where the data input format, compression, and resulting columns are defined.

PropertyPurpose
is_defaultIf set to 1, then this transform's table will apply this transform to a streaming ingest that doesn't otherwise specify a transform. (Has no effect on batch ingests.)
output_columnsAn array of JSON objects with a description of every column in the target table that the incoming data will populate. See Column definitions
compressionA string describing the compression applied to the data (or "none", the default). See Handling compression
format_detailsA JSON object containing configuration information specific to the data's overall format, just as field-delimiter characters with CSV files. See Message/File encoding,

Did this page help you?