Self Described Messaging

Self Described Messages

Alternatively, instead of pre-publishing a transform, Hydrolix can ingest data using "self described" messages which contain their own embedded schemas.

To stream batches of self described messages to Hydrolix, a POST message is sent to the streaming_url for the deployment.

Required HTTP Headers

NameValue
x-hdx-streaming-formatjson-batch
x-hdx-streaming-typedescribed

Message attributes

Self described messages must be associated with a destination table:

AttributePurpose
namespace_idTo be provided by Hydrolix.
settingsContains the format of the data elements later in the document.
dataAn array of individual messages conforming to the schema described in settings
{
  "namespace_id": "to be given by Hydrolix",
  "settings": {
    "schema": []
    }
    "data": []
}

schema element

Under settings is a schema. This is an array of data elements to be expected in this message, along with a description of how to treat the field. For example:

"schema": [
      {
        "name": "timestamp",
        "datatype": {
          "type": "datetime",
          "format": "2006-01-02 15:04:03 MST",
          "primary": true
        }
      },
      {
        "name": "clientId",
        "datatype": {
          "type": "uint64"
        }
      }
     ]

See the transform page for more details on what we support for the data schema.

We support multiple data type see the list of data type to use in the schema here

data

The data element is an array containing individual messages. Each message is represented as an array, contains values in the order they were described in the schema. If a message in this particular batch contains a null or empty value, it will be replaced with the "default" value if one is declared.

"data": [
    [ "2020-02-26 16:01:27 PST", 29991, "1.2.3.4/24", 1223, "1.4.5.7", 1.234 ],
    [ "2020-02-26 16:01:28 PST", 29989, "1.2.3.5/24", 9190, "1.4.5.7", 1.324 ],
    [ "2020-02-26 16:01:28 PST", 29990, "1.2.3.5/24", null, "1.4.5.7", 12.34 ]
  ]

Example API Call

$ curl -s \
     -H "x-hdx-streaming-format: json-batch" \
     -H "x-hdx-streaming-type: described" \
     ${streaming_url} -X POST -d '
{
    "namespace_id": "to be provided",
    "settings": {
        "schema": [
            {
                "name": "timestamp",
                "datatype": {
                    "type": "datetime",
                    "format": "2006-01-02 15:04:03 MST",
                    "primary": true
                }
            },
            {
                "name": "clientId",
                "datatype": {
                    "type": "uint64"
                }
            },
            {
                "name": "clientIp",
                "datatype": {
                    "type": "string"
                }
            },
            {
                "name": "clientCityCode",
                "datatype": {
                    "type": "uint64"
                }
            },
            {
                "name": "resolverIp",
                "datatype": {
                    "type": "string"
                }
            },
            {
                "name": "resolveDuration",
                "datatype": {
                    "type": "double"
                }
            }
        ]
    },
    "data": [
        [ "2020-02-26 16:01:27 PST", 29991, "1.2.3.4/24", 1223, "1.4.5.7", 1.234 ],
        [ "2020-02-26 16:01:28 PST", 29989, "1.2.3.5/24", 9190, "1.4.5.7", 1.324 ],
        [ "2020-02-26 16:01:28 PST", 29990, "1.2.3.5/24", null, "1.4.5.7", 12.34 ]
    ]
}'

Did this page help you?