Getting Started

The HTTP Streaming API is used to continuously consume events via HTTP to be loaded into the Hydrolix platform. Event messages are sent via HTTP POST to the /ingest/event API endpoint.

An event's structure can be of two types:

• Predefined: the event has a predefined transform loaded into Hydrolix prior to loading data.
• Self-Described: the event contains the transform within the message itself. This can be useful if you need to define your message structure at time of creation.

HTTP headers are used to provide information to the system about which table the data should be loaded into, the content-type of the incoming data, authentication information, and the transform used to prepare and store the data.

👍

Basic Steps

1. Create a Project/Table
2. Create a Transform
3. Scale the Stream Architecture .
4. HTTP POST messages to the /ingest/event API endpoint .

It is assumed that the project, table, and transform are all already configured. More information on how to set these up can be found on the Projects & Tables and Write Transforms pages.

The API Endpoint

The HTTP endpoint uses the URI path /ingest/event and is made available by default on port 443 (HTTPS) to receive data. IP allowlists and authentication information can be used to grant/restrict access to the endpoint (see IP Allow List, User Authentication).

The POST method is used to send data with HTTP Headers used to define the characteristics of the message.

For example:

$ curl -s \
     -H 'content-type: application/json' \
     -H 'x-hdx-table: <project.table>' \
     -H 'x-hdx-transform: <pre-existing transform>' \
     -H 'Authorization: Bearer <token>`
     https://<myhost>/ingest/event -X POST -d @<my_data_file>

HTTP Headers

Most HTTP headers required for streaming ingest are prefixed with x-hdx-. Firewall rules should be adjusted where necessary to allow for the additional headers x-hdx- and Authorization. If x-hdx- headers are omitted, a 4xx-style response will be issued by the endpoint.

Optional headers all have the default of NONE if they are not present.

The OAuth Bearer Token

Get the bearer token, which is good for the next 24 hours, to authenticate future API calls. This command assumes you've set the $HDX_HOSTNAME, $HDX_USER and $HDX_PASSWORD environment variables:

export HDX_TOKEN=$(
  curl -v -X POST -H "Content-Type: application/json" \
  https://$HDX_HOSTNAME/config/v1/login/ \
  -d "{
    \"username\":\"$HDX_USER\",
    \"password\":\"$HDX_PASSWORD\"  
  }" | jq -r ".auth_token.access_token"
)

Using Query Strings

As an alternative to using HTTP headers to specify the table and transform, the following query string parameters can be used within the URI string:

For example:

curl 'https://my.hydrolix.net/ingest/event?table=myproject.mytable&transform=transform'  -H 'content-type: application/json' -H 'Authorization: Bearer <token>

Supported Payloads

The payload for the POST can be:

• a single JSON object.
• an array of JSON objects.
• new-line delimited (ND_JSON) format.
• a CSV (Character Separated) file.

Native HTTP compression is supported, with specific payload compression being specified within the transform (write schema). If another format is needed, please contact Hydrolix Support

JSON Payloads

JSON can be sent as a single JSON object, an array of JSON object or ND_JSON format.

Single JSON object

{
   "timestamp": "2020-03-01 00:00:00 PST",
   "a_metric": 45.2,
   "a_dimension": "My First Dimension"
}

Array of JSON objects

[
  {
    "timestamp": "2020-03-01 00:00:00 PST",
    "a_metric": 45.2,
    "a_dimension": "My First Dimension"
  },
  {
    "timestamp": "2020-03-01 00:00:15 PST",
    "a_metric": 12.6,
    "a_dimension": "My Second Dimension"
  }
]

ND_JSON

{"timestamp": "2020-03-01 00:00:00 PST", "a_metric": 45.2,"a_tag": "My First Dimension"}
{"timestamp": "2020-03-01 00:00:15 PST", "a_metric": 12.6,"a_tag": "My Second Dimension"}

CSV Payloads

CSVs, or character-separated values, are sent as data structured as CSV. The transform schema must have:

• the same delimiter defined in its properties as the character used as the delimiter in the data.
• a "source": { "from_input_index": 0 } of each column that matches the structure of the data and has the position of the column.

The header for content-type should be as follows: content-type: text/csv.

$ curl -s \
     -H "x-hdx-table: the_project.the_table" \
     -H "x-hdx-transform: my_transform_schema" \
     -H "content-type: text/csv" \
     -H "Authorization: Bearer <token>" \
     https://demo.hydrolix.io/ingest/event -X POST -d '
    "2020-03-01 00:00:00 PST", 45.2, "My First Dimension"
    "2020-03-01 00:00:15 PST", 12.6, "My Second Dimension"
		'

Compression

If the message is compressed, the content-encoding: value should be set to the compression type. More detail on the content-encoding header can be found here. If this value is not declared, Hydrolix will attempt to infer any applied compression.

See Compression Algorithms for a complete list of supported compression types. If no compression is used on the request document, the content-encoding header is not required.

📘

Multiple Compression Options

It should be noted that the HTTP request document can be compressed using multiple methods if required before being ingested to Hydrolix.

If data was encoded with compression A, then B, then C,

Text

# Original Encoding
encoded_data = C(B(A(data)))

# Header
content-encoding: A, B, C

# Decoded Data
decoded_data = decodeA(decodeB(decodeC(encoded_data)))

For example:

cURL

$ gzip mydata.json
$ curl -s \
     -H "x-hdx-table: my_table" \
     -H "x-hdx-transform: my_transform_schema" \
     -H "content-type: application/json" \
     -H "content-encoding: gzip, bzip2, zip" \
     -H "Authorization: Bearer <token> \
     https://demo.hydrolix.io/ingest/event -X POST --data-binary @mydata.json.gz.bzip2.zip

Event Structure and Transforms

As discussed earlier, there are two types of event structure that can be sent to Hydrolix for load.

• Predefined: events that have a transform predefined and configured in the platform prior to loading data.
• Self-Described: events that contain the transform in the message itself.

Predefined

Predefined events require the Transform to be defined already within the platform. Transforms are directly linked to tables and define how data should be ingested. When using predefined events, the transform name is included in the x-hdx-transform header sent to the streaming end-point.

Default Transform

The user has the ability to mark a transform as the default transform for a table. This means that all streaming traffic for that table will be processed with that default transform, unless a different transform is selected by the HTTP header value.

Example :
x-hdx-transform: <pre-existing transform>

In the example below we are assuming:

• a project and table already exists called the_project.the_table
• an existing schema attached to that table named my_transform_schema that has:
  • timestamp defined as "type":"datetime","primary":true
  • a_metric defined as "type":"double"
  • a_dimension defeined as "type":"string".

$ curl -s \
  -H "x-hdx-table: the_project.the_table" \
  -H "x-hdx-transform: my_transform" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  https://demo.hydrolix.io/ingest/event -X POST -d '
  [
    {
       "timestamp": "2020-03-01 00:00:00 PST",
       "a_metric": 45.2,
       "a_dimension": "My first Dimension"
    },
    {
       "timestamp": "2020-03-01 00:00:15 PST",
       "the_metric": 12.6,
       "a_dimension": "My Second Dimension"
    }
]'

Self-Described

In some cases, an ingest transform may not yet exist in the system. In this case, it can be included in the HTTP Request document. In these cases, the HTTP request document will have two parts:

$ curl -s \
    -H 'content-type: <data type being passed>' \
    -H 'x-hdx-table: <project.table>' \
    -H 'Authorization: Bearer <token>' \
    https://demo.hydrolix.io/ingest/event -X POST -d 
    '{
      "transform": <ingest Transform doc>,
      "data": <either JSON or CSV data payload>
    }'

• transform - contains the Hydrolix transform telling the system how to interpret and manage the incoming data.

• data - the data that is to be inserted into the table.

For example, below we have defined the transform, with the data to be sent as JSON, and the values are provided as arrays.

$ curl -s \
     -H 'content-type: application/json' \
     -H 'x-hdx-table: <project.table>' \
     -H 'Authorization: Bearer <token>' \
     https://demo.hydrolix.io/ingest/event -X POST -d 
'{
  "transform": {
    "type": "json",
    "settings":{
      "output_columns": [
        {
          "name": "timestamp",
          "datatype": {
            "type": "datetime",
            "primary": true,
            "format": "2006-01-02 15:04:05 MST"
          }
        },
        {
          "name": "clientId",
          "datatype": {
            "type": "uint64"
          }
        },
        {
          "name": "clientIp",
          "datatype": {
            "type": "string",
            "index": true,
            "default": "0.0.0.0"
          }
        },
        {
          "name": "clientCityCode",
          "datatype": {
            "type": "uint32"
          }
        },
        {
          "name": "resolverIp",
          "datatype": {
            "type": "string",
            "index": true,
            "default": "0.0.0.0"
          }
        },
        {
          "name": "resolveDuration",
          "datatype": {
            "type": "double",
            "default": -1.0
          }
        }
      ]
    },
  }
  "data": [
    ["2020-02-26 16:01:27 PST", 29991, "1.2.3.4/24", 1223, "1.4.5.7", 1.234],
    ["2020-02-26 16:01:28 PST", 29989, "1.2.3.5/24", 9190, "1.4.5.7", 1.324],
    ["2020-02-26 16:01:28 PST", 29990, "1.2.3.5/24", null, "1.4.5.7", 12.34]
	]
}'

The data element above is an array containing individual messages. Each message is represented as an array, containing values in the order they were described in the schema. If a message in this particular batch contains a null or empty value, it will be replaced with the default value if one is declared.