Skip to content

Complex Data Types

Hydrolix supports storing and indexing arrays and maps.

Type Description Indexed by default? Indexing exceptions Stored as type Nullable
array An array of any one of the primitive or complex data types supported by Hydrolix No (can be enabled using configuration) Array of Arrays,
Array of Doubles, Array of Maps		 array Yes
map A map containing key/value pairs of primitive or complex data types supported by Hydrolix Yes Map of Arrays,
Map of Doubles,
Map of Maps		 map Yes

In the transform schema, these types have an additional elements object nested under the datatype property. See the example transform schemas for Maps and Arrays.

Elements object⚓︎

The elements object defines the structure of the map or array and the data types it contains. It uses the same settings as the parent object (type, format, index, and other fields) to define how data should be treated.

Map column⚓︎

A map column stores key-value pairs, where keys and values are each a consistent type across all entries in the map.

Map keys and values must be homogeneous types

While key type and value type can differ within a map, all keys and all values stored in a map must be of the same basic or complex type.

Invalid map schemas⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// cannot create a map which accepts both (String, String) and (String, Int32) data
[
  {
    "name": "invalid_map_heterogenous",
    "datatype": {
      "type": "map",
      "elements": [
        {
          "type": "string",
          "index": true
        },
        {
          "type": "string",
          "index": true
        },
        {
          "type": "string",
          "index": true
        },
        {
          "type": "int32",
          "index": true
        }
      ]
    }
  }
]

1
2
3
4
5
6
7
8
[
  {
    "invalid_map_heterogenous": {
      "key1": "value1",
      "key2": 2
    }
  }
]

Valid map schemas⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// can instead create two maps within the transform, one which accepts (String, String)
// and another that accepts (String, Int32) data
[
  {
    "name": "map_string_string",
    "datatype": {
      "type": "map",
      "elements": [
        {
          "type": "string"
        },
        {
          "type": "string"
        }
      ]
    }
  },
  {
    "name": "map_string_int32",
    "datatype": {
      "type": "map",
      "elements": [
        {
          "type": "string"
        },
        {
          "type": "int32"
        }
      ]
    }
  }
]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[
  {
    "map_string_string": {
      "key1": "value1"
    },
    "map_string_int32": {
      "key2": 2
    }
  }
]

Maps are defined as structures containing {key:value} pairs with the key and the value requiring discrete configurations. These example transform schemas define columns of type map.

Sample transform, maps⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
[
  {
    "name": "map_string_epoch",
    "datatype": {
      "type": "map",
      "elements": [
        {
          "name": "string_key",
          "type": "string"
        },
        {
          "name": "epoch_value",
          "type": "epoch",
          "format": "us",
          "resolution": "ms"
        }
      ]
    }
  }
]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[
  {
    "name": "map_string_array",
    "datatype": {
      "type": "map",
      "elements": [
        {
          "name": "string_key",
          "type": "string"
        },
        {
          "name": "array_value",
          "type": "array",
          "elements": [
            {
              "type": "string"
            }
          ]
        }
      ]
    }
  }
]

Sample query against map⚓︎

This example queries a map column of type map(string,uint64):

Map Query: uint64
1
select mymap['{key}'] from {column} where mymap['{key}'] = 6288

Arrays⚓︎

An array column stores an ordered collection of values, all of the same type. Arrays are useful for fields that represent multiple values. For example, tags, IP addresses, or event codes. Hydrolix supports arrays of all primitive types and some complex types.

Arrays aren't indexed by default. Indexing can be enabled to accelerate element lookup and whole-array comparison queries. See Indexing and Array operations for details.

Sample transform, arrays⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
[
  {
    "name": "tags",
    "datatype": {
      "type": "array",
      "elements": [
        {
          "type": "string"
        }
      ]
    }
  }
]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
[
  {
    "name": "error_codes",
    "datatype": {
      "type": "array",
      "elements": [
        {
          "type": "int32"
        }
      ]
    }
  }
]

Arrays can only store one type

Values stored in an array must all be of the same basic or complex type.

Array of Maps⚓︎

Hydrolix supports the use of the Array of Maps data type. These example transform schemas define columns of type array.

Sample transform, array of maps⚓︎

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
[
  {
    "name": "array_map_string_string",
    "datatype": {
      "type": "array",
      "elements": [
        {
          "name": "map_string_string",
          "type": "map",
          "elements": [
            {
              "type": "string"
            },
            {
              "type": "string"
            }
          ]
        }
      ]
    }
  }
]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
[
  {
    "name": "array_map_int_string",
    "datatype": {
      "type": "array",
      "elements": [
        {
          "name": "map_int_string",
          "type": "map",
          "elements": [
            {
              "type": "int32"
            },
            {
              "type": "string"
            }
          ]
        }
      ]
    }
  }
]

Indexing⚓︎

Hydrolix indexes columns by default in most cases. Turn off indexing of a column only when necessary. For more information, see the indexing at Hydrolix blog post.

Enable or disable indexing⚓︎

Set "index": true or "index": false in the output columns of the transform schema to enable or disable indexing. For more information about output columns, see Transform Structure. This example demonstrates how to enable or disable indexing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
"settings": {
   "output_columns": [
        {
            "name": "indexed_array_column",
            "datatype": {
                "type": "array",
                "index": true
            }
        },
        {
            "name": "not_indexed_array_column",
            "datatype": {
                "type": "array",
                "index": false
            }
        }
    ]
}

Array operations⚓︎

Supported index lookup operations⚓︎

Lookup Type Array Type Supported Functions
Array element lookup
(example: arr_column[2] = ‘v1’)		 String =, like, iLike, in, empty(), isNull(), isNotNull()
Integer =, >, >=, <, <=, in, isNull(), isNotNull()
Datetime =, !=, >, >=, <, <=, in, isNull(), isNotNull()
Array lookup
(example: arr_column = [‘v1’, ‘v2’, ‘v3’])		 All =, in, empty(), notEmpty(), hasAll(), hasAny(), has()