Auto Values

Auto Values

Hydrolix has some automatic values that can be included within a transform:

Auto Value Name

Description

Stream

Kafka

Batch

current_time

Current time

Yes

Yes

Yes

index_pool

Pool name used by the indexer (stream peer, kafka peer, batch peer)

Yes

Yes

Yes

input_filename

For batch only, specify the filename indexed. The filename is the full s3 path:
s3://bucket/folder/subfolder/filename.csv

No

No

Yes

intake_method

Specify the intake method (Batch, Stream / Kafka)

Yes

Yes

Yes

intake_pool

Pool name used by the intake service (stream head, batch head)

Yes

Yes

Yes

kafka_topic

Specify the Kafka topic used to index the data

Yes

No

No

project_name

Name of the project where the data will be indexed

Yes

Yes

Yes

project_revision

Revision number of the project where the data will be indexed

Yes

Yes

Yes

project_uuid

UUID of the project where the data will be indexed

Yes

Yes

Yes

receive_time

Time when the data is received

Yes

Yes

Yes

table_name

Name of the table where the data will be indexed

Yes

Yes

Yes

table_revision

Revision number of the table where the data will be indexed.

Yes

Yes

Yes

table_uuid

UUID of the table where the data will be indexed

Yes

Yes

Yes

transform_name

Name of the transform used to index the data

Yes

Yes

Yes

transform_revision

Revision number of the transform used to index the data

Yes

Yes

Yes

transform_type

Transform type (csv or json).

Yes

Yes

Yes

transform_uuid

UUID of the transform used to index the data.

Yes

Yes

Yes

input_filename_segment_[0-12]

Extract part of the filename with batch/autoingest where the separator is /
Example:
s3://bucket/folder/subfolder/filename.csv
s3 = input_filename_segment_0
bucket = input_filename_segment_2
folder = input_filename_segment_3
subfolder = input_filename_segment_4
filename.csv = input_filename_segment_5

No

No

Yes

To use those automatic value you can add the following in your transform:

{
    "name": "filename",
    "datatype": {
        "type": "string",
        "index": true,
        "source": {
            "from_automatic_value": "input_filename"
        }
    },
},

The source should be from_automatic_value and then the value name that you can find in the table above.


Did this page help you?