Rejects
During ingestion, Hydrolix sometimes receives incorrect or malformed data. You can specify a set of criteria for correct data and selectively ignore data that doesn't meet the criteria. We call these pieces of data rejects. Hydrolix stores rejected files and rows in a separate reject directory so you can review which data was rejected and, if necessary, adjust your transform to accommodate and re-ingest the rejected data.
Only values that come directly from a field in the incoming data can produce rejects. Values produced by a SQL transform are never rejected, but some conversion failures in SQL transform output discard a whole partition without writing a reject. See SQL transforms and rejects.
Storage Location⚓︎
Hydrolix stores reject files in cloud storage at the following location:
The following example shows 3 reject files in a GKE deployed architecture. They represent data rejected from a project with UUID 3470db40-bf27-44f9-bc0b-a0890ba2fea8 and a table with UUID 61d30656-dc13-49eb-ae44-234b37e2b2e4:
To find the UUIDs for your projects and tables, use the Get org summary endpoint.
To see individual messages with complete pathnames of reject files, set the log_level of the stream-head service to a log_level of info. You will find these messages in the stream-head service's logs. Note that this can cause high log output for your cluster, and should only be used temporarily. See more information about Log Levels.
File Format⚓︎
Hydrolix writes rejects files as JSON objects with the following format:
The project_id, table_id and transform_id record the UUIDs of the project, table, and transform the data was originally ingested into (before being rejected). When you use the default transform to ingest a piece of data, the transform_id field won't contain a value.
The data object contains the malformed data that Hydrolix rejected.
Finally the reason object contains the reason for the rejection. The following table contains some common failures with some additional details to help understand what failed:
| Failure | Reason |
|---|---|
| "strconv.ParseInt: parsing \"text\": invalid syntax" | The field has been given a string value when a numeric Int was expected. The error can contain the name of the column. Often it's easiest to search through the data to see what has been supplied and match this in the transform. |
| "strconv.ParseUInt: parsing \"unknown\": invalid syntax" | The field has been given a string value of "unknown" when and UInt was expected. This can happen with custom null values, to enable custom nulls use the null_values attribute within the output columns of your transform. |
| "reason": "unexpected EOF" | Often occurs when a message body arrives compressed and the compression isn't handled within the Transform. |
| "reason":"event primary too old" | Occurs when an event's primary timestamp is older than the table's cold_data_max_age_days. |
SQL transforms and rejects⚓︎
Hydrolix doesn't reject values produced by a SQL transform, because SQL transforms run after Hydrolix checks incoming values against their column types.
Hydrolix converts each value a SQL transform outputs to the column's type. When the value doesn't match that type, Hydrolix doesn't write a reject or report an error, and the stored value depends on the SQL transform output. For example, in a uint8 column, the string '1.5' becomes NULL, the number 1.5 is truncated to 1, and the number -1 wraps to 255. Numeric conversions follow ClickHouse type conversion rules, which truncate fractional values toward zero and wrap values that are outside the column's range.
Conversion failures that discard a partition⚓︎
The primary column and columns of type json, array, or map can't store NULL. If a SQL transform produces a value that can't be converted for one of these columns, Hydrolix can't index the partition that contains the row.
A partition collects rows for a table over a time window. It can include rows from many ingest requests and from every transform writing to that table. When indexing fails, Hydrolix discards the entire partition, including rows from other transforms, and doesn't retry it. No reject file is written for the discarded partition.
The failure is recorded only by the ingest service that built the partition, for example stream-head or intake-head. That service logs an error-level Sink error. message with the underlying database error in the indexer_error field, and increments the hdx_sink_error_count metric.