Limiters

The transform datatype has an optional feature called limiting. This allows the taking of different actions on inspection of incoming values. A “limiter” is a structure that differs slightly depending on type. With the exception of a primary column, a limiter will either reject the row or clamp the value.

Numeric limiters

A numeric limiter has a min value, a max value, and an action keyword. For example:

"datatype": {
  "type": "uint64",
  "limits": {
    "min": 0,
    "action": "reject"
  }
}

The above will reject any incoming event with a negative value for this unsigned type. Any of min, max, or action may be omitted. If action is omitted, it defaults to reject. If either bound is omitted, it remains untested. The format is the same for all integral types and for floating point values.

String limiters

Like numeric limiters, a string has a min, a max, and an action. However, a string may also include pad, a padding string to use. Since strings are tested on length (not value), the padding character is used to bring a string up to the minimum length when clamping, whereas a long string will be truncated. So

"datatype": {
  "type": "string",
  "limits": {
    "min": 10,
    "max": 20,
    "pad": "foo",
    "action": "clamp"
  }
}

with the following inputs yields the following outputs:

Array limiters

Arrays work exactly like strings, except that the padding may be anything. I’m not sure how much type checking of padding characters goes on at this point, so this should be used with caution. Note that since this is a property of the datatype (not the column), the limiter can be pushed down as far into the type (into elements or further) as desired. Or, the limiter can work on the top-level array itself. For example,

"datatype": {
  "type": "array",
  "limits": {
    "max": 5,
    "action": "clamp"
  }
}

will truncate any arrays longer than 5 elements.

Datetime limiters

Datetime values (which epoch types also use) are a bit tricky. There are two sets of limits: absolute and relative, and these may be mixed and matched but are tested in a strict order. The full form looks like

"datatype": {
  "type": "datetime",
  "limits": {
    "past": "5m",
    "future": "10m",
    "min": "2022-09-13T12:00:00.123456789Z",
    "max": "2022-09-13T13:00:00.987654321Z",
    "action": "clamp"
  }
}

and limits are tested in the above-listed order. That is,

1. Relative past: is the value too far in the past?
2. Relative future: is the value too far in the future?
3. Absolute minimum: is the value before the min value?
4. Absolute maximum: is the value after the max value?

The relative bounds (past and future) are in Go’s duration format (magnitude-unit), so 3h2m1s is “three hours, two minutes, and one second”. The absolute bounds are in RFC3339 with nanosecond precision format.

Since all fields are optional, specifying:

"datatype": {
  "type": "datetime",
  "limits": {
    "future": "8766h"
  }
}

will reject any value over a year in the future. This is the only form that a primary column recognizes (all other fields are ignored for primary columns).

Maps, Tuples, and Bools limiters

There is support for limiting a bool column, but this has limited use. There is no support for limiting (at top-level) a map or tuple type, though limiters may appear as part of any embedded data type within those structured types.