Transform¶

The transform example builds on the filter example and extends the example.trickle by adding a transformation that modifies the incoming event. The produced event from this query statement has a different structure than the incoming event.

Environment¶

It connects to the pipeline example in the example.trickle file using the tremor script language to change the json for the log.

All other configuration is the same as per the passthrough example, and is elided here for brevity.

Business Logic¶

select {                                    # 1. We can inline new json-like document structures
    "hello": "hi there #{event.hello}",      # 2. Tremor supports flexible string interpolation useful for templating
    "world": event.hello
} from in where event.selected into out

Command line testing during logic development¶

Run a the passthrough query against a sample input.json

$ tremor run -i input.json ./etc/tremor/config/example.trickle
{"hello":"hi there world","world":"world"}

Change the input.json and toggle the selected filed to true and run again.

Deploy the solution into a docker environment

$ docker-compose up
>> {"hello":"hi there again","world":"again"}

Inject test messages via websocat

Note

Can be installed via cargo install websocat for the lazy/impatient amongst us

$ cat inputs.txt | websocat ws://localhost:4242
...

Discussion¶

Transformations in tremor query ( trickle ) can be any legal type / value supported by the tremor family of languages:

A boolean value
An integer
A floating point value
A UTF-8 encoded string
An array of any legal value
A record of field / value pairs where the field name is a string, and the value is any legal value

Examples¶

Templating percentile estimates from HDR Histogram¶

In this example, we restructure output from the tremor aggr::stats::hdr aggregate function and use string interpolation to generate record templates with a field naming scheme and structure this is compatible with tremor's influx data offramp.

A nice advantage of tremor, is that the business logic is separate from any externalising factors. However, one drawback with unstructured transformations is there is no explicit validation by schema supported by tremor out of the box - although, there are patterns in use to validate against external schema formats in use in production.

select {
  "measurement":  event.measurement,
  "tags":  event.tags,
  "timestamp": event.timestamp,
  "fields":  {
    # the following fields are generated by templating / string interpolation
    "count_#{event.class}":  event.stats.count,
    "min_#{event.class}":  event.stats.min,
    "max_#{event.class}":  event.stats.max,
    "mean_#{event.class}":  event.stats.mean,
    "stdev_#{event.class}":  event.stats.stdev,
    "var_#{event.class}":  event.stats.var,
    "p50_#{event.class}":  event.stats.percentiles["0.5"],
    "p90_#{event.class}":  event.stats.percentiles["0.9"],
    "p99_#{event.class}":  event.stats.percentiles["0.99"],
    "p99.9_#{event.class}":  event.stats.percentiles["0.999"]
  }
}
from normalize
into batch;

Tip

Not all tremor script ideoms are allowed in the select statement. Most notably we do not allow any mutating operations such as let or control flow such as emit or drop. Those constructs can however still be used inside a script block on their own.