vector: [ansi_stripper] stripper doesn't strip color?

Hi,

ansi_stripper transform doesn’t remove ansi color from field?

For the following test:

[transforms.ansi_stripper_simple_yo]
  inputs = []
  type = "ansi_stripper"
  field = "message"
[[tests]]
  name = "ansi_stripper_simple_yo"
  [tests.input]
    insert_at = "ansi_stripper_simple_yo"
    type = "raw"
    value = "\t\r\\033[31mHelloWorld"
  [[tests.outputs]]
    extract_from = "ansi_stripper_simple_yo"
    [[tests.outputs.conditions]]
      "message.equals" = "HelloWorld"

It outputs:

check transform 'ansi_stripper_simple_yo' failed conditions:
  condition[0]: predicates failed: [ message.equals: "HelloWorld" ]
payloads (events encoded as JSON):
  input: {"message":"\t\r\\033[31mHelloWorld","timestamp":"2020-04-02T03:52:42.774856684Z"}
  output: {"message":"\\033[31mHelloWorld","timestamp":"2020-04-02T03:52:42.774856684Z"}

Maybe it’s a wanted behavior and we should write ansi_color_stripper? WDYT?

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (20 by maintainers)

Most upvoted comments

Regarding the octal escape characters, it’s actually working as intended. ansi_stripper is removing not printable and ANSI Escape sequences, so ansi_stripper actually doesn’t deal with any \ based escapes but with the actual characters like newline, tab, escape, etc. So it doesn’t remove \t, \r, \x1b, etc.

The confusion arouse from different approaches that ansi_stripper, Rust, and serde, have towards escape characters. Rust will in all but raw mode convert \t, \r, \x1b, etc. to the actual characters, so ansi_stripper will remove them, but won’t \033 since Rust wont convert it. And the issue with serde is covered with #2492 .

In short, ansi_stripper is stripping the color. There is no bug here, just a missing feature #2492 which only affects toml configuration, and because of that the test in the config is failing . As such, I’ll close this issue.

ktff on Apr 30, 2020