ANSI control character in captured strings break MDSD ingestion #79
Description
ANSI control characters that may be used to colorize a TTY, when captured by fluentd and forwarded to mdsd cause the ingestion to fail with an ACK_DECODE_ERROR.
Such strings are for instance seen when containers workloads such as bitnami/openresty docker images start up and log strings such as the following to stdout/stderr:
\u001b[38;5;6mopenresty \u001b[38;5;5m21:25:41.40 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Initializing OpenResty
Note that the above representation is an encoded string that escapes the ANSI control characters.
In order to avoid ingestion to fail, custom filters such as the following have to be added to sanitize the string.
<filter *.*>
@type record_transformer
enable_ruby
<record>
log ${ record["log"].gsub(/[\u001b\u009b][\[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/, '') }
</record>
</filter>As a user of the mdsd plugin I expect the plugin to sanitize or escape any invalid characters out from an FT_STRING before it is passed to mdsd based on what mdsd deems valid, without requiring custom filters to be written.