[confgenerator] Add parse_multiline Otel Logging experimental support.#2110
Open
franciscovalentecastro wants to merge 23 commits intomasterfrom
Open
[confgenerator] Add parse_multiline Otel Logging experimental support.#2110franciscovalentecastro wants to merge 23 commits intomasterfrom
parse_multiline Otel Logging experimental support.#2110franciscovalentecastro wants to merge 23 commits intomasterfrom
Conversation
franciscovalentecastro
changed the title
parse_multiline experimental support.parse_multiline Otel Logging experimental support.
Contributor
Author
|
Closing draft. I'll reopen later after discussing implementation details. |
franciscovalentecastro
requested review from
a team,
braydonk,
jefferbrecht and
jinghan-ma
and removed request for
a team and
braydonk
LujieDuan
reviewed
Mar 6, 2026
Comment on lines
+200
to
+202
| continue | ||
| } | ||
| return nil, errors.New("unsupported in otel") |
Contributor
There was a problem hiding this comment.
Suggested change
| continue | |
| } | |
| return nil, errors.New("unsupported in otel") | |
| } |
yaml validation should reject anything else
| // Below is the working java rules provided by fluentbit team: https://github.com/fluent/fluent-bit/issues/4611 | ||
| // Move to built-in java support, when upstream fixes the issue | ||
| "java": []MultilineRule{ | ||
| {"start_state, java_start_exception", `/(?:Exception|Error|Throwable|V8 errors stack trace)[:\r\n]/`, "java_after_exception"}, |
Contributor
There was a problem hiding this comment.
Just noticed that we actually never used java_start_exception state - we replaced it with java_nested_exception
| secondExprSlice = append(secondExprSlice, "("+strings.Join(expr, " and ")+")") | ||
| } | ||
| } | ||
| firstExpr := strings.Join(firstExprSlice, " or ") + " or (" + strings.Join(secondExprSlice, " and ") + ")" |
Contributor
There was a problem hiding this comment.
I am not sure I follow this line:
F - first line exp joined by OR, S - second line exp joined by AND
does this line gives us
(F and S) OR S?
is this special handling for cases where user specify multiple languages?
Comment on lines
+221
to
+229
| javaTransitionsExpr := strings.Join(transitionLinesExpr["java"], " and ") | ||
| pythonTransitionsExpr := strings.Join(transitionLinesExpr["python"], " and ") | ||
| transitionsExprSlice := []string{} | ||
| if len(transitionLinesExpr["java"]) > 0 { | ||
| transitionsExprSlice = append(transitionsExprSlice, javaTransitionsExpr) | ||
| } | ||
| if len(transitionLinesExpr["python"]) > 0 { | ||
| transitionsExprSlice = append(transitionsExprSlice, pythonTransitionsExpr) | ||
| } |
Contributor
There was a problem hiding this comment.
Can we loop over language here?
for lang, expr := range transitionLinesExpr {
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is "close enough" implementation of the
parse_multilineprocessor that uses Fluent-bit state machine solutions. The Otel recombine uses ais_first_entryoris_last_entry(only one can be set) to create multi-line log groups.Limitations
Summary : The solution works nicely with individual languages
go,pythonorjavaused separately. Since there are possible overlapping of the format of "stacktrace" log lines, the interactions of setting multiple languages may be to fragile to implement. A "state machine" solutions avoids this issues since being in ago,pythonorjavalanguage starting state reduces the possible "continuing" states.The current implemented solution tries to separate all possible log lines into 4 sets / categories :
Java Log EntryJava First Line Log Entryto start a multiline group.nestedmutli-line groups.Python Log EntryPython First Line Log Entryto start a multiline group.nestedmutli-line groups.Go Log EntryGo First Line Log Entryto start a multiline group.nestedmutli-line groups.Not a Go or Python or Java Log EntryThis solution depends completely in how accurate an descriptive are the "regex" patterns that describe each category. The main edge case that won't solvable (or may be too fragile) is the interactions between languages.
For example, using multiple languages causes a lot of edge cases.
Related issue
b/413446918
How has this been tested?
Checklist: