Fix the AsyncLoggerConfig ring buffer size property name in the manual#4038
Conversation
The docs for the AsyncLoggerConfig properties, which are relevant for mixed sync and async loggers, specify the wrong property name for the config that controls the Disruptor ring buffer size. Interestingly the right property name is used in the anchor ID linking to the exact section in the docs, but if you copy the property name from the docs themselves (like me), you are gonna have a bad time. I found this after noticing a too big ring buffer in a heap dump, and ended up chasing it with the StatusLogger debug logging and the log line in AsyncLoggerConfigDisruptor.start dumping the configured size. The small fix was also verified this way. In hindsight, this should have been obvious, but it really wasn't (for me).
vy
left a comment
There was a problem hiding this comment.
@dimitarndimitrov, thanks so much for the fix.
Would you mind elaborating on your use case a bit, please? In particular,
- Why do you use async. loggers? If it is for performance reasons, was your decision justified with performance figures?
- Why do you mix sync. and async. loggers?
- What kind of an application do you use this configuration for?
AsyncLoggerConfig ring buffer size property name in the manual
|
Thanks for the review, @vy ! Regarding your questions, our use-case is for a relatively new, asynchronous event loop subsystem, which is part of a bigger, pre-existing, Java NIO system that's a node in a distributed system - you can imagine something like an Apache Cassandra node or an Apache Kafka broker. We could see the event loops stuck logging for more than our tolerable window in some rare, but consistently behaving cases, and asynchronous logging seemed like a great fit. After enabling mixed-mode asynchronous logging, we could see a nice little improvement in these cases for the new subsystem, visible in both tail latency figures and CPU flamegraphs. Enabling asynchronous logging for the whole system however would require more validation, especially as in some deployment form factors there are indications that the sole logger thread and maybe more importantly, the sole disruptor ring buffer, might be a bigger issue than what we currently have. |
|
@dimitarndimitrov, CI is stuck, and I am not able to give it a kick. Would you mind pushing some changes (e.g., fix a typo 😅, merge changes from
There are several ways to achieve this, and they are all covered in Asynchronous logging. Have you also experimented with asynchronous appenders? If so, have you also tried customizing its queue? |
Sure, pushed an empty commit, and it looks like it's now waiting for your approval to run again.
No, we haven't tried asynchronous appenders, because their blocking queue component didn't seem to fit our event loop use-case that well. |
Thanks — approved it.
Note the customizing its queue link I've shared earlier. It explains how you can replace its queue backend, and there are non-blocking alternatives. Would it be possible to give this a try and share the outcome with us, please? I'm insisting on this subject, because asynchronous loggers constitute the biggest complexity in the entire Log4j code base. Judging from my personal experience, many users choose this setup because they want their logging backend to be "fast". Though many times this decision lacks rigorous experiments backed with numbers, and exclude simpler alternatives, e.g., using asynchronous appender in combination with a Conversant Disruptor queue. In the long run, we really want to move away from this complexity, or, at least, confine it to an isolated module. We need help from community to drive this simplification effort. |
The docs for the AsyncLoggerConfig properties, which are relevant for mixed sync and async loggers, specify the wrong property name for the config that controls the Disruptor ring buffer size. Interestingly the right property name is used in the anchor ID linking to the exact section in the docs, but if you copy the property name from the docs themselves (like me), you are gonna have a bad time.
I found this after noticing a too big ring buffer in a heap dump, and ended up chasing it with the
StatusLoggerdebug logging and the log line inAsyncLoggerConfigDisruptor.startdumping the configured size. The small fix was also verified this way.In hindsight, this should have been obvious, but it really wasn't (for me).
Checklist
2.xbranch if you are targeting Log4j 2; usemainotherwise./mvnw verifysucceeds (the build instructions)src/changelog/.2.x.xdirectory