[improve][broker] Add counter to topic stats for delayed messages which exceed TTL at publish time

[zjxxzjwang]

Motivation

Currently, Pulsar Broker lacks a metric to track instances where delayed messages exceed their TTL. This results in messages set with delayed delivery times exceeding the TTL expiring before being consumed by users, with no mechanism to detect this occurrence. Therefore, a counter is needed to track such instances.

Modifications

1. Add a new member variable Rate exceedTTLDelayMessage = new Rate() to PersistentTopic to track the number of delayed messages exceeding the TTL.
2. Within the message sending logic of handleSend, add a check specifically for delayed messages whose delay time exceeds the TTL. If exceeded, call exceedTTLDelayMessage.recordEvent().
3. In TopicStatsImpl, add the exceedTTLDelayMessages variable to retrieve and display the count of delayed messages exceeding the TTL.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[Denovo1998]

Left some comments.

[zjxxzjwang]

@Denovo1998 Hello, could you please help me review again

[zjxxzjwang]

@Denovo1998 Hello, could you please help me review again

[lhotari]

Please check the review comments.
In addition, please revisit the PR title. It's currently [improve][broker]Increase the number of delayed messages sent whose delay time exceeds the TTL statistical metric which is not very clear. Something like [improve][broker] Add counter to topic stats for delayed messages which exceed TTL at publish time would be more accurate.

[zjxxzjwang]

@lhotari Hello, I have made the modifications. Can you help me review them

[lhotari]

Currently, Pulsar Broker lacks a metric to track instances where delayed messages exceed their TTL. This results in messages set with delayed delivery times exceeding the TTL expiring before being consumed by users, with no mechanism to detect this occurrence. Consequently, there is a significant risk of message loss.

The implementation doesn't seem to match the goal of the change ("track instances where delayed messages exceed their TTL").
It currently target the case where the message has already been expired when it gets published. How common is that case? What TTL value are you using?

[zjxxzjwang]

Currently, Pulsar Broker lacks a metric to track instances where delayed messages exceed their TTL. This results in messages set with delayed delivery times exceeding the TTL expiring before being consumed by users, with no mechanism to detect this occurrence. Consequently, there is a significant risk of message loss.

The implementation doesn't seem to match the goal of the change ("track instances where delayed messages exceed their TTL"). It currently target the case where the message has already been expired when it gets published. How common is that case? What TTL value are you using?

I have made the necessary modifications, and the current logic will not introduce additional overhead. Based on extensive user usage patterns, this counter is critical and highly valuable. Without it, users would remain unaware when messages expire due to delayed delivery exceeding the TTL. With this counter in place, we can implement alerting mechanisms to notify users and adjust the TTL accordingly.

[zjxxzjwang]

@lhotari Hello, I have made the modifications. Can you help me review them

[zjxxzjwang]

I have made the modifications. Please review them. I have broken down the method and reduced unnecessary metadata parsing. @lhotari