History service memory usage upward trend

[hafiz-qasim]

Hi Temporal team and community,

I’m trying to understand whether it’s realistically possible to run Temporal on Kubernetes in a way that keeps the history service memory usage stable and within limits, without eventually hitting OOM kills.

Environment
• Temporal version: 1.29.1
• Persistence: PostgreSQL
• Deployment: Kubernetes
• History service replicas: 1
• Shards: tried multiple values
• Monitoring: Grafana community dashboard for Temporal server

Load model
• Execute 100 workflows per minute, evenly distributed
• After each minute of execution: 1 minute sleep
• After 15 cycles, there is a 30-minute idle period
• This pattern repeats

What I’ve tried, adjusting:
• Number of shards
• History and events cache settings
• Database connection pool sizes
• Verifying via Grafana that traffic drops during idle periods

Observed behavior
• History pod memory usage continuously accumulates over time
• Memory does not stabilize or decrease, even during idle periods
• Eventually the history pod is OOM killed
• I don’t see a clear plateau in memory usage in Grafana

Main question
Has anyone successfully deployed Temporal on Kubernetes such that the history service memory remains stable over time (especially under cyclical or bursty workloads), without relying on restarts or OOM recovery?

If so:
• What configuration patterns made the difference?
• Is multiple history replicas required for memory stability?
• Are there known limitations or expected behavior around cache eviction / GC in this scenario?

I’d appreciate any guidance, references, or confirmation of whether this is expected behavior or a misconfiguration on my side.

Thanks in advance!

[andropler]

I'm also interested in this issue.
I was able to configure the history cache settings to handle traffic stably without OOM by setting the mutable cache in count units, the event cache in capacity units, and applying the GOMEMLIMIT setting. However, as soon as I set the mutable cache in capacity units, the history pod significantly exceeds the configured cache capacity, even surpasses the GOMEMLIMIT, and gets restarted due to OOM. This was on the latest version 1.29.1, and I'd like to know the solution to this issue.
I've been struggling with this issue for several weeks now, and although I posted about it in the community, I haven't received any particularly helpful responses.
Link: https://community.temporal.io/t/memory-oom-issues-with-history-pod-and-size-based-cache-configuration/18787
I would greatly appreciate it if the official Temporal developers could take an interest in this issue.