syncstorage-rs memory consumption

[erkolson]

We are seeing syncstorage pods killed/evicted in production due to consuming too much memory and starving the host. It has been difficult to pin down when/why this is happening, as the memory utilization reported by monitoring stats is always below 75%. The application must be allocating a large amount of memory in a very short period of time, otherwise, we would see the 100% memory utilization in monitoring stats.

Is it possible to bound the memory footprint of rust applications?

Are there metrics available from the application concerning memory allocation?

[erkolson]

We've had 6 pods evicted in the last 12 hours. This time, the metrics show abrupt memory utilization around the time the pods get evicted: