Memory usage grows indefinitely when repeatedly calling TALibFFI.xxx (memory leak?)

[tmac]

Summary

When repeatedly calling TALibFFI.xxx in a long-running process, memory usage grows steadily and never stabilizes, suggesting a potential memory leak.

Environment

• OS: macOS 14.5
• Ruby version: 3.4
• ta-lib-ffi version: 0.3.0
• TA-Lib version: 0.6.4 (installed via Homebrew)

Reproduction

require 'ta_lib_ffi'

count = 0
100_000.times { 
  TALibFFI.ema([1,2,3,4,5,6,7,8,9,10,11,12,13,14], time_period: 14) 
  count         += 1
  rss           = `ps -o rss= -p #{Process.pid}`.to_i
  pointer_count = ObjectSpace.each_object(Fiddle::Pointer).count

  printf("\r%-60s", "Records: #{count}, Pointers: #{pointer_count}, Memory: #{rss} KB") 
}

• Pointer count climbs, then drops (suggesting GC is cleaning them up)
• However, overall RSS continues growing and never stabilizes

###Notes

• The same test with ta-lib-python shows no memory growth.
• Possibly related to TA-Lib or Fiddle holding references internally?

Would appreciate your thoughts. Thanks!

[azeemh]

memory management in ruby has this issue.

i use ruby on rails and script in ruby and love it but also use python.

in my rails server, i have to optimize all memory queries.

for example .find_in_batches instead of .all lets you cut memory usage on model queries.

according to this useful.codes article we might need explicit mem management?

https://useful.codes/foreign-function-interfaces-ffi-in-ruby/

also the latest python version handles streaming an indicator value which may be faster than a function when working with live streaming data, such as from a marketplace/exchange.

maybe we can stream the data and thus reduce memory allocations?

and does jemalloc give any benefit?

if this can match the cython implementation or even work faster that would be cool...

[Youngv]

On Windows, I do not see memory growth.

Environment:

• OS: Windows 11 (10.0.26100)
• Ruby: ruby 3.1.6p260 (2024-05-29 revision a777087be6) [x64-mingw-ucrt]
• ta-lib-ffi: 0.3.0
• TA-Lib: 0.6.4

Repro (uses sys-proctable to read working set):

require 'ta_lib_ffi'
require 'sys/proctable'
include Sys

count = 0
100_000.times {
  TALibFFI.ema([1,2,3,4,5,6,7,8,9,10,11,12,13,14], time_period: 14)
  count += 1
  proc_info = ProcTable.ps.find { |p| p.pid == Process.pid }
  memory_kb = proc_info.working_set_size / 1024
  pointer_count = ObjectSpace.each_object(Fiddle::Pointer).count
  printf("\r%-60s", "Records: #{count}, Pointers: #{pointer_count}, Memory: #{memory_kb} KB")
}

Observed example during the run:
Records: 13883, Pointers: 1251, Memory: 52536 KB

But I can reproduce memory growth on Ubuntu 24.04.

require 'ta_lib_ffi'
require 'sys/proctable'
include Sys

count = 0
page_size = 4096

100_000.times do
  TALibFFI.ema([1,2,3,4,5,6,7,8,9,10,11,12,13,14], time_period: 14)
  count += 1
  proc_info = ProcTable.ps(pid: Process.pid)
  memory_kb = proc_info.rss * (page_size / 1024)
  pointer_count = ObjectSpace.each_object(Fiddle::Pointer).count
  printf("\r%-60s", "Records: #{count}, Pointers: #{pointer_count}, Memory: #{memory_kb} KB")
end

Observed example:
Records: 10569, Pointers: 1251, Memory: 22772 KB

I may not be able to fully resolve a low-level FFI/allocator issue immediately, but I will try.