C API: Add rocksdb_memory_allocator_create for custom allocators#14403
C API: Add rocksdb_memory_allocator_create for custom allocators#14403varun29ankuS wants to merge 2 commits intofacebook:mainfrom
Conversation
Add a callback-based factory function for creating custom MemoryAllocator instances through the C API. This follows the same pattern used by rocksdb_comparator_create, rocksdb_mergeoperator_create, and other callback-based C API wrappers (state + destructor + function pointers). The new API allows language bindings (Rust, Go, Python, Node.js) to plug in allocators like mimalloc, tcmalloc, or snmalloc at runtime, without requiring them to be linked at RocksDB build time. This is particularly important on Windows where jemalloc is not supported and the CRT allocator does not return freed memory to the OS. Resolves: facebook#14367
|
Hi @varun29ankuS! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
Platform-gated rocksdb dependency: - Linux/macOS: jemalloc feature replaces glibc/system malloc inside RocksDB's C++ code, preventing memory fragmentation and RSS bloat - Windows: unchanged (jemalloc incompatible with MSVC, awaits facebook/rocksdb#14403 for custom allocator API) Confirmed fix per rust-rocksdb maintainer (zaidoon1) in facebook/rocksdb#12364. PRs #989 and #1026 fixed jemalloc linking in rust-rocksdb v0.24.0.
…#102) Platform-gated rocksdb dependency: - Linux/macOS: jemalloc feature replaces glibc/system malloc inside RocksDB's C++ code, preventing memory fragmentation and RSS bloat - Windows: unchanged (jemalloc incompatible with MSVC, awaits facebook/rocksdb#14403 for custom allocator API) Confirmed fix per rust-rocksdb maintainer (zaidoon1) in facebook/rocksdb#12364. PRs #989 and #1026 fixed jemalloc linking in rust-rocksdb v0.24.0.
Summary
Add a callback-based
rocksdb_memory_allocator_create()to the C API, enablinglanguage bindings to plug in custom memory allocators (mimalloc, tcmalloc, snmalloc)
at runtime without requiring them to be linked at RocksDB build time.
Resolves #14367
Motivation
The C API currently only exposes
rocksdb_jemalloc_nodump_allocator_create().On Windows (MSVC), jemalloc is not supported, and the CRT allocator never
decommits freed heap segments — causing unbounded memory growth (#4112, #12364).
Language bindings (Rust, Go, Python) have no way to plug in an alternative.
Implementation
Follows the exact callback pattern used by
rocksdb_comparator_create,rocksdb_mergeoperator_create, androcksdb_compactionfilter_create:CustomMemoryAllocatorstruct indb/c.ccinheritingMemoryAllocatorallocate,deallocate,usable_size(optional),name(optional)state+destructorfor user-managed lifetimestd::shared_ptrvia existingrocksdb_memory_allocator_t::repChanges
include/rocksdb/c.h: Addrocksdb_memory_allocator_create()declarationdb/c.cc: AddCustomMemoryAllocatorproxy class and factory functiondb/c_test.c: Add test creating a custom allocator with malloc/free callbacks, wiring it to an LRU cacheunreleased_history/public_api_changes/custom_memory_allocator.md: Release noteTest Plan
StartPhase("custom_memory_allocator")test indb/c_test.c