Skip to content

C API: Add rocksdb_memory_allocator_create for custom allocators#14403

Open
varun29ankuS wants to merge 2 commits intofacebook:mainfrom
varun29ankuS:c-api-custom-memory-allocator
Open

C API: Add rocksdb_memory_allocator_create for custom allocators#14403
varun29ankuS wants to merge 2 commits intofacebook:mainfrom
varun29ankuS:c-api-custom-memory-allocator

Conversation

@varun29ankuS
Copy link

Summary

Add a callback-based rocksdb_memory_allocator_create() to the C API, enabling
language bindings to plug in custom memory allocators (mimalloc, tcmalloc, snmalloc)
at runtime without requiring them to be linked at RocksDB build time.

Resolves #14367

Motivation

The C API currently only exposes rocksdb_jemalloc_nodump_allocator_create().
On Windows (MSVC), jemalloc is not supported, and the CRT allocator never
decommits freed heap segments — causing unbounded memory growth (#4112, #12364).
Language bindings (Rust, Go, Python) have no way to plug in an alternative.

Implementation

Follows the exact callback pattern used by rocksdb_comparator_create,
rocksdb_mergeoperator_create, and rocksdb_compactionfilter_create:

  • CustomMemoryAllocator struct in db/c.cc inheriting MemoryAllocator
  • Function pointers: allocate, deallocate, usable_size (optional), name (optional)
  • state + destructor for user-managed lifetime
  • Wrapped in std::shared_ptr via existing rocksdb_memory_allocator_t::rep

Changes

  • include/rocksdb/c.h: Add rocksdb_memory_allocator_create() declaration
  • db/c.cc: Add CustomMemoryAllocator proxy class and factory function
  • db/c_test.c: Add test creating a custom allocator with malloc/free callbacks, wiring it to an LRU cache
  • unreleased_history/public_api_changes/custom_memory_allocator.md: Release note

Test Plan

  • Added StartPhase("custom_memory_allocator") test in db/c_test.c
  • Test creates allocator → sets on LRU cache options → creates cache → destroys in order
  • Verifies no crashes or leaks through the callback dispatch path

Add a callback-based factory function for creating custom MemoryAllocator
instances through the C API. This follows the same pattern used by
rocksdb_comparator_create, rocksdb_mergeoperator_create, and other
callback-based C API wrappers (state + destructor + function pointers).

The new API allows language bindings (Rust, Go, Python, Node.js) to plug
in allocators like mimalloc, tcmalloc, or snmalloc at runtime, without
requiring them to be linked at RocksDB build time. This is particularly
important on Windows where jemalloc is not supported and the CRT allocator
does not return freed memory to the OS.

Resolves: facebook#14367
@meta-cla
Copy link

meta-cla bot commented Feb 28, 2026

Hi @varun29ankuS!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@meta-cla meta-cla bot added the CLA Signed label Feb 28, 2026
varun29ankuS added a commit to varun29ankuS/shodh-memory that referenced this pull request Mar 9, 2026
Platform-gated rocksdb dependency:
- Linux/macOS: jemalloc feature replaces glibc/system malloc inside
  RocksDB's C++ code, preventing memory fragmentation and RSS bloat
- Windows: unchanged (jemalloc incompatible with MSVC, awaits
  facebook/rocksdb#14403 for custom allocator API)

Confirmed fix per rust-rocksdb maintainer (zaidoon1) in
facebook/rocksdb#12364. PRs #989 and #1026 fixed jemalloc linking
in rust-rocksdb v0.24.0.
varun29ankuS added a commit to varun29ankuS/shodh-memory that referenced this pull request Mar 9, 2026
…#102)

Platform-gated rocksdb dependency:
- Linux/macOS: jemalloc feature replaces glibc/system malloc inside
  RocksDB's C++ code, preventing memory fragmentation and RSS bloat
- Windows: unchanged (jemalloc incompatible with MSVC, awaits
  facebook/rocksdb#14403 for custom allocator API)

Confirmed fix per rust-rocksdb maintainer (zaidoon1) in
facebook/rocksdb#12364. PRs #989 and #1026 fixed jemalloc linking
in rust-rocksdb v0.24.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

C API: Add callback-based custom MemoryAllocator (rocksdb_memory_allocator_create_custom)

1 participant