Static memory leak in ICudaEngine::createExecutionContext

[mariusv-github]

Describe the bug
Creating an IExecutionContext with ICudaEngine::createExecutionContext will induce a GPU memory leak from 0x400000 (4 MBytes for small NV example on the page "Using the TensorRT-RTX Runtime API") up to 0xE00000 (14 MBytes for a big Yolo11-X/ONNX model), function of network complexity (but independent from network precision float/half). Destroying the newly creating IExecutionContext will not free this block of GPU memory, and any other new creation of an IExecutionContext will not increase the memory leak.
This memory block can be release only by NV driver when calling cudaDeviceReset.

It looks like a "static" GPU memory allocation (pointer kept in a static variable or static attribute) in ICudaEngine::createExecutionContext when creating an IExecutionContext object in the depths of "tensorrt_rtx_1_1.dll" (not "tensorrt_onnxparser_rtx_1_1.dll", nor "cudart.dll") and non-released by IExecutionContext::~IExecutionContext destructor nor by ICudaEngine::~ICudaEngine destructor.

And the call to method IExecutionContext::enqueueV3 or IExecutionContext::executeV2 will eventually induce another static memory leak up to 0x600000 bytes (6 MBytes for Yolo11-X model, both methods produce the same memory leak) so the max total leak can be equal to 0x1400000 bytes (20 MBytes). Here there is no new object created (from the user point of view) so nothing to destroy... Same behavior as ICudaEngine::createExecutionContext (fixed size for a given network, dependent of network complexity but independent of data precision and released only by the final cudaDeviceReset).

Steps to reproduce
Take any TensorRT-RTX example (or personal code) creating an IExecutionContext and modify the main function:

size_t ShowGPUmem()
{
  size_t memFree, memTotal;
  if(cudaMemGetInfo(&memFree, &memTotal)==::cudaSuccess) printf("Free: %016I64X, Total: %016I64X\n", memFree, memTotal);
  return memFree;
}

int main()
{
  size_t init=ShowGPUmem();
  for(int i=0; i<5; i++) // the size of memory leak is constant and independent of number of loops
  try
  {
    // call any test creating an IExecutionContext with ICudaEngine::createExecutionContext: after-before == 0x400000 ... 0xE00000
    // call any test calling ICudaEngine::createExecutionContext and IExecutionContext::enqueueV3: after-before == 0x400000 ... 0x1400000
    // call any test neutralized before ICudaEngine::createExecutionContext : after-before will be == 0
  }
  catch(MyExcept_& e) { std::cerr<<e; }
  size_t before=ShowGPUmem();
  cudaDeviceReset();
  size_t after=ShowGPUmem();
  printf("cudaDeviceReset: %s, Memory leaks of ICudaEngine::createExecutionContext = %016I64X bytes\n", (after==init?"OK":"ERROR"), after-before);
  return 0;
}

Expected behavior
The IExecutionContext::~IExecutionContext destructor must release any GPU memory allocated during the creation of the object. And the user can have a way to release the extra GPU memory allocated when calling executeV2/enqueueV3 methods.

Environment (same results)

• TensorRT-RTX version: 1.1.1.26
• GPU: RTX 5090 Laptop / RTX 3080
• Operating system: W10 LTSC, W10 22H2, W11 LTSC 24H2, drivers NV 576.52, VS 15/16/17
• CUDA version: 12.9
• CPU architecture: Intel(R) Core(TM) Ultra 9 275HX

Screenshots

Additional context

[davemichael]

Thanks for filing. Do you know if you're using the runtime cache feature? We've seen something similar that happens when the runtime cache is enabled, but we hadn't observed this yet without the runtime cache. Either way it's a bug, just hoping to narrow down further. If you could provide a fully complete reproducer, that might help us (e.g. replace the "// call any test" with some minimal reproducing code.

Thank you for your help!

[mariusv-github]

Thanks for your fast answer. The first "small" example is your example from "https://docs.nvidia.com/deeplearning/tensorrt-rtx/latest/installing-tensorrt-rtx/runtime-api.html", just cleaned and corrected by me. I think it do not use runtime cache feature (as any basic example, it uses the default configuration).

Here is the small version of the code (only a header and main, ready to test, compatible with VS15/16/17). Header TrtHelper.h :

#ifndef _TRT_HELPER_
#define _TRT_HELPER_
#include "NvInfer.h"
#include "NvInferRuntime.h"
#include "NvOnnxParser.h"

#include <cassert>
#include <cstdlib>
#include <cuda_runtime.h>
#include <iostream>
#include <iomanip>
#include <memory>
#include <vector>
#include <fstream>
#include <sstream>
#include <algorithm>

//! helper collection to be used with any TensorRT-RTX application
namespace nvinfer1 {
//! TensorRT-RTX applications are responsible for implementing the nvinfer1::ILogger interface.
//! This is used to log messages from the TensorRT-RTX library.
class Logger: public ILogger
{
protected:
public:
  Logger(std::ostream& _os): os(_os) {};
  ~Logger() override = default;
  std::ostream& os;
private:
  static std::string severityToString(ILogger::Severity severity)
  {
    switch(severity)
    {
    case ILogger::Severity::kVERBOSE:        return "VERBOSE";
    case ILogger::Severity::kINFO:           return "INFO";
    case ILogger::Severity::kWARNING:        return "WARNING";
    case ILogger::Severity::kERROR:          return "ERROR";
    case ILogger::Severity::kINTERNAL_ERROR: return "INTERNAL_ERROR";
    default:                                 return "UNKNOWN";
    }
  }
  void log(ILogger::Severity severity, const char* msg) noexcept override
  { os<<std::setiosflags(std::ios::left)<<std::setw(13)<<severityToString(severity)<<": "<<msg<<'\n'; }
};

//! Enhance the basic Logger with a local ofstream (eventually)
class OStream: public Logger
{
private:
  bool ownOs=false;
public:
  OStream(): Logger(std::cout) {}
  OStream(std::ostream &_os): Logger(_os) {}
  OStream(const char* fn_log):Logger(*(new std::ofstream(fn_log))), ownOs(true) {}
  ~OStream() { if(ownOs) delete &os; }
};

//! Dim64 helper
class Dims64_: public Dims64
{
public:
  std::ostream& Show(std::ostream& os) const
  {
    if(nbDims==-1 && d[0]==0) return os<<"[invalid dims]";
    else if(nbDims==-1 && d[0]==-1) return os<<"[unknown rank]";
    else if(nbDims>0 && nbDims<=MAX_DIMS)
    {
      os<<'['<<d[0];
      for(unsigned i=1; i<nbDims; i++) os<<" x "<<d[i];
      os<<"] = ";
      int64_t sz=Size();
      return sz==-1 ? (os<<"dynamic size") : (os<<sz);
    }
    else return os<<"[??????]";
  }
  int64_t Size() const {
    if(nbDims<=0||nbDims>MAX_DIMS) return 0;
    int64_t sz=1;
    for(int32_t i=0; i<nbDims; i++)
      if(d[i]==-1) return -1;
      else sz*=d[i];
    return sz;
  }
};
//! DataType helpers
inline const char* GetTypeName(const DataType& type)
{
  const char* names[]={"FLOAT","HALF","INT8","INT32","BOOL","UINT8","FP8","BF16","INT64","INT4","FP4","E8M0","Unknown"};
  return names[std::min(int32_t(type),EnumMax<DataType>())];
}
inline uint32_t GetTypeBits(const DataType& type)
{
  uint32_t bits[]={32,16,8,32,8,8,8,16,64,4,4,8,0};
  return bits[std::min(int32_t(type), EnumMax<DataType>())];
}

} // namespace nvinfer1

inline std::ostream& operator<<(std::ostream& os, const nvinfer1::Dims64& dims) { return ((nvinfer1::Dims64_&)dims).Show(os); }
inline int64_t Size(const nvinfer1::Dims64& dims) {return ((nvinfer1::Dims64_&)dims).Size();}
inline std::vector<int64_t> SignificantSize(const nvinfer1::Dims64& dims)
{
  // keep only significant dimmensions
  std::vector<int64_t> _dims;
  for(int32_t j=0; j<dims.nbDims; j++) if(dims.d[j]!=1) _dims.push_back(dims.d[j]);
  if(_dims.empty()) _dims.push_back(1);
  return _dims;
}
//! printBuffer is used by some NV examples like MNIST
template <typename T> inline void printBuffer(std::ostream& os, const std::string& name, const T& buffer, size_t lineSz=0)
{
  os<<name<<": ";
  for(const auto& value:buffer) os<<((lineSz!=0 && ((&value-&buffer[0])%lineSz)==0)?"\n":" ")<<value;
  os<<'\n';
}
//! helper to be used as first line of a function or method
struct _BeginEnd
{
  const char* const func;
  std::ostream& os;
  _BeginEnd(std::ostream& _os, const char* _func):os(_os), func(_func) { os<<"************* Begin of "<<func<<" **************\n"; }
  ~_BeginEnd() { os<<"************* End of "<<func<<" **************\n"; }
};
#define BeginEndOs(os) _BeginEnd be(os,__FUNCTION__)
#define BeginEnd _BeginEnd be(std::cout,__FUNCTION__)

// Exception to be used as output stream while is thrown (don't add "\n" at the end of message):
// if(...) throw MyExcept << "Exception due of this thing equal to " << thing_value;
class MyExcept_
{
  const char *const file;
  unsigned line;
public:
  MyExcept_(const MyExcept_& e) noexcept: _os(e._os.str()), file(e.file), line(e.line) {}
  std::ostringstream _os;
  MyExcept_(const char *const _file="", unsigned _line=0): line(_line), file(_file) {}
  std::ostream& Show(std::ostream& os) const { return os<<"Error \""<<_os.str()<<"\" in file "<<file<<" at line "<<line<<"\n"; }
};
template <typename T> inline MyExcept_& operator<<(MyExcept_& e, const T& t) { e._os<<t; return e; }
inline std::ostream& operator<<(std::ostream& os, const MyExcept_& e) { return e.Show(os); }
#define MyExcept MyExcept_(__FILE__, __LINE__)

#ifndef TestCudaErr
inline bool _TestCudaErr(cudaError_t err, char const *const Expr, const char *const file, unsigned line, bool doExcept)
{
  if(err==cudaSuccess) return true;
  if(doExcept) throw MyExcept_(file,line)<<"Cuda Error #"<<int(err)<<" ("<<cudaGetErrorName(err)<<":"<<cudaGetErrorString(err)<<") "<<Expr;
#ifdef _IOSTREAM_
  std::cerr<<"Cuda Error #"<<int(err)<<" ("<<cudaGetErrorName(err)<<":"<<cudaGetErrorString(err)<<") "<<Expr<<" in file \""<<file<<"\" line "<<line<<"\n";
#else
  printf("Cuda Error #%lu (%s:%s) %s in file \"%s\" line %u\n", err, cudaGetErrorName(err), cudaGetErrorString(err), Expr, file, line);
#endif
  return false;
}
#define TestCudaErr(Expr) (_TestCudaErr(cudaGetLastError(),"before "#Expr,__FILE__, __LINE__,false) && _TestCudaErr((Expr),"after "#Expr,__FILE__, __LINE__,false))
#define TestCudaLastErr() _TestCudaErr(cudaGetLastError(),"before this point",__FILE__, __LINE__,false)
#define ThrowCudaErr(Expr) (_TestCudaErr(cudaGetLastError(),"before "#Expr,__FILE__, __LINE__,true) && _TestCudaErr((Expr),"after "#Expr,__FILE__, __LINE__,true))
#define ThrowCudaLastErr() _TestCudaErr(cudaGetLastError(),"before this point",__FILE__, __LINE__,true)
#define ReturnOnCudaErr(Expr) { if(!TestCudaErr(Expr)) return false; }
#endif

//! basic data buffer on host/device
template<typename T> class DataBuf
{
  DataBuf(const DataBuf&) {}
  DataBuf& operator=(const DataBuf&) { return *this; }
  virtual void Free()=0;
  virtual void Alloc(size_t _size)=0;
public:
  T* pBuf;
  size_t size; // size in Elements
  DataBuf(): pBuf(0), size(0) {}
  virtual ~DataBuf() { }
  bool resize(size_t _size, bool usePrec=true)
  {
    if(usePrec && _size<=size) return true;
    Free(); Alloc(_size);
    return true;
  }
  T* data() { return pBuf; }
  const T* data() const { return pBuf; }
  T& operator[](size_t idx) { if(idx>=size) throw MyExcept<<"Index override buffer size "<<idx<<" >= "<<size; return pBuf[idx]; }
  const T& operator[](size_t idx) const { if(idx>=size) throw MyExcept<<"Index override buffer size "<<idx<<" >= "<<size; return pBuf[idx]; }
};
template<typename T> class DDataBuf: public DataBuf<T>
{
  virtual void Free() { if(pBuf) { ThrowCudaErr(cudaFree(pBuf)); pBuf=0; }}
  virtual void Alloc(size_t _size) { if(_size) ThrowCudaErr(cudaMalloc((void **)&pBuf, (size=_size)*sizeof(T))); };
public:
  virtual ~DDataBuf() { Free(); }
};
template<typename T> class HDataBuf: public DataBuf<T>
{
  virtual void Free() { if(pBuf) { ThrowCudaErr(cudaFreeHost(pBuf)); pBuf=0; }}
  virtual void Alloc(size_t _size) { if(_size) ThrowCudaErr(cudaMallocHost((void **)&pBuf, (size=_size)*sizeof(T))); };
public:
  virtual ~HDataBuf() { Free(); }
};

//! helper to detect Gpu/NV memory leaks
inline size_t ShowGPUmem()
{
  size_t memFree, memTotal;
  if(cudaMemGetInfo(&memFree, &memTotal)==::cudaSuccess) printf("Free: %016I64X, Total: %016I64X\n", memFree, memTotal);
  return memFree;
}
inline int TestGpuMemLeak(int(*func)(), size_t times=1)
{
  size_t init=ShowGPUmem();
  for(int i=0; i<times; i++) func();
  size_t before=ShowGPUmem();
  cudaDeviceReset();
  size_t after=ShowGPUmem();
  printf("cudaDeviceReset: %s, Memory leaks of ICudaEngine::createExecutionContext = %016I64X bytes\n", (after==init?"OK":"ERROR"), after-before);
  return 0;
}

#endif //_TRT_HELPER_

and main.cpp:

#include "TrtHelper.h"
using namespace nvinfer1;
using namespace std;

// 1st example from "https://docs.nvidia.com/deeplearning/tensorrt-rtx/latest/installing-tensorrt-rtx/runtime-api.html"
// and modified by me to track the memory leak in ICudaEngine::createExecutionContext and IExecuteContext::executeV2/enqueueV3

// Build a CUDA engine and save it to an file <- NO memory leaks here
void buildEngine(const char* engineName, int64_t N, std::ostream& os)
{
  Logger logger(os);
  // Create builder
  auto builder = createInferBuilder(logger);
  auto network = builder->createNetworkV2(0U);
  // Input tensor of shape {1, 1, N, N}
  auto input = network->addInput("input", DataType::kFLOAT, Dims4{1,1,N,N});
  // Prepare convolution weights for NxN kernel, 1 input, 1 output channel
  std::vector<float> weightValues(N*N, 1.f);
  std::vector<float> biasValues(1, 0.f);
  Weights W{DataType::kFLOAT, weightValues.data(), N*N};
  Weights B{DataType::kFLOAT, biasValues.data(), 1};
  // Add convolution layer
  auto conv = network->addConvolutionNd(*input, 1, DimsHW{N,N}, W, B);
  conv->setStrideNd(DimsHW{1,1});
  // Mark network output
  auto output = conv->getOutput(0);
  output->setName("output");
  network->markOutput(*output);
  // Build the engine and serialize it in one step
  //builder->setMaxBatchSize(1); // incompatible with TensorRT-RTX, works only with TensorRT, TODO NV: change original example in TensorRT-RTX doc
  auto config = builder->createBuilderConfig();
  //config->setMaxWorkspaceSize(1<<20); // incompatible with TensorRT-RTX, works only with TensorRT, TODO NV: change original example in TensorRT-RTX doc
  auto engine = builder->buildSerializedNetwork(*network, *config);
  // Save the serialized engine to a file
  std::ofstream out{engineName, ios::binary};
  out.write((char*)engine->data(), engine->size());
  out.close();
  // Clean up
  delete engine;
  delete network;
  delete config;
  delete builder;
}

// Load the engine and perform inference
void performInference(char const* engineName, const vector<float>& inputData, std::ostream& os)
{
  Logger logger(os);
  auto runtime = createInferRuntime(logger);
  // Deserialize the engine from file
  ICudaEngine* engine{};
  ifstream is(engineName, ios::binary);
  is.seekg(0, is.end);
  size_t nbBytes = is.tellg();
  is.seekg(0, is.beg);
  vector<uint8_t> data(nbBytes);
  is.read((char*)&data[0], nbBytes);
  is.close();
  engine = runtime->deserializeCudaEngine(&data[0], nbBytes);
#if 1 // here executeV2 or enqueueV3 don't produce memory leaks
  // Create an execution context for inference
  auto context = engine->createExecutionContext(); // <-- here first memory leak of 0x400000 bytes
  void* bindings[2];
  size_t inputSize = inputData.size();
  size_t outputSize = 1;
  size_t inputNbBytes = inputSize*sizeof(float);
  size_t outputNbBytes = outputSize*sizeof(float);
  ThrowCudaErr(cudaMalloc(&bindings[0], inputNbBytes));
  ThrowCudaErr(cudaMalloc(&bindings[1], outputNbBytes));
  // Populate input binding on device
  ThrowCudaErr(cudaMemcpy(bindings[0], &inputData[0], inputNbBytes, cudaMemcpyHostToDevice));
#if 0 //asynchronous execution: NO memory leak here, only for more complex networks
  cudaStream_t stream;
  ThrowCudaErr(cudaStreamCreate(&stream));
  for(size_t i=0; i<2; i++) context->setTensorAddress(engine->getIOTensorName(i), bindings[i]);
  context->enqueueV3(stream);
  ThrowCudaErr(cudaStreamSynchronize(stream));
  ThrowCudaErr(cudaStreamDestroy(stream));
#elif 1 // synchronous inference: NO memory leak here, only for more complex networks
  context->executeV2(bindings);
#else  // do nothing
#endif
  // Copy result into host memory space
  float result;
  ThrowCudaErr(cudaMemcpy(&result, bindings[1], outputNbBytes, cudaMemcpyDeviceToHost));
  os<<"Result = "<<result<<'\n';
  ThrowCudaErr(cudaFree(bindings[0])); ThrowCudaErr(cudaFree(bindings[1])); // TODO NV: add it to the TensorRT-RTX doc !!!
  delete context;
#endif
  delete engine;
}

int test1()
{
  const char* engName=__FUNCTION__".trt";
  const int64_t N=316; // N*N < 100000 !!!
  std::ostream& os(std::cout);
  try {
    buildEngine(engName, N, os); // build and save
    vector<float> inputData(size_t(N*N), 2.0f);
    performInference(engName, inputData, os); // load and execute
  } catch(MyExcept_& e) { std::cerr<<e; }
  return 0;
}

//int testHelloWorld(int argc, const char** argv);
//int testMNIST();
//int testApiUsage();
//int testYolo11_v1();

int main(int argc, const char* argv[])
{
  //return test1();
  //return testHelloWorld(argc, argv);
  //const char* args[]={"","--onnx=helloWorld.onnx"}; return testHelloWorld(2, args);
  //return testMNIST();
  //return testApiUsage();
  return TestGpuMemLeak(test1); // constant memory leak of 0x40000 bytes in createExecutionContext, independent of N
  //return TestGpuMemLeak(testYolo11_v1); // constant memeory leak of 0xE00000+0x600000  bytes in createExecutionContext and enqueueV3/executeV2
}

To get the correct values, don't launch nor use any other application, tool, profiler or debugger using the GPU memory.

Then, I tried compute-sanitizer but it's not very useful here, it seems it only sees the tip of the iceberg. In this case it finds only 2 small memory leaks (4 and 15744 bytes), maybe the rest is allocated dynamically in Cuda kernels (inside TensorRT-RTX DLL library):

========= Leaked 4 bytes at 0xdf6e00000
=========     Saved host backtrace up to driver entry point at allocation time
=========         Host Frame: cuMemHostAlloc [0x7fff5d00d5f5] in nvcuda64.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b3f6035] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b3e1a26] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b3deb4d] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b5b4ee6] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b5b5c9c] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6d00f0] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6cad1e] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6d3770] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6d2d24] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b53a501] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b408781] in tensorrt_rtx_1_1.dll
=========         Host Frame: getInferLibVersion [0x7fff4b067292] in tensorrt_rtx_1_1.dll
=========         Host Frame: getInferLibVersion [0x7fff4b0803c4] in tensorrt_rtx_1_1.dll
=========         Host Frame: fmt::cudnn_fmt_v7::detail::uint128_wrapper::uint128_wrapper [0x7fff4b023a84] in tensorrt_rtx_1_1.dll
=========         Host Frame: fmt::cudnn_fmt_v7::detail::uint128_wrapper::uint128_wrapper [0x7fff4b019ea0] in tensorrt_rtx_1_1.dll
=========         Host Frame: fmt::cudnn_fmt_v7::detail::uint128_wrapper::high [0x7fff4b006d18] in tensorrt_rtx_1_1.dll
=========         Host Frame:  [0xfa02] in netProc.exe
=========         Host Frame:  [0x11980] in netProc.exe
=========         Host Frame:  [0x8755] in netProc.exe
=========         Host Frame:  [0x1d960] in netProc.exe
=========         Host Frame: BaseThreadInitThunk [0x7fffa87cdbe7] in KERNEL32.DLL
=========         Host Frame: RtlUserThreadStart [0x7fffa9385a6c] in ntdll.dll
=========
========= Leaked 15 744 bytes at 0x131b000000
=========     Saved host backtrace up to driver entry point at allocation time
=========         Host Frame: cuMemAlloc_v2 [0x7fff5d00c4e5] in nvcuda64.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b3f6085] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b3e1d61] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b3df0de] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b5b4e2c] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b5b5c9c] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6d00f0] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6cad1e] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6d3770] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b6d2d24] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b53a501] in tensorrt_rtx_1_1.dll
=========         Host Frame: getBuilderPluginRegistry [0x7fff4b408781] in tensorrt_rtx_1_1.dll
=========         Host Frame: getInferLibVersion [0x7fff4b067292] in tensorrt_rtx_1_1.dll
=========         Host Frame: getInferLibVersion [0x7fff4b0803c4] in tensorrt_rtx_1_1.dll
=========         Host Frame: fmt::cudnn_fmt_v7::detail::uint128_wrapper::uint128_wrapper [0x7fff4b023a84] in tensorrt_rtx_1_1.dll
=========         Host Frame: fmt::cudnn_fmt_v7::detail::uint128_wrapper::uint128_wrapper [0x7fff4b019ea0] in tensorrt_rtx_1_1.dll
=========         Host Frame: fmt::cudnn_fmt_v7::detail::uint128_wrapper::high [0x7fff4b006d18] in tensorrt_rtx_1_1.dll
=========         Host Frame:  [0xfa02] in netProc.exe
=========         Host Frame:  [0x11980] in netProc.exe
=========         Host Frame:  [0x8755] in netProc.exe
=========         Host Frame:  [0x1d960] in netProc.exe
=========         Host Frame: BaseThreadInitThunk [0x7fffa87cdbe7] in KERNEL32.DLL
=========         Host Frame: RtlUserThreadStart [0x7fffa9385a6c] in ntdll.dll
=========

I can provide you my full project with the big example using Yolo11 models (memory leaks go up to 0x1400000 bytes) if you provide me a private space to upload all the stuff (code, binaries and models)...

Thank you Michael for your support and happy debugging!

[vincejhan98]

Hi @mariusv-github, thank you for bringing this issue up and providing the details. We have addressed and fixed this issue in our latest release, TensorRT for RTX v1.2. See download link: https://developer.nvidia.com/tensorrt-rtx

Please let us know if the issue is resolved, so we can close this issue thread. Thanks again!

[mariusv-github]

Hi @vincejhan98,

Thank you for your answer.
I summarize here the measured memory leaks on different networks using different TensorRT versions and the same source code:

Model / Total leaks	RT 10.13.3.9	RT-RTX 1.1.1.26	RT-RTX-1.2.0.54
Small RT-RTX example	0x1800000	0x0400000	0x0400000
Yolo11-X Pose FP16/32	0x3600000	0x1400000	0x0E00000
Sam2.0 hiera large FP32	0x5600000-0x5E00000	0x2200000	0x1C00000
Sam2.1 hiera large FP16	0x4400000-0x5200000	0x2200000	0x1C00000

Model / RT 10.13.3.9	createExecCtx	enqueueV3/execV2	TOTAL
Small RT-RTX example	0x1800000	0x0000000	0x1800000
Yolo11-X Pose	0x3600000	0x0000000	0x3600000
Sam2.0 hiera large [MIN]	2*0x2B00000	0x0000000	0x5600000
Sam2.0 hiera large [MAX]	2*0x2D00000	0x0400000	0x5E00000
Sam2.1 hiera large [MIN]	2*0x2200000	0x0000000	0x4400000
Sam2.1 hiera large [MAX]	2*0x2600000	0x0600000	0x5200000

Model / RT-RTX 1.1.1.26	createExecCtx	enqueueV3/execV2	TOTAL
Small RT-RTX example	0x0400000	0x0000000	0x0400000
Yolo11-X Pose	0x0E00000	0x0600000	0x1400000
Sam2.0 hiera large FP32	2*0x0E00000	0x0600000	0x2200000
Sam2.1 hiera large FP16	2*0x0E00000	0x0400000	0x2000000

Model / RT-RTX-1.2.0.54	createExecCtx	enqueueV3/execV2	TOTAL
Small RT-RTX example	0x0400000	0x0000000	0x0400000
Yolo11-X Pose	0x0E00000	0x0000000	0x0E00000
Sam2.0 hiera large FP32	2*0x0E00000	0x0000000	0x1C00000
Sam2.1 hiera large FP16	2*0x0E00000	0x0000000	0x1C00000

The small example uses the code above, Yolo11 uses the pose X version and there is no difference between FP32 and FP16, while the Sam2.X uses the Hiera large version and it is split in video encoder and video decoder so it calls createExecutionContext twice (for those 2 different networks), providing more memory leaks.

• Both RT-RTX versions provide a very small parse and compiling times with respect of all RT 10.x versions.
• Memory leaks produced by RT-RTX versions are always reproducible while those produced by RT version have a random range for large networks as SAM 2.X.
• RT 10.13.3.9 generates the largest memory leaks while the last RTX version (1.2.0.54) generates the smallest memory leaks.
• Inference memory leaks (when calling enqueueV3/executeV2) are triggered only by large networks while the createExecutionContext produces always memory leaks (larger for larger networks).
• RT-RTX-1.2.0.54 produces the sames memory leaks as RT-RTX 1.1.1.26 when calling ICudaEngine::createExecutionContext
• RT-RTX-1.2.0.54 has no more memory leaks when calling enqueueV3/executeV2 methods.

Conclusion: the version RT-RTX-1.2.0.54 corrected the enqueueV3 and executeV2 methods (no more memory leaks here) BUT the createExecutionContext keeps the memory leak bug as before.

I remain at your disposal for further details.