how to debug this

[De-Novo-Research]

Hello.

I am running a multithreaded application performing some regression on a many observations. The regression is an expectation maximization so is a little complex. I break that data into blocks and pass each block to a different thread. Each thread gets its own instance of the regression object that is doing the math.

If I run this in release mode, without fulldebug, the entire process completes and the max memory usage according to windows is 41 Mb.
If I run with full debug mode, the memory increases until I run out of memory. If I decrease the size of my dataset such that the regression completes (can only use a few thousand observations, of the million or so I need to do) then I exit the program, it takes a long time to exit, but no memory leaks are reported. I did a test to create a purposeful leak and the report did come up with the leak, so I know its working.

The thing is that this is being run in a dunitx test case. So when the test ends, windows is still saying 8GB of memory is used, but the testing GUI is setting there. All my objects are freed by the time the test completed, so closing down the GUI itself should not somehow "clean up" my memory in any way.

I am unclear how to debug this. My guess is somehow the logs associated with the full debug mode are increasing and increasing. But I cannot figure out why they would be doing this if after the regression completes there are no leaks reported.

[jdavison-mo]

There is a note in Usage Instructions section in FastMM5.pas and Readme.md which may answer your question. https://github.com/pleriche/FastMM5/blob/master/FastMM5.pas#L57

In debug mode freed memory blocks will be filled with the byte pattern $808080... so that usage of a freed memory block or object, as well as corruption of the block header and/or footer will likely be detected.

So it may be that you may not really have any leaks, it may be just that you create a lot of objects over the life of the program, and because FastMM doesn't really free the memory in debug mode, because it is keeping them as marked as freed, from a process perspective it appears to leak.

[De-Novo-Research]

@jdavison-mo - that might have something to do with it. I ran FastMM_LogStateToFile before and after the algorithm. There are one or two more heap allocated objects. Those all make sense. However, the main key difference seems to be

Before:
Usage Summary:
4443K Allocated
13092K Overhead
25% Efficiency

After
Usage Summary:
3357K Allocated
2363106K Overhead
0% Efficiency

The "Overhead" has grown tremendously. I have double checked the algorithm and minimized the number of objects constructed within the thread loops as much as I can. As far as I can tell, each thread is creating and freeing a few objects at the very beginning and end of the thread. In the part that is looping, no objects are being constructed. no strings are being used in the threads except exception messages (which are never raised as the algorithm is not crashing). Each of the 12 threads basically

1. Allocates some heap memory for its objects it needs
2. performs regression on its block of data (no new memory being allocated here, except some local var of simple types (integer float etc)
3. frees the objects created in step 1

Is there a way to track down what is happening with the "overhead"

[pleriche]

Hi,

When a medium block (between roughly 3K and 260K) is freed in debug mode, the block will never be coalesced with adjacent medium blocks that are not in use. This is in order to catch "use after free" errors.

I suspect you may have a fairly large list or dynamic array that is constantly growing and every time a new entry is added a new slightly larger block is allocated, and the old block is freed. Since the old freed block will always be smaller than any new request (due to coalescing being disabled) it will never be reused and so the overhead grows. There are ways to mitigate this: The key is to use ReallocMem or SetLength (if it is a dynamic array) instead of manually allocating a new block and moving the data.

You can confirm that the overhead is going into medium blocks by looking at the output of FastMM_GetMemoryManagerState, in particular the ReservedMediumBlockAddressSpace field.

You can also call FastMM_WalkBlocks and check the sizes of all free medium blocks. If you see multitudes of free medium blocks of various sizes then what I described above is likely what is happening.

I am looking into breaking debug mode into separate options, so debug mode is not an "all or nothing" affair. That would enable you to use all the other debug mode features while still allowing the coalescing of free medium blocks. It's not ready yet though.

[De-Novo-Research]

Due to all your help, I tracked down the issue. It turns out a 3rd party library was needlessly allocating/freeing memory within a tight loop of the optimization. It was doing so with a raw GetMem/FreeMem call so was not easy to track down.

I am not sure if this exists, but maybe an idea of having an event in FastMM5 like OnNewMemoryAllocated. Maybe this event is only available in FullDebugMode. The idea would be that in situations where you expect no memory to be allocated you could set this event, and set a breakpoint within your handler. Then, if the program breaks, you will get a call stack where your code (or 3rd party code) was allocating memory.