Runtime determination of Arena size

[Strahbehry]

Hey Pierre,

We have an application that's very computationally intensive. But our customers can use a wide range of CPU's. Ranging from low-end i3's to some of the highest end i7's. We mostly see potential gains for the high-end i7's, especially the newer models that ship with Efficiency and Performance cores. Furthermore we also have some use cases where they run on Xeon processors but that's another story.

We also use a lot of TParallel.For's in our application and we see a lot of benefit on personal devices tweaking the block arena count. (few million contentions in a 100 second benchmark)

It's hard for us to ship with an one size fits all. Do you have any plans / ideas for determining these runtime?

Also for an example on my i7-12700h using:
CFastMM_SmallBlockArenaCount = 14;
CFastMM_MediumBlockArenaCount = 8;
CFastMM_LargeBlockArenaCount = 6;

Is about 8% faster.

[pleriche]

Hi Mitch,

Runtime adjustment of the number of arenas is on the to-do list. The plan is to allocate the arena management structures instead of using gobal variables for them. This way I can vary their number and also guarantee 64 byte alignment. I was delaying it in the hope that Delphi would add support for 64 byte alignment so it would not be necessary, but alas it has not happened. I'll make a note to revisit this again.

Do you find that specifying many arenas hurts performance on lower spec machines? I would expect it to mostly drive up address space usage, but performance should not be too badly impacted. This is what my tests have shown, but of course your workloads won't be exactly the same.

Pierre

[Strahbehry]

If you have specific requests for what I could test I wouldn't mind doing so.

For the lower spec machines, I'm noticing quite of an area within which results are comparable. As soon as I stop running into thread contention the performance stays about the same even if I add more arenas after. I have to roughly double the amount of arena's t start losing 1 or 2% performance. And have to go to something absurd like 100 arena's of each to lose my original gain. I'm seeing similar results with the higher spec machine, but it's a lot more sensitive, which makes sense because my benchmark is probably not challenging enough for my new laptop.

I don't have a device anymore with incredibly slow ram though which might play into this. For a random ballpark.

Intel Core i3-6157U:
Sweet spot hit at
Small - 4
Medium - 4
Large - 2

If I double small/medium/large I lose about 1.5%

Intel Core i7-12700h:
Sweet spot hit at
Small - 14
Medium - 8
Large - 3

I can basically also double them here and lose like 2%.

I'm basically seeing a performance increase that's steep and becomes more flat as I reach the sweet spot and after that I very slowly lose performance. You've put a recommendation of 0.5-1x amount of cores as arenas. My anecdotal suggestion would be that setting normal and medium to the amount of threads would be fine. Perhaps when you pass a count of 8 only increase the small by 1 for every 2 additional threads. And medium with 1 per 4 additional. And that the large arena's could be 1/5th only of the small/medium.

[pleriche]

Those percentages are quite small. I fear that the additional costs of dynamic management of arenas might counter any gains to be had.

However, I will see what I can do.

[amancini]

hello any news for this issue ?

[pleriche]

I have taken another look at this today. The biggest issue with dynamic adjustment of the number of arenas is that it can always only be in one direction - an increase. That is because once an arena has been used to allocate memory you have to keep that arena around. If you wanted to reduce the number of arenas then you would need to merge the blocks managed by the arenas that are about to fall away into the remaining arenas. That's not a trivial task.

The biggest cost of having an excessive number of arenas is a potential increase in internal fragmentation as things get busy: If an arena with available space is busy then another arena must be used, and if the only arenas that are available do not have space then it has to obtain some. The performance cost of managing an excessive number of arenas is generally quite low, in my tests less than 2%.

While I can add support for a dynamic increase in the number of arenas, it will inevitably affect performance a little bit - perhaps by half a percent or so (I'm guessing). That hit is close to the performance hit you get when just picking an excessive number of arenas, so then you have to wonder whether picking a static number of arenas that is at the upper limit of your use case wouldn't be a good enough solution. Performance will be slightly less than optimal on systems with too few cores to make use of all the arenas, but I don't expect it to be much worse than the dynamic solution.

[jdavison-mo]

If I may add my two cents to this, similar to @Strahbehry our application can be used with a wide range of workloads, so one size does not fit all, and by default I would not configure the arenas high unless the workload warranted it, and the collision count indicated it was necessary. Dynamically increasing arenas, during runtime seems reasonable without the need to downscale during the lifetime process. It means 1 executable can be released, rather than having multiple.

Perhaps a middle ground is to leave the existing conditional defines e.g. FastMM_5Arenas...FastMM_16Arenas, but allow another one FastMM_DynamicArenas? Then it could be left to the application programmer to decide the option that gives the best performance in their scenario.

The application programmer could even set up some logic that monitors the
FastMM_[Small/Medium/Large]BlockThreadContentionCount variables and decide to dynamically increase it on demand.

[pleriche]

I'm not keen on developing and maintaining a separate code path for dynamic arenas, but it could be a last resort. First prize would be to find a cheap enough implementation that can be enabled all the time.

The critical area is inside the small block allocator, and having the arena count as a constant simplifies things. I'll spend some more time playing with options, but I cannot make any promises at this stage.