Skip to content

Preprocess sliding window #192

@ATATC

Description

@ATATC

BRaTS Results Summary (136 MB volumes, 4 modalities)

Method Time Speedup
Blosc2 mmap patch (nnUNet) 1.67s 4.55x
NIfTI + MONAI Thread 2.11s 3.61x
NIfTI read + crop patch 6.07s 1.25x
NIfTI + PyTorch (workers=0) 7.61s 1.00x
Blosc2 full read 20.17s 0.38x
NIfTI + PyTorch (workers=4) 51.09s 0.15x
  1. MIPCandy scenario (full volume): MONAI ThreadDataLoader is optimal - 3.61x faster than baseline
  2. nnUNet's real advantage: Not the data loader itself, but the ability to read only patches (8 MB) instead of full volumes (136 MB) via mmap - that's 17x less data per sample
  3. PyTorch multiprocess is counterproductive: Pickle serialization of 136 MB tensors creates massive IPC overhead (0.15x = 6.7x slower)
  4. Blosc2 full read is slow: Decompression overhead makes it slower than raw NIfTI for full volumes
  5. The mmap advantage: nnUNet preprocesses data to blosc2 format, then uses mmap to read only the required patch region directly - achieving 3.6x speedup over NIfTI read+crop

This confirms that for MIPCandy's current architecture (full volume loading), MONAI ThreadDataLoader is the correct solution.

Originally posted by @perctrix in #133

Metadata

Metadata

Labels

enhancementNew feature or requesttodoNew task or assignment

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions