Multi-Patient Dataset Export System

[LucaLumetti]

Overview

Implement a feature allowing researchers to export patient data in bulk. Users can select multiple patients, choose which medical imaging modalities to include, and download the result as a structured ZIP file.
The system must also track all exports, enabling users to download or manage them later.

Goal

Deliver a complete export system with three parts:

1. Export Configuration Page – User interface for export setup
2. Backend Processing – Server-side logic for data gathering and ZIP creation
3. Export Management Page – Displays past exports, download links, and management tools

---

Technical Architecture

Data Model

Create a new model, e.g. DatasetExport, to represent export jobs.

Fields:

• User (creator)
• Status (pending, processing, completed, failed)
• Filter criteria (patients, modalities, folders)
• ZIP file path, size, hash
• Creation and completion timestamps
• Error message (if any)
• Others info that might be important

Follow existing patterns from FileRegistry for consistency.

---

Frontend

Export Configuration Page

A new page where users can define what to export.

Patient Selection

• Tree view of patient folders
• Select folders or individual patients
• Filter by tags
• Dynamic patient count

Modality Selection

• Checkboxes for modalities (CBCT, IOS, intraoral-photo, etc.)
• Option to include raw and processed files
• Show estimated export size

Export Options

• Export name (auto-generated, editable)
• Optional notes
• Submit to create export job

---

Backend

API Endpoint

POST /api/<project_slug>/exports/create/

Steps:

1. Validate Request

   • Check user permissions
   • Verify modalities exist
   • Enforce size limits

2. Create Export Job

   • Save new DatasetExport with status='pending'
   • Return export ID immediately

3. Asynchronous Processing

   • Use background job system to handle file gathering and ZIP creation

Processing Outline:

export_2025-10-20_143022/
  patient_001/
    ios/
      upper.stl
      lower.stl
    intraoral-photo/
      photo_1.jpg
      photo_2.jpg
    cbct/
      scan.nii.gz
  patient_002/
    ...
  metadata.json

Steps:

• Collect matching files from FileRegistry
• Copy them into a structured temp directory (or find a better solution that doesn't make a full copy of everything)
• Generate metadata.json with:
  • Export metadata (creator, date, patients, modalities)
  • File checksums
• Zip the directory
• Update DatasetExport with file path and status
• Remove temp files

---

Error Handling

• On failure: mark as failed, log details in DatasetExport
• Include readable error messages for debugging

---

Performance

• Background jobs prevent request timeouts
• (optional) Account for very large exports (e.g., 1000+ patients with CBCTs)
• Monitor disk space and enforce quotas

---

Security

• Strict permission checks for all patients
• Prevent directory traversal or unauthorized exports
• Sanitize paths and input

---

User Experience

• Progress indicators for long exports
• Option to cancel ongoing exports
• Email notification on completion or failure
• Export Management Page to re-download or delete past exports