Add SAM2 few-shot segmentation worker

[arjunrajlab]

Summary

This PR introduces a new SAM2 few-shot segmentation worker that enables automatic polygon segmentation based on training annotations. The worker uses SAM2's image encoder to extract feature vectors from training examples, creates a prototype representation, and then applies similarity-based filtering to automatically segment similar objects in other frames.

Key Changes

• New Worker Implementation (workers/annotations/sam2_fewshot_segmentation/)

  • entrypoint.py: Main worker logic implementing two-phase pipeline:
    • Phase 1: Extract and average feature vectors from training annotations to create a prototype
    • Phase 2: Generate candidate masks using SAM2's automatic mask generator, filter by cosine similarity to prototype, and convert to polygon annotations
  • environment.yml: Conda environment with required dependencies (Python 3.10, SAM2, scikit-image, etc.)
  • Dockerfile and Dockerfile_M1: Container definitions for standard and ARM64 architectures

• Build Configuration

  • Updated build_machine_learning_workers.sh to include SAM2 few-shot segmentation worker build step

• Comprehensive Test Suite (workers/annotations/sam2_fewshot_segmentation/tests/)

  • Unit tests for core utility functions:
    • extract_crop_with_context: Context-aware crop extraction with occupancy targeting
    • pool_features_with_mask: Weighted feature pooling using binary masks
    • ensure_rgb: Image format normalization
    • annotation_to_mask: Polygon to binary mask conversion
    • interface: Worker interface configuration

Notable Implementation Details

• Feature Extraction: Uses SAM2's image encoder (image_embed) at 256 channels, 64x64 resolution for semantic feature representation
• Prototype Learning: Averages normalized feature vectors from all training annotations to create a robust prototype
• Similarity Filtering: Applies cosine similarity threshold to filter candidate masks, with configurable threshold (default 0.5)
• Context-Aware Cropping: Extracts crops with configurable occupancy ratio to provide appropriate context for feature extraction
• Configurable Parameters: User-facing interface includes model selection, similarity threshold, target occupancy, points per side, mask area bounds, and smoothing
• Batch Processing: Supports processing multiple XY, Z, and Time coordinates in a single run
• Tag-Based Training: Supports filtering training annotations by tags with exclusive/inclusive modes

https://claude.ai/code/session_01SiA9ktGYkhfqo1c4YBqPKw