+ 
+ 
+
+
+ 
+
+
+
+ WASP2
+Allele-Specific Analysis Pipeline
+ + + + +
-
-
-
+
+ 
+
+ Documentation • + Podcast • + McVicker Lab • + Original WASP +
+--- -## Installation -Recommended installation through conda, and given environment -```shell script -conda env create -f environment.yml -``` - - -## Allelic Imbalance Analysis -Analysis pipeline currently consists of two tools (Count and Analysis) - - -### Count Tool -Process allele specific read counts per SNP.\ -Sample names can be provided in order to filter out non-heterozygous SNPs. -Genes and ATAC-seq peaks can also be provided to include SNPs that overlap regions of interest.\ -Providing samples and regions is highly recommended for allelic-imbalance analysis - -**Usage** -```shell script -python WASP2/src/counting count-variants [BAM] [VCF] {OPTIONS} -``` - -**Required Arguments** -- BAM file containing aligned reads. -- VCF file containing SNP info - - -**Optional Arguments** -- -s/--samples: Filter SNPs whose genotypes are heterozygous in one or more samples. Accepts comma delimited string, or file with one sample per line. -- -r/--region: Filter SNPs that overlap peaks/regions of interest. Accepts files in narrowPeak, BED, gtf and gff3 format. -- -o/--out_file: Output file for counts. Defaults to counts.tsv -- -t/--temp_loc: Write intermediary files to a directory instead of deleting. Useful for debugging issues. -- --use_region_names: If regions are provided use region names as identifiers instead of coordinates. Names are denoted in fourth column of BED. Ignored if no name column in BED file. - - -**RNA-Seq Specific Arguments** -- --gene_feature: Feature type in gtf/gff3 for counting intersecting SNPs. Defaults to 'exon' for snp counting. -- --gene_attribute: Attribute name from gtf/gff3 attribute column to use as ID. Defaults to 'ACGTACGT"] + R2["Read with ALT allele
ACGTGCGT"] + + R1 -->|"Maps perfectly"| M1["✓ High MAPQ
Correct position"] + R2 -->|"Mismatch penalty"| M2["✗ Lower MAPQ
May mismap or fail"] + end + + style M1 fill:#90EE90 + style M2 fill:#FFB6C1 +``` + +## The WASP Solution: Allele Swap & Filter + +```mermaid +flowchart LR + subgraph "Step 1: Find Overlapping Reads" + BAM["BAM File"] --> FIND["Find reads
at het sites"] + VCF["VCF File"] --> FIND + end + + subgraph "Step 2: Create Alternate Reads" + FIND --> ORIG["Original Read
ACGTACGT"] + ORIG --> SWAP["Swap allele"] + SWAP --> ALT["Alternate Read
ACGTGCGT"] + end + + subgraph "Step 3: Remap Both" + ORIG --> ALIGN1["Align"] + ALT --> ALIGN2["Align"] + ALIGN1 --> POS1["Position 1
MAPQ 60"] + ALIGN2 --> POS2["Position 2
MAPQ 55"] + end + + subgraph "Step 4: Compare & Filter" + POS1 --> COMP{"Same position
& quality?"} + POS2 --> COMP + COMP -->|"Yes"| KEEP["✓ KEEP
Unbiased read"] + COMP -->|"No"| DISCARD["✗ DISCARD
Biased read"] + end + + style KEEP fill:#90EE90 + style DISCARD fill:#FFB6C1 +``` + +## The Combined Haplotype Test (CHT) + +```mermaid +graph TB + subgraph "Two Sources of Signal" + RD["Read Depth Signal
(across all individuals)"] + AI["Allelic Imbalance Signal
(within heterozygotes)"] + end + + subgraph "Combined Haplotype Test" + RD --> CHT["Integrate both signals
in likelihood framework"] + AI --> CHT + CHT --> BB["Beta-binomial model
handles overdispersion"] + BB --> LRT["Likelihood Ratio Test"] + LRT --> PVAL["p-value for QTL"] + end + + style CHT fill:#87CEEB + style PVAL fill:#FFD700 +``` + +## The Original WASP Pipeline + +```mermaid +flowchart TB + subgraph "Input" + VCF["VCF files"] + BAM["BAM files"] + end + + subgraph "Preparation" + VCF --> SNP2H5["snp2h5
(convert to HDF5)"] + SNP2H5 --> H5["HDF5 database"] + end + + subgraph "WASP Filtering" + BAM --> FIND["find_intersecting_snps.py"] + H5 --> FIND + FIND --> REMAP["Remap with alternate alleles"] + REMAP --> FILTER["filter_remapped_reads.py"] + FILTER --> CLEAN["Filtered BAM
(bias removed)"] + end + + subgraph "Analysis" + CLEAN --> COUNT["Count alleles"] + COUNT --> CHT2["combined_test.py
(CHT)"] + CHT2 --> QTL["QTL Results"] + end + + style H5 fill:#FFA07A + style CLEAN fill:#90EE90 + style QTL fill:#FFD700 +``` + +## Key Insight + +```mermaid +graph LR + A["If a read maps differently
depending on which allele
it carries..."] --> B["...that read is
BIASED
by definition"] + B --> C["Remove it!"] + + style A fill:#FFB6C1 + style B fill:#FF6347 + style C fill:#90EE90 +``` + +--- + +## Episode Reference +- **Episode**: 001 - The Origin Swarm +- **Topic**: Original WASP mapping bias correction (2015) +- **Paper**: van de Geijn et al., Nature Methods 2015 diff --git a/docs/source/_static/podcast/illuminations/illumination-002-architecture.md b/docs/source/_static/podcast/illuminations/illumination-002-architecture.md new file mode 100644 index 0000000..549a986 --- /dev/null +++ b/docs/source/_static/podcast/illuminations/illumination-002-architecture.md @@ -0,0 +1,199 @@ +# Illumination: WASP2 Architecture +# Episode: 002 - Building the New Hive + +## The Modernization: Old vs New + +```mermaid +graph LR + subgraph "Original WASP (2015)" + O_VCF["VCF"] --> O_CONV["snp2h5
(conversion)"] + O_CONV --> O_H5["HDF5"] + O_H5 --> O_SCRIPTS["Multiple
Python scripts"] + O_SCRIPTS --> O_OUT["Output"] + end + + subgraph "WASP2 (2021+)" + N_VCF["VCF/BCF"] --> N_CLI["Unified CLI
(wasp2-*)"] + N_CLI --> N_OUT["Parquet/AnnData"] + end + + style O_CONV fill:#FFB6C1 + style O_H5 fill:#FFA07A + style N_CLI fill:#90EE90 +``` + +## Module Organization + +```mermaid +graph TB + subgraph "src/wasp2/" + CLI["cli/
Typer-based commands"] + COUNT["counting/
Allele counting"] + MAP["mapping/
Read filtering"] + ANAL["analysis/
Statistical tests"] + IO["io/
Format handlers"] + end + + CLI --> COUNT + CLI --> MAP + CLI --> ANAL + COUNT --> IO + MAP --> IO + ANAL --> IO + + style CLI fill:#87CEEB + style COUNT fill:#98FB98 + style MAP fill:#DDA0DD + style ANAL fill:#FFD700 + style IO fill:#F0E68C +``` + +## The Unified CLI + +```mermaid +flowchart LR + subgraph "Command Structure" + WASP2["wasp2"] + WASP2 --> COUNT["wasp2-count
Allele counting"] + WASP2 --> MAP["wasp2-map
Bias correction"] + WASP2 --> ANALYZE["wasp2-analyze
QTL discovery"] + end + + subgraph "Features" + COUNT --> F1["• VCF/BCF native
• No conversion
• Parquet output"] + MAP --> F2["• WASP filtering
• Multi-sample
• Remapping"] + ANALYZE --> F3["• CHT
• Beta-binomial
• Single-cell"] + end + + style COUNT fill:#98FB98 + style MAP fill:#DDA0DD + style ANALYZE fill:#FFD700 +``` + +## Data Flow + +```mermaid +flowchart TB + subgraph "Inputs" + BAM["BAM/CRAM
Alignments"] + VCF["VCF/BCF
Variants"] + META["Sample
Metadata"] + end + + subgraph "WASP2 Processing" + BAM --> WASP["WASP2"] + VCF --> WASP + META --> WASP + + WASP --> FILT["Filtered reads
(bias removed)"] + WASP --> COUNTS["Allele counts
(per variant)"] + WASP --> STATS["Statistical tests
(QTL calls)"] + end + + subgraph "Outputs" + FILT --> O_BAM["Filtered BAM"] + COUNTS --> O_PQ["Parquet tables"] + STATS --> O_RES["Results TSV"] + COUNTS --> O_AD["AnnData
(single-cell)"] + end + + style WASP fill:#87CEEB + style O_PQ fill:#90EE90 + style O_AD fill:#FFD700 +``` + +## Technology Stack Comparison + +```mermaid +graph TB + subgraph "Original WASP" + O1["Python 3.x"] + O2["C extensions"] + O3["HDF5/PyTables"] + O4["NumPy/SciPy"] + O5["pysam"] + O1 --> O2 + O2 --> O3 + O3 --> O4 + O4 --> O5 + end + + subgraph "WASP2" + N1["Python 3.8+"] + N2["Typer CLI"] + N3["Rich terminal"] + N4["Parquet/Arrow"] + N5["AnnData"] + N6["cyvcf2"] + N1 --> N2 + N2 --> N3 + N1 --> N4 + N4 --> N5 + N1 --> N6 + end + + style O3 fill:#FFB6C1 + style N2 fill:#90EE90 + style N5 fill:#90EE90 +``` + +## Single-Cell Integration + +```mermaid +flowchart LR + subgraph "WASP2 Output" + COUNTS["Allele counts
per cell × variant"] + end + + subgraph "AnnData Structure" + X["X: count matrix"] + VAR["var: variant info"] + OBS["obs: cell metadata"] + LAYERS["layers: ref/alt counts"] + end + + subgraph "Downstream" + SCANPY["scanpy"] + DIFF["Differential AI
analysis"] + end + + COUNTS --> X + COUNTS --> VAR + COUNTS --> LAYERS + X --> SCANPY + LAYERS --> DIFF + + style COUNTS fill:#87CEEB + style SCANPY fill:#90EE90 + style DIFF fill:#FFD700 +``` + +## Design Principles + +```mermaid +mindmap + root((WASP2)) + No Conversion + VCF/BCF native + tabix indexing + No HDF5 step + Unified CLI + wasp2-count + wasp2-map + wasp2-analyze + Modern Stack + Typer + Rich + Parquet + Single-Cell + AnnData + scanpy integration + Millions of cells +``` + +--- + +## Episode Reference +- **Episode**: 002 - Building the New Hive +- **Topic**: WASP2 modernization and architecture (2021) +- **Repository**: mcvickerlab/WASP2 diff --git a/docs/source/_static/podcast/illuminations/illumination-003-performance.md b/docs/source/_static/podcast/illuminations/illumination-003-performance.md new file mode 100644 index 0000000..2dfc3af --- /dev/null +++ b/docs/source/_static/podcast/illuminations/illumination-003-performance.md @@ -0,0 +1,352 @@ +# Illumination: Performance Transformation +# Episode: 003 - The Rust Metamorphosis + +## Performance Comparison + +```mermaid +xychart-beta + title "WASP2 Performance Gains (seconds)" + x-axis ["BAM-BED Intersect", "Statistical Analysis", "Full Pipeline"] + y-axis "Time (seconds)" 0 --> 550 + bar [152, 2.7, 500] + bar [3, 0.5, 50] +``` + +## The Speedup Table + +```mermaid +graph LR + subgraph "Before: Python" + P1["BAM-BED: 152s"] + P2["Analysis: 2.7s"] + P3["Pipeline: ~500s"] + end + + subgraph "After: Rust" + R1["BAM-BED: 2-3s"] + R2["Analysis: 0.5s"] + R3["Pipeline: ~50s"] + end + + subgraph "Speedup" + S1["50-75x"] + S2["5x"] + S3["10x"] + end + + P1 -.-> S1 + P2 -.-> S2 + P3 -.-> S3 + S1 -.-> R1 + S2 -.-> R2 + S3 -.-> R3 + + style P1 fill:#FFB6C1 + style P2 fill:#FFB6C1 + style P3 fill:#FFB6C1 + style R1 fill:#90EE90 + style R2 fill:#90EE90 + style R3 fill:#90EE90 + style S1 fill:#FFD700 + style S2 fill:#FFD700 + style S3 fill:#FFD700 +``` + +## Rust Module Architecture + +```mermaid +graph TB + subgraph "Python Layer" + CLI["CLI
(Typer)"] + ORCH["Orchestration"] + IO["I/O dispatch"] + end + + subgraph "Rust Layer (via PyO3)" + BAM_INT["bam_intersect.rs
COITree intervals"] + BAM_CNT["bam_counter.rs
Parallel counting"] + BAM_RMP["bam_remapper.rs
CIGAR manipulation"] + ANAL["analysis.rs
Beta-binomial"] + end + + CLI --> ORCH + ORCH --> IO + IO --> BAM_INT + IO --> BAM_CNT + IO --> BAM_RMP + ORCH --> ANAL + + style CLI fill:#87CEEB + style BAM_INT fill:#FF8C00 + style BAM_CNT fill:#FF8C00 + style BAM_RMP fill:#FF8C00 + style ANAL fill:#FF8C00 +``` + +## The COITree Secret Weapon + +```mermaid +graph TD + subgraph "Old: pybedtools" + OLD1["BAM file"] --> OLD2["Write temp BED"] + OLD2 --> OLD3["bedtools intersect
(subprocess)"] + OLD3 --> OLD4["Parse output"] + OLD4 --> OLD5["152 seconds"] + end + + subgraph "New: COITree" + NEW1["BAM file"] --> NEW2["Build interval tree
(O(n log n))"] + NEW2 --> NEW3["Query per read
(O(log n + k))"] + NEW3 --> NEW4["2-3 seconds"] + end + + style OLD5 fill:#FFB6C1 + style NEW4 fill:#90EE90 +``` + +## Parallel Processing Architecture + +```mermaid +flowchart TB + subgraph "Input" + BAM["BAM File"] + end + + subgraph "Chunking" + BAM --> C1["Chunk 1
chr1:1-10M"] + BAM --> C2["Chunk 2
chr1:10M-20M"] + BAM --> C3["Chunk 3
chr1:20M-30M"] + BAM --> C4["..."] + end + + subgraph "Parallel Workers (Rayon)" + C1 --> W1["Worker 1"] + C2 --> W2["Worker 2"] + C3 --> W3["Worker 3"] + C4 --> W4["Worker N"] + end + + subgraph "Aggregation" + W1 --> AGG["Lock-free
aggregation"] + W2 --> AGG + W3 --> AGG + W4 --> AGG + AGG --> OUT["Final counts"] + end + + style W1 fill:#FF8C00 + style W2 fill:#FF8C00 + style W3 fill:#FF8C00 + style W4 fill:#FF8C00 + style OUT fill:#90EE90 +``` + +## The Python/Rust Boundary + +```mermaid +graph TB + subgraph "Stays in Python" + P1["CLI argument parsing"] + P2["Configuration handling"] + P3["High-level workflow"] + P4["User messages"] + P5["I/O format detection"] + end + + subgraph "Moves to Rust" + R1["Inner loops over reads"] + R2["Interval tree operations"] + R3["Log-likelihood calculations"] + R4["CIGAR string parsing"] + R5["Allele swapping"] + end + + P3 --> R1 + P3 --> R2 + P3 --> R3 + + style P1 fill:#87CEEB + style P2 fill:#87CEEB + style P3 fill:#87CEEB + style P4 fill:#87CEEB + style P5 fill:#87CEEB + style R1 fill:#FF8C00 + style R2 fill:#FF8C00 + style R3 fill:#FF8C00 + style R4 fill:#FF8C00 + style R5 fill:#FF8C00 +``` + +## New Capabilities Enabled + +```mermaid +mindmap + root((Rust
Metamorphosis)) + INDEL Support + Full insertions + Full deletions + Not just SNPs + Multi-Format + VCF native + BCF native + PGEN native + Auto-detection + Scale + Millions of cells + Streaming processing + Constant memory + Statistics + Beta-binomial + More accurate + Proper overdispersion +``` + +## Format Speedups + +```mermaid +graph LR + subgraph "VCF Parsing" + V1["Standard Python
1x baseline"] + V2["cyvcf2 (C-backed)
6.9x faster"] + end + + subgraph "Genotype Format" + G1["VCF/BCF
1x baseline"] + G2["PGEN format
25x faster"] + end + + V1 -.->|"6.9x"| V2 + G1 -.->|"25x"| G2 + + style V2 fill:#90EE90 + style G2 fill:#90EE90 +``` + +## The 80/20 Principle Applied + +```mermaid +pie title "Code Distribution vs Runtime Impact" + "Python (90% of code)" : 5 + "Rust (10% of code)" : 95 +``` + +*10% of the code was responsible for 95% of the runtime. Rewrite those 10%.* + +--- + +## Deployment Ecosystem + +```mermaid +graph TB + subgraph "Source" + CODE["WASP2
Python + Rust"] + end + + subgraph "Build Systems" + MATURIN["maturin
(Rust→Python)"] + DOCKER["Docker
Multi-stage"] + end + + subgraph "Distribution" + PYPI["PyPI
pip install wasp2"] + BIOCONDA["Bioconda
conda install"] + DOCKERHUB["Docker Hub
jaureguy760/wasp2"] + SINGULARITY["Singularity
HPC clusters"] + end + + subgraph "Workflows" + NF_RNA["nf-rnaseq"] + NF_ATAC["nf-atacseq"] + NF_SC["nf-scatac"] + NF_OUT["nf-outrider"] + end + + CODE --> MATURIN + CODE --> DOCKER + MATURIN --> PYPI + MATURIN --> BIOCONDA + DOCKER --> DOCKERHUB + DOCKERHUB --> SINGULARITY + + PYPI --> NF_RNA + DOCKERHUB --> NF_RNA + PYPI --> NF_ATAC + DOCKERHUB --> NF_ATAC + PYPI --> NF_SC + DOCKERHUB --> NF_SC + PYPI --> NF_OUT + DOCKERHUB --> NF_OUT + + style CODE fill:#87CEEB + style PYPI fill:#90EE90 + style BIOCONDA fill:#90EE90 + style DOCKERHUB fill:#FF8C00 + style SINGULARITY fill:#FF8C00 +``` + +## Nextflow Pipeline Architecture + +```mermaid +flowchart LR + subgraph "Input" + BAM["BAM files"] + VCF["VCF/BCF/PGEN"] + META["Sample sheet"] + end + + subgraph "Nextflow Pipeline" + NF["nextflow run
wasp2/nf-rnaseq"] + + subgraph "Processes" + P1["WASP2_COUNT"] + P2["WASP2_MAP"] + P3["WASP2_ANALYZE"] + end + end + + subgraph "Execution" + LOCAL["Local"] + SLURM["SLURM"] + AWS["AWS Batch"] + DOCKER2["Docker"] + SING["Singularity"] + end + + subgraph "Output" + COUNTS["Allele counts"] + FILTERED["Filtered BAMs"] + RESULTS["QTL results"] + REPORT["MultiQC report"] + end + + BAM --> NF + VCF --> NF + META --> NF + NF --> P1 + P1 --> P2 + P2 --> P3 + + NF --> LOCAL + NF --> SLURM + NF --> AWS + NF --> DOCKER2 + NF --> SING + + P1 --> COUNTS + P2 --> FILTERED + P3 --> RESULTS + P3 --> REPORT + + style NF fill:#87CEEB + style SLURM fill:#FFD700 + style SING fill:#FF8C00 +``` + +--- + +## Episode Reference +- **Episode**: 003 - The Rust Metamorphosis +- **Topic**: Rust acceleration and deployment ecosystem (2024-2026) +- **Version**: 1.3.0 +- **Rust Lines**: 10,551+ +- **Pipelines**: nf-rnaseq, nf-atacseq, nf-scatac, nf-outrider diff --git a/docs/source/_static/podcast/index.html b/docs/source/_static/podcast/index.html new file mode 100644 index 0000000..ec69f74 --- /dev/null +++ b/docs/source/_static/podcast/index.html @@ -0,0 +1,1772 @@ + + + + + +
+
+ 
+
+
+ The WASP's Nest
+Buzz from the Hive
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/source/_static/podcast/manifest.yml b/docs/source/_static/podcast/manifest.yml
new file mode 100644
index 0000000..cec3a88
--- /dev/null
+++ b/docs/source/_static/podcast/manifest.yml
@@ -0,0 +1,157 @@
+# The WASP's Nest - Changelog Podcast
+# "Buzz from the Hive"
+#
+# 🐝 Dispatches from WASP2 Development 🐝
+
+realm:
+ name: "wasp2"
+ title: "The WASP's Nest"
+ tagline: "Buzz from the Hive"
+ description: |
+ Welcome to the Hive! The Queen Bee documents every release,
+ every feature, every bug squashed in WASP2 - the allele-specific
+ pipeline for unbiased read mapping and allelic imbalance analysis.
+
+ Each episode (Buzz Report) chronicles what's new in the hive,
+ from foraging expeditions into new features to defending against
+ pesky bugs. Join the swarm and stay informed!
+ author: "The Queen Bee"
+ website: "https://github.com/Jaureguy760/WASP2-final"
+ language: "en-us"
+ logo: "artwork/wasp2_logo.png"
+ logo_source: "doc/wasp2_hex_logo_v1.png"
+
+# Voice/narrator configuration
+voice:
+ persona: "queen_bee"
+ engine: "chatterbox"
+ emotion_level: 0.5
+
+# When to generate new episodes
+trigger:
+ type: "tag"
+ tag_pattern: "v*.*.*"
+ branch: "main"
+
+# Theming and style
+style:
+ theme: "hive"
+ sound_effects: true
+ diagrams: true
+ logo_symbolism: |
+ Two wasps facing each other represent paired alleles.
+ Red/blue colored bands symbolize allelic variants.
+ Hexagonal frame = honeycomb = the hive.
+ terminology:
+ narrator: "The Queen Bee"
+ realm: "The Hive"
+ episodes: "Buzz Reports"
+ updates: "Swarm Updates"
+ bugs_fixed: "Squashed Bugs"
+ celebration: "Buzz buzz!"
+ opening: "Welcome to the Hive, fellow worker bees..."
+ closing: "Keep building, keep buzzing. Buzz out!"
+
+# Episode chapters follow bee activities
+chapter_themes:
+ - name: "Foraging"
+ description: "New features and explorations"
+ - name: "Building"
+ description: "Infrastructure and improvements"
+ - name: "Defending"
+ description: "Bug fixes and security updates"
+ - name: "Pollinating"
+ description: "Community contributions and integrations"
+
+# Podcast metadata
+volume: 1
+episode_count: 3
+
+# Special series: The WASP Chronicles
+# A 3-part history of WASP's evolution from 2015 to 2026
+series:
+ - name: "The WASP Chronicles"
+ description: |
+ A special 3-episode series tracing the evolution of WASP from
+ its origins in 2015 to the modern Rust-accelerated implementation.
+ Perfect for new users wanting to understand the project's history
+ and design philosophy.
+ episodes: [1, 2, 3]
+
+episodes:
+ - number: 1
+ title: "The Origin Swarm"
+ subtitle: "Original WASP (2015)"
+ series: "The WASP Chronicles"
+ file: "chronicles/episode-001-origin-swarm.md"
+ illumination: "illuminations/illumination-001-wasp-mapping.md"
+ date: "2026-02-03"
+ duration_estimate: "8-10 minutes"
+ description: |
+ The story of the original WASP published in Nature Methods 2015.
+ Learn how van de Geijn, McVicker, Gilad, and Pritchard solved the
+ mapping bias problem that plagued allele-specific analysis.
+ topics:
+ - mapping bias
+ - allele swapping
+ - Combined Haplotype Test
+ - HDF5 format
+ references:
+ paper:
+ title: "WASP: allele-specific software for robust molecular QTL discovery"
+ authors: ["van de Geijn B", "McVicker G", "Gilad Y", "Pritchard JK"]
+ journal: "Nature Methods"
+ year: 2015
+ pmid: 26366987
+ repo: "https://github.com/bmvdgeijn/WASP"
+
+ - number: 2
+ title: "Building the New Hive"
+ subtitle: "McVicker Lab WASP2"
+ series: "The WASP Chronicles"
+ file: "chronicles/episode-002-new-hive.md"
+ illumination: "illuminations/illumination-002-architecture.md"
+ date: "2026-02-03"
+ duration_estimate: "8-10 minutes"
+ description: |
+ How the McVicker Lab rebuilt WASP for the modern era. No more
+ HDF5 conversion, unified CLI, single-cell support, and clean
+ Python architecture.
+ topics:
+ - modernization
+ - VCF/BCF native
+ - Typer CLI
+ - AnnData integration
+ - single-cell support
+ references:
+ repo: "https://github.com/mcvickerlab/WASP2"
+ timeline:
+ established: "2021-12"
+ v1_release: "2024-09"
+
+ - number: 3
+ title: "The Rust Metamorphosis"
+ subtitle: "WASP2-exp High Performance"
+ series: "The WASP Chronicles"
+ file: "chronicles/episode-003-rust-metamorphosis.md"
+ illumination: "illuminations/illumination-003-performance.md"
+ date: "2026-02-03"
+ duration_estimate: "10-12 minutes"
+ description: |
+ The transformation to Rust-accelerated performance. 50-100x speedups,
+ full INDEL support, beta-binomial statistics, and the philosophy of
+ surgical optimization.
+ topics:
+ - Rust acceleration
+ - COITree
+ - INDEL support
+ - beta-binomial model
+ - PyO3 integration
+ - performance benchmarks
+ references:
+ version: "1.2.1"
+ rust_lines: 10551
+ speedups:
+ bam_bed_intersect: "50-75x"
+ statistical_analysis: "5x"
+ full_pipeline: "10x"
diff --git a/docs/source/_static/podcast/voice-config.yml b/docs/source/_static/podcast/voice-config.yml
new file mode 100644
index 0000000..fbc4e0d
--- /dev/null
+++ b/docs/source/_static/podcast/voice-config.yml
@@ -0,0 +1,81 @@
+# Voice Configuration for The Queen Bee
+# The WASP's Nest Changelog Podcast
+#
+# 🐝 "Buzz from the Hive" 🐝
+
+persona: "queen_bee"
+
+# Primary TTS engine configuration
+engine: "xtts-v2"
+
+xtts:
+ model: "tts_models/multilingual/multi-dataset/xtts_v2"
+ language: "en"
+ device: "cpu"
+ # Voice should be warm, knowledgeable, with slight scientific precision
+ temperature: 0.7
+ speed: 1.0
+
+# Fallback TTS engine
+edge_tts:
+ voice: "en-US-JennyNeural" # Friendly, warm voice
+ rate: "-3%"
+ pitch: "+2Hz" # Slightly higher for Queen Bee character
+
+# Audio production settings
+audio:
+ format: "mp3"
+ bitrate: "192k"
+ sample_rate: 44100
+ normalize: true
+
+# Narrative emotion tags for script markup
+tags:
+ celebration: "[happy buzz]"
+ emphasis: "[pause]"
+ concern: "[worried hum]"
+ excitement: "[excited waggle]"
+ technical: "[precise tone]"
+ humor: "[playful buzz]"
+
+# Opening sequence
+opening:
+ music: true
+ music_file: "hive_intro.mp3"
+ fade_in_seconds: 2
+ greeting: |
+ Welcome to the Hive, fellow worker bees!
+
+ I'm the Queen Bee, and this is The WASP's Nest -
+ your source for the latest buzz from WASP2 development.
+
+ Today's Buzz Report brings news from the colony...
+
+# Closing sequence
+closing:
+ music: true
+ music_file: "hive_outro.mp3"
+ fade_out_seconds: 3
+ farewell: |
+ And that's the buzz for today, worker bees!
+
+ Keep building, keep buzzing!
+ May your reads map true and your alleles balance.
+
+ From the WASP's Nest, this is the Queen Bee.
+ Buzz out! 🐝
+
+# Chapter transition sounds
+transitions:
+ foraging: "wing_flutter.mp3"
+ building: "comb_construction.mp3"
+ defending: "defensive_buzz.mp3"
+ pollinating: "happy_waggle.mp3"
+
+# Special phrases and their delivery
+phrase_styling:
+ "WASP2": "emphasized, proud"
+ "allelic imbalance": "technical precision"
+ "beta-binomial": "scientific authority"
+ "bug squashed": "satisfied celebration"
+ "new feature": "excited anticipation"
diff --git a/docs/source/api/analysis.rst b/docs/source/api/analysis.rst
new file mode 100644
index 0000000..a4e09d3
--- /dev/null
+++ b/docs/source/api/analysis.rst
@@ -0,0 +1,69 @@
+Analysis Module API
+===================
+
+The analysis module provides statistical detection of allelic imbalance using beta-binomial models.
+
+Core Statistical Engine
+-----------------------
+
+as_analysis
+~~~~~~~~~~~
+
+.. automodule:: analysis.as_analysis
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+as_analysis_sc
+~~~~~~~~~~~~~~
+
+.. automodule:: analysis.as_analysis_sc
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+Group Comparison
+----------------
+
+compare_ai
+~~~~~~~~~~
+
+.. automodule:: analysis.compare_ai
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+Analysis Runners
+----------------
+
+run_analysis
+~~~~~~~~~~~~
+
+.. automodule:: analysis.run_analysis
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+run_analysis_sc
+~~~~~~~~~~~~~~~
+
+.. automodule:: analysis.run_analysis_sc
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+run_compare_ai
+~~~~~~~~~~~~~~
+
+.. automodule:: analysis.run_compare_ai
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+CLI Entry Point
+---------------
+
+.. automodule:: analysis.__main__
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/source/api/counting.rst b/docs/source/api/counting.rst
new file mode 100644
index 0000000..6bf40dc
--- /dev/null
+++ b/docs/source/api/counting.rst
@@ -0,0 +1,61 @@
+Counting Module API
+===================
+
+The counting module provides functions for allele-specific read counting from BAM files.
+
+count_alleles
+-------------
+
+.. automodule:: counting.count_alleles
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+count_alleles_sc
+----------------
+
+.. automodule:: counting.count_alleles_sc
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+filter_variant_data
+-------------------
+
+.. automodule:: counting.filter_variant_data
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+parse_gene_data
+---------------
+
+.. automodule:: counting.parse_gene_data
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+run_counting
+------------
+
+.. automodule:: counting.run_counting
+ :members:
+ :undoc-members:
+ :show-inheritance:
+ :exclude-members: bam_file, variant_file, region_file, samples, out_file, temp_loc, variant_prefix, vcf_bed, skip_vcf_to_bed, region_type, is_gene_file, use_region_names, intersect_file, skip_intersect, gtf_bed
+
+run_counting_sc
+---------------
+
+.. automodule:: counting.run_counting_sc
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+CLI Entry Point
+---------------
+
+.. automodule:: counting.__main__
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/source/api/mapping.rst b/docs/source/api/mapping.rst
new file mode 100644
index 0000000..dc90ca9
--- /dev/null
+++ b/docs/source/api/mapping.rst
@@ -0,0 +1,60 @@
+Mapping Module API
+==================
+
+The mapping module implements the WASP algorithm for unbiased read remapping to correct reference bias.
+
+filter_remap_reads
+------------------
+
+.. automodule:: mapping.filter_remap_reads
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+intersect_variant_data
+----------------------
+
+.. automodule:: mapping.intersect_variant_data
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+make_remap_reads
+----------------
+
+.. automodule:: mapping.make_remap_reads
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+remap_utils
+-----------
+
+.. automodule:: mapping.remap_utils
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+run_mapping
+-----------
+
+.. automodule:: mapping.run_mapping
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+wasp_data_files
+---------------
+
+.. automodule:: mapping.wasp_data_files
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+CLI Entry Point
+---------------
+
+.. automodule:: mapping.__main__
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst
new file mode 100644
index 0000000..6877a9b
--- /dev/null
+++ b/docs/source/changelog.rst
@@ -0,0 +1,41 @@
+Changelog
+=========
+
+Version 1.0.0 (2025-11-17)
+--------------------------
+
+Initial Release
+~~~~~~~~~~~~~~~
+
+**Features:**
+
+* Complete type hint coverage (24 files, 5,500 lines)
+* PyPI package available (pip install wasp2)
+* CI/CD pipeline with GitHub Actions
+* Pre-commit hooks for code quality
+* Comprehensive documentation on ReadTheDocs
+
+**Modules:**
+
+* **Counting**: Allele-specific read counting from BAM files
+* **Mapping**: WASP algorithm for unbiased read remapping
+* **Analysis**: Statistical detection of allelic imbalance
+
+**Type Hints:**
+
+* TH-1: Counting module (7 files)
+* TH-2: Analysis module (10 files)
+* TH-3: Mapping module (7 files)
+
+**Testing:**
+
+* Regression tests (memory, performance)
+* Full pipeline validation with real genomic data
+* All tests passing in CI
+
+**Documentation:**
+
+* API documentation auto-generated from type hints
+* User guides for each module
+* Installation and quickstart guides
+* Development and contributing guides
diff --git a/docs/source/conf.py b/docs/source/conf.py
new file mode 100644
index 0000000..fab87fb
--- /dev/null
+++ b/docs/source/conf.py
@@ -0,0 +1,193 @@
+# Configuration file for the Sphinx documentation builder.
+# WASP2 Documentation
+
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../../src"))
+
+# Mock imports for modules that require compiled extensions
+autodoc_mock_imports = [
+ "wasp2_rust",
+ "pysam",
+ "pybedtools",
+ "anndata",
+ "scanpy",
+]
+
+# -- Project information -----------------------------------------------------
+project = "WASP2"
+copyright = "2025, Aaron Ho, Jeff Jaureguy, McVicker Lab"
+author = "Aaron Ho, Jeff Jaureguy, McVicker Lab"
+
+# The short X.Y version
+version = "1.3"
+# The full version, including alpha/beta/rc tags
+release = "1.3.0"
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings
+extensions = [
+ "sphinx.ext.autodoc", # Auto-generate from docstrings
+ "sphinx.ext.napoleon", # Google/NumPy docstring support
+ "sphinx.ext.viewcode", # Add source code links
+ "sphinx.ext.intersphinx", # Link to other docs
+ "sphinx_autodoc_typehints", # Use our type hints!
+ "sphinx.ext.autosummary", # Generate summary tables
+ "sphinx.ext.coverage", # Coverage checker
+ "sphinx.ext.todo", # Support TODO items
+ "sphinx.ext.mathjax", # MathJax for equation rendering
+ "nbsphinx", # Jupyter notebook support
+ "myst_parser", # Markdown file support
+ "sphinx_copybutton", # Copy button for code blocks
+ "sphinx_design", # Cards, grids, and other UI components
+]
+
+# Add any paths that contain templates here, relative to this directory
+templates_path = ["_templates"]
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files
+exclude_patterns = []
+
+# The suffix(es) of source filenames
+source_suffix = {
+ ".rst": "restructuredtext",
+ ".md": "markdown",
+}
+
+# The master toctree document
+master_doc = "index"
+
+# The language for content autogenerated by Sphinx
+language = "en"
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages
+html_theme = "pydata_sphinx_theme"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+html_theme_options = {
+ "navigation_depth": 4,
+ "show_nav_level": 2,
+ "show_toc_level": 2,
+ "navbar_align": "left",
+ "icon_links": [
+ {
+ "name": "GitHub",
+ "url": "https://github.com/Jaureguy760/WASP2-final",
+ "icon": "fa-brands fa-github",
+ },
+ {
+ "name": "PyPI",
+ "url": "https://test.pypi.org/project/wasp2-rust/",
+ "icon": "fa-solid fa-box",
+ },
+ ],
+ "use_edit_page_button": True,
+ "announcement": "WASP2 v1.3.0 with Nextflow pipelines is now available!",
+}
+
+html_context = {
+ "github_user": "Jaureguy760",
+ "github_repo": "WASP2-final",
+ "github_version": "main",
+ "doc_path": "docs/source",
+}
+
+# Logo configuration
+html_logo = "_static/logo.png"
+html_favicon = "_static/logo.png"
+
+# Add any paths that contain custom static files (such as style sheets)
+html_static_path = ["_static"]
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names
+html_sidebars = {}
+
+# -- Extension configuration -------------------------------------------------
+
+# -- Options for autodoc extension -------------------------------------------
+
+# This value controls how to represent typehints
+autodoc_typehints = "description" # Show types in parameter descriptions
+autodoc_typehints_description_target = "documented"
+
+# This value selects what content will be inserted into the main body
+autodoc_default_options = {
+ "members": True,
+ "member-order": "bysource",
+ "special-members": "__init__",
+ "undoc-members": True,
+ "exclude-members": "__weakref__",
+ "show-inheritance": True,
+}
+
+# Automatically extract typehints when specified
+autodoc_typehints_format = "short"
+
+# -- Options for intersphinx extension ---------------------------------------
+
+# Example configuration for intersphinx: refer to the Python standard library
+intersphinx_mapping = {
+ "python": ("https://docs.python.org/3/", None),
+ "numpy": ("https://numpy.org/doc/stable/", None),
+ "pandas": ("https://pandas.pydata.org/docs/", None),
+ "scipy": ("https://docs.scipy.org/doc/scipy/", None),
+}
+
+# -- Options for napoleon extension ------------------------------------------
+
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_init_with_doc = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
+napoleon_preprocess_types = False
+napoleon_type_aliases = None
+napoleon_attr_annotations = True
+
+# -- Options for todo extension ----------------------------------------------
+
+# If true, `todo` and `todoList` produce output, else they produce nothing
+todo_include_todos = True
+
+# -- Options for MathJax extension -------------------------------------------
+
+# MathJax configuration for equation rendering in statistical methods docs
+mathjax3_config = {
+ "tex": {
+ "macros": {
+ "text": [r"\textrm{#1}", 1],
+ },
+ },
+}
+
+# -- Options for nbsphinx extension ------------------------------------------
+
+# Don't execute notebooks during build (they should be pre-executed)
+nbsphinx_execute = "never"
+
+# -- Options for myst_parser extension ---------------------------------------
+
+myst_enable_extensions = [
+ "colon_fence", # ::: directive syntax
+ "deflist", # Definition lists
+ "dollarmath", # $math$ syntax
+ "tasklist", # Task lists
+]
+
+# -- Options for sphinx_copybutton extension ---------------------------------
+
+# Don't copy prompts from code blocks
+copybutton_prompt_text = r">>> |\.\.\. |\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: "
+copybutton_prompt_is_regexp = True
diff --git a/docs/source/development.rst b/docs/source/development.rst
new file mode 100644
index 0000000..d88d15d
--- /dev/null
+++ b/docs/source/development.rst
@@ -0,0 +1,272 @@
+Development Guide
+=================
+
+Contributing to WASP2
+---------------------
+
+We welcome contributions! This guide helps you get started.
+
+Development Setup
+-----------------
+
+Clone Repository
+~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+ git clone https://github.com/Jaureguy760/WASP2-exp
+ cd WASP2-exp
+
+Install Development Dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+ pip install -e ".[dev]"
+
+This installs:
+* pytest (testing)
+* mypy (type checking)
+* black (code formatting)
+* flake8 (linting)
+* pre-commit (git hooks)
+
+Install Pre-commit Hooks
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+ pre-commit install
+
+Hooks run automatically before each commit:
+* Black formatting
+* Flake8 linting
+* mypy type checking
+* Quick tests
+
+Code Standards
+--------------
+
+Type Hints
+~~~~~~~~~~
+
+WASP2 has 100% type hint coverage. All new code must include type hints:
+
+.. code-block:: python
+
+ def count_alleles(
+ bam_file: str,
+ vcf_file: str,
+ min_count: int = 10
+ ) -> pd.DataFrame:
+ """Count alleles from BAM file."""
+ ...
+
+Formatting
+~~~~~~~~~~
+
+Use Black with 100-character lines:
+
+.. code-block:: bash
+
+ black src/ --line-length=100
+
+Linting
+~~~~~~~
+
+Pass Flake8 checks:
+
+.. code-block:: bash
+
+ flake8 src/ --max-line-length=100
+
+Testing
+-------
+
+Run Tests Locally
+~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+ # All tests
+ pytest tests/ -v
+
+ # Fast tests only (skip slow integration tests)
+ pytest tests/ -v -m "not slow"
+
+ # With coverage
+ pytest tests/ --cov=src --cov-report=html
+
+Test Requirements
+~~~~~~~~~~~~~~~~~
+
+* All new features need tests
+* Maintain >80% code coverage
+* Tests must pass in CI before merge
+
+Type Checking
+-------------
+
+Run mypy:
+
+.. code-block:: bash
+
+ mypy src/counting/ src/mapping/ src/analysis/
+
+All code must pass mypy with 0 errors.
+
+CI/CD Pipeline
+--------------
+
+GitHub Actions
+~~~~~~~~~~~~~~
+
+Tests run automatically on every push:
+* Python 3.10 and 3.11
+* Type checking (mypy)
+* Unit tests (pytest)
+* Full pipeline validation
+* Documentation build
+
+CI must pass before PR can be merged.
+
+Pre-commit Hooks
+~~~~~~~~~~~~~~~~
+
+Local checks before commit:
+* Code formatting
+* Type checking
+* Quick tests
+
+To bypass (not recommended):
+
+.. code-block:: bash
+
+ git commit --no-verify
+
+Pull Request Process
+--------------------
+
+1. Fork & Branch
+~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+ git checkout -b feature/my-feature
+
+2. Develop & Test
+~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+ # Make changes
+ vim src/analysis/my_feature.py
+
+ # Add type hints
+ # Write tests
+ # Run locally
+ pytest tests/ -v
+ mypy src/
+
+3. Commit
+~~~~~~~~~
+
+.. code-block:: bash
+
+ git add src/analysis/my_feature.py tests/test_my_feature.py
+ git commit -m "Add my feature"
+
+ # Pre-commit hooks run automatically
+
+4. Push & PR
+~~~~~~~~~~~~
+
+.. code-block:: bash
+
+ git push origin feature/my-feature
+
+ # Open PR on GitHub
+ # CI will run automatically
+ # Request review
+
+Code Review
+-----------
+
+PRs are reviewed for:
+* Correctness
+* Type safety
+* Test coverage
+* Documentation
+* Code style
+
+Project Structure
+-----------------
+
+.. code-block:: text
+
+ WASP2-exp/
+ ├── src/
+ │ ├── counting/ # Allele counting
+ │ ├── mapping/ # WASP remapping
+ │ └── analysis/ # Statistical analysis
+ ├── tests/
+ │ └── regression/ # Regression tests
+ ├── docs/ # Sphinx documentation
+ ├── scripts/ # Utility scripts
+ ├── baselines/ # Test baselines
+ └── test_data/ # Example data
+
+Building Documentation
+----------------------
+
+.. code-block:: bash
+
+ cd docs
+ make html
+ open build/html/index.html
+
+Documentation must build without warnings.
+
+Release Process
+---------------
+
+1. Update version in ``pyproject.toml``
+2. Update ``docs/source/changelog.rst``
+3. Merge to main
+4. Tag release: ``git tag v1.1.0``
+5. Push tag: ``git push origin v1.1.0``
+6. Publish to PyPI: ``python -m build && twine upload dist/*``
+
+AI-Assisted Development
+-----------------------
+
+WASP2 pipeline development benefits from AI tooling. See the full integration guide:
+:doc:`/seqera_ai_integration`
+
+Recommended Workflow
+~~~~~~~~~~~~~~~~~~~~
+
+1. **Design**: Use Claude Code for architecture and complex logic
+2. **Generate**: Use Seqera AI for DSL2 syntax and nf-test templates
+3. **Validate**: Use Anthropic life-sciences scripts for environment checks
+4. **Review**: Use Claude Code for code review and optimization
+
+Tool Selection
+~~~~~~~~~~~~~~
+
+* **Architecture and design** → Claude Code
+* **Nextflow DSL2 syntax** → Seqera AI
+* **nf-test generation** → Seqera AI
+* **Environment validation** → ``nextflow run . -profile test -preview``
+
+Getting Help
+------------
+
+* **Issues**: https://github.com/Jaureguy760/WASP2-exp/issues
+* **Discussions**: GitHub Discussions
+* **Email**: Contact maintainers
+
+License
+-------
+
+WASP2 is released under the MIT License. See LICENSE file.
diff --git a/docs/source/index.rst b/docs/source/index.rst
new file mode 100644
index 0000000..f0a742d
--- /dev/null
+++ b/docs/source/index.rst
@@ -0,0 +1,109 @@
+WASP2: Allele-Specific Analysis
+================================
+
+.. image:: https://img.shields.io/pypi/v/wasp2
+ :target: https://pypi.org/project/wasp2/
+ :alt: PyPI
+
+.. image:: https://github.com/Jaureguy760/WASP2-exp/workflows/WASP2%20Tests/badge.svg
+ :target: https://github.com/Jaureguy760/WASP2-exp/actions
+ :alt: Tests
+
+WASP2 is a comprehensive suite of tools for unbiased allele-specific analysis of next-generation sequencing data. It addresses reference bias in read mapping and provides statistical methods for detecting allelic imbalance.
+
+Features
+--------
+
+* **Unbiased Mapping**: WASP algorithm for correcting reference bias
+* **Allele Counting**: Count allele-specific reads from BAM files
+* **Statistical Analysis**: Beta-binomial models for allelic imbalance detection
+* **Single-Cell Support**: Specialized tools for single-cell RNA-seq
+* **Type-Safe**: 100% type hint coverage for robust code
+* **Well-Tested**: Comprehensive regression and integration tests
+
+Quick Start
+-----------
+
+Install via pip:
+
+.. code-block:: bash
+
+ pip install wasp2
+
+Count alleles from a BAM file:
+
+.. code-block:: bash
+
+ wasp2-count count-variants sample.bam variants.vcf
+
+Analyze allelic imbalance:
+
+.. code-block:: bash
+
+ wasp2-analyze find-imbalance counts.tsv
+
+Documentation
+-------------
+
+.. toctree::
+ :maxdepth: 2
+ :caption: Getting Started
+
+ installation
+ quickstart
+
+.. toctree::
+ :maxdepth: 2
+ :caption: User Guide
+
+ user_guide/counting
+ user_guide/mapping
+ user_guide/analysis
+ user_guide/single_cell
+
+.. toctree::
+ :maxdepth: 2
+ :caption: Tutorials
+
+ tutorials/quickstart_counting
+ tutorials/quickstart_mapping
+ tutorials/rna_seq
+ tutorials/scrna_seq
+ tutorials/scatac_workflow
+ tutorials/comparative_imbalance
+ tutorials/atac_seq_workflow
+ tutorials/statistical_methods_tutorial
+
+.. toctree::
+ :maxdepth: 2
+ :caption: Statistical Methods
+
+ methods/index
+ methods/counting_algorithm
+ methods/mapping_filter
+ methods/statistical_models
+ methods/dispersion_estimation
+ methods/fdr_correction
+
+.. toctree::
+ :maxdepth: 2
+ :caption: API Reference
+
+ api/counting
+ api/mapping
+ api/analysis
+
+.. toctree::
+ :maxdepth: 1
+ :caption: Development
+
+ development
+ seqera_ai_integration
+ changelog
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
diff --git a/docs/source/installation.rst b/docs/source/installation.rst
new file mode 100644
index 0000000..a7dad9a
--- /dev/null
+++ b/docs/source/installation.rst
@@ -0,0 +1,169 @@
+Installation
+============
+
+Requirements
+------------
+
+System Dependencies
+~~~~~~~~~~~~~~~~~~~
+
+WASP2 requires:
+
+* bcftools >= 1.10
+* bedtools >= 2.29
+* samtools >= 1.10
+
+On Ubuntu/Debian:
+
+.. code-block:: bash
+
+ sudo apt-get install bcftools bedtools samtools
+
+On macOS with Homebrew:
+
+.. code-block:: bash
+
+ brew install bcftools bedtools samtools
+
+Compiling pgenlib
+~~~~~~~~~~~~~~~~~
+
+WASP2 uses `pgenlib
+
+
+
+
+
+
+ ACGTACGT"] + R2["Read with ALT allele
ACGTGCGT"] + + R1 -->|"Maps perfectly"| M1["✓ High MAPQ
Correct position"] + R2 -->|"Mismatch penalty"| M2["✗ Lower MAPQ
May mismap or fail"] + end + + style M1 fill:#90EE90 + style M2 fill:#FFB6C1 +``` + +## The WASP Solution: Allele Swap & Filter + +```mermaid +flowchart LR + subgraph "Step 1: Find Overlapping Reads" + BAM["BAM File"] --> FIND["Find reads
at het sites"] + VCF["VCF File"] --> FIND + end + + subgraph "Step 2: Create Alternate Reads" + FIND --> ORIG["Original Read
ACGTACGT"] + ORIG --> SWAP["Swap allele"] + SWAP --> ALT["Alternate Read
ACGTGCGT"] + end + + subgraph "Step 3: Remap Both" + ORIG --> ALIGN1["Align"] + ALT --> ALIGN2["Align"] + ALIGN1 --> POS1["Position 1
MAPQ 60"] + ALIGN2 --> POS2["Position 2
MAPQ 55"] + end + + subgraph "Step 4: Compare & Filter" + POS1 --> COMP{"Same position
& quality?"} + POS2 --> COMP + COMP -->|"Yes"| KEEP["✓ KEEP
Unbiased read"] + COMP -->|"No"| DISCARD["✗ DISCARD
Biased read"] + end + + style KEEP fill:#90EE90 + style DISCARD fill:#FFB6C1 +``` + +## The Combined Haplotype Test (CHT) + +```mermaid +graph TB + subgraph "Two Sources of Signal" + RD["Read Depth Signal
(across all individuals)"] + AI["Allelic Imbalance Signal
(within heterozygotes)"] + end + + subgraph "Combined Haplotype Test" + RD --> CHT["Integrate both signals
in likelihood framework"] + AI --> CHT + CHT --> BB["Beta-binomial model
handles overdispersion"] + BB --> LRT["Likelihood Ratio Test"] + LRT --> PVAL["p-value for QTL"] + end + + style CHT fill:#87CEEB + style PVAL fill:#FFD700 +``` + +## The Original WASP Pipeline + +```mermaid +flowchart TB + subgraph "Input" + VCF["VCF files"] + BAM["BAM files"] + end + + subgraph "Preparation" + VCF --> SNP2H5["snp2h5
(convert to HDF5)"] + SNP2H5 --> H5["HDF5 database"] + end + + subgraph "WASP Filtering" + BAM --> FIND["find_intersecting_snps.py"] + H5 --> FIND + FIND --> REMAP["Remap with alternate alleles"] + REMAP --> FILTER["filter_remapped_reads.py"] + FILTER --> CLEAN["Filtered BAM
(bias removed)"] + end + + subgraph "Analysis" + CLEAN --> COUNT["Count alleles"] + COUNT --> CHT2["combined_test.py
(CHT)"] + CHT2 --> QTL["QTL Results"] + end + + style H5 fill:#FFA07A + style CLEAN fill:#90EE90 + style QTL fill:#FFD700 +``` + +## Key Insight + +```mermaid +graph LR + A["If a read maps differently
depending on which allele
it carries..."] --> B["...that read is
BIASED
by definition"] + B --> C["Remove it!"] + + style A fill:#FFB6C1 + style B fill:#FF6347 + style C fill:#90EE90 +``` + +--- + +## Episode Reference +- **Episode**: 001 - The Origin Swarm +- **Topic**: Original WASP mapping bias correction (2015) +- **Paper**: van de Geijn et al., Nature Methods 2015 diff --git a/podcast/illuminations/illumination-002-architecture.md b/podcast/illuminations/illumination-002-architecture.md new file mode 100644 index 0000000..549a986 --- /dev/null +++ b/podcast/illuminations/illumination-002-architecture.md @@ -0,0 +1,199 @@ +# Illumination: WASP2 Architecture +# Episode: 002 - Building the New Hive + +## The Modernization: Old vs New + +```mermaid +graph LR + subgraph "Original WASP (2015)" + O_VCF["VCF"] --> O_CONV["snp2h5
(conversion)"] + O_CONV --> O_H5["HDF5"] + O_H5 --> O_SCRIPTS["Multiple
Python scripts"] + O_SCRIPTS --> O_OUT["Output"] + end + + subgraph "WASP2 (2021+)" + N_VCF["VCF/BCF"] --> N_CLI["Unified CLI
(wasp2-*)"] + N_CLI --> N_OUT["Parquet/AnnData"] + end + + style O_CONV fill:#FFB6C1 + style O_H5 fill:#FFA07A + style N_CLI fill:#90EE90 +``` + +## Module Organization + +```mermaid +graph TB + subgraph "src/wasp2/" + CLI["cli/
Typer-based commands"] + COUNT["counting/
Allele counting"] + MAP["mapping/
Read filtering"] + ANAL["analysis/
Statistical tests"] + IO["io/
Format handlers"] + end + + CLI --> COUNT + CLI --> MAP + CLI --> ANAL + COUNT --> IO + MAP --> IO + ANAL --> IO + + style CLI fill:#87CEEB + style COUNT fill:#98FB98 + style MAP fill:#DDA0DD + style ANAL fill:#FFD700 + style IO fill:#F0E68C +``` + +## The Unified CLI + +```mermaid +flowchart LR + subgraph "Command Structure" + WASP2["wasp2"] + WASP2 --> COUNT["wasp2-count
Allele counting"] + WASP2 --> MAP["wasp2-map
Bias correction"] + WASP2 --> ANALYZE["wasp2-analyze
QTL discovery"] + end + + subgraph "Features" + COUNT --> F1["• VCF/BCF native
• No conversion
• Parquet output"] + MAP --> F2["• WASP filtering
• Multi-sample
• Remapping"] + ANALYZE --> F3["• CHT
• Beta-binomial
• Single-cell"] + end + + style COUNT fill:#98FB98 + style MAP fill:#DDA0DD + style ANALYZE fill:#FFD700 +``` + +## Data Flow + +```mermaid +flowchart TB + subgraph "Inputs" + BAM["BAM/CRAM
Alignments"] + VCF["VCF/BCF
Variants"] + META["Sample
Metadata"] + end + + subgraph "WASP2 Processing" + BAM --> WASP["WASP2"] + VCF --> WASP + META --> WASP + + WASP --> FILT["Filtered reads
(bias removed)"] + WASP --> COUNTS["Allele counts
(per variant)"] + WASP --> STATS["Statistical tests
(QTL calls)"] + end + + subgraph "Outputs" + FILT --> O_BAM["Filtered BAM"] + COUNTS --> O_PQ["Parquet tables"] + STATS --> O_RES["Results TSV"] + COUNTS --> O_AD["AnnData
(single-cell)"] + end + + style WASP fill:#87CEEB + style O_PQ fill:#90EE90 + style O_AD fill:#FFD700 +``` + +## Technology Stack Comparison + +```mermaid +graph TB + subgraph "Original WASP" + O1["Python 3.x"] + O2["C extensions"] + O3["HDF5/PyTables"] + O4["NumPy/SciPy"] + O5["pysam"] + O1 --> O2 + O2 --> O3 + O3 --> O4 + O4 --> O5 + end + + subgraph "WASP2" + N1["Python 3.8+"] + N2["Typer CLI"] + N3["Rich terminal"] + N4["Parquet/Arrow"] + N5["AnnData"] + N6["cyvcf2"] + N1 --> N2 + N2 --> N3 + N1 --> N4 + N4 --> N5 + N1 --> N6 + end + + style O3 fill:#FFB6C1 + style N2 fill:#90EE90 + style N5 fill:#90EE90 +``` + +## Single-Cell Integration + +```mermaid +flowchart LR + subgraph "WASP2 Output" + COUNTS["Allele counts
per cell × variant"] + end + + subgraph "AnnData Structure" + X["X: count matrix"] + VAR["var: variant info"] + OBS["obs: cell metadata"] + LAYERS["layers: ref/alt counts"] + end + + subgraph "Downstream" + SCANPY["scanpy"] + DIFF["Differential AI
analysis"] + end + + COUNTS --> X + COUNTS --> VAR + COUNTS --> LAYERS + X --> SCANPY + LAYERS --> DIFF + + style COUNTS fill:#87CEEB + style SCANPY fill:#90EE90 + style DIFF fill:#FFD700 +``` + +## Design Principles + +```mermaid +mindmap + root((WASP2)) + No Conversion + VCF/BCF native + tabix indexing + No HDF5 step + Unified CLI + wasp2-count + wasp2-map + wasp2-analyze + Modern Stack + Typer + Rich + Parquet + Single-Cell + AnnData + scanpy integration + Millions of cells +``` + +--- + +## Episode Reference +- **Episode**: 002 - Building the New Hive +- **Topic**: WASP2 modernization and architecture (2021) +- **Repository**: mcvickerlab/WASP2 diff --git a/podcast/illuminations/illumination-003-performance.md b/podcast/illuminations/illumination-003-performance.md new file mode 100644 index 0000000..2dfc3af --- /dev/null +++ b/podcast/illuminations/illumination-003-performance.md @@ -0,0 +1,352 @@ +# Illumination: Performance Transformation +# Episode: 003 - The Rust Metamorphosis + +## Performance Comparison + +```mermaid +xychart-beta + title "WASP2 Performance Gains (seconds)" + x-axis ["BAM-BED Intersect", "Statistical Analysis", "Full Pipeline"] + y-axis "Time (seconds)" 0 --> 550 + bar [152, 2.7, 500] + bar [3, 0.5, 50] +``` + +## The Speedup Table + +```mermaid +graph LR + subgraph "Before: Python" + P1["BAM-BED: 152s"] + P2["Analysis: 2.7s"] + P3["Pipeline: ~500s"] + end + + subgraph "After: Rust" + R1["BAM-BED: 2-3s"] + R2["Analysis: 0.5s"] + R3["Pipeline: ~50s"] + end + + subgraph "Speedup" + S1["50-75x"] + S2["5x"] + S3["10x"] + end + + P1 -.-> S1 + P2 -.-> S2 + P3 -.-> S3 + S1 -.-> R1 + S2 -.-> R2 + S3 -.-> R3 + + style P1 fill:#FFB6C1 + style P2 fill:#FFB6C1 + style P3 fill:#FFB6C1 + style R1 fill:#90EE90 + style R2 fill:#90EE90 + style R3 fill:#90EE90 + style S1 fill:#FFD700 + style S2 fill:#FFD700 + style S3 fill:#FFD700 +``` + +## Rust Module Architecture + +```mermaid +graph TB + subgraph "Python Layer" + CLI["CLI
(Typer)"] + ORCH["Orchestration"] + IO["I/O dispatch"] + end + + subgraph "Rust Layer (via PyO3)" + BAM_INT["bam_intersect.rs
COITree intervals"] + BAM_CNT["bam_counter.rs
Parallel counting"] + BAM_RMP["bam_remapper.rs
CIGAR manipulation"] + ANAL["analysis.rs
Beta-binomial"] + end + + CLI --> ORCH + ORCH --> IO + IO --> BAM_INT + IO --> BAM_CNT + IO --> BAM_RMP + ORCH --> ANAL + + style CLI fill:#87CEEB + style BAM_INT fill:#FF8C00 + style BAM_CNT fill:#FF8C00 + style BAM_RMP fill:#FF8C00 + style ANAL fill:#FF8C00 +``` + +## The COITree Secret Weapon + +```mermaid +graph TD + subgraph "Old: pybedtools" + OLD1["BAM file"] --> OLD2["Write temp BED"] + OLD2 --> OLD3["bedtools intersect
(subprocess)"] + OLD3 --> OLD4["Parse output"] + OLD4 --> OLD5["152 seconds"] + end + + subgraph "New: COITree" + NEW1["BAM file"] --> NEW2["Build interval tree
(O(n log n))"] + NEW2 --> NEW3["Query per read
(O(log n + k))"] + NEW3 --> NEW4["2-3 seconds"] + end + + style OLD5 fill:#FFB6C1 + style NEW4 fill:#90EE90 +``` + +## Parallel Processing Architecture + +```mermaid +flowchart TB + subgraph "Input" + BAM["BAM File"] + end + + subgraph "Chunking" + BAM --> C1["Chunk 1
chr1:1-10M"] + BAM --> C2["Chunk 2
chr1:10M-20M"] + BAM --> C3["Chunk 3
chr1:20M-30M"] + BAM --> C4["..."] + end + + subgraph "Parallel Workers (Rayon)" + C1 --> W1["Worker 1"] + C2 --> W2["Worker 2"] + C3 --> W3["Worker 3"] + C4 --> W4["Worker N"] + end + + subgraph "Aggregation" + W1 --> AGG["Lock-free
aggregation"] + W2 --> AGG + W3 --> AGG + W4 --> AGG + AGG --> OUT["Final counts"] + end + + style W1 fill:#FF8C00 + style W2 fill:#FF8C00 + style W3 fill:#FF8C00 + style W4 fill:#FF8C00 + style OUT fill:#90EE90 +``` + +## The Python/Rust Boundary + +```mermaid +graph TB + subgraph "Stays in Python" + P1["CLI argument parsing"] + P2["Configuration handling"] + P3["High-level workflow"] + P4["User messages"] + P5["I/O format detection"] + end + + subgraph "Moves to Rust" + R1["Inner loops over reads"] + R2["Interval tree operations"] + R3["Log-likelihood calculations"] + R4["CIGAR string parsing"] + R5["Allele swapping"] + end + + P3 --> R1 + P3 --> R2 + P3 --> R3 + + style P1 fill:#87CEEB + style P2 fill:#87CEEB + style P3 fill:#87CEEB + style P4 fill:#87CEEB + style P5 fill:#87CEEB + style R1 fill:#FF8C00 + style R2 fill:#FF8C00 + style R3 fill:#FF8C00 + style R4 fill:#FF8C00 + style R5 fill:#FF8C00 +``` + +## New Capabilities Enabled + +```mermaid +mindmap + root((Rust
Metamorphosis)) + INDEL Support + Full insertions + Full deletions + Not just SNPs + Multi-Format + VCF native + BCF native + PGEN native + Auto-detection + Scale + Millions of cells + Streaming processing + Constant memory + Statistics + Beta-binomial + More accurate + Proper overdispersion +``` + +## Format Speedups + +```mermaid +graph LR + subgraph "VCF Parsing" + V1["Standard Python
1x baseline"] + V2["cyvcf2 (C-backed)
6.9x faster"] + end + + subgraph "Genotype Format" + G1["VCF/BCF
1x baseline"] + G2["PGEN format
25x faster"] + end + + V1 -.->|"6.9x"| V2 + G1 -.->|"25x"| G2 + + style V2 fill:#90EE90 + style G2 fill:#90EE90 +``` + +## The 80/20 Principle Applied + +```mermaid +pie title "Code Distribution vs Runtime Impact" + "Python (90% of code)" : 5 + "Rust (10% of code)" : 95 +``` + +*10% of the code was responsible for 95% of the runtime. Rewrite those 10%.* + +--- + +## Deployment Ecosystem + +```mermaid +graph TB + subgraph "Source" + CODE["WASP2
Python + Rust"] + end + + subgraph "Build Systems" + MATURIN["maturin
(Rust→Python)"] + DOCKER["Docker
Multi-stage"] + end + + subgraph "Distribution" + PYPI["PyPI
pip install wasp2"] + BIOCONDA["Bioconda
conda install"] + DOCKERHUB["Docker Hub
jaureguy760/wasp2"] + SINGULARITY["Singularity
HPC clusters"] + end + + subgraph "Workflows" + NF_RNA["nf-rnaseq"] + NF_ATAC["nf-atacseq"] + NF_SC["nf-scatac"] + NF_OUT["nf-outrider"] + end + + CODE --> MATURIN + CODE --> DOCKER + MATURIN --> PYPI + MATURIN --> BIOCONDA + DOCKER --> DOCKERHUB + DOCKERHUB --> SINGULARITY + + PYPI --> NF_RNA + DOCKERHUB --> NF_RNA + PYPI --> NF_ATAC + DOCKERHUB --> NF_ATAC + PYPI --> NF_SC + DOCKERHUB --> NF_SC + PYPI --> NF_OUT + DOCKERHUB --> NF_OUT + + style CODE fill:#87CEEB + style PYPI fill:#90EE90 + style BIOCONDA fill:#90EE90 + style DOCKERHUB fill:#FF8C00 + style SINGULARITY fill:#FF8C00 +``` + +## Nextflow Pipeline Architecture + +```mermaid +flowchart LR + subgraph "Input" + BAM["BAM files"] + VCF["VCF/BCF/PGEN"] + META["Sample sheet"] + end + + subgraph "Nextflow Pipeline" + NF["nextflow run
wasp2/nf-rnaseq"] + + subgraph "Processes" + P1["WASP2_COUNT"] + P2["WASP2_MAP"] + P3["WASP2_ANALYZE"] + end + end + + subgraph "Execution" + LOCAL["Local"] + SLURM["SLURM"] + AWS["AWS Batch"] + DOCKER2["Docker"] + SING["Singularity"] + end + + subgraph "Output" + COUNTS["Allele counts"] + FILTERED["Filtered BAMs"] + RESULTS["QTL results"] + REPORT["MultiQC report"] + end + + BAM --> NF + VCF --> NF + META --> NF + NF --> P1 + P1 --> P2 + P2 --> P3 + + NF --> LOCAL + NF --> SLURM + NF --> AWS + NF --> DOCKER2 + NF --> SING + + P1 --> COUNTS + P2 --> FILTERED + P3 --> RESULTS + P3 --> REPORT + + style NF fill:#87CEEB + style SLURM fill:#FFD700 + style SING fill:#FF8C00 +``` + +--- + +## Episode Reference +- **Episode**: 003 - The Rust Metamorphosis +- **Topic**: Rust acceleration and deployment ecosystem (2024-2026) +- **Version**: 1.3.0 +- **Rust Lines**: 10,551+ +- **Pipelines**: nf-rnaseq, nf-atacseq, nf-scatac, nf-outrider diff --git a/podcast/index.html b/podcast/index.html new file mode 100644 index 0000000..ec69f74 --- /dev/null +++ b/podcast/index.html @@ -0,0 +1,1772 @@ + + + + + +
+
+
+
+
+ The WASP's Nest
+Buzz from the Hive
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/podcast/manifest.yml b/podcast/manifest.yml
new file mode 100644
index 0000000..cec3a88
--- /dev/null
+++ b/podcast/manifest.yml
@@ -0,0 +1,157 @@
+# The WASP's Nest - Changelog Podcast
+# "Buzz from the Hive"
+#
+# 🐝 Dispatches from WASP2 Development 🐝
+
+realm:
+ name: "wasp2"
+ title: "The WASP's Nest"
+ tagline: "Buzz from the Hive"
+ description: |
+ Welcome to the Hive! The Queen Bee documents every release,
+ every feature, every bug squashed in WASP2 - the allele-specific
+ pipeline for unbiased read mapping and allelic imbalance analysis.
+
+ Each episode (Buzz Report) chronicles what's new in the hive,
+ from foraging expeditions into new features to defending against
+ pesky bugs. Join the swarm and stay informed!
+ author: "The Queen Bee"
+ website: "https://github.com/Jaureguy760/WASP2-final"
+ language: "en-us"
+ logo: "artwork/wasp2_logo.png"
+ logo_source: "doc/wasp2_hex_logo_v1.png"
+
+# Voice/narrator configuration
+voice:
+ persona: "queen_bee"
+ engine: "chatterbox"
+ emotion_level: 0.5
+
+# When to generate new episodes
+trigger:
+ type: "tag"
+ tag_pattern: "v*.*.*"
+ branch: "main"
+
+# Theming and style
+style:
+ theme: "hive"
+ sound_effects: true
+ diagrams: true
+ logo_symbolism: |
+ Two wasps facing each other represent paired alleles.
+ Red/blue colored bands symbolize allelic variants.
+ Hexagonal frame = honeycomb = the hive.
+ terminology:
+ narrator: "The Queen Bee"
+ realm: "The Hive"
+ episodes: "Buzz Reports"
+ updates: "Swarm Updates"
+ bugs_fixed: "Squashed Bugs"
+ celebration: "Buzz buzz!"
+ opening: "Welcome to the Hive, fellow worker bees..."
+ closing: "Keep building, keep buzzing. Buzz out!"
+
+# Episode chapters follow bee activities
+chapter_themes:
+ - name: "Foraging"
+ description: "New features and explorations"
+ - name: "Building"
+ description: "Infrastructure and improvements"
+ - name: "Defending"
+ description: "Bug fixes and security updates"
+ - name: "Pollinating"
+ description: "Community contributions and integrations"
+
+# Podcast metadata
+volume: 1
+episode_count: 3
+
+# Special series: The WASP Chronicles
+# A 3-part history of WASP's evolution from 2015 to 2026
+series:
+ - name: "The WASP Chronicles"
+ description: |
+ A special 3-episode series tracing the evolution of WASP from
+ its origins in 2015 to the modern Rust-accelerated implementation.
+ Perfect for new users wanting to understand the project's history
+ and design philosophy.
+ episodes: [1, 2, 3]
+
+episodes:
+ - number: 1
+ title: "The Origin Swarm"
+ subtitle: "Original WASP (2015)"
+ series: "The WASP Chronicles"
+ file: "chronicles/episode-001-origin-swarm.md"
+ illumination: "illuminations/illumination-001-wasp-mapping.md"
+ date: "2026-02-03"
+ duration_estimate: "8-10 minutes"
+ description: |
+ The story of the original WASP published in Nature Methods 2015.
+ Learn how van de Geijn, McVicker, Gilad, and Pritchard solved the
+ mapping bias problem that plagued allele-specific analysis.
+ topics:
+ - mapping bias
+ - allele swapping
+ - Combined Haplotype Test
+ - HDF5 format
+ references:
+ paper:
+ title: "WASP: allele-specific software for robust molecular QTL discovery"
+ authors: ["van de Geijn B", "McVicker G", "Gilad Y", "Pritchard JK"]
+ journal: "Nature Methods"
+ year: 2015
+ pmid: 26366987
+ repo: "https://github.com/bmvdgeijn/WASP"
+
+ - number: 2
+ title: "Building the New Hive"
+ subtitle: "McVicker Lab WASP2"
+ series: "The WASP Chronicles"
+ file: "chronicles/episode-002-new-hive.md"
+ illumination: "illuminations/illumination-002-architecture.md"
+ date: "2026-02-03"
+ duration_estimate: "8-10 minutes"
+ description: |
+ How the McVicker Lab rebuilt WASP for the modern era. No more
+ HDF5 conversion, unified CLI, single-cell support, and clean
+ Python architecture.
+ topics:
+ - modernization
+ - VCF/BCF native
+ - Typer CLI
+ - AnnData integration
+ - single-cell support
+ references:
+ repo: "https://github.com/mcvickerlab/WASP2"
+ timeline:
+ established: "2021-12"
+ v1_release: "2024-09"
+
+ - number: 3
+ title: "The Rust Metamorphosis"
+ subtitle: "WASP2-exp High Performance"
+ series: "The WASP Chronicles"
+ file: "chronicles/episode-003-rust-metamorphosis.md"
+ illumination: "illuminations/illumination-003-performance.md"
+ date: "2026-02-03"
+ duration_estimate: "10-12 minutes"
+ description: |
+ The transformation to Rust-accelerated performance. 50-100x speedups,
+ full INDEL support, beta-binomial statistics, and the philosophy of
+ surgical optimization.
+ topics:
+ - Rust acceleration
+ - COITree
+ - INDEL support
+ - beta-binomial model
+ - PyO3 integration
+ - performance benchmarks
+ references:
+ version: "1.2.1"
+ rust_lines: 10551
+ speedups:
+ bam_bed_intersect: "50-75x"
+ statistical_analysis: "5x"
+ full_pipeline: "10x"
diff --git a/podcast/voice-config.yml b/podcast/voice-config.yml
new file mode 100644
index 0000000..fbc4e0d
--- /dev/null
+++ b/podcast/voice-config.yml
@@ -0,0 +1,81 @@
+# Voice Configuration for The Queen Bee
+# The WASP's Nest Changelog Podcast
+#
+# 🐝 "Buzz from the Hive" 🐝
+
+persona: "queen_bee"
+
+# Primary TTS engine configuration
+engine: "xtts-v2"
+
+xtts:
+ model: "tts_models/multilingual/multi-dataset/xtts_v2"
+ language: "en"
+ device: "cpu"
+ # Voice should be warm, knowledgeable, with slight scientific precision
+ temperature: 0.7
+ speed: 1.0
+
+# Fallback TTS engine
+edge_tts:
+ voice: "en-US-JennyNeural" # Friendly, warm voice
+ rate: "-3%"
+ pitch: "+2Hz" # Slightly higher for Queen Bee character
+
+# Audio production settings
+audio:
+ format: "mp3"
+ bitrate: "192k"
+ sample_rate: 44100
+ normalize: true
+
+# Narrative emotion tags for script markup
+tags:
+ celebration: "[happy buzz]"
+ emphasis: "[pause]"
+ concern: "[worried hum]"
+ excitement: "[excited waggle]"
+ technical: "[precise tone]"
+ humor: "[playful buzz]"
+
+# Opening sequence
+opening:
+ music: true
+ music_file: "hive_intro.mp3"
+ fade_in_seconds: 2
+ greeting: |
+ Welcome to the Hive, fellow worker bees!
+
+ I'm the Queen Bee, and this is The WASP's Nest -
+ your source for the latest buzz from WASP2 development.
+
+ Today's Buzz Report brings news from the colony...
+
+# Closing sequence
+closing:
+ music: true
+ music_file: "hive_outro.mp3"
+ fade_out_seconds: 3
+ farewell: |
+ And that's the buzz for today, worker bees!
+
+ Keep building, keep buzzing!
+ May your reads map true and your alleles balance.
+
+ From the WASP's Nest, this is the Queen Bee.
+ Buzz out! 🐝
+
+# Chapter transition sounds
+transitions:
+ foraging: "wing_flutter.mp3"
+ building: "comb_construction.mp3"
+ defending: "defensive_buzz.mp3"
+ pollinating: "happy_waggle.mp3"
+
+# Special phrases and their delivery
+phrase_styling:
+ "WASP2": "emphasized, proud"
+ "allelic imbalance": "technical precision"
+ "beta-binomial": "scientific authority"
+ "bug squashed": "satisfied celebration"
+ "new feature": "excited anticipation"
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..0219d05
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,233 @@
+[build-system]
+requires = ["maturin>=1.6,<2.0"]
+build-backend = "maturin"
+
+[project]
+name = "wasp2"
+dynamic = ["version"] # Version sourced from rust/Cargo.toml via maturin
+description = "Allele-specific analysis of next-generation sequencing data with high-performance multi-format variant support (VCF/cyvcf2/PGEN)"
+readme = "README.md"
+authors = [
+ {name = "Aaron Ho"},
+ {name = "Jeff Jaureguy", email = "jeffpjaureguy@gmail.com"},
+ {name = "McVicker Lab"},
+]
+license = {text = "MIT"}
+requires-python = ">=3.10"
+keywords = [
+ "bioinformatics",
+ "genomics",
+ "allele-specific",
+ "ngs",
+ "sequencing",
+ "wasp",
+ "allelic-imbalance",
+ "plink2",
+ "pgen",
+ "vcf",
+ "cyvcf2",
+ "high-performance",
+]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Intended Audience :: Science/Research",
+ "License :: OSI Approved :: MIT License",
+ "Operating System :: POSIX :: Linux",
+ "Operating System :: MacOS",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: Scientific/Engineering :: Bio-Informatics",
+ "Typing :: Typed",
+]
+
+dependencies = [
+ "numpy>=1.21.0",
+ "pandas>=1.5.0,<3.0.0", # Pin <2.0 for anndata compatibility
+ "polars>=0.19.0",
+ "scipy>=1.10.0",
+ "pysam>=0.21.0",
+ "pybedtools>=0.9.0",
+ "anndata>=0.8.0,<0.12.0", # Pin <0.10 for pandas <2.0 compatibility
+ "scanpy>=1.9.0",
+ "typer>=0.12.0",
+ "rich>=13.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+ "pytest>=7.0",
+ "pytest-cov>=4.0",
+ "pytest-benchmark>=4.0",
+ "memory-profiler>=0.61",
+ "mypy>=1.0",
+ "basedpyright>=1.18.0",
+ "ruff>=0.9.0",
+ "pre-commit>=3.0",
+ "build>=0.10",
+ "twine>=4.0",
+ "maturin>=1.4",
+ # Security tools
+ "bandit[toml]>=1.8.0",
+ "pip-audit>=2.7.0",
+]
+benchmark = [
+ "pytest-benchmark>=4.0",
+ "memory-profiler>=0.61",
+ "matplotlib>=3.7.0",
+ "seaborn>=0.12.0",
+]
+docs = [
+ "sphinx>=7.0",
+ "pydata-sphinx-theme>=0.15",
+ "sphinx-autodoc-typehints>=1.25",
+ "nbsphinx>=0.9",
+ "myst-parser>=2.0",
+ "sphinx-copybutton>=0.5",
+ "sphinx-design>=0.5",
+ "ipython>=8.0",
+]
+rust = [
+ "maturin>=1.0",
+]
+plink = [
+ "Pgenlib>=0.90",
+]
+cyvcf2 = [
+ "cyvcf2>=0.31.0",
+]
+
+[project.scripts]
+wasp2-count = "counting.__main__:app"
+wasp2-map = "mapping.__main__:app"
+wasp2-analyze = "analysis.__main__:app"
+wasp2-ipscore = "ipscore.__main__:app"
+
+[project.urls]
+Homepage = "https://github.com/Jaureguy760/WASP2-final"
+Documentation = "https://Jaureguy760.github.io/WASP2-final/"
+Repository = "https://github.com/Jaureguy760/WASP2-final"
+Issues = "https://github.com/Jaureguy760/WASP2-final/issues"
+
+# Note: [tool.setuptools] is not used when building with maturin
+# Maturin handles package discovery via [tool.maturin.python-packages]
+# The setuptools config below is kept as reference if ever switching to pure Python builds:
+# [tool.setuptools]
+# package-dir = {"" = "src"}
+# [tool.setuptools.packages.find]
+# where = ["src"]
+# include = ["counting*", "mapping*", "analysis*", "wasp2*"]
+
+[tool.maturin]
+manifest-path = "rust/Cargo.toml"
+python-source = "src"
+python-packages = ["counting", "mapping", "analysis", "wasp2", "ipscore"]
+bindings = "pyo3"
+strip = true
+include = ["LICENSE", "README.md"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = "test_*.py"
+python_classes = "Test*"
+python_functions = "test_*"
+addopts = "-v --strict-markers --tb=short"
+markers = [
+ "unit: unit tests",
+ "integration: integration tests",
+ "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+ "rust: tests requiring Rust backend",
+ "benchmark: performance benchmark tests",
+ "requires_plink2: marks tests that require plink2",
+ "requires_bcftools: marks tests that require bcftools",
+ "sanity: Sanity tests using real chr21 HG00731 data",
+ "slow_sanity: Slow sanity tests (>30 seconds)",
+]
+
+[tool.mypy]
+python_version = "3.10"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = false
+ignore_missing_imports = true
+explicit_package_bases = true
+files = ["src"]
+
+[tool.basedpyright]
+pythonVersion = "3.10"
+include = ["src"]
+exclude = ["**/__pycache__", "build", "dist", ".venv"]
+typeCheckingMode = "standard"
+reportMissingImports = "warning"
+reportMissingTypeStubs = false
+reportUnusedImport = false # Handled by Ruff
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+src = ["src", "tests"]
+exclude = [
+ ".git",
+ ".venv",
+ "__pycache__",
+ "build",
+ "dist",
+ "*.egg-info",
+]
+
+[tool.ruff.lint]
+select = [
+ "E", # pycodestyle errors
+ "F", # Pyflakes
+ "I", # isort
+ "UP", # pyupgrade
+ "B", # flake8-bugbear
+ "SIM", # flake8-simplify
+ "C4", # flake8-comprehensions
+]
+ignore = [
+ "E501", # line too long (handled by formatter)
+ "E712", # comparison to True/False (pandas boolean indexing style)
+ "E722", # bare except (reviewed case-by-case)
+ "E731", # lambda assignment (sometimes clearer)
+ "E741", # ambiguous variable name (l, O, I) - common in math/bio code
+ "B008", # function calls in argument defaults (needed for typer)
+ "B905", # zip without strict (would break existing logic)
+ "SIM102", # collapsible-if (sometimes less readable)
+ "SIM105", # suppressible-exception (contextlib.suppress less explicit)
+ "SIM108", # ternary instead of if-else (sometimes less readable)
+ "SIM115", # context manager for open (already handled in most cases)
+ "SIM116", # dict lookup instead of if-else (case-dependent)
+ "SIM117", # nested with statements (pytest context managers)
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["F841"] # unused variables OK in tests
+
+[tool.ruff.lint.isort]
+known-first-party = ["wasp2", "counting", "mapping", "analysis", "ipscore"]
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+docstring-code-format = true
+
+[tool.coverage.run]
+source = ["src"]
+omit = ["*/tests/*", "*/__pycache__/*"]
+
+[tool.coverage.report]
+precision = 2
+show_missing = true
+
+# Bandit security linter configuration
+[tool.bandit]
+exclude_dirs = ["tests", ".venv", "build", "dist"]
+skips = [
+ "B101", # assert_used - OK in tests and CLI validation
+ "B404", # import_subprocess - needed for external tool calls
+ "B603", # subprocess_without_shell_equals_true - intentional for safety
+ "B607", # start_process_with_partial_path - expected for samtools/bcftools CLI calls
+]
+targets = ["src"]
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..bf7196f
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,25 @@
+# WASP2 Python Dependencies
+# Install with: pip install -r requirements.txt
+# Note: System dependencies (bcftools, samtools, bedtools) must be installed separately
+
+# Data processing
+numpy>=1.21.0
+pandas>=2.0.0,<2.4.0 # Requires anndata>=0.10.7 for compatibility
+polars>=0.19.0
+scipy>=1.9.0
+
+# Bioinformatics
+pysam>=0.21.0
+pybedtools>=0.9.0
+anndata>=0.10.7,<0.12.0 # 0.10.7+ has pandas 2.x compatibility fixes
+
+# CLI
+typer>=0.9.0
+typing-extensions>=4.0.0
+
+# Testing
+pytest>=7.0.0
+pytest-cov>=4.0.0
+
+# Type checking
+mypy>=1.0.0
diff --git a/rust/AUDIT.md b/rust/AUDIT.md
new file mode 100644
index 0000000..391e2d7
--- /dev/null
+++ b/rust/AUDIT.md
@@ -0,0 +1,254 @@
+# Rust Code Quality & Safety Audit Report
+
+**Issue:** #199
+**Scope:** 15 Rust source files in `rust/src/` (10,679 lines)
+**Date:** 2026-02-02
+
+---
+
+## Executive Summary
+
+The WASP2 Rust codebase is well-structured and demonstrates strong engineering practices. **Zero `unsafe` blocks** were found across all 10,679 lines. Error handling is consistent, PyO3 bindings are correctly implemented, and memory safety is upheld through Rust's ownership system. The audit identified **4 bugs**, **3 moderate concerns**, and **10 warnings** to address.
+
+**Overall Risk:** LOW — no data corruption vectors or memory safety issues found.
+
+---
+
+## 1. Unsafe Code Audit
+
+**Result: ZERO `unsafe` blocks found.**
+
+All performance-critical operations use safe abstractions:
+- `rust-htslib` wraps the C `htslib` library internally (unsafe is contained within the crate)
+- `coitrees` uses safe Rust with SIMD optimizations
+- `rayon` provides safe parallelism via `par_iter()`
+- `crossbeam-channel` provides safe MPMC channels
+
+No `transmute`, raw pointer dereference, or `unsafe impl Send/Sync` found anywhere.
+
+---
+
+## 2. Error Handling Review
+
+### Pattern Used
+All modules consistently use `anyhow::Result
+
+
+
+
+
+
+