[Data] Data acquisition scripts (spot history, traces, incident archives)

[JasonEran]

Epic: #11

Goal

Collect public datasets for replay/backtesting with clear licensing.

Acceptance Criteria

• Scripts + README.
• Data provenance documented.

[JasonEran]

#34 implementation started (2026-02-22 UTC)

Initial implementation is complete locally:

Delivered

• Data acquisition scripts (spot history / traces / incident archives):
  • scripts/data_acquisition/fetch_spot_history.py
  • scripts/data_acquisition/fetch_cluster_traces.py
  • scripts/data_acquisition/fetch_incident_archives.py
• Script usage docs:
  • scripts/data_acquisition/README.md
• Data provenance docs (license/terms + manifest conventions):
  • docs/Data-Provenance-v2.3-M2.md
• README docs index now includes M2 data docs.

Local validation

• Syntax check:
  • python -m py_compile scripts/data_acquisition/fetch_spot_history.py scripts/data_acquisition/fetch_cluster_traces.py scripts/data_acquisition/fetch_incident_archives.py
• Smoke runs:
  • python scripts/data_acquisition/fetch_spot_history.py --source spot-js --max-records 2000 --output-dir .tmp/data-acq-smoke
  • python scripts/data_acquisition/fetch_cluster_traces.py --max-files 1 --output-dir .tmp/data-acq-smoke
  • python scripts/data_acquisition/fetch_incident_archives.py --max-items-per-feed 5 --output-dir .tmp/data-acq-smoke
• All scripts generated provenance/*_manifest.json.

Next: merge into PR #49 and close #34 after final evidence update.

[JasonEran]

#34 final evidence update (2026-02-22 UTC)

Acceptance criteria are satisfied and changes are pushed on feature/v2.3:

• Commit: 44ced1c (feat(v2.3): add M2 data acquisition scripts and provenance docs)
• PR: [v2.3] Full delivery M1-M4 + release readiness docs #49

Delivered files

• scripts/data_acquisition/fetch_spot_history.py
• scripts/data_acquisition/fetch_cluster_traces.py
• scripts/data_acquisition/fetch_incident_archives.py
• scripts/data_acquisition/README.md
• docs/Data-Provenance-v2.3-M2.md
• README.md (added M2 docs index entries)

Validation logs

• Syntax check:
  • python -m py_compile scripts/data_acquisition/fetch_spot_history.py scripts/data_acquisition/fetch_cluster_traces.py scripts/data_acquisition/fetch_incident_archives.py
• Smoke runs:
  • python scripts/data_acquisition/fetch_spot_history.py --source spot-js --max-records 500 --output-dir .tmp/data-acq-smoke
    • wrote 500 rows + spot_history_manifest.json
  • python scripts/data_acquisition/fetch_cluster_traces.py --max-files 1 --skip-existing --output-dir .tmp/data-acq-smoke
    • generated cluster_traces_manifest.json
  • python scripts/data_acquisition/fetch_incident_archives.py --max-items-per-feed 5 --output-dir .tmp/data-acq-smoke
    • generated incident_archives_manifest.json

Acceptance checklist

• Scripts + README
• Data provenance documented

#34 is ready to close.