Skip to content

Add full regression mode to integration test runner #372

@JayVDZ

Description

@JayVDZ

Summary

Add an -Scenario All option to Run-IntegrationTests.ps1 that runs every implemented integration test scenario in sequence, performing lightweight resets between scenarios. This enables a full regression test in a single invocation without redundant build or infrastructure startup steps.

Problem

Each integration test scenario must currently be run individually. Running a full regression suite requires manually invoking the script multiple times, each time paying the cost of Docker image builds, Samba AD startup, and service health checks. For 6 implemented scenarios, this means ~15-20 minutes of redundant infrastructure overhead.

Proposed Approach

  1. Build once — Docker images built at the start and reused
  2. Start all external systems upfront — All Samba AD instances started once
  3. Lightweight reset between scenarios — Reset JIM database and clean Samba AD OUs without rebuilding containers
  4. Aggregated results — Combined pass/fail summary with per-scenario breakdown
  5. Continue on failure — A failing scenario doesn't abort the remaining tests

Plan Document

See docs/plans/INTEGRATION_TEST_FULL_REGRESSION.md for the full design including infrastructure grouping, reset strategy, results format, and implementation notes.

Scope

  • Modify test/integration/Run-IntegrationTests.ps1 to support -Scenario All
  • Add "All Scenarios" to the interactive scenario menu
  • Implement lightweight reset function between scenarios
  • Aggregate results into a single JSON output

Acceptance Criteria

  • ./Run-IntegrationTests.ps1 -Scenario All -Template Nano runs all implemented scenarios sequentially
  • Interactive menu includes "All Scenarios (full regression)" option
  • JIM database is fully reset between scenarios (clean state)
  • Samba AD test data is cleaned between scenarios
  • Docker images are only built once at the start
  • External system containers are only started once
  • A failing scenario does not prevent subsequent scenarios from running
  • Aggregated results JSON saved to test/integration/results/
  • Exit code is non-zero if any scenario failed

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingSomething needs testing

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions