Skip to content

feat: unify prerequisite validation via IRequirementCheck + IPrerequisiteRunner (#106)#312

Open
sellakumaran wants to merge 3 commits intomainfrom
users/sellak/failEarly
Open

feat: unify prerequisite validation via IRequirementCheck + IPrerequisiteRunner (#106)#312
sellakumaran wants to merge 3 commits intomainfrom
users/sellak/failEarly

Conversation

@sellakumaran
Copy link
Contributor

Commands now declare prerequisites using IRequirementCheck and fail early with actionable messages before any side effects occur.

Phase 1 - pure reorganization (zero behavioral change):

  • Add AzureAuthRequirementCheck and InfrastructureRequirementCheck adapters
  • Add IPrerequisiteRunner / PrerequisiteRunner to run checks in order
  • Route AllSubcommand, BlueprintSubcommand, InfrastructureSubcommand, and DeployCommand through the shared runner instead of ad-hoc validators
  • Delete dead code: ISubCommand.ValidateAsync, IAzureValidator/AzureValidator
  • Make AzureAuthValidator.ValidateAuthenticationAsync virtual for testability

Phase 2 - minimal early-fail additions:

  • cleanup azure: auth check before preview display
  • deploy mcp: explicit early guards for agentBlueprintId and agenticAppId before any Graph/network calls

Closes #106

…siteRunner (#106)

Commands now declare prerequisites using IRequirementCheck and fail early
with actionable messages before any side effects occur.

Phase 1 - pure reorganization (zero behavioral change):
- Add AzureAuthRequirementCheck and InfrastructureRequirementCheck adapters
- Add IPrerequisiteRunner / PrerequisiteRunner to run checks in order
- Route AllSubcommand, BlueprintSubcommand, InfrastructureSubcommand,
  and DeployCommand through the shared runner instead of ad-hoc validators
- Delete dead code: ISubCommand.ValidateAsync, IAzureValidator/AzureValidator
- Make AzureAuthValidator.ValidateAuthenticationAsync virtual for testability

Phase 2 - minimal early-fail additions:
- cleanup azure: auth check before preview display
- deploy mcp: explicit early guards for agentBlueprintId and agenticAppId
  before any Graph/network calls

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@sellakumaran sellakumaran requested review from a team as code owners March 6, 2026 23:36
Copilot AI review requested due to automatic review settings March 6, 2026 23:36
@github-actions
Copy link

github-actions bot commented Mar 6, 2026

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR standardizes CLI prerequisite validation by introducing a shared IPrerequisiteRunner that executes IRequirementCheck implementations, replacing ad-hoc validators and enabling consistent early-fail behavior across commands.

Changes:

  • Added IPrerequisiteRunner/PrerequisiteRunner plus new requirement checks (AzureAuthRequirementCheck, InfrastructureRequirementCheck) and corresponding unit tests.
  • Routed setup and deploy flows through the shared runner; removed legacy IAzureValidator/AzureValidator and ISubCommand.ValidateAsync.
  • Added additional early guards for deploy mcp to fail before Graph/network calls when required IDs are missing.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/Microsoft.Agents.A365.DevTools.Cli/Services/Requirements/RequirementChecks/InfrastructureRequirementCheck.cs Adds config-only infrastructure field validation as a reusable requirement check.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Requirements/RequirementChecks/AzureAuthRequirementCheck.cs Adds an adapter requirement check over AzureAuthValidator.
src/Microsoft.Agents.A365.DevTools.Cli/Services/IPrerequisiteRunner.cs Introduces interface for running prerequisite checks.
src/Microsoft.Agents.A365.DevTools.Cli/Services/PrerequisiteRunner.cs Implements centralized prerequisite execution and logging.
src/Microsoft.Agents.A365.DevTools.Cli/Services/AzureAuthValidator.cs Makes ValidateAuthenticationAsync virtual for testability.
src/Microsoft.Agents.A365.DevTools.Cli/Program.cs Rewires DI: removes IAzureValidator, registers IPrerequisiteRunner, passes new deps to commands.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupCommand.cs Updates setup command wiring to use prerequisite runner + auth/environment validators.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/AllSubcommand.cs Replaces manual validation collection with requirement checks + runner; invokes environment validator.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/InfrastructureSubcommand.cs Uses prerequisite runner for Azure auth + infra config checks before provisioning.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/BlueprintSubcommand.cs Uses prerequisite runner for Azure auth prior to blueprint creation.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/DeployCommand.cs Uses prerequisite runner for Azure auth in deploy flows; adds early guards in deploy mcp.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/CleanupCommand.cs Adds prerequisite runner + auth validator to Azure cleanup subcommand.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/CreateInstanceCommand.cs Removes Azure validation dependency from deprecated command (still exits immediately).
src/Microsoft.Agents.A365.DevTools.Cli/Services/AzureValidator.cs Removes legacy unified Azure validator.
src/Microsoft.Agents.A365.DevTools.Cli/Services/ISubCommand.cs Removes legacy subcommand validation interface.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Requirements/InfrastructureRequirementCheckTests.cs Adds unit tests for InfrastructureRequirementCheck.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Requirements/AzureAuthRequirementCheckTests.cs Adds unit tests for AzureAuthRequirementCheck.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/PrerequisiteRunnerTests.cs Adds unit tests for PrerequisiteRunner behavior (pass/fail/warn/order).
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/SubcommandValidationTests.cs Updates infra validation tests to use requirement checks instead of removed validators.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/SetupCommandTests.cs Updates command construction/tests for new prerequisite runner + validators.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/DeployCommandTests.cs Updates deploy command tests for new prerequisite runner + auth validator injection.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/CreateInstanceCommandTests.cs Updates test wiring due to removed Azure validator parameter.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/CleanupCommandTests.cs Updates cleanup tests for new prerequisite runner + auth validator injection.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/CleanupCommandBotEndpointTests.cs Updates bot-endpoint cleanup tests for new prerequisite runner + auth validator injection.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/BlueprintSubcommandTests.cs Updates blueprint subcommand tests to use prerequisite runner + auth validator instead of removed Azure validator.

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +340 to +341
if (!await prerequisiteRunner.RunAsync(authChecks, config, logger, CancellationToken.None))
return;
Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the Azure auth prerequisite fails, this handler currently just returns, which will likely result in a zero exit code even though cleanup could not proceed. Consider exiting with a non-zero code (e.g., ExceptionHandler.ExitWithCleanup(1)) after prerequisiteRunner reports failure so scripts can detect the error.

Suggested change
if (!await prerequisiteRunner.RunAsync(authChecks, config, logger, CancellationToken.None))
return;
if (!await prerequisiteRunner.RunAsync(authChecks, config, logger, CancellationToken.None))
{
ExceptionHandler.ExitWithCleanup(1);
return;
}

Copilot uses AI. Check for mistakes.
Comment on lines +38 to +40
else if (result.IsWarning && !string.IsNullOrWhiteSpace(result.Details))
{
logger.LogWarning("{CheckName}: {Details}", check.Name, result.Details);
Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PrerequisiteRunner only logs warnings when IsWarning is true AND Details is non-empty, and it ignores the warning message (RequirementCheckResult.Warning sets ErrorMessage). This means warnings with no Details will be silently dropped, and even when Details is present the main warning message is lost. Consider logging the warning message (ErrorMessage) and falling back to Details, and logging whenever IsWarning is true (even if Details is empty).

Suggested change
else if (result.IsWarning && !string.IsNullOrWhiteSpace(result.Details))
{
logger.LogWarning("{CheckName}: {Details}", check.Name, result.Details);
else if (result.IsWarning)
{
var warningMessage = !string.IsNullOrWhiteSpace(result.ErrorMessage)
? result.ErrorMessage
: result.Details;
if (!string.IsNullOrWhiteSpace(warningMessage))
{
logger.LogWarning("{CheckName}: {WarningMessage}", check.Name, warningMessage);
}
else
{
logger.LogWarning("{CheckName}: A warning was reported but no message was provided.", check.Name);
}

Copilot uses AI. Check for mistakes.
: config.AppServicePlanSku;

if (!IsValidAppServicePlanSku(sku))
errors.Add($"Invalid appServicePlanSku '{sku}'. Valid SKUs: F1 (Free), B1/B2/B3 (Basic), S1/S2/S3 (Standard), P1V2/P2V2/P3V2 (Premium V2), P1V3/P2V3/P3V3 (Premium V3)");
Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InfrastructureRequirementCheck accepts isolated SKUs (I1/I2/I3/I1V2/I2V2/I3V2) as valid, but the failure message shown for invalid SKUs does not list these as valid options. This is misleading for users; either include the isolated SKUs in the message or remove them from the accepted SKU list to keep validation and guidance consistent.

Suggested change
errors.Add($"Invalid appServicePlanSku '{sku}'. Valid SKUs: F1 (Free), B1/B2/B3 (Basic), S1/S2/S3 (Standard), P1V2/P2V2/P3V2 (Premium V2), P1V3/P2V3/P3V3 (Premium V3)");
errors.Add($"Invalid appServicePlanSku '{sku}'. Valid SKUs: F1 (Free), B1/B2/B3 (Basic), S1/S2/S3 (Standard), P1V2/P2V2/P3V2 (Premium V2), P1V3/P2V3/P3V3 (Premium V3), I1/I2/I3/I1V2/I2V2/I3V2 (Isolated)");

Copilot uses AI. Check for mistakes.
Comment on lines +228 to +233
return;
}
if (string.IsNullOrWhiteSpace(updateConfig.AgenticAppId))
{
logger.LogError("agenticAppId is not configured. Run 'a365 setup all' to complete setup.");
return;
Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These early-guard failures log an error and then return from the handler, which will likely produce a zero exit code even though the command failed. To make this an actionable early-fail, exit with a non-zero code (e.g., ExceptionHandler.ExitWithCleanup(1) / Environment.Exit(1), or set the invocation exit code) after logging.

Suggested change
return;
}
if (string.IsNullOrWhiteSpace(updateConfig.AgenticAppId))
{
logger.LogError("agenticAppId is not configured. Run 'a365 setup all' to complete setup.");
return;
Environment.Exit(1);
}
if (string.IsNullOrWhiteSpace(updateConfig.AgenticAppId))
{
logger.LogError("agenticAppId is not configured. Run 'a365 setup all' to complete setup.");
Environment.Exit(1);

Copilot uses AI. Check for mistakes.
Comment on lines +228 to +233
return;
}
if (string.IsNullOrWhiteSpace(updateConfig.AgenticAppId))
{
logger.LogError("agenticAppId is not configured. Run 'a365 setup all' to complete setup.");
return;
Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue here: logging an error and returning will likely exit with code 0. This should terminate the command with a non-zero exit code so automation/scripts can detect the failure.

Suggested change
return;
}
if (string.IsNullOrWhiteSpace(updateConfig.AgenticAppId))
{
logger.LogError("agenticAppId is not configured. Run 'a365 setup all' to complete setup.");
return;
Environment.Exit(1);
}
if (string.IsNullOrWhiteSpace(updateConfig.AgenticAppId))
{
logger.LogError("agenticAppId is not configured. Run 'a365 setup all' to complete setup.");
Environment.Exit(1);

Copilot uses AI. Check for mistakes.
Comment on lines 273 to +276
// Validate Azure CLI authentication, subscription, and environment
if (!await azureValidator.ValidateAllAsync(configData.SubscriptionId))
var checks = new List<Services.Requirements.IRequirementCheck>
{
new AzureAuthRequirementCheck(authValidator)
Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says this validates Azure CLI authentication, subscription, and environment, but the current check list only runs AzureAuthRequirementCheck and does not run the AzureEnvironmentValidator anymore. Please update the comment to match the actual behavior (or re-add the environment validation if it’s still intended here).

Copilot uses AI. Check for mistakes.
Comment on lines +221 to 223
// Run advisory environment check (warning only, never blocks)
await environmentValidator.ValidateEnvironmentAsync();

Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AllSubcommand now runs environmentValidator.ValidateEnvironmentAsync() (which executes az --version on Windows) before running the prerequisite checks. Previously, the environment check was only reached after Azure auth succeeded via AzureValidator. Consider running the environment check after AzureAuthRequirementCheck passes (or modeling it as a warning-only IRequirementCheck) to avoid invoking Azure CLI when auth has already failed.

Copilot uses AI. Check for mistakes.
sellakumaran and others added 2 commits March 6, 2026 16:08
- ConfigFileNotFoundException now extends Agent365Exception so missing
  config errors surface as clean user messages (no stack trace) on all
  commands, not just those with local catch blocks. Removes ad-hoc
  FileNotFoundException catches in CleanupCommand and CreateInstanceCommand.

- config init: expand relative/dot deployment paths to absolute before
  saving so the stored value is portable across directories. Update help
  text to clarify relative paths are accepted.

- config init: drop platform-specific parenthetical from 'Allow public
  client flows' log message -- the setting is required on all platforms.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move "Running all setup steps..." to after requirements check output
- Remove redundant "Agent 365 Setup" header (user already knows the command)
- Change CorrelationId log to LogDebug for setup all and blueprint; surface
  as TraceId inline on the action line ("Running all setup steps... (TraceId: ...)")
  so it is always captured in setup.log as [INF] and visible on console
- Demote PlatformDetector internal logs to LogDebug; single "Detected project
  platform: X" line remains as the user-facing output
- Add AzureAuthRequirementCheck to GetConfigRequirementChecks so Azure auth
  appears in requirements output for all setup subcommands
- Remove redundant mid-execution auth gate from BlueprintSubcommand that caused
  duplicate [PASS] Azure Authentication output
- Fix RequirementCheck base class: use LogInformation for all check result lines
  to avoid WARNING:/ERROR: prefix doubling from logger formatter
- Collapse verbose requirements summary to single line:
  "Requirements: X passed, Y warnings, Z failed"
- Update tests to match new message text and log level assertions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings March 7, 2026 02:37
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 7 comments.


You can also share your feedback on Copilot code review. Take the survey.

{
logger.LogWarning(" Details: {Details}", details);
}
logger.LogInformation("[WARN] {Name}{Details}", Name,
Copy link

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LogCheckWarning is emitting warnings at Information level. This makes warning checks indistinguishable from normal output in many logging configurations, and it also creates inconsistent behavior versus PrerequisiteRunner (which logs warnings at Warning). Consider logging warnings at LogLevel.Warning here to keep severity consistent across runners/commands.

Suggested change
logger.LogInformation("[WARN] {Name}{Details}", Name,
logger.LogWarning("[WARN] {Name}{Details}", Name,

Copilot uses AI. Check for mistakes.
Comment on lines +49 to +51
logger.LogInformation("[FAIL] {Name}", Name);
logger.LogInformation(" Issue: {ErrorMessage}", errorMessage);
logger.LogInformation(" Resolution: {ResolutionGuidance}", resolutionGuidance);
Copy link

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LogCheckFailure is using LogInformation for failures. In contexts where checks are run directly (e.g., setup requirements), failed prerequisites may not surface as errors and could be missed in logs. Consider using LogError (or at least LogWarning) here so failures keep the correct severity without requiring a separate runner to re-log them.

Suggested change
logger.LogInformation("[FAIL] {Name}", Name);
logger.LogInformation(" Issue: {ErrorMessage}", errorMessage);
logger.LogInformation(" Resolution: {ResolutionGuidance}", resolutionGuidance);
logger.LogError("[FAIL] {Name}", Name);
logger.LogError(" Issue: {ErrorMessage}", errorMessage);
logger.LogError(" Resolution: {ResolutionGuidance}", resolutionGuidance);

Copilot uses AI. Check for mistakes.
var validatedConfig = await ValidateDeploymentPrerequisitesAsync(
config.FullName, configService, azureValidator, executor, logger);
config.FullName, configService, prerequisiteRunner, authValidator, executor, logger);
if (validatedConfig == null) return;
Copy link

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When ValidateDeploymentPrerequisitesAsync returns null (e.g., failed Azure auth / missing web app), the handler simply returns, which will usually produce exit code 0. Consider setting a non-zero exit code when prerequisites fail so automated callers can detect deployment failure reliably.

Suggested change
if (validatedConfig == null) return;
if (validatedConfig == null)
{
System.Environment.ExitCode = 1;
return;
}

Copilot uses AI. Check for mistakes.
if (!await azureValidator.ValidateAllAsync(setupConfig.SubscriptionId))
var checks = new List<Services.Requirements.IRequirementCheck>
{
new AzureAuthRequirementCheck(authValidator),
Copy link

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prereq flow replaced azureValidator.ValidateAllAsync, but it no longer runs the advisory IAzureEnvironmentValidator.ValidateEnvironmentAsync() checks (32-bit Python on Windows, etc.). If that diagnostic is still desired for setup infrastructure, consider running the environment validator after auth (non-blocking) or adding an environment requirement-check adapter.

Suggested change
new AzureAuthRequirementCheck(authValidator),
new AzureAuthRequirementCheck(authValidator),
new EnvironmentRequirementCheck(),

Copilot uses AI. Check for mistakes.
Comment on lines +341 to +342
if (!await prerequisiteRunner.RunAsync(authChecks, config, logger, CancellationToken.None))
return;
Copy link

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the Azure auth prerequisite check fails, the handler currently just returns. Similar to other commands, this should likely produce a non-zero exit code so CI/scripts can detect failure (e.g., call the exit helper or set the invocation exit code).

Suggested change
if (!await prerequisiteRunner.RunAsync(authChecks, config, logger, CancellationToken.None))
return;
if (!await prerequisiteRunner.RunAsync(authChecks, config, logger, CancellationToken.None))
{
System.Environment.ExitCode = 1;
return;
}

Copilot uses AI. Check for mistakes.
Comment on lines +31 to +41

if (!string.IsNullOrWhiteSpace(result.ErrorMessage))
logger.LogError("{CheckName}: {ErrorMessage}", check.Name, result.ErrorMessage);

if (!string.IsNullOrWhiteSpace(result.ResolutionGuidance))
logger.LogError(" Resolution: {ResolutionGuidance}", result.ResolutionGuidance);
}
else if (result.IsWarning && !string.IsNullOrWhiteSpace(result.Details))
{
logger.LogWarning("{CheckName}: {Details}", check.Name, result.Details);
}
Copy link

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PrerequisiteRunner calls check.CheckAsync(config, logger, ...) and then logs failures/warnings itself. Since many checks already emit PASS/WARN/FAIL output via RequirementCheck.ExecuteCheckWithLoggingAsync, this can lead to duplicated or mixed-severity output. Consider having the runner either rely on the check's logging (just return a boolean), or run checks with a non-output logger and centralize logging in the runner.

Suggested change
if (!string.IsNullOrWhiteSpace(result.ErrorMessage))
logger.LogError("{CheckName}: {ErrorMessage}", check.Name, result.ErrorMessage);
if (!string.IsNullOrWhiteSpace(result.ResolutionGuidance))
logger.LogError(" Resolution: {ResolutionGuidance}", result.ResolutionGuidance);
}
else if (result.IsWarning && !string.IsNullOrWhiteSpace(result.Details))
{
logger.LogWarning("{CheckName}: {Details}", check.Name, result.Details);
}
}

Copilot uses AI. Check for mistakes.
Comment on lines +274 to +278
var checks = new List<Services.Requirements.IRequirementCheck>
{
new AzureAuthRequirementCheck(authValidator)
};
if (!await prerequisiteRunner.RunAsync(checks, configData, logger))
Copy link

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This auth validation replaced azureValidator.ValidateAllAsync, but it no longer runs the advisory Azure environment validation (Azure CLI architecture/version warnings). If that diagnostic is still useful during deploy, consider invoking IAzureEnvironmentValidator.ValidateEnvironmentAsync() (non-blocking) after auth succeeds.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

All commands should validate and fail early if needed

2 participants