-
Notifications
You must be signed in to change notification settings - Fork 0
chore: add strict codecov and zero-issue quality gates #98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
55bf3e1
a63ce4a
3663b79
9dcc31b
3eaafc5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| name: Codacy Zero | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main, master] | ||
| pull_request: | ||
| branches: [main, master] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| codacy-zero: | ||
| name: Codacy Zero | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| - name: Assert Codacy zero-open gate | ||
| env: | ||
| CODACY_API_TOKEN: ${{ secrets.CODACY_API_TOKEN }} | ||
| run: | | ||
| python3 scripts/quality/check_codacy_zero.py \ | ||
| --owner "${GITHUB_REPOSITORY_OWNER}" \ | ||
| --repo "${GITHUB_REPOSITORY#*/}" \ | ||
| --out-json "codacy-zero/codacy.json" \ | ||
| --out-md "codacy-zero/codacy.md" | ||
| - name: Upload Codacy artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: codacy-zero | ||
| path: codacy-zero |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| name: Codecov Analytics | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main, master] | ||
| pull_request: | ||
| branches: [main, master] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| codecov-analytics: | ||
| name: Codecov Analytics | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| - uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.12' | ||
| - uses: actions/setup-node@v6 | ||
| with: | ||
| node-version: '20' | ||
|
|
||
| - uses: actions/setup-dotnet@v4 | ||
| with: | ||
| dotnet-version: '8.0.x' | ||
| - name: Run tests with coverage | ||
| run: | | ||
| mkdir -p coverage | ||
| dotnet restore | ||
| dotnet test tests/SwfocTrainer.Tests/SwfocTrainer.Tests.csproj -c Release /p:CollectCoverage=true /p:CoverletOutput=./TestResults/coverage.cobertura.xml /p:CoverletOutputFormat=cobertura | ||
| - name: Upload coverage to Codecov | ||
| uses: codecov/codecov-action@v5 | ||
|
Check warning on line 37 in .github/workflows/codecov-analytics.yml
|
||
| with: | ||
| files: tests/SwfocTrainer.Tests/TestResults/coverage.cobertura.xml | ||
|
Comment on lines
+31
to
+39
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 4. Coverlet coverage not wired The workflows pass MSBuild Coverlet properties (CollectCoverage, CoverletOutput), but the test project/repo does not reference any Coverlet package/config, so the coverage file likely won’t be generated and downstream steps will fail. Agent Prompt
|
||
| flags: dotnet | ||
| fail_ci_if_error: true | ||
| verbose: true | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| name: Coverage 100 | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main, master] | ||
| pull_request: | ||
| branches: [main, master] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| coverage-100: | ||
| name: Coverage 100 Gate | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| - uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.12' | ||
| - uses: actions/setup-node@v6 | ||
| with: | ||
| node-version: '20' | ||
|
Comment on lines
+22
to
+24
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: cat .github/workflows/coverage-100.ymlRepository: Prekzursil/SWFOC-Mod-Menu Length of output: 1476 🏁 Script executed: rg -n -C2 -e 'setup-node|node-version|npm|pnpm|yarn|\\bnode\\b' .github/workflows/coverage-100.ymlRepository: Prekzursil/SWFOC-Mod-Menu Length of output: 254 Remove unused Node setup from this workflow. Line 22 adds 🤖 Prompt for AI Agents |
||
|
|
||
| - uses: actions/setup-dotnet@v4 | ||
| with: | ||
| dotnet-version: '8.0.x' | ||
| - name: Run tests with coverage | ||
| run: | | ||
| mkdir -p coverage | ||
| dotnet restore | ||
| dotnet test tests/SwfocTrainer.Tests/SwfocTrainer.Tests.csproj -c Release /p:CollectCoverage=true /p:CoverletOutput=./TestResults/coverage.cobertura.xml /p:CoverletOutputFormat=cobertura | ||
|
|
||
|
Comment on lines
+14
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 3. Linux runner for windows tests The new coverage workflows run dotnet test on ubuntu-latest, but the test project targets net8.0-windows, so these jobs are likely to fail to build/run and will block merges. Agent Prompt
|
||
| - name: Enforce 100% coverage | ||
| run: | | ||
| python3 scripts/quality/assert_coverage_100.py \ | ||
| --xml "dotnet=tests/SwfocTrainer.Tests/TestResults/coverage.cobertura.xml" \ | ||
| --out-json "coverage-100/coverage.json" \ | ||
| --out-md "coverage-100/coverage.md" | ||
| - name: Upload coverage artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: coverage-100 | ||
| path: | | ||
| coverage | ||
| **/coverage | ||
| **/TestResults | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| name: DeepScan Zero | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main, master] | ||
| pull_request: | ||
| branches: [main, master] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
| checks: read | ||
|
|
||
| jobs: | ||
| deepscan-zero: | ||
| name: DeepScan Zero | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: # First, let's examine the actual content of the workflow file
cat -n .github/workflows/deepscan-zero.yml | head -40Repository: Prekzursil/SWFOC-Mod-Menu Length of output: 1108 🏁 Script executed: # Run the suggested verification command to find mutable action references
rg -n '^\s*-\s*uses:\s*[^@]+@v[0-9]+' .github/workflowsRepository: Prekzursil/SWFOC-Mod-Menu Length of output: 1502 🌐 Web query:
💡 Result: Recommendation (security-first)Pin third‑party actions to a full-length commit SHA (e.g., When version tags are acceptableIf you need human-readable versions, pin to a tag that is part of an immutable release (e.g., Practical best practice pattern (security + maintainability)
Org/enterprise enforcementGitHub Actions policies can now enforce SHA pinning (workflows fail if an action isn’t pinned to a full commit SHA) and also support blocking specific actions/versions. [3] Sources[1] GitHub Well-Architected Framework – “Securing GitHub Actions Workflows” (Pin versions of actions) (wellarchitected.github.com) Citations:
Pin GitHub Actions to full commit SHAs instead of mutable version tags. Lines 21 and 29 use mutable version tags ( This pattern is used throughout the repository's workflows and should be addressed consistently. 🤖 Prompt for AI Agents |
||
| - name: Assert DeepScan vendor check is green | ||
| run: | | ||
| python3 scripts/quality/check_required_checks.py \ | ||
| --repo "${GITHUB_REPOSITORY}" \ | ||
| --sha "${GITHUB_SHA}" \ | ||
| --required-context "DeepScan" \ | ||
| --timeout-seconds 1200 \ | ||
| --poll-seconds 20 \ | ||
| --out-json "deepscan-zero/deepscan.json" \ | ||
| --out-md "deepscan-zero/deepscan.md" | ||
| - name: Upload DeepScan artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: deepscan-zero | ||
| path: deepscan-zero | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,82 @@ | ||||||||||||||||||||||||||||||||||||||
| name: Quality Zero Gate | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| on: | ||||||||||||||||||||||||||||||||||||||
| push: | ||||||||||||||||||||||||||||||||||||||
| branches: [main, master] | ||||||||||||||||||||||||||||||||||||||
| pull_request: | ||||||||||||||||||||||||||||||||||||||
| branches: [main, master] | ||||||||||||||||||||||||||||||||||||||
| workflow_dispatch: | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| permissions: | ||||||||||||||||||||||||||||||||||||||
| contents: read | ||||||||||||||||||||||||||||||||||||||
Check noticeCode scanning / SonarCloud Read permissions should be defined at the job level Low
Move this read permission from workflow level to job level. See more on SonarQube Cloud
|
||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+10
to
+12
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: cat -n .github/workflows/quality-zero-gate.ymlRepository: Prekzursil/SWFOC-Mod-Menu Length of output: 3700 🏁 Script executed: cat -n scripts/quality/check_required_checks.pyRepository: Prekzursil/SWFOC-Mod-Menu Length of output: 8355 🌐 Web query:
💡 Result: GitHub REST API “check-runs” endpoints — required permissionsKey rule: Write access to Checks (creating/updating/rerequesting check runs) is only available to GitHub Apps; OAuth apps and user tokens can view check runs but can’t create/update them. [1]
Tip: GitHub also returns Sources: GitHub REST “Check runs” docs [1]; “Permissions required for GitHub Apps” matrix [2]. 🌐 Web query:
💡 Result: GitHub Status endpoint (service health)
GitHub REST API commit statuses (per-commit CI/status contexts)Key REST endpoints
Permission requirements
Repo access note
Sources: [1] GitHub Status API docs, [2] Commit Statuses REST endpoints, [3] Fine-grained PAT permissions table, [4] OAuth scopes ( Grant explicit read scopes for checks and status APIs. Lines 10-12 only grant Suggested fix permissions:
contents: read
+ checks: read
+ statuses: read📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||||||||||
| jobs: | ||||||||||||||||||||||||||||||||||||||
| secrets-preflight: | ||||||||||||||||||||||||||||||||||||||
| name: Quality Secrets Preflight | ||||||||||||||||||||||||||||||||||||||
| runs-on: ubuntu-latest | ||||||||||||||||||||||||||||||||||||||
| env: | ||||||||||||||||||||||||||||||||||||||
| SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} | ||||||||||||||||||||||||||||||||||||||
| CODACY_API_TOKEN: ${{ secrets.CODACY_API_TOKEN }} | ||||||||||||||||||||||||||||||||||||||
| CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} | ||||||||||||||||||||||||||||||||||||||
| SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} | ||||||||||||||||||||||||||||||||||||||
| SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }} | ||||||||||||||||||||||||||||||||||||||
| SENTRY_ORG: ${{ vars.SENTRY_ORG }} | ||||||||||||||||||||||||||||||||||||||
| SENTRY_PROJECT: ${{ vars.SENTRY_PROJECT }} | ||||||||||||||||||||||||||||||||||||||
| DEEPSCAN_POLICY_MODE: ${{ vars.DEEPSCAN_POLICY_MODE }} | ||||||||||||||||||||||||||||||||||||||
| DEEPSCAN_OPEN_ISSUES_URL: ${{ vars.DEEPSCAN_OPEN_ISSUES_URL }} | ||||||||||||||||||||||||||||||||||||||
| DEEPSCAN_API_TOKEN: ${{ secrets.DEEPSCAN_API_TOKEN }} | ||||||||||||||||||||||||||||||||||||||
| steps: | ||||||||||||||||||||||||||||||||||||||
| - uses: actions/checkout@v6 | ||||||||||||||||||||||||||||||||||||||
| - name: Run quality secrets preflight | ||||||||||||||||||||||||||||||||||||||
| run: | | ||||||||||||||||||||||||||||||||||||||
| python3 scripts/quality/check_quality_secrets.py \ | ||||||||||||||||||||||||||||||||||||||
| --out-json quality-secrets/secrets.json \ | ||||||||||||||||||||||||||||||||||||||
| --out-md quality-secrets/secrets.md | ||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+30
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 1. Tooling changes lack test evidence This PR introduces new CI/quality tooling but does not include deterministic test evidence artifacts for these changes nor an explicit, documented skip justification in-repo. This reduces auditability/reproducibility of the claimed quality enforcement behavior. Agent Prompt
|
||||||||||||||||||||||||||||||||||||||
| - name: Upload secrets preflight artifact | ||||||||||||||||||||||||||||||||||||||
| if: always() | ||||||||||||||||||||||||||||||||||||||
| uses: actions/upload-artifact@v4 | ||||||||||||||||||||||||||||||||||||||
| with: | ||||||||||||||||||||||||||||||||||||||
| name: quality-secrets | ||||||||||||||||||||||||||||||||||||||
| path: quality-secrets | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| quality-zero-gate: | ||||||||||||||||||||||||||||||||||||||
| name: Quality Zero Gate | ||||||||||||||||||||||||||||||||||||||
| if: always() | ||||||||||||||||||||||||||||||||||||||
| runs-on: ubuntu-latest | ||||||||||||||||||||||||||||||||||||||
| needs: | ||||||||||||||||||||||||||||||||||||||
| - secrets-preflight | ||||||||||||||||||||||||||||||||||||||
| env: | ||||||||||||||||||||||||||||||||||||||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||||||||||||||||||||||||||||||||||||||
| steps: | ||||||||||||||||||||||||||||||||||||||
| - uses: actions/checkout@v6 | ||||||||||||||||||||||||||||||||||||||
| - name: Assert secrets preflight succeeded | ||||||||||||||||||||||||||||||||||||||
| run: | | ||||||||||||||||||||||||||||||||||||||
| if [ "${{ needs.secrets-preflight.result }}" != "success" ]; then | ||||||||||||||||||||||||||||||||||||||
| echo "Quality Secrets Preflight failed or was not successful." >&2 | ||||||||||||||||||||||||||||||||||||||
| exit 1 | ||||||||||||||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||||||||||||||
| - name: Assert required quality contexts are green | ||||||||||||||||||||||||||||||||||||||
| run: | | ||||||||||||||||||||||||||||||||||||||
| python3 scripts/quality/check_required_checks.py \ | ||||||||||||||||||||||||||||||||||||||
| --repo "${GITHUB_REPOSITORY}" \ | ||||||||||||||||||||||||||||||||||||||
| --sha "${GITHUB_SHA}" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "Coverage 100 Gate" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "Codecov Analytics" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "Sonar Zero" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "Codacy Zero" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "Snyk Zero" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "Sentry Zero" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "DeepScan Zero" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "SonarCloud Code Analysis" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "Codacy Static Code Analysis" \ | ||||||||||||||||||||||||||||||||||||||
| --required-context "DeepScan" \ | ||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+60
to
+72
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 6. Sonar context name mismatch Quality Zero Gate waits for a required context named "SonarCloud Code Analysis", but this repository’s Sonar workflow job is named sonarcloud; the required-checks poller will never observe the expected context and will fail. Agent Prompt
|
||||||||||||||||||||||||||||||||||||||
| --timeout-seconds 1500 \ | ||||||||||||||||||||||||||||||||||||||
| --poll-seconds 20 \ | ||||||||||||||||||||||||||||||||||||||
| --out-json quality-zero-gate/required-checks.json \ | ||||||||||||||||||||||||||||||||||||||
| --out-md quality-zero-gate/required-checks.md | ||||||||||||||||||||||||||||||||||||||
| - name: Upload aggregate artifact | ||||||||||||||||||||||||||||||||||||||
| if: always() | ||||||||||||||||||||||||||||||||||||||
| uses: actions/upload-artifact@v4 | ||||||||||||||||||||||||||||||||||||||
| with: | ||||||||||||||||||||||||||||||||||||||
| name: quality-zero-gate | ||||||||||||||||||||||||||||||||||||||
| path: quality-zero-gate | ||||||||||||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| name: Sentry Zero | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main, master] | ||
| pull_request: | ||
| branches: [main, master] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| sentry-zero: | ||
| name: Sentry Zero | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }} | ||
| SENTRY_ORG: ${{ vars.SENTRY_ORG }} | ||
| SENTRY_PROJECT: ${{ vars.SENTRY_PROJECT }} | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| - name: Assert Sentry unresolved issues are zero | ||
| run: | | ||
| python3 scripts/quality/check_sentry_zero.py \ | ||
| --project "${SENTRY_PROJECT}" \ | ||
| --out-json "sentry-zero/sentry.json" \ | ||
| --out-md "sentry-zero/sentry.md" | ||
| - name: Upload Sentry artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: sentry-zero | ||
| path: sentry-zero |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❓ Verification inconclusive
Script executed:
Repository: Prekzursil/SWFOC-Mod-Menu
Repository: Prekzursil/SWFOC-Mod-Menu
Exit code: 0
stdout:
GitHub Actions pinning best practices security mutable tags SHA
GitHub Actions pinning (security) — best practices
uses:to make the reference immutable and reduce supply‑chain risk from moved tags/branches. Example:uses: actions/checkout@<sha> # vX.Y.Z. [1][2]@v1/@main(or even@v1.2.3if it’s not protected by immutability), because tags can be moved/deleted if a repo is compromised. [1]Sources
[1] GitHub Docs – Security hardening for GitHub Actions (third‑party actions, SHA vs tags) (docs.github.com)
[2] GitHub Well‑Architected – Actions security recommendations (SHA pinning, comments, mutable deps, enforcement) (wellarchitected.github.com)
[3] GitHub Docs – Immutable releases (what they protect) (docs.github.com)
[4] GitHub Docs – Using immutable releases and tags for actions (docs.github.com)
[5] GitHub Docs – Keeping your actions up to date with Dependabot (docs.github.com)
[6] GitHub Changelog – Dependabot updates support reusable workflows (github.blog)
[7] GitHub Docs (Enterprise) – Policies: “Require actions to be pinned to a full-length commit SHA” (docs.github.com)
[8] GitHub Changelog (2025‑08‑15) – Policy supports SHA pinning enforcement (github.blog)
Citations:
Pin all GitHub Actions to immutable commit SHAs.
Lines 20, 21, 24, and 43 use mutable version tags (
@v6,@v5), which can be moved, deleted, or compromised. According to GitHub security hardening guidance, pin to full-length commit SHAs (40 chars) and add a comment with the human-readable version for maintainability and auditability. Use Dependabot to automate SHA updates.Example hardening pattern
🤖 Prompt for AI Agents