Add unit tests for checklist-runner module by nikolasdehor · Pull Request #250 · SynkraAI/aiox-core

nikolasdehor · 2026-02-18T19:42:56Z

Summary

Add 37 unit tests for the checklist-runner orchestration module
Cover checklist loading, YAML/markdown parsing, item normalization, validation execution, and summary generation
All file system operations properly mocked with fs-extra

Test Coverage

Area	Tests	Key Scenarios
constructor	1	Path initialization
loadChecklist	4	File loading, .md extension, missing files
parseChecklistItems	5	Markdown checkboxes, YAML blocks, invalid YAML, deduplication
normalizeItem	4	String/object normalization, blocker flags
evaluateItem	4	Manual items, validation, errors, custom messages
executeValidation	11	file exists, directory exists, not empty, contains, min size
run (integration)	3	Missing checklist, pass, blocker fail
getSummary	2	Missing checklist, category summary
Total	37

Test plan

All 37 tests pass
No external dependencies needed
Mocks fs-extra and js-yaml

Closes #310

vercel · 2026-02-18T19:43:01Z

@nikolasdehor is attempting to deploy a commit to the Pedro Valério Lopez's projects Team on Vercel.

A member of the Team first needs to authorize it.

coderabbitai · 2026-02-18T19:43:06Z

Warning

Rate limit exceeded

@nikolasdehor has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 0 minutes and 48 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

Adds a comprehensive Jest unit test suite for the .aios-core/core/orchestration/checklist-runner module to validate checklist loading/parsing and deterministic validation execution behavior.

Changes:

Introduces 37 unit/integration-style tests covering ChecklistRunner constructor, checklist loading, markdown/YAML parsing, item normalization, evaluation, validation rules, run(), and getSummary().
Mocks fs-extra and js-yaml to keep tests deterministic and filesystem-independent.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-18T19:48:08Z

tests/core/orchestration/checklist-runner.test.js

+    test('items with validation execute it', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      const item = {
+        description: 'File exists',
+        tipo: 'pre-conditions',
+        blocker: true,
+        validation: 'file exists',
+      };
+
+      const result = await runner.evaluateItem(item, '/src/index.js');
+      expect(result.passed).toBe(true);
+    });


Most tests pass absolute target paths (e.g., "/src/app.js"). In ChecklistRunner.executeValidation the code does path.join(projectRoot, targetPath); when targetPath is absolute, projectRoot is ignored, so these tests don’t exercise the intended “projectRoot + relative path” behavior. Use relative paths (e.g., "src/app.js") and/or assert fs.pathExists/fs.stat were called with the joined full path to better match real workflow usage (phase.creates values are relative).

Copilot · 2026-02-18T19:48:08Z

tests/core/orchestration/checklist-runner.test.js

+      expect(await runner.executeValidation('file exists', '/src/app.js')).toBe(true);
+    });
+
+    test('file exists - fails', async () => {
+      fs.pathExists.mockResolvedValue(false);
+      expect(await runner.executeValidation('file exists', '/missing.js')).toBe(false);
+    });
+
+    test('file exists with array paths', async () => {
+      fs.pathExists.mockResolvedValueOnce(true).mockResolvedValueOnce(true);
+      expect(await runner.executeValidation('file exists', ['/a.js', '/b.js'])).toBe(true);
+    });
+
+    test('directory exists - passes', async () => {
+      fs.stat.mockResolvedValue({ isDirectory: () => true });
+      expect(await runner.executeValidation('directory exists', '/src')).toBe(true);
+    });
+
+    test('directory exists - fails for file', async () => {
+      fs.stat.mockResolvedValue({ isDirectory: () => false });
+      expect(await runner.executeValidation('directory exists', '/file.txt')).toBe(false);
+    });
+
+    test('not empty - passes for non-empty file', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      fs.readFile.mockResolvedValue('content here');
+      expect(await runner.executeValidation('not empty', '/file.md')).toBe(true);
+    });
+
+    test('not empty - fails for empty file', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      fs.readFile.mockResolvedValue('   ');
+      expect(await runner.executeValidation('not empty', '/file.md')).toBe(false);
+    });
+
+    test('contains check - passes', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      fs.readFile.mockResolvedValue('# architecture\n## components');
+      expect(await runner.executeValidation("contains 'architecture'", '/doc.md')).toBe(true);
+    });
+
+    test('contains check - fails', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      fs.readFile.mockResolvedValue('# empty doc');
+      expect(await runner.executeValidation("contains 'architecture'", '/doc.md')).toBe(false);
+    });
+
+    test('contains check - fails for missing file', async () => {
+      fs.pathExists.mockResolvedValue(false);
+      expect(await runner.executeValidation("contains 'something'", '/missing.md')).toBe(false);
+    });
+
+    test('minimum size check - passes', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      fs.stat.mockResolvedValue({ size: 500 });
+      expect(await runner.executeValidation('minimum size: 100', '/file.md')).toBe(true);
+    });
+
+    test('minimum size check - fails', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      fs.stat.mockResolvedValue({ size: 50 });
+      expect(await runner.executeValidation('min size: 100', '/file.md')).toBe(false);
+    });
+
+    test('minimum size - fails for missing file', async () => {
+      fs.pathExists.mockResolvedValue(false);
+      expect(await runner.executeValidation('min size: 100', '/missing.md')).toBe(false);
+    });
+
+    test('unknown validation defaults to true', async () => {
+      expect(await runner.executeValidation('some human description', '/file')).toBe(true);


executeValidation tests use absolute paths ("/src", "/missing.js", etc.), which bypass projectRoot due to path.join semantics and can hide regressions in path handling. Switch these to relative paths and add expectations on fs.pathExists/fs.stat arguments so the tests validate the constructed full paths.

Suggested change

expect(await runner.executeValidation('file exists', '/src/app.js')).toBe(true);

});

test('file exists - fails', async () => {

fs.pathExists.mockResolvedValue(false);

expect(await runner.executeValidation('file exists', '/missing.js')).toBe(false);

});

test('file exists with array paths', async () => {

fs.pathExists.mockResolvedValueOnce(true).mockResolvedValueOnce(true);

expect(await runner.executeValidation('file exists', ['/a.js', '/b.js'])).toBe(true);

});

test('directory exists - passes', async () => {

fs.stat.mockResolvedValue({ isDirectory: () => true });

expect(await runner.executeValidation('directory exists', '/src')).toBe(true);

});

test('directory exists - fails for file', async () => {

fs.stat.mockResolvedValue({ isDirectory: () => false });

expect(await runner.executeValidation('directory exists', '/file.txt')).toBe(false);

});

test('not empty - passes for non-empty file', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue('content here');

expect(await runner.executeValidation('not empty', '/file.md')).toBe(true);

});

test('not empty - fails for empty file', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue(' ');

expect(await runner.executeValidation('not empty', '/file.md')).toBe(false);

});

test('contains check - passes', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue('# architecture\n## components');

expect(await runner.executeValidation("contains 'architecture'", '/doc.md')).toBe(true);

});

test('contains check - fails', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue('# empty doc');

expect(await runner.executeValidation("contains 'architecture'", '/doc.md')).toBe(false);

});

test('contains check - fails for missing file', async () => {

fs.pathExists.mockResolvedValue(false);

expect(await runner.executeValidation("contains 'something'", '/missing.md')).toBe(false);

});

test('minimum size check - passes', async () => {

fs.pathExists.mockResolvedValue(true);

fs.stat.mockResolvedValue({ size: 500 });

expect(await runner.executeValidation('minimum size: 100', '/file.md')).toBe(true);

});

test('minimum size check - fails', async () => {

fs.pathExists.mockResolvedValue(true);

fs.stat.mockResolvedValue({ size: 50 });

expect(await runner.executeValidation('min size: 100', '/file.md')).toBe(false);

});

test('minimum size - fails for missing file', async () => {

fs.pathExists.mockResolvedValue(false);

expect(await runner.executeValidation('min size: 100', '/missing.md')).toBe(false);

});

test('unknown validation defaults to true', async () => {

expect(await runner.executeValidation('some human description', '/file')).toBe(true);

expect(await runner.executeValidation('file exists', 'src/app.js')).toBe(true);

expect(fs.pathExists).toHaveBeenCalledWith('/project/src/app.js');

});

test('file exists - fails', async () => {

fs.pathExists.mockResolvedValue(false);

expect(await runner.executeValidation('file exists', 'missing.js')).toBe(false);

expect(fs.pathExists).toHaveBeenCalledWith('/project/missing.js');

});

test('file exists with array paths', async () => {

fs.pathExists.mockResolvedValueOnce(true).mockResolvedValueOnce(true);

expect(await runner.executeValidation('file exists', ['a.js', 'b.js'])).toBe(true);

expect(fs.pathExists).toHaveBeenNthCalledWith(1, '/project/a.js');

expect(fs.pathExists).toHaveBeenNthCalledWith(2, '/project/b.js');

});

test('directory exists - passes', async () => {

fs.stat.mockResolvedValue({ isDirectory: () => true });

expect(await runner.executeValidation('directory exists', 'src')).toBe(true);

expect(fs.stat).toHaveBeenCalledWith('/project/src');

});

test('directory exists - fails for file', async () => {

fs.stat.mockResolvedValue({ isDirectory: () => false });

expect(await runner.executeValidation('directory exists', 'file.txt')).toBe(false);

expect(fs.stat).toHaveBeenCalledWith('/project/file.txt');

});

test('not empty - passes for non-empty file', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue('content here');

expect(await runner.executeValidation('not empty', 'file.md')).toBe(true);

expect(fs.pathExists).toHaveBeenCalledWith('/project/file.md');

});

test('not empty - fails for empty file', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue(' ');

expect(await runner.executeValidation('not empty', 'file.md')).toBe(false);

expect(fs.pathExists).toHaveBeenCalledWith('/project/file.md');

});

test('contains check - passes', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue('# architecture\n## components');

expect(await runner.executeValidation("contains 'architecture'", 'doc.md')).toBe(true);

expect(fs.pathExists).toHaveBeenCalledWith('/project/doc.md');

});

test('contains check - fails', async () => {

fs.pathExists.mockResolvedValue(true);

fs.readFile.mockResolvedValue('# empty doc');

expect(await runner.executeValidation("contains 'architecture'", 'doc.md')).toBe(false);

expect(fs.pathExists).toHaveBeenCalledWith('/project/doc.md');

});

test('contains check - fails for missing file', async () => {

fs.pathExists.mockResolvedValue(false);

expect(await runner.executeValidation("contains 'something'", 'missing.md')).toBe(false);

expect(fs.pathExists).toHaveBeenCalledWith('/project/missing.md');

});

test('minimum size check - passes', async () => {

fs.pathExists.mockResolvedValue(true);

fs.stat.mockResolvedValue({ size: 500 });

expect(await runner.executeValidation('minimum size: 100', 'file.md')).toBe(true);

expect(fs.pathExists).toHaveBeenCalledWith('/project/file.md');

expect(fs.stat).toHaveBeenCalledWith('/project/file.md');

});

test('minimum size check - fails', async () => {

fs.pathExists.mockResolvedValue(true);

fs.stat.mockResolvedValue({ size: 50 });

expect(await runner.executeValidation('min size: 100', 'file.md')).toBe(false);

expect(fs.pathExists).toHaveBeenCalledWith('/project/file.md');

expect(fs.stat).toHaveBeenCalledWith('/project/file.md');

});

test('minimum size - fails for missing file', async () => {

fs.pathExists.mockResolvedValue(false);

expect(await runner.executeValidation('min size: 100', 'missing.md')).toBe(false);

expect(fs.pathExists).toHaveBeenCalledWith('/project/missing.md');

});

test('unknown validation defaults to true', async () => {

expect(await runner.executeValidation('some human description', 'file')).toBe(true);

Copilot · 2026-02-18T19:48:09Z

tests/core/orchestration/checklist-runner.test.js

+      fs.readFile.mockResolvedValue('   ');
+      expect(await runner.executeValidation('not empty', '/file.md')).toBe(false);
+    });
+


There’s no test covering executeValidation('not empty', ). The current implementation returns true when the file does not exist (it only fails if the file exists and is empty), which is inconsistent with the 'contains' and 'min size' rules that fail on missing files. Add a test for the missing-file case to lock in the intended behavior (and adjust implementation if missing should be treated as failure).

Suggested change

test('not empty - fails for missing file', async () => {

fs.pathExists.mockResolvedValue(false);

expect(await runner.executeValidation('not empty', '/missing.md')).toBe(false);

});

Copilot · 2026-02-18T19:48:09Z

tests/core/orchestration/checklist-runner.test.js

+    test('runs checklist and passes', async () => {
+      fs.pathExists.mockResolvedValue(true);
+      fs.readFile.mockResolvedValue('- [ ] Review code\n- [ ] Check formatting');
+
+      const result = await runner.run('quality', '/src/app.js');
+
+      expect(result.passed).toBe(true);
+      expect(result.items).toHaveLength(2);
+      expect(result.timestamp).toBeDefined();
+    });
+
+    test('fails when blocker item fails', async () => {
+      fs.pathExists.mockImplementation((p) => {
+        if (p.includes('checklists')) return true;
+        return false;
+      });
+      fs.readFile.mockResolvedValue('```yaml\npre-conditions:\n  - Output exists\n```');
+      yaml.load.mockReturnValue({
+        'pre-conditions': [{
+          'Output exists': null,
+          validation: 'file exists',
+          blocker: true,
+        }],
+      });
+
+      const result = await runner.run('pre-check', '/output.md');
+
+      expect(result.passed).toBe(false);
+      expect(result.errors.length).toBeGreaterThan(0);
+    });


The run() integration tests also pass absolute target paths (e.g., "/src/app.js", "/output.md"), so they don’t validate the common usage where target paths are relative to projectRoot (as provided by workflow phase.creates). Prefer relative paths here too, and consider asserting that the blocker failure is specifically the expected "Blocker failed: …" error to make the test more precise.

nikolasdehor · 2026-02-19T19:16:30Z

Consolidated into #426

test: add unit tests for checklist-runner module

82fce9c

Copilot AI review requested due to automatic review settings February 18, 2026 19:42

Copilot started reviewing on behalf of nikolasdehor February 18, 2026 19:43 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

nikolasdehor closed this Feb 19, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add unit tests for checklist-runner module#250

Add unit tests for checklist-runner module#250
nikolasdehor wants to merge 1 commit intoSynkraAI:mainfrom
nikolasdehor:test/checklist-runner-coverage

nikolasdehor commented Feb 18, 2026 •

edited

Loading

Uh oh!

vercel bot commented Feb 18, 2026

Uh oh!

coderabbitai bot commented Feb 18, 2026

Rate limit exceeded

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

nikolasdehor commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

+    test('not empty - fails for missing file', async () => {
+      fs.pathExists.mockResolvedValue(false);
+      expect(await runner.executeValidation('not empty', '/missing.md')).toBe(false);
+    });

Uh oh!

Conversation

nikolasdehor commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Coverage

Test plan

Uh oh!

vercel bot commented Feb 18, 2026

Uh oh!

coderabbitai bot commented Feb 18, 2026

Rate limit exceeded

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

nikolasdehor commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nikolasdehor commented Feb 18, 2026 •

edited

Loading