Skip to content

feat(squad-creator): add internalization quality system #539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Pahe-Digital wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(squad-creator): add internalization quality system #539

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .aios-core/development/agents/squad-creator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,28 @@ core_principles:
- CRITICAL: Use JSON Schema for manifest validation
- CRITICAL: Support 3-level distribution (Local, aios-squads, Synkra API)
- CRITICAL: Integrate with existing squad-loader and squad-validator
- CRITICAL: NEVER create dependencies.context in agent definitions — all intelligence must be internalized in the agent YAML
- CRITICAL: When creating agents from external references (PDF/DOCX/MD), apply knowledge_extraction_process to embed knowledge — never point the agent back to the source file

knowledge_extraction_process:
description: "When creating an agent from an external reference document, follow this 6-step distillation process to ensure all intelligence is fully internalized."
steps:
MAP: "Identify all sections, rules, frameworks, and concepts in the source document"
EXTRACT: "Extract only actionable intelligence — discard prose, keep operational directives"
TRANSFORM: "Convert free-form content into structured YAML (core_principles, commands, workflows, vocabulary)"
STRUCTURE: "Organize into agent sections: persona, core_principles, operational_data, commands"
VERIFY: "Cross-check that every key concept from the source is represented in the YAML — nothing lost"
DISCARD: "Source file is no longer needed — the agent YAML is the single source of truth"
quality_signals:
- "Agent can operate fully without the source document"
- "Every command has concrete behavioral instructions, not vague descriptions"
- "Vocabulary, tone, and anti-patterns are explicitly enumerated"
- "No reference to loading external files at runtime"
anti_patterns:
- "Copy-pasting raw document sections into YAML values (still unstructured)"
- "Creating a dependencies.context field pointing back to the source"
- "Summarizing instead of extracting (loses critical operational detail)"
- "Embedding prose paragraphs instead of actionable operational directives"

# All commands require * prefix when used (e.g., *help)
commands:
Expand Down
25 changes: 25 additions & 0 deletions .aios-core/development/tasks/squad-creator-analyze.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ Checklist:
- "[ ] Load squad.yaml manifest"
- "[ ] Inventory components by type"
- "[ ] Calculate coverage metrics"
- "[ ] Calculate internalization score (scan agents for dependencies.context)"
- "[ ] Generate improvement suggestions"
- "[ ] Format and display report"
---
Expand Down Expand Up @@ -170,6 +171,29 @@ const coverage = analyzer.calculateCoverage(inventory, manifest);
// }
```

### Step 4.5: Calculate Internalization Score

```javascript
const internalization = await analyzer.calculateInternalizationScore(squadPath, inventory);

// Scans each agent file for:
// 1. Presence of 'dependencies.context' key (CRITICAL violation — score = 0 for that agent)
// 2. core_principles count (< 3 = shallow, >= 5 = good, >= 8 = excellent)
// 3. persona.role/identity length (< 30 chars = skeletal)
// 4. commands with substantive descriptions (< 10 chars = missing guidance)

// Expected structure:
// {
// total_agents: 3,
// fully_internalized: 3,
// has_external_deps: 0, // agents with dependencies.context
// shallow_principles: 0, // agents with < 3 core_principles
// skeletal_personas: 0, // agents with brief role/identity
// score_percentage: 100, // 0-100
// violations: [] // list of agent files with issues
// }
```

Comment on lines +174 to +196
Copy link

@coderabbitai coderabbitai bot Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

find . -name "squad-analyzer.js" -type f

Repository: SynkraAI/aios-core

Length of output: 118

🏁 Script executed:

ast-grep --pattern $'calculateInternalizationScore($$$) {
  $$$
}'

Repository: SynkraAI/aios-core

Length of output: 44

🏁 Script executed:

rg "calculateInternalizationScore" .aios-core/development/scripts/squad/squad-analyzer.js

Repository: SynkraAI/aios-core

Length of output: 44

🏁 Script executed:

cat .aios-core/development/scripts/squad/squad-analyzer.js

Repository: SynkraAI/aios-core

Length of output: 18240

Implement calculateInternalizationScore method in squad-analyzer.js.

Step 4.5 references analyzer.calculateInternalizationScore(squadPath, inventory), but this method does not exist in .aios-core/development/scripts/squad/squad-analyzer.js. The SquadAnalyzer class includes calculateCoverage() but not calculateInternalizationScore(). Either implement this method with the documented signature and return structure, or update the task step to use an existing method.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.aios-core/development/tasks/squad-creator-analyze.md around lines 174 -
196, The review points out that SquadAnalyzer is missing the
calculateInternalizationScore(squadPath, inventory) method referenced by
analyzer.calculateInternalizationScore; add this method to the SquadAnalyzer
class in squad-analyzer.js (or alternatively update the task to call the
existing calculateCoverage), implementing the documented behavior and return
shape: iterate agent files under the given squadPath using the inventory, detect
dependencies.context (count as has_external_deps and zero-score for that agent),
count core_principles, check persona.role/identity length, inspect commands
descriptions, accumulate total_agents, fully_internalized, shallow_principles,
skeletal_personas, violations array and compute score_percentage (0–100) before
returning the object; ensure the method signature matches
calculateInternalizationScore(squadPath, inventory) so existing callers work.

### Step 5: Generate Suggestions

```javascript
Expand Down Expand Up @@ -235,6 +259,7 @@ Coverage
Tasks: {bar} {percentage}% ({details})
Config: {bar} {percentage}% ({details})
Docs: {bar} {percentage}% ({details})
Internalization: {bar} {score_percentage}% ({fully_internalized}/{total_agents} agents) {violations_indicator}

Suggestions
1. {suggestion-1}
Expand Down
1 change: 1 addition & 0 deletions .aios-core/development/tasks/squad-creator-create.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ tags:
├── Generate squad.yaml from template
├── Generate config files
├── Generate example agent (if requested)
│ └── RULE: If agent is based on external reference, apply knowledge_extraction_process first
├── Generate example task (if requested)
└── Add .gitkeep to empty directories

Expand Down
32 changes: 31 additions & 1 deletion .aios-core/development/tasks/squad-creator-extend.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,10 @@ Checklist:
- "[ ] Validate squad exists"
- "[ ] Collect component type"
- "[ ] Collect component name and metadata"
- "[ ] If adding agent with external reference: apply knowledge_extraction_process before generating"
- "[ ] Create file from template"
- "[ ] Update squad.yaml manifest"
- "[ ] Run validation"
- "[ ] Run validation (includes internalization gate)"
- "[ ] Display result and next steps"
---

Expand Down Expand Up @@ -190,6 +191,35 @@ if (componentType === 'task' && !agentId) {
}
```

### Step 2.5: Internalization Gate (Agents Only)

When the component type is `agent`, ask if the user has an external reference document:

```
? Are you creating this agent based on an external reference file (PDF, DOCX, MD)?
1. No — I'll define the agent from scratch
2. Yes — I have a source document to extract from

> 2

? Provide path or paste the key content sections you want to internalize:
> [user provides content]
```

If the user provides a reference, apply `knowledge_extraction_process` BEFORE generating the file:

```
1. MAP → List all key concepts, rules, frameworks in the source
2. EXTRACT → Identify actionable intelligence (vocabulary, core_principles, commands, workflows)
3. TRANSFORM → Convert to YAML structures
4. STRUCTURE → Place in correct agent sections (persona, core_principles, operational_data)
5. VERIFY → Confirm all key concepts are represented — nothing lost
6. DISCARD → Source file is NOT referenced — agent YAML is the sole source of truth
```

**HARD RULE:** The generated agent file MUST NOT contain a `dependencies.context` key.
If you are tempted to add one, that is a signal that EXTRACT/TRANSFORM steps were skipped.

### Step 3: Create Component File

```javascript
Expand Down
13 changes: 12 additions & 1 deletion .aios-core/development/tasks/squad-creator-validate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,13 @@ Validates a squad against the JSON Schema and TASK-FORMAT-SPECIFICATION-V1.
- Warns if project-level reference doesn't exist
- Errors if local reference doesn't exist

### 6. Internalization Gate
- Scans all agent files in `agents/` for presence of `dependencies.context` key
- **CRITICAL ERROR** if any agent contains `dependencies.context` — external runtime dependencies are forbidden
- Checks that agent YAML has substantive `core_principles` (not empty or single-item)
- Checks that agent `commands` have descriptions longer than 10 chars (signals real behavioral guidance)
- Warns if agent `persona` sections are skeletal (role/identity under 30 chars)

Comment on lines +70 to +76
Copy link

@coderabbitai coderabbitai bot Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Define an explicit error code for the command-description quality check.

Line 74 introduces a new check (commands descriptions > 10 chars), but there is no corresponding code in the Error Codes table. Please add a dedicated warning code (and severity/description) so validator output stays deterministic and machine-consumable.

Suggested doc patch 
 ### 6. Internalization Gate
 ...
-- Checks that agent `commands` have descriptions longer than 10 chars (signals real behavioral guidance)
+- Checks that agent `commands` have descriptions longer than 10 chars (signals real behavioral guidance)
+  - Emits `AGENT_SHALLOW_COMMAND_DESC` (Warning) when command descriptions are too brief
 ...
 | `AGENT_EXTERNAL_DEP` | Error | Agent contains dependencies.context (forbidden — must internalize) |
 | `AGENT_SHALLOW_PRINCIPLES` | Warning | Agent core_principles is empty or has fewer than 3 entries |
+| `AGENT_SHALLOW_COMMAND_DESC` | Warning | Agent command descriptions are too brief to be operational |
 | `AGENT_SKELETAL_PERSONA` | Warning | Agent persona.role/identity are too brief to be operational |

As per coding guidelines, "Check that deliverables are well-defined."

Also applies to: 131-133

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.aios-core/development/tasks/squad-creator-validate.md around lines 70 - 76,
Add a deterministic error code entry for the "commands descriptions > 10 chars"
check in the Error Codes table: create a unique code (e.g., W_CMD_DESC_SHORT),
set severity to WARNING, and provide a concise description like "Command
description shorter than 10 characters; indicates insufficient behavioral
guidance." Update the Error Codes table row and any validator mapping or
documentation references for the Internalization Gate and the "commands"
description check (mentioned near "Internalization Gate" and lines referenced
131-133) so validator output is machine-consumable and consistent.

## Flow

```
Expand All @@ -79,7 +86,8 @@ Validates a squad against the JSON Schema and TASK-FORMAT-SPECIFICATION-V1.
├── validateStructure() → Directory check
├── validateTasks() → Task format check
├── validateAgents() → Agent format check
└── validateConfigReferences() → Config path check (SQS-10)
├── validateConfigReferences() → Config path check (SQS-10)
└── validateInternalization() → Internalization gate (forbidden dependencies.context)
Comment on lines +89 to +90
Copy link

@coderabbitai coderabbitai bot Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Locating squad validator implementation files..."
fd "squad-validator" .aios-core/development/scripts 2>/dev/null || echo "No squad-validator files found in .aios-core/development/scripts"

echo
echo "Searching for validateInternalization definition and usage context..."
rg -n -C3 "validateInternalization\s*\(" .aios-core/development/scripts 2>/dev/null || echo "No validateInternalization found in .aios-core/development/scripts"

echo
echo "Searching validation pipeline ordering around config references/internalization..."
rg -n -C4 "validateConfigReferences|validateInternalization|validate\s*\(" .aios-core/development/scripts 2>/dev/null || echo "No matches found in .aios-core/development/scripts"

echo
echo "Listing files in squad-creator-validate.md directory..."
ls -la .aios-core/development/tasks/ 2>/dev/null | head -20 || echo "Directory not found"

echo
echo "Checking the actual content of squad-creator-validate.md around lines 89-90..."
cat -n .aios-core/development/tasks/squad-creator-validate.md 2>/dev/null | sed -n '85,95p' || echo "File not found"

Repository: SynkraAI/aios-core

Length of output: 22177

validateInternalization() is missing from SquadValidator implementation.

The task documentation references this validation method at line 90, but it does not exist in .aios-core/development/scripts/squad/squad-validator.js. The main validate() method calls validateConfigReferences() and validateWorkflows(), but has no step for internalization validation. Either implement the missing validation method or remove it from the documented flow.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.aios-core/development/tasks/squad-creator-validate.md around lines 89 - 90,
Implement the missing validateInternalization() method in the SquadValidator
class in squad-validator.js: add a new method named validateInternalization()
that enforces the documented internalization gate (e.g., checks for forbidden
dependencies.context and fails/returns errors when internalization violations
are found), and call this method from the main validate() flow alongside
validateConfigReferences() and validateWorkflows(); ensure the method signature
and error/reporting style match existing methods (e.g., same return
type/structure and use of validator helpers) so it integrates cleanly with the
SquadValidator implementation.


3. Format and display result
├── Show errors (if any)
Expand Down Expand Up @@ -120,6 +128,9 @@ Result: VALID (with warnings)
| `TASK_MISSING_FIELD` | Warning | Task missing recommended field |
| `AGENT_INVALID_FORMAT` | Warning | Agent file may not follow format |
| `INVALID_NAMING` | Warning | Filename not in kebab-case |
| `AGENT_EXTERNAL_DEP` | Error | Agent contains dependencies.context (forbidden — must internalize) |
| `AGENT_SHALLOW_PRINCIPLES` | Warning | Agent core_principles is empty or has fewer than 3 entries |
| `AGENT_SKELETAL_PERSONA` | Warning | Agent persona.role/identity are too brief to be operational |

## Implementation

Expand Down
3 changes: 3 additions & 0 deletions .aios-core/development/templates/squad/agent-template.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ commands:
description: "Exit agent mode"

dependencies:
# INTERNALIZATION RULE: NEVER add a 'context:' key here.
# External documents (PDF, DOCX, MD) must be fully extracted and embedded
# in this agent YAML using knowledge_extraction_process — never referenced at runtime.
tasks: []
templates: []
checklists: []
Expand Down
24 changes: 22 additions & 2 deletions .claude/commands/AIOS/agents/squad-creator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,28 @@ core_principles:
- CRITICAL: Use JSON Schema for manifest validation
- CRITICAL: Support 3-level distribution (Local, aios-squads, Synkra API)
- CRITICAL: Integrate with existing squad-loader and squad-validator
- CRITICAL: NEVER create dependencies.context in agent definitions — all intelligence must be internalized in the agent YAML
- CRITICAL: When creating agents from external references (PDF/DOCX/MD), apply knowledge_extraction_process to embed knowledge — never point the agent back to the source file

knowledge_extraction_process:
description: "When creating an agent from an external reference document, follow this 6-step distillation process to ensure all intelligence is fully internalized."
steps:
MAP: "Identify all sections, rules, frameworks, and concepts in the source document"
EXTRACT: "Extract only actionable intelligence — discard prose, keep operational directives"
TRANSFORM: "Convert free-form content into structured YAML (core_principles, commands, workflows, vocabulary)"
STRUCTURE: "Organize into agent sections: persona, core_principles, operational_data, commands"
VERIFY: "Cross-check that every key concept from the source is represented in the YAML — nothing lost"
DISCARD: "Source file is no longer needed — the agent YAML is the single source of truth"
quality_signals:
- "Agent can operate fully without the source document"
- "Every command has concrete behavioral instructions, not vague descriptions"
- "Vocabulary, tone, and anti-patterns are explicitly enumerated"
- "No reference to loading external files at runtime"
anti_patterns:
- "Copy-pasting raw document sections into YAML values (still unstructured)"
- "Creating a dependencies.context field pointing back to the source"
- "Summarizing instead of extracting (loses critical operational detail)"
- "Embedding prose paragraphs instead of actionable operational directives"

# All commands require * prefix when used (e.g., *help)
commands:
Expand Down Expand Up @@ -340,5 +362,3 @@ Type `*help` to see all commands, or `*guide` for detailed usage.
- **@devops (Gage)** - Handles deployment

---
---
*AIOS Agent - Synced from .aios-core/development/agents/squad-creator.md*
24 changes: 22 additions & 2 deletions .codex/agents/squad-creator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,28 @@ core_principles:
- CRITICAL: Use JSON Schema for manifest validation
- CRITICAL: Support 3-level distribution (Local, aios-squads, Synkra API)
- CRITICAL: Integrate with existing squad-loader and squad-validator
- CRITICAL: NEVER create dependencies.context in agent definitions — all intelligence must be internalized in the agent YAML
- CRITICAL: When creating agents from external references (PDF/DOCX/MD), apply knowledge_extraction_process to embed knowledge — never point the agent back to the source file

knowledge_extraction_process:
description: "When creating an agent from an external reference document, follow this 6-step distillation process to ensure all intelligence is fully internalized."
steps:
MAP: "Identify all sections, rules, frameworks, and concepts in the source document"
EXTRACT: "Extract only actionable intelligence — discard prose, keep operational directives"
TRANSFORM: "Convert free-form content into structured YAML (core_principles, commands, workflows, vocabulary)"
STRUCTURE: "Organize into agent sections: persona, core_principles, operational_data, commands"
VERIFY: "Cross-check that every key concept from the source is represented in the YAML — nothing lost"
DISCARD: "Source file is no longer needed — the agent YAML is the single source of truth"
quality_signals:
- "Agent can operate fully without the source document"
- "Every command has concrete behavioral instructions, not vague descriptions"
- "Vocabulary, tone, and anti-patterns are explicitly enumerated"
- "No reference to loading external files at runtime"
anti_patterns:
- "Copy-pasting raw document sections into YAML values (still unstructured)"
- "Creating a dependencies.context field pointing back to the source"
- "Summarizing instead of extracting (loses critical operational detail)"
- "Embedding prose paragraphs instead of actionable operational directives"

# All commands require * prefix when used (e.g., *help)
commands:
Expand Down Expand Up @@ -340,5 +362,3 @@ Type `*help` to see all commands, or `*guide` for detailed usage.
- **@devops (Gage)** - Handles deployment

---
---
*AIOS Agent - Synced from .aios-core/development/agents/squad-creator.md*
24 changes: 22 additions & 2 deletions .gemini/rules/AIOS/agents/squad-creator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,28 @@ core_principles:
- CRITICAL: Use JSON Schema for manifest validation
- CRITICAL: Support 3-level distribution (Local, aios-squads, Synkra API)
- CRITICAL: Integrate with existing squad-loader and squad-validator
- CRITICAL: NEVER create dependencies.context in agent definitions — all intelligence must be internalized in the agent YAML
- CRITICAL: When creating agents from external references (PDF/DOCX/MD), apply knowledge_extraction_process to embed knowledge — never point the agent back to the source file

knowledge_extraction_process:
description: "When creating an agent from an external reference document, follow this 6-step distillation process to ensure all intelligence is fully internalized."
steps:
MAP: "Identify all sections, rules, frameworks, and concepts in the source document"
EXTRACT: "Extract only actionable intelligence — discard prose, keep operational directives"
TRANSFORM: "Convert free-form content into structured YAML (core_principles, commands, workflows, vocabulary)"
STRUCTURE: "Organize into agent sections: persona, core_principles, operational_data, commands"
VERIFY: "Cross-check that every key concept from the source is represented in the YAML — nothing lost"
DISCARD: "Source file is no longer needed — the agent YAML is the single source of truth"
quality_signals:
- "Agent can operate fully without the source document"
- "Every command has concrete behavioral instructions, not vague descriptions"
- "Vocabulary, tone, and anti-patterns are explicitly enumerated"
- "No reference to loading external files at runtime"
anti_patterns:
- "Copy-pasting raw document sections into YAML values (still unstructured)"
- "Creating a dependencies.context field pointing back to the source"
- "Summarizing instead of extracting (loses critical operational detail)"
- "Embedding prose paragraphs instead of actionable operational directives"

# All commands require * prefix when used (e.g., *help)
commands:
Expand Down Expand Up @@ -340,5 +362,3 @@ Type `*help` to see all commands, or `*guide` for detailed usage.
- **@devops (Gage)** - Handles deployment

---
---
*AIOS Agent - Synced from .aios-core/development/agents/squad-creator.md*