feat(squad-creator): add internalization quality system#539
feat(squad-creator): add internalization quality system#539Pahe-Digital wants to merge 1 commit intoSynkraAI:mainfrom
Conversation
Problem: When creating agents from external reference documents (PDF, DOCX, MD), squad-creator would create agents with a `dependencies.context` field that loaded the source file at runtime instead of embedding the intelligence in the agent YAML. This caused agents to break when source files were moved/deleted, and created inconsistent behavior on repeated runs. Solution: 4-layer system to enforce and verify internalization. Changes: - squad-creator (agent): add 2 CRITICAL core_principles blocking dependencies.context; add knowledge_extraction_process section with 6-step distillation (MAP→EXTRACT→TRANSFORM→STRUCTURE→ VERIFY→DISCARD), quality_signals, and anti_patterns - agent-template.md: add INTERNALIZATION RULE comment in dependencies block — blocks addition of context: key at template level, making the constraint visible at creation time - squad-creator-extend.md: add Step 2.5 Internalization Gate with interactive flow for agents created from external references; hard rule: generated agent MUST NOT contain dependencies.context - squad-creator-create.md: add RULE note in flow step 4 for agent generation from external references - squad-creator-validate.md: add Validation Check SynkraAI#6 (Internalization Gate) with validateInternalization(); add 3 new error codes: AGENT_EXTERNAL_DEP (Error), AGENT_SHALLOW_PRINCIPLES (Warning), AGENT_SKELETAL_PERSONA (Warning) - squad-creator-analyze.md: add Step 4.5 internalization score calculation; add Internalization metric line to coverage report Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pahe-Digital
requested review from
Pedrovaleriolopez and
oalanicolas
as code owners
|
@Pahe-Digital is attempting to deploy a commit to the Pedro Valério Lopez's projects Team on Vercel. A member of the Team first needs to authorize it. |
WalkthroughThis change introduces a formal knowledge_extraction_process and internalization requirements across agent configuration files and task workflows. The updates enforce that agents must embed all intelligence within their YAML definitions rather than referencing external sources, with a documented 6-step distillation process (MAP, EXTRACT, TRANSFORM, STRUCTURE, VERIFY, DISCARD) and new validation gates. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs). Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Welcome to aios-core! Thanks for your first pull request.
What happens next?
- Automated checks will run on your PR
- A maintainer will review your changes
- Once approved, we'll merge your contribution!
PR Checklist:
- Tests pass (
npm test) - Linting passes (
npm run lint) - Commit messages follow Conventional Commits
Thanks for contributing!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.aios-core/development/tasks/squad-creator-analyze.md:
- Around line 174-196: The review points out that SquadAnalyzer is missing the
calculateInternalizationScore(squadPath, inventory) method referenced by
analyzer.calculateInternalizationScore; add this method to the SquadAnalyzer
class in squad-analyzer.js (or alternatively update the task to call the
existing calculateCoverage), implementing the documented behavior and return
shape: iterate agent files under the given squadPath using the inventory, detect
dependencies.context (count as has_external_deps and zero-score for that agent),
count core_principles, check persona.role/identity length, inspect commands
descriptions, accumulate total_agents, fully_internalized, shallow_principles,
skeletal_personas, violations array and compute score_percentage (0–100) before
returning the object; ensure the method signature matches
calculateInternalizationScore(squadPath, inventory) so existing callers work.
In @.aios-core/development/tasks/squad-creator-validate.md:
- Around line 70-76: Add a deterministic error code entry for the "commands
descriptions > 10 chars" check in the Error Codes table: create a unique code
(e.g., W_CMD_DESC_SHORT), set severity to WARNING, and provide a concise
description like "Command description shorter than 10 characters; indicates
insufficient behavioral guidance." Update the Error Codes table row and any
validator mapping or documentation references for the Internalization Gate and
the "commands" description check (mentioned near "Internalization Gate" and
lines referenced 131-133) so validator output is machine-consumable and
consistent.
- Around line 89-90: Implement the missing validateInternalization() method in
the SquadValidator class in squad-validator.js: add a new method named
validateInternalization() that enforces the documented internalization gate
(e.g., checks for forbidden dependencies.context and fails/returns errors when
internalization violations are found), and call this method from the main
validate() flow alongside validateConfigReferences() and validateWorkflows();
ensure the method signature and error/reporting style match existing methods
(e.g., same return type/structure and use of validator helpers) so it integrates
cleanly with the SquadValidator implementation.
ℹ️ Review info
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
.aios-core/development/agents/squad-creator.md.aios-core/development/tasks/squad-creator-analyze.md.aios-core/development/tasks/squad-creator-create.md.aios-core/development/tasks/squad-creator-extend.md.aios-core/development/tasks/squad-creator-validate.md.aios-core/development/templates/squad/agent-template.md.claude/commands/AIOS/agents/squad-creator.md.codex/agents/squad-creator.md.gemini/rules/AIOS/agents/squad-creator.md
| ### Step 4.5: Calculate Internalization Score | ||
|
|
||
| ```javascript | ||
| const internalization = await analyzer.calculateInternalizationScore(squadPath, inventory); | ||
|
|
||
| // Scans each agent file for: | ||
| // 1. Presence of 'dependencies.context' key (CRITICAL violation — score = 0 for that agent) | ||
| // 2. core_principles count (< 3 = shallow, >= 5 = good, >= 8 = excellent) | ||
| // 3. persona.role/identity length (< 30 chars = skeletal) | ||
| // 4. commands with substantive descriptions (< 10 chars = missing guidance) | ||
|
|
||
| // Expected structure: | ||
| // { | ||
| // total_agents: 3, | ||
| // fully_internalized: 3, | ||
| // has_external_deps: 0, // agents with dependencies.context | ||
| // shallow_principles: 0, // agents with < 3 core_principles | ||
| // skeletal_personas: 0, // agents with brief role/identity | ||
| // score_percentage: 100, // 0-100 | ||
| // violations: [] // list of agent files with issues | ||
| // } | ||
| ``` | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -name "squad-analyzer.js" -type fRepository: SynkraAI/aios-core
Length of output: 118
🏁 Script executed:
ast-grep --pattern $'calculateInternalizationScore($$$) {
$$$
}'Repository: SynkraAI/aios-core
Length of output: 44
🏁 Script executed:
rg "calculateInternalizationScore" .aios-core/development/scripts/squad/squad-analyzer.jsRepository: SynkraAI/aios-core
Length of output: 44
🏁 Script executed:
cat .aios-core/development/scripts/squad/squad-analyzer.jsRepository: SynkraAI/aios-core
Length of output: 18240
Implement calculateInternalizationScore method in squad-analyzer.js.
Step 4.5 references analyzer.calculateInternalizationScore(squadPath, inventory), but this method does not exist in .aios-core/development/scripts/squad/squad-analyzer.js. The SquadAnalyzer class includes calculateCoverage() but not calculateInternalizationScore(). Either implement this method with the documented signature and return structure, or update the task step to use an existing method.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.aios-core/development/tasks/squad-creator-analyze.md around lines 174 -
196, The review points out that SquadAnalyzer is missing the
calculateInternalizationScore(squadPath, inventory) method referenced by
analyzer.calculateInternalizationScore; add this method to the SquadAnalyzer
class in squad-analyzer.js (or alternatively update the task to call the
existing calculateCoverage), implementing the documented behavior and return
shape: iterate agent files under the given squadPath using the inventory, detect
dependencies.context (count as has_external_deps and zero-score for that agent),
count core_principles, check persona.role/identity length, inspect commands
descriptions, accumulate total_agents, fully_internalized, shallow_principles,
skeletal_personas, violations array and compute score_percentage (0–100) before
returning the object; ensure the method signature matches
calculateInternalizationScore(squadPath, inventory) so existing callers work.
| ### 6. Internalization Gate | ||
| - Scans all agent files in `agents/` for presence of `dependencies.context` key | ||
| - **CRITICAL ERROR** if any agent contains `dependencies.context` — external runtime dependencies are forbidden | ||
| - Checks that agent YAML has substantive `core_principles` (not empty or single-item) | ||
| - Checks that agent `commands` have descriptions longer than 10 chars (signals real behavioral guidance) | ||
| - Warns if agent `persona` sections are skeletal (role/identity under 30 chars) | ||
|
|
There was a problem hiding this comment.
Define an explicit error code for the command-description quality check.
Line 74 introduces a new check (commands descriptions > 10 chars), but there is no corresponding code in the Error Codes table. Please add a dedicated warning code (and severity/description) so validator output stays deterministic and machine-consumable.
Suggested doc patch
### 6. Internalization Gate
...
-- Checks that agent `commands` have descriptions longer than 10 chars (signals real behavioral guidance)
+- Checks that agent `commands` have descriptions longer than 10 chars (signals real behavioral guidance)
+ - Emits `AGENT_SHALLOW_COMMAND_DESC` (Warning) when command descriptions are too brief
...
| `AGENT_EXTERNAL_DEP` | Error | Agent contains dependencies.context (forbidden — must internalize) |
| `AGENT_SHALLOW_PRINCIPLES` | Warning | Agent core_principles is empty or has fewer than 3 entries |
+| `AGENT_SHALLOW_COMMAND_DESC` | Warning | Agent command descriptions are too brief to be operational |
| `AGENT_SKELETAL_PERSONA` | Warning | Agent persona.role/identity are too brief to be operational |As per coding guidelines, "Check that deliverables are well-defined."
Also applies to: 131-133
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.aios-core/development/tasks/squad-creator-validate.md around lines 70 - 76,
Add a deterministic error code entry for the "commands descriptions > 10 chars"
check in the Error Codes table: create a unique code (e.g., W_CMD_DESC_SHORT),
set severity to WARNING, and provide a concise description like "Command
description shorter than 10 characters; indicates insufficient behavioral
guidance." Update the Error Codes table row and any validator mapping or
documentation references for the Internalization Gate and the "commands"
description check (mentioned near "Internalization Gate" and lines referenced
131-133) so validator output is machine-consumable and consistent.
| ├── validateConfigReferences() → Config path check (SQS-10) | ||
| └── validateInternalization() → Internalization gate (forbidden dependencies.context) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locating squad validator implementation files..."
fd "squad-validator" .aios-core/development/scripts 2>/dev/null || echo "No squad-validator files found in .aios-core/development/scripts"
echo
echo "Searching for validateInternalization definition and usage context..."
rg -n -C3 "validateInternalization\s*\(" .aios-core/development/scripts 2>/dev/null || echo "No validateInternalization found in .aios-core/development/scripts"
echo
echo "Searching validation pipeline ordering around config references/internalization..."
rg -n -C4 "validateConfigReferences|validateInternalization|validate\s*\(" .aios-core/development/scripts 2>/dev/null || echo "No matches found in .aios-core/development/scripts"
echo
echo "Listing files in squad-creator-validate.md directory..."
ls -la .aios-core/development/tasks/ 2>/dev/null | head -20 || echo "Directory not found"
echo
echo "Checking the actual content of squad-creator-validate.md around lines 89-90..."
cat -n .aios-core/development/tasks/squad-creator-validate.md 2>/dev/null | sed -n '85,95p' || echo "File not found"Repository: SynkraAI/aios-core
Length of output: 22177
validateInternalization() is missing from SquadValidator implementation.
The task documentation references this validation method at line 90, but it does not exist in .aios-core/development/scripts/squad/squad-validator.js. The main validate() method calls validateConfigReferences() and validateWorkflows(), but has no step for internalization validation. Either implement the missing validation method or remove it from the documented flow.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.aios-core/development/tasks/squad-creator-validate.md around lines 89 - 90,
Implement the missing validateInternalization() method in the SquadValidator
class in squad-validator.js: add a new method named validateInternalization()
that enforces the documented internalization gate (e.g., checks for forbidden
dependencies.context and fails/returns errors when internalization violations
are found), and call this method from the main validate() flow alongside
validateConfigReferences() and validateWorkflows(); ensure the method signature
and error/reporting style match existing methods (e.g., same return
type/structure and use of validator helpers) so it integrates cleanly with the
SquadValidator implementation.
nikolasdehor
left a comment
nikolasdehor
left a comment
There was a problem hiding this comment.
Review: feat(squad-creator): add internalization quality system
Resumo
Este PR adiciona um sistema de 4 camadas para impedir que agentes criados pelo squad-creator mantenham dependencias externas via dependencies.context. A motivacao e valida: um caso real onde um agente criado a partir de um documento externo ficou apontando para o arquivo-fonte em runtime, quebrando quando o arquivo era movido/renomeado.
As 4 camadas sao:
- Template (
agent-template.md) — comentario de alerta no blocodependencies - Agent (
squad-creator.md) — 2 novosCRITICALcore_principles + blocoknowledge_extraction_process - Task gates (
squad-creator-extend.md,squad-creator-create.md) — Step interativo + regra no fluxo de criacao - Validacao + Analise (
squad-creator-validate.md,squad-creator-analyze.md) — gate de validacao + score de internalizacao
O design conceitual e solido. Porem, existem problemas concretos que precisam ser resolvidos antes do merge.
Problemas encontrados
1. calculateInternalizationScore() nao existe em squad-analyzer.js (Critico)
Arquivo: .aios-core/development/tasks/squad-creator-analyze.md (Step 4.5)
O task documenta:
const internalization = await analyzer.calculateInternalizationScore(squadPath, inventory);Porem, SquadAnalyzer em squad-analyzer.js so possui calculateCoverage(). O metodo nao existe. Qualquer implementacao que siga este task como spec vai falhar com TypeError.
Acao necessaria: Implementar o metodo em squad-analyzer.js OU criar uma issue linkada rastreando a implementacao.
2. validateInternalization() nao existe em squad-validator.js (Critico)
Arquivo: .aios-core/development/tasks/squad-creator-validate.md (Flow diagram)
O diagrama mostra validateInternalization() como chamada, mas o metodo nao existe em SquadValidator. Os novos error codes (AGENT_EXTERNAL_DEP, AGENT_SHALLOW_PRINCIPLES, AGENT_SKELETAL_PERSONA) tambem nao existem no ValidationErrorCodes enum.
Acao necessaria: Mesma situacao do item 1 — implementar ou rastrear explicitamente.
3. Error code faltante para checagem de descricao de comandos (Medio)
O Internalization Gate define 4 checagens, mas so 3 tem error code:
dependencies.contextpresente →AGENT_EXTERNAL_DEP(Error) ✓core_principles< 3 →AGENT_SHALLOW_PRINCIPLES(Warning) ✓persona.role/identity< 30 chars →AGENT_SKELETAL_PERSONA(Warning) ✓commandscom descricoes < 10 chars → ??? Sem error code
Sugestao: Adicionar AGENT_SHALLOW_COMMANDS (Warning) na tabela.
4. score_percentage — formula nao definida (Medio)
O Step 4.5 documenta score_percentage: 100 // 0-100, mas nao define a formula de calculo. Como os diferentes fatores sao ponderados? Sem formula definida, cada implementacao vai inventar a sua.
5. Thresholds arbitrarios sem justificativa (Menor)
30 chars para persona "skeletal" e 10 chars para commands — de onde vieram esses valores? Sugiro adicionar um comentario breve explicando a razao.
O que esta bem feito
- Design de 4 camadas e robusto: template, agent, task gate, validacao. Cobre o ciclo completo
- Anti-patterns bem documentados — copy-paste de sections, summarizing em vez de extracting. Parte mais util do PR
- Quality signals verificaveis: "Agent can operate fully without the source document" e um criterio claro
- Origem real-world documentada na PR description
- HARD RULE no extend: "Se voce esta tentado a adicionar
dependencies.context, e sinal de que pulou EXTRACT/TRANSFORM" — boa decisao de design
Confirmo os findings do CodeRabbit
| # | Finding | Valido? |
|---|---|---|
| 1 | calculateInternalizationScore() nao existe |
Sim, critico |
| 2 | Error code faltante para checagem de commands | Sim, medio |
| 3 | validateInternalization() nao existe |
Sim, critico |
Veredicto: COMMENT
O design conceitual e solido e resolve um problema real. Porem, este PR e puramente documentacao/spec que referencia metodos e error codes que nao existem na implementacao. Isso cria um gap perigoso entre spec e codigo.
Recomendacao: Este PR pode ser mergeado SE houver uma issue aberta rastreando a implementacao de:
calculateInternalizationScore()emsquad-analyzer.jsvalidateInternalization()emsquad-validator.js- Error codes no
ValidationErrorCodesenum (incluindo o faltante para commands) - Formula explicita para
score_percentage
Se esses itens nao forem rastreados, a documentacao vira dead spec.
Revisado por @nikolasdehor
2 participants
Problem
When using
*extend-squador*create-squadto add agents based on external reference documents (PDF, DOCX, MD files), the squad-creator would generate agents with adependencies.contextfield that loaded the source file at runtime.This caused:
Solution
A 4-layer system that enforces and verifies knowledge internalization at every touchpoint.
Changes
Layer 1 — Template (
agent-template.md)Added
# INTERNALIZATION RULEcomment in thedependenciesblock. Makes the constraint visible at the moment of creation — developers see it before they can add acontext:key.Layer 2 — Agent Process (
squad-creator.md)CRITICALcore_principles blockingdependencies.contextknowledge_extraction_processsection with a 6-step distillation workflow:MAP → EXTRACT → TRANSFORM → STRUCTURE → VERIFY → DISCARDquality_signals(how to verify internalization is complete) andanti_patterns(common failure modes)Layer 3 — Task Gates
squad-creator-extend.md: New Step 2.5 — interactive Internalization Gate when adding an agent from an external reference. Includes the 6-step flow inline and a hard rule: the generated file MUST NOT containdependencies.context.squad-creator-create.md: Added RULE note in the agent generation step.Layer 4 — Validation + Analysis
squad-creator-validate.md: New Validation Check feat(llm-routing): add DeepSeek usage tracking system #6 (Internalization Gate) +validateInternalization()in the flow + 3 new error codes:AGENT_EXTERNAL_DEP(Error) — agent containsdependencies.contextAGENT_SHALLOW_PRINCIPLES(Warning) — fewer than 3 core_principlesAGENT_SKELETAL_PERSONA(Warning) — too-brief role/identitysquad-creator-analyze.md: New Step 4.5 calculates an internalization score per agent; newInternalization:coverage line in the report output.Real-world origin
This was discovered in a production squad where an agent created from a methodology document (SKILL_v2.md) was pointing back to the source on every run. The root cause was traced to the absence of any structural constraint preventing
dependencies.contextfrom being used. This PR adds those constraints at all 4 levels.Testing
*validate-squad <name>now reportsAGENT_EXTERNAL_DEPfor any agent withdependencies.context*analyze-squad <name>now shows Internalization score in coverage output*extend-squad <name>now asks about external references when adding agents and guides through the distillation process🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Chores