Agent Skill and dynamic tool loading

[spdr870]

Native Agent Skill support made it into Agent Framework. This is great!

I have been working on skill support myself and, although the implementation is pretty similar, I did it slightly differently and want to discuss the differences.

The implementation in Agent Framework adds the skill content to the context window (chat history) of the LLM. This has some benefits, but also some issues:

• Pro: it allows loading multiple skills
• Pro: the skill remains loaded in sequential agent runs
• Con: skills cannot be unloaded
• Con: when history is reduced, the skill is reduced
• Con: hard to implement dynamic loading of additional tools (because additional tools are only for the current run, the skill keeps in the context causing a mismatch)

I took a slightly different approach. Instead of loading the skill content into the chat history, I added it to the system instructions. This has different benefits:

• Pro: it allows loading multiple skills
• Pro: skills + tools can be unloaded (very useful when the user switches topics during chat)
• Pro: no chat history pollution, smaller context window
• Pro: easy to implement dynamic loading of additional tools
• Pro: easy to filter available tools based on user permissions
• Con: skills need to be (re)loaded on each agent run
• Neutral: skill loading prompt is slightly more comprehensive

At first, this might not seem like a good idea, because on each run the skill is removed from the instructions. But in practice it works really well. The LLM understands perfectly that it needs to reload the skill each run. It even understands which skill it loaded before, because the tool call remains in the chat history. The performance penalty for the extra tool call each run is compensated with the smaller context size (chat history).

The main benefit for me is being able to progressively load and unload tools. Since my use case involves a large number of tools, limiting the available tools is important.

The below snippet shows my 'load_skill' tool implementation. It includes using the current user identity to filter tools based on user authorization. The last line demonstrates how the instructions are progressively expanded each time a skill is loaded. Multiple skill loadings in a single agent run will append each skill to the system instructions.

[Description(@"Load skill instructions and tools.")]
public void LoadSkill(string skillName)
{
	var identity = _identityProvider.GetIdentity();

	var skill = _skillsStore.GetSkill(skillName);

	var skillContent = _skillLoader.ReadSkillContent(skill);

	var functionService = FunctionInvokingChatClient.CurrentContext!;

	var filteredTools = _toolStore.GetAllTools().Where(t => skill.AllowedTools?.Any(a => a.Matches(t.Name)) == true);

	foreach (var tool in filteredTools)
	{
		if (ToolAuthorizedForIdentity(identity, tool))
			functionService.Options!.Tools!.Add(tool);
	}

	functionService!.Options!.Instructions = skillContent + "\n\n\n" + functionService.Options.Instructions;
}

The system instructions used to select the skill is in the snippet below.

public const string SystemPromptTemplate = """
      # Skills System

      Select appropriate skills to assist with the user's request. Multiple skills can be loaded.

      **Available Skills:**
      {skills_list}

      ---

      **MANDATORY Workflow:**

      1. **Recognize when a skill applies**: Check if the user's task matches any skill's description above
      2. **A skill needs to be (re)loaded when there is no detailed description in the system prompt.**
      3. **Use the LoadSkill function to load the skill using the exact skill name.**
      4. **Use the skill and if more skills are needed, then load the next skill.**
      5. **Skills can be loaded at any time, to extend the agents capabilities and knowledge.**
      6. **All answers should follow the skills instructions.**
      7. **Always (re)load skills after each user message.**

      These are all currently loaded skills:
      """;

[moonbox3]

Adding @SergeyMenshykh