SelectionStrategy and its issues

[arafattehsin]

Hey team. Recently, I have been focusing a lot on the Agent Framework of SK and I have found it really interesting so far. However, I have recently encountered a few issues while using SelectionStrategy. I want to know what are some of the best examples of SelectionStrategy as I usually encounter issues. Below is one of the examples which I just encountered so I thought I should ask for the expert advice here.

Note: I'm just extending on the examples which have been provided earlier in the Learn docs.

Background

I have got 4 agents.

1. WeatherAgent
2. FoodAgent
3. TravelAgent
4. ConciergeAgent (Termination Agent with an approved plan)

Here's my SelectionFunction prompt:

`KernelFunction selectionFunction =
AgentGroupChat.CreatePromptFunctionForStrategy(
$$$"""
Determine which participant takes the next turn in a conversation based on the the most recent participant.
State only the name of the participant to take the next turn.
No participant should take more than one turn in a row.

    Choose only from these participants in the following order:
    - {{{weatherAgent.Name}}}
    - {{{foodAgent.Name}}}
    - {{{travelAgent.Name}}}
    - {{{conciergeAgent.Name}}}
    
    After {{{weatherAgent.Name}}} has responded, select the {{{foodAgent.Name}}} agent for the next response.
    After {{{foodAgent.Name}}} has responded, select the {{{travelAgent.Name}}} agent for the next response.
    After {{{travelAgent.Name}}} has responded, select the {{{conciergeAgent.Name}}} agent for the next response.

    Based upon the feedback of ConciergeAgent's feedback, select the appropriate agent for the response. 

    History:
    {{$history}}
    """,
    safeParameterNames: "history");`

Output I received

[crickman]

This looks like the model is returning a list of agent names for the failed response:

FoodAgent
TravelAgent
ConciergeAgent

Perhaps it had decided to anticipate the next 3 turns? I'd be curious how often this occurs and which model you are targeting for the selection strategy.

In anycase, a couple thoughts come to mind:

• When parsing the result, you could take only the first name as the selection result (as a resiliency measure).
• Since your stategy is only based on the previous respondant, I might use a truncation history reducer to only include the most recent message (which will also save tokens). (https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/GettingStartedWithAgents/Step04_KernelFunctionStrategies.cs#L88)
• Consider using structured outputs to constrain the model to more reliable (JSON) result. As part of this, I might ask the model to include the selected agent's name as well as its reasoning for selecting that agent:

  {
    "selectedagent": "...",
    "selectionreason": "..."
  }

  Including the reason serves two purposes. 1) provides insight into why a selection occurred (when unexpected) for potential prompt tuning 2) encourages the model to engage in "chain of thought" in order to influence the result. For the JSON schema, I'd utilize the description property for selectedagent to reenforce the direction to only select a single agent. (I've included the JSON schema at the end of this response.)
• Even prior to the error, agent selection doesn't appear to be matching the defined order. I wonder if the model isn't over keying on the term "after" (since at a certain point all messages are after each of the agents have responded). Some prompt tuning may help with this (which can be somewhat model dependant). Also limiting the amount of history being evaluated (previous point on reducer) may also aid in focusing the model.
• If you simply want to select each agent in order, perhaps consider the SequentialSelectionStrategy?
• As agent coordination becomes more complex, consider using our newly introduced Process Framework: https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/GettingStartedWithProcesses. This provides a graph based appoach that may be used for coordinating agent selection.

Note: Here's an outline of the suggested JSON schema:

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
      "selectedagent": {
          "type": "string",
          "description": "the name of a single agent being selected"
      },
      "selectionreason": {
          "type": "string",
          "description": "the reason for selecting the specified agent"
      }
  },
  "required": [
      "selectedagent",
      "selectionreason"
  ]
}

[arafattehsin]

Thank you @crickman for this detailed explanation. I have now got the idea, and it gave me some other thoughts on improving it as well with the json schema.

To answer your questions: I'm was GPT-4 but my idea was to use multiple models. The occurrences of multiple turns were very frequent until I played with prompts to restrict them.