Citations from a Plugin

[malv007]

We have several Plugins that provide information and functionality to our IChatCompletionService (AzureOpenAIChatCompletionService).

We would like for some of these Plugins to provide citation information to the IChatCompletionService, basically be able to retrieve the citation information from the context by doing something like this

      var message = chatMessageContent.InnerContent as OpenAI.Chat.ChatCompletion;
      var citations = message.GetMessageContext().Citations;

Is this even possible? We have tried for the Plugin functions to return a KernelSearchResults enumeration, but that didn't work.

What are the options for getting citations in the IChatCompletionService? The only example we have seen is the one that uses the AzureChatDataSource property.

semantic-kernel/dotnet/samples/Concepts/ChatCompletion/AzureOpenAIWithData_ChatCompletion.cs

Line 51 in ed1fb08

var promptExecutionSettings = new AzureOpenAIPromptExecutionSettings { AzureChatDataSource = dataSource };

Is implementing our own ChatDataSource the only option for getting Citations in the IChatCompletionService?

Thanks!

[dmytrostruk]

Hi @malv007, thanks for your question!

Just to clarify, what would be the purpose of this citation information in AzureOpenAIChatCompletionService? As far as I understand, you want your Plugins to provide a citation information, which should be used in AzureOpenAIChatCompletionService, is that correct? If yes, then I assume you are working with function calling, since plugins/functions are used in chat completion service when they are invoked within function calling process. So, if you want to get that information from Plugin and send it to AI model for processing, you can return any result type from your Plugin/Function. This result type will be serialized and passed to AI model in string format. When AI will process the data and return some response, you will receive that response within ChatMessageContent object.

Let me know if this is something you are looking for. I will be happy to assist further. Thanks!

[malv007]

Thank you so much for your response!

The idea behind using citations is to be able to inform the end user what information the AI model used in order to generate that answer.

These are the main components of our application:

Storage > Kernel Memory > Semantic Kernel > ai-chat-protocol > web application

• Kernel Memory to extract text from documents, partition them, generate embeddings, store the information, and perform searches
• Semantic Kernel for their chat completion (AzureOpenAIChatCompletionService), chat history, function invocation (Plugin). Right now, the integration between Semantic Kernel and Kernel Memory is done via a Plugin (function invocation)
• ai-chat-protocol to send the chat messages back and forth from the web client

These three libraries use their own way for representing citations.

• Kernel Memory has the Citation class which you can access from the SearchResult object.
• Semantic Kernel has the Citations property which you can access from the context of the ChatCompletion object.
• ai-chat-protocol typically represents these citations as data_points in the context property.

What would be the best way to integrate Kernel Memory and Semantic Kernel so the citations produced the Kernel Memory are transferred to Semantic Kernel. We have tried for the function (Plugin) to return the citations information in different properties, but this doesn't make the ChatCompletion object to have its Citations property populated.

But perhaps function invocation isn't the best way to integrate Kernel Memory with Semantic Kernel and have these citations transferred.

I hope my explanation makes sense, but feel free to ask for any additional information.

What are your thoughts?

[malv007]

We were thinking that perhaps the only way to to have the Citations property (ChatCitation class) populated in the ChatCompletion object is by using a ChatDataSource (AzureChatDataSource property in the AzureOpenAIPromptExecutionSettings class). Could that be?

That means that we would have to wrap Kernel Memory with a ChatDataSource class, or something like that.

[dmytrostruk]

@malv007 Thanks for providing more information, it's really helpful!

I think that's fine to integrate Kernel Memory with Semantic Kernel via Plugin. Just want to mention, that it's not really important which integration is used via Plugin, as chat completion service can work with any kind of Plugin output.

For example, imagine you are working with following data model:

public sealed class DataModel
{
    public string Content { get; set; }
    public string Citation { get; set; }
}

And you have a plugin which returns some data using that model:

public sealed class MyPlugin
{
    [KernelFunction]
    [Description("Provides data with citations.")]
    public static List<DataModel> GetData()
    {
        return
        [
            new DataModel { Content = "Content #1", Citation = "file1.pdf" },
            new DataModel { Content = "Content #2", Citation = "file2.pdf" }
        ];
    }
}

When user wants to get an information with citations, you can provide it by using following code snippet:

Kernel kernel = Kernel.CreateBuilder()
    .AddOpenAIChatCompletion(
        modelId: "gpt-4o",
        apiKey: "apikey")
    .Build();

kernel.ImportPluginFromType<MyPlugin>();

var executionSettings = new OpenAIPromptExecutionSettings
{
    FunctionChoiceBehavior = FunctionChoiceBehavior.Auto()
};

var result = await kernel.InvokePromptAsync("Get data with citations", new(executionSettings));

Console.WriteLine(result);

The output will look like the following:

Here is the data retrieved with citations:

1. **Content**: Content #1
   - **Citation**: file1.pdf

2. **Content**: Content #2
   - **Citation**: file2.pdf

As we can see, the model was able to process the data provided by the plugin and return a result with citations to the user.

Now, if you don't want to include citations in the same response message together with content, but you want to add it to your UI separately, you would need a programmatic access to the Citation property in the code. For that, I would recommend using OpenAI Structured Outputs. The code with look almost the same, with additional ResponseFormat property in execution settings and result model for your data:

public sealed class DataModelResult
{
    public List<DataModel> Items { get; set; }
}

// ...

var executionSettings = new OpenAIPromptExecutionSettings
{
    FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(),
    ResponseFormat = typeof(DataModelResult)
};

var result = await kernel.InvokePromptAsync("Get data with citations", new(executionSettings));

// Since ResponseFormat is specified, now it's possible to deserialize the response to get access to citations
var data = JsonSerializer.Deserialize<DataModelResult>(result.ToString());

foreach (var item in data.Items)
{
    Console.WriteLine(item.Content);
    Console.WriteLine(item.Citation);
}

More examples with Structured Outputs can be found here.

That means that we would have to wrap Kernel Memory with a ChatDataSource class, or something like that.

Property AzureOpenAIPromptExecutionSettings.AzureChatDataSource has type AzureSearchChatDataSource, which means that it supports Azure AI Search as a data source only. At some point we will update this property to be ChatDataSource, so it can support other data sources available, like Cosmos DB, Mongo DB and so on. But even in this case, I'm not sure if it supports custom ChatDataSource implementations, that needs to be investigated in more detail.

Let me know if code examples above will work in your scenario, thanks!

[malv007]

Thanks for the info about ChatDataSource, good to know.

Thanks for the sample code too. We understand how to retrieve structured outputs when invoking a prompt, but I think our use case is slightly different, we are trying to use an IChatCompletionService and leverage the ChatHistory object (you didn't use them in your example). Our kernel has been configured with many different functions that can return different data structures, and each of those data structures can have citations or not (depending on the function). And the AI model decides when it needs to invoke a particular function.

The code e are using to process chat messages and convert them to the ai-chat-protocol format is the following:

      var chatFunction = _kernel.GetRequiredService<IChatCompletionService>();
      var userInput = message.Messages.Last();
      var history = await _stateStore.GetStateAsync(Id) ?? new ChatHistory(systemMessage);
      history.AddUserMessage(userInput.Content);

      var botResponse = await chatFunction.GetChatMessageContentAsync(
          history,
          executionSettings: _openAIPromptExecutionSettings,
          kernel: _kernel);

      history.AddAssistantMessage(botResponse.ToString());
      await _stateStore.SetStateAsync(Id, history);
      return new AIChatCompletion(Message: new AIChatMessage
      {
        Role = AIChatRole.Assistant,
        Content = $"{botResponse}",
      })
      {
        SessionState = Id,
        Context = GetCitations(botResponse)
      };

But the return chat messages never have citation information, even if the functions return that information in their data structures.

Does that make sense? Perhaps we aren't understanding how the IChatCompletionService works.

[dmytrostruk]

we are trying to use an IChatCompletionService and leverage the ChatHistory object (you didn't use them in your example)

@malv007 Got it, thanks for clarification. Here is the same code example using IChatCompletionService:

public sealed class DataModel
{
    public string Content { get; set; }
    public string Citation { get; set; }
}

public sealed class DataModelResult
{
    public List<DataModel> Items { get; set; }
}

public sealed class MyPlugin
{
    [KernelFunction]
    [Description("Provides data with citations.")]
    public static List<DataModel> GetData()
    {
        return
        [
            new DataModel { Content = "Content #1", Citation = "file1.pdf" },
            new DataModel { Content = "Content #2", Citation = "file2.pdf" }
        ];
    }
}

[Fact]
public async Task TestAsync()
{
    Kernel kernel = Kernel.CreateBuilder()
        .AddOpenAIChatCompletion(
            modelId: "gpt-4o-2024-08-06",
            apiKey: "apikey")
        .Build();

    kernel.ImportPluginFromType<MyPlugin>();

    var executionSettings = new OpenAIPromptExecutionSettings
    {
        FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(),
        ResponseFormat = typeof(DataModelResult)
    };

    var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

    var chatHistory = new ChatHistory();

    chatHistory.AddUserMessage("Get data with citations");

    var result = await chatCompletionService.GetChatMessageContentAsync(chatHistory, executionSettings, kernel);

    var data = JsonSerializer.Deserialize<DataModelResult>(result.ToString());

    foreach (var item in data.Items)
    {
        Console.WriteLine(item.Content);
        Console.WriteLine(item.Citation);
    }
}

But the return chat messages never have citation information, even if the functions return that information in their data structures.

I would recommend checking your function result models, for example are they serializable? If you are sure that your plugins provide citations and they are serializable, try to check your HTTP request body and see if this information is passed to AI model. If this information is passed, try to ask AI model to return the citations back to you. If you don't want to use structured outputs, you can specify it in system message. Otherwise, by using structured outputs, the model will understand that citations should be returned if Citation property exists in your response format model. You can also take the example I shared above, run it in your environment, and instead of my test plugin, try to import your plugins and see how that works.

Thanks!

[dmytrostruk]

@malv007 We also have a console RAG example, where we use a vector store as a source of information, and we output citation information to the user:

• README: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Demos/VectorStoreRAG/README.md
• Code: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Demos/VectorStoreRAG/RAGChatService.cs

I think this example should also help you to achieve your scenario.

[malv007]

Thanks again for all your help!

Sorry, I didn't get a chance to try your suggestion until now, but it seems like the model we are using doesn't support structured responses, even though the documentation seems to state that it should be supported.

This is the error we are getting:

Invalid parameter: 'response_format' of type 'json_schema' is not supported with this model. Learn more about supported models at the Structured Outputs guide: https://platform.openai.com/docs/guides/structured-outputs
   at Azure.AI.OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
   at Azure.AI.OpenAI.Chat.AzureChatClient.CompleteChatAsync(BinaryContent content, RequestOptions options)
   at OpenAI.Chat.ChatClient.<>c__DisplayClass12_0.<<CompleteChatStreamingAsync>g__sendRequestAsync|0>d.MoveNext()

And this is the model we are using (gpt-4o-mini 2024-07-18):

And the documentation mentions that it should be supported:

https://platform.openai.com/docs/guides/structured-outputs#supported-models

Any idea why this might be?