.Net: Bug: Crash in OnnxRuntimeGenAIChatCompletionService

[lilhoser]

Describe the bug
Application crashes with "bad allocation" nebulous message when iterating over vector search results:

var searchResult = await collection.VectorizedSearchAsync(searchVector, searchOptions);
await foreach (var item in searchResult.Results)
{
    return item.Record.Description;
}
return string.Empty;

Call stack:

   at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr nativeResult)
   at Microsoft.ML.OnnxRuntimeGenAI.Generator..ctor(Model model, GeneratorParams generatorParams)
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<RunInferenceAsync>d__13.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/Connectors/Connectors.Onnx/OnnxRuntimeGenAIChatCompletionService.cs:line 99
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<RunInferenceAsync>d__13.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<GetStreamingChatMessageContentsAsync>d__12.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/Connectors/Connectors.Onnx/OnnxRuntimeGenAIChatCompletionService.cs:line 82
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<GetStreamingChatMessageContentsAsync>d__12.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/Connectors/Connectors.Onnx/OnnxRuntimeGenAIChatCompletionService.cs:line 82
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<GetStreamingChatMessageContentsAsync>d__12.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.<InvokeStreamingCoreAsync>d__7`1.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/SemanticKernel.Core/Functions/KernelFunctionFromPrompt.cs:line 304
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.<InvokeStreamingCoreAsync>d__7`1.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/SemanticKernel.Core/Functions/KernelFunctionFromPrompt.cs:line 304
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.<InvokeStreamingCoreAsync>d__7`1.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Microsoft.SemanticKernel.KernelFunction.<InvokeStreamingAsync>d__30`1.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/SemanticKernel.Abstractions/Functions/KernelFunction.cs:line 418
   at Microsoft.SemanticKernel.KernelFunction.<InvokeStreamingAsync>d__30`1.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/SemanticKernel.Abstractions/Functions/KernelFunction.cs:line 430
   at Microsoft.SemanticKernel.KernelFunction.<InvokeStreamingAsync>d__30`1.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)

Repro steps might be a little complicated because the data set is large.

[f2bo]

I don't know if your issue is the same, but my experience of the "bad allocation" error message with ONNX is when failing to specify MaxTokens in the OnnxRuntimeGenAIPromptExecutionSettings. If the model you are using supports a large context length, (e.g. 128K), it will attempt to allocate too much memory and fail.

I reported the issue about the "nebulous message" in the ONNX runtime genai repo (microsoft/onnxruntime-genai#980). There's a PR in progress that makes the message slightly less obscure.