[playground][dogfooding] initial playground dogfooding check list

[Parker-Stafford]

Datasets

(Tony + Xander)

• test on large datasets
• streaming
• non streaming (ui side)
• [stretch] test behind proxy with some timeout

Tools

(Roger + Parker)

• tool switching for calls and schemas between providers

Tool calling

Does the playground allow you to simulate an actual tool call (user message with tool, assistant message with tool calls, tool message with results, proper response from llm)

• tool calling for anthropic
• tool calling for openai

Tool use

Does the playground allow you to add tools that can be appropriately picked out by an llm (add a tool with a relevant message)

• tool use for anthropic
• tool use for openai

Template application

• mustache (datasets / normal)
• fstring (datasets / normal)

Span replay

(Xander + Parker)

• test going from auto instrumentor to playground
• test going from playground to span to playground
• input
• output
• model
• invocation params
• tools
• tool calls

Structured output

Parker + Roger

Known issues / WIP

Playground spans

• [playground][instrumentation] message.tool.id is not instrumented messages on playground spans  #5370
• [playground][instrumentation] tool calls in input messages are only properly instrumented for openai #5369

UI

• [playground][ui] add experiment run details slideover with reference input, reference output, and playground run output #5286
• [playground][ui] display "no variable" errors for dataset examples before and after running #5284
• [playground][ui] support anthropic tool response messages #5267
• [playground] llm span to playground parameter parsing utils #4886 - missing prompt template variable parsing

Server

New Issues

• [playground][ui] cannot paginate datasets table on playground page #5382
• [playground][ui] need to add errors into output display so they are persisted longer than just the notification for debugging #5383
• [playground][ui] bug- output schema is cleared every time you click the model config button #5384
• [playground][ui] output schema is available on anthropic but doesn't work #5385
• [playground][ui] bug - invocation params form makes a fetch on every key press when typing in an azure deployment name #5386
• azure invocation param serving needs investigation, do we call out to get them or just serve a default set #5387
• [playground][ui] bug - azure model config saving is broken #5388
• [playground][instrumentation] add azure as provider on playground spans for azure runs #5389
• [plagyround][server] invocation parameter defaults are missing for most models which leads to inconsistent ux #5390
• [playground][ui] surface that you don't have to set api keys in the browser can be added as environment variables for server #5391
• [playground][server] bug - not setting an api key in the browser for playground causes a swallowed error #5392
• [playground][server] investigate strategy for batching llm calls on dataset runs in playground #5394

Lower prio

• [playground][instrumentation] strip out internal graphql / playground param from input.value in playground spans #5393