RAG Document Chat Not Working with Agents

[ogipogi]

What happened?

When using agents with RAG (Retrieval-Augmented Generation), the AI chat doesn't receive the file content when asking questions, even though document upload works successfully. This is due to a conditional check in the code that prevents context handlers from being created when using agents.

Steps to Reproduce

Steps to Reproduce:

1. Configure and enable RAG
2. Configure and enable agents
3. Start a chat with an agent
4. Upload a document to the chat
5. Ask questions about the document content
6. Observe that the agent responses don't include information from the uploaded document

Expected behavior:

The agent should be able to access and utilize the content from uploaded documents when responding to questions, similar to how it works with non-agent endpoints.

Actual behavior:

The agent cannot access the document content due to the isAgentsEndpoint condition blocking the creation of context handlers.

Code location:

LibreChat/api/server/controllers/agents/client.js

Lines 280 to 285 in f150355

	if (this.message_file_map && !isAgentsEndpoint(this.options.endpoint)) {
	this.contextHandlers = createContextHandlers(
	this.options.req,
	orderedMessages[orderedMessages.length - 1].text,
	);
	}

Workaround:

Remove the !isAgentsEndpoint(this.options.endpoint) condition to allow context handlers to be created for agent endpoints:

if (this.message_file_map) {
  this.contextHandlers = createContextHandlers(
    this.options.req,
    orderedMessages[orderedMessages.length - 1].text,
  );
}

• LibreChat version: main #c2e0789a

Thanks for checking 🙏

What browsers are you seeing the problem on?

Chrome

Relevant log output

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

For agents, it is intended to utilize the RAG API with the File Search capability: https://www.librechat.ai/docs/features/agents#file-search

[ogipogi]

That means, is it also for documents that the user is uploading?

I thought that file search is for static files uploaded upfront that agent can access...

[danny-avila]

Yes, that will allow the user to upload files, for which the agent will be aware of them if uploaded correctly:

[ogipogi]

Thanks for the insights, I tested it and it worked now 🙏