Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicit the "Failed to parse output. Returning None." of RagasoutputParser or add a way to catch it. #1681

Open
Gwenn-LR opened this issue Nov 15, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@Gwenn-LR
Copy link
Contributor

Gwenn-LR commented Nov 15, 2024

Describe the Feature
Hi @shahules786 @jjmachan, I'm back in Ragas business ^^

I've recently stumble upon a Failed to parse output. Returning None. while trying to evaluate the faithfulness of answers generated from a model based on a generated dataset.

As explained in the warning, it seems to come from a ill structured output. After looking in your repository, I've noticed many issues related to this warning and its consequences.

Some users have found different solutions as changing the evaluation prompt (#1150) or the context/output max token (#1120) but for me it was thanks to @jjmachan answer that I've managed to correctly evaluate my rag by changing the llm metric from llama3.1:8b to llama3.2:3b (I use LangChain wrapper to wrap a model provided by Ollama with ChatOllama).

However, there still are many opened issues concerning this same warning (#1630, #1545, #1358, #1274, #1228, #1206, #1186, #1150, #1120, #966, #957, #955, #859)

That's why I was wondering if you could explicit a bit more this warning to guide users along some way to improve their evaluation method or change the warning into an exception so we can raise it if needed and explicit the issue ourselves.

Why is the feature important for you?
So others won't spend so much time to debug their evaluation step and avoid adding useless issues to your repository ^^

In any case, thanks again for your amazing work, keep it up as long as possible ! Have a nice day :)

@Gwenn-LR Gwenn-LR added the enhancement New feature or request label Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant