You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've recently stumble upon a Failed to parse output. Returning None. while trying to evaluate the faithfulness of answers generated from a model based on a generated dataset.
As explained in the warning, it seems to come from a ill structured output. After looking in your repository, I've noticed many issues related to this warning and its consequences.
Some users have found different solutions as changing the evaluation prompt (#1150) or the context/output max token (#1120) but for me it was thanks to @jjmachananswer that I've managed to correctly evaluate my rag by changing the llm metric from llama3.1:8b to llama3.2:3b (I use LangChain wrapper to wrap a model provided by Ollama with ChatOllama).
That's why I was wondering if you could explicit a bit more this warning to guide users along some way to improve their evaluation method or change the warning into an exception so we can raise it if needed and explicit the issue ourselves.
Why is the feature important for you?
So others won't spend so much time to debug their evaluation step and avoid adding useless issues to your repository ^^
In any case, thanks again for your amazing work, keep it up as long as possible ! Have a nice day :)
The text was updated successfully, but these errors were encountered:
Describe the Feature
Hi @shahules786 @jjmachan, I'm back in Ragas business ^^
I've recently stumble upon a
Failed to parse output. Returning None.
while trying to evaluate the faithfulness of answers generated from a model based on a generated dataset.As explained in the warning, it seems to come from a ill structured output. After looking in your repository, I've noticed many issues related to this warning and its consequences.
Some users have found different solutions as changing the evaluation prompt (#1150) or the context/output max token (#1120) but for me it was thanks to @jjmachan answer that I've managed to correctly evaluate my rag by changing the llm metric from
llama3.1:8b
tollama3.2:3b
(I use LangChain wrapper to wrap a model provided by Ollama with ChatOllama).However, there still are many opened issues concerning this same warning (#1630, #1545, #1358, #1274, #1228, #1206, #1186, #1150, #1120, #966, #957, #955, #859)
That's why I was wondering if you could explicit a bit more this warning to guide users along some way to improve their evaluation method or change the warning into an exception so we can raise it if needed and explicit the issue ourselves.
Why is the feature important for you?
So others won't spend so much time to debug their evaluation step and avoid adding useless issues to your repository ^^
In any case, thanks again for your amazing work, keep it up as long as possible ! Have a nice day :)
The text was updated successfully, but these errors were encountered: