FaithfulnesswithHHEM
doesn't match original prompt, leading to inconsistent scores
#1648
Labels
FaithfulnesswithHHEM
doesn't match original prompt, leading to inconsistent scores
#1648
[x] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug
FaithfulnesswithHHEM
class for the faithfulness metric (using Vectara's model) doesn't lead to the same scores when compared with running the original implementation (on HF).I've figured 2 reasons for this mismatch:
Ragas version: 0.2.4 (latest)
Python version: 3.10
Code to Reproduce
Error trace
No error
Expected behavior
I expect both the scores to be the same i.e. RAGAS implementation of HHEM resulting in same score as HHEM's original implementation on HF.
The text was updated successfully, but these errors were encountered: