Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ToolCorrectnessMetric raises ZeroDivisionError when expected_tools is empty #1234

Open
j-mesnil opened this issue Dec 18, 2024 · 0 comments
Open

Comments

@j-mesnil
Copy link

Describe the bug
The ToolCorrectnessMetric raises a ZeroDivisionError when called with an LLMTestCase for which expected_tools is empty. In my use case, I want to benchmark that the LLM only calls a tool when necessary, i.e. with an empty expected_tools list.

To Reproduce

  1. Run minimal example:
from deepeval.metrics import ToolCorrectnessMetric
from deepeval.test_case import LLMTestCase

metric = ToolCorrectnessMetric()
test_case = LLMTestCase(
    input="What is an elephant?",
    actual_output="(...)",
    tools_called=[],
    expected_tools=[]
)

metric.measure(test_case)
print(metric.score)
  1. See error ZeroDivisionError: division by zero raised by metric.measure(test_case). This is caused by the line score = len(used_expected_tools) / len(self.expected_tools) in _calculate_score() of the ToolCorrectnessMetric.

Expected behavior
I expected a perfect score of 1 for the minimal example, which represents a test case where no tool call was executed and no tool call was expected.

Software versions:

  • OS: macOS 15.2 (24C101)
  • Python 3.12
  • deepeval 2.0.5

Additional context
This can be mitigated by using should_exact_match=True, but I did not find this workaround to be documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant