Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass objects to custom oracles #473

Merged
merged 2 commits into from
Nov 29, 2023

Conversation

niknetniko
Copy link
Member

@niknetniko niknetniko commented Nov 28, 2023

This replaces the expected/actual values and provides some more context for the oracles.

Fixes #472 and closes #470.

A check function must now look like this:

# noinspection PyUnresolvedReferences
from evaluation_utils import EvaluationResult


def evaluate_no_arguments(context):
    pass

def evaluate_no_arguments(context, arg1, arg2):
    pass

When passing additional arguments in the test suite, they are added to the function call as arg1, arg2, etc.

The context is a class with the following fields:

class ConvertedOracleContext:
    expected: Any  # Expected value from the test suite
    actual: Any        # Value generated by the submission
    execution_directory: str  # Directory where the execution took place (this will be a subdirectory of the workdir)
    evaluation_directory: str  # Path to the evaluation directory (in the exercise directory)
    programming_language: str  # Name of the programming language of the submission
    natural_language: str  # Natural language

This replaces the expected/actual values and provides some more context
for the oracles.

This is currently a dictionary, but if #472 were done, we could
replace it with an attrs class.
@niknetniko niknetniko marked this pull request as ready for review November 28, 2023 19:33
@niknetniko niknetniko merged commit d71fbdf into master Nov 29, 2023
7 checks passed
@niknetniko niknetniko deleted the enhance/better-custom-check-functions branch November 29, 2023 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant