Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation errors are not shown correctly during autograding #1933

Open
tmetzl opened this issue Nov 5, 2024 · 0 comments
Open

Validation errors are not shown correctly during autograding #1933

tmetzl opened this issue Nov 5, 2024 · 0 comments

Comments

@tmetzl
Copy link
Contributor

tmetzl commented Nov 5, 2024

nbgrader_version=0.9.3 and older

When autograding a notebook with metadata errors, the error messages are not helpful and/or misleading.
Here is an example notebook with a duplicate grade_id:

from nbformat.v4 import new_notebook, new_markdown_cell
from nbgrader.preprocessors import CheckCellMetadata
from nbgrader.nbgraderformat import MetadataValidator

nb = new_notebook()

cell = new_markdown_cell(
    metadata=dict(
        nbgrader=dict(
            grade_id="my_id",
            grade=False,
            solution=False,
            locked=True,
            schema_version=3
        )
    )
)

nb.cells.append(
    cell
)

nb.cells.append(
    cell
)

Now when the MetadataValidator is used to validate the notebook there is a descriptive error message that gives the correct information about the error:

MetadataValidator().validate_nb(nb)

Output:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
---> 27 MetadataValidator().validate_nb(nb)

File /opt/conda/lib/python3.11/site-packages/nbgrader/nbgraderformat/v3.py:110, in MetadataValidatorV3.validate_nb(self, nb)
    108 grade_id = cell.metadata['nbgrader']['grade_id']
    109 if grade_id in ids:
--> 110     raise ValidationError("Duplicate grade id: {}".format(grade_id))
    111 ids.add(grade_id)

ValidationError: Duplicate grade id: my_id

When the CheckCellMetadata preprocessor is used the error is undescriptive:

CheckCellMetadata().preprocess(nb, {})

Output:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
---> 28 CheckCellMetadata().preprocess(nb, {})

File /opt/conda/lib/python3.11/site-packages/nbgrader/preprocessors/checkcellmetadata.py:18, in CheckCellMetadata.preprocess(self, nb, resources)
     16     msg = "Notebook failed to validate; the nbgrader metadata may be corrupted."
     17     self.log.error(msg)
---> 18     raise ValidationError(msg)
     20 return nb, resources

ValidationError: Notebook failed to validate; the nbgrader metadata may be corrupted.

When autograding a notebook like this I actually get this misleading error originating from here: https://github.com/jupyter/nbgrader/blob/main/nbgrader/converters/base.py#L421

AutogradeApp | ERROR] One or more notebooks in the assignment use an old version 
    of the nbgrader metadata format. Please **back up your class files 
    directory** and then update the metadata using:
    
    nbgrader update .

This might be related to #1629.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant