-
Notifications
You must be signed in to change notification settings - Fork 584
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Teacher tool: first pass of nested validators #9862
Conversation
…ach loop to easily check children
…n successful blocks
…erk-nested-validators
localtypings/pxteditor.d.ts
Outdated
@@ -298,6 +298,7 @@ declare namespace pxt.editor { | |||
export interface EditorMessageRunEvalRequest extends EditorMessageRequest { | |||
action: "runeval"; | |||
validatorPlan: pxt.blocks.ValidatorPlan; | |||
planBank: pxt.blocks.ValidatorPlan[]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small naming suggestion for clarity.
planBank: pxt.blocks.ValidatorPlan[]; | |
planLib: pxt.blocks.ValidatorPlan[]; |
blockResults.missingBlocks.length === 0 && | ||
blockResults.insufficientBlocks.length === 0; | ||
return success; | ||
const blockId = Object.keys(inputs.blockCounts)[0]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method has several array accesses without range checks. Are these arrays guaranteed to be non-empty?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For blockId
in particular, if Object.keys(inputs.blockCounts)
returned nothing, then the validation would be essentially pointless. With this array access, I'm getting the block that we're checking exists for that validator run.
So, for example, if you have this validator:
{ ".desc": "A custom variable's value is set to a random number", "name": "variable_set_random", "threshold": 1, "checks": [ { "validator": "blocksExist", "blockCounts": { "variables_set": 1 }, "childValidatorPlans": ["device_random_used"] } ] }
The block that we're actually wanting to check for is variables_set
, and you can only know this in this function by looking in the JSON. For tutorial validation, you can have multiple different blocks in the blockCounts
object, but for our validation, we should only ever have one entry in the blockCounts
object. Either way, if Object.keys(inputs.blockCounts)
was empty at the point that we do an array access in this function, validation would be essentially pointless. However, you're right in that I should check whether the array returned from Object.keys()
has length before accessing.
const childPassed = runValidatorPlanAsync(blocksToUse, childPlan, planBank); | ||
timesPassed += childPassed ? 1 : 0; | ||
} | ||
checkPassed = timesPassed > 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Potential bug here. In the course of iterating over child validator plans, checkPassed
can become false then true again with no consequences. Only the last-assigned value of checkPassed
is used when computing checksSucceeded
below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, good catch. I think "anding" this will do the trick..
If we have any of the children validators fail, then the whole validator plan should fail, so, if at any moment, checkPassed
is false, then that false value should be maintained.
…ocks exist passed
…erk-nested-validators
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great!
This sets up the ability to test for nested blocks in student projects. I haven't tested this extensively yet, but in order to do that, I need to create a lot of new validators, and I didn't want to put all of that work in one PR.
In this PR:
runValidatorPlanAsync
to use a for loop. It was easier for me to conceptualize the nested checks this way. Also, the in the switch statement in the anonymous function inpromisePool
, therun
function results were getting returned, and I wasn't sure how to work within that pattern while running nested checks. So, checks are now run sequentially.runBlocks__ExistValidation
functions inrunValidatorPlanAysnc
to be a tuple of the blocks that were successful with the validation and the boolean pass/fail of the validation. We can arguably simplify this by just returning the list and checking its length, but I was planning on leaving the validators where we don't want to allow nesting to just return booleans, and I would like there to be some commonality between all of therun
functions. I didn't change the return types of the comment validators because thespecific block comments
validator returns the blocks that don't have comments on them, so there's a discrepancy there.Some screenshots: