Matching a regex partially can lead to generating a token which causes the whole generated sequence to be invalid.

When using partial=True, we ensure we don't generate invalid output, but also this makes it impossible to generate certain output sequences. Therefore it's necessary to allow generating tokens which match only partially, and then take the substring of that token which matches the regex.