Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #55 for numeric looking tokens.
I have some concerns over performance (relative to the previous implementation), as I constructed a
String
to capture relevant part of the chars iterator. This can probably be improved, but it wasnt obvious to me that I had the right data to do it. Also it is checking i32 || f64 conversion to str as a means of deciding. It's possible a custom int/float lexer could shortcut by identifying the more trivial cases before resorting to the conversion routines.Some tests were added with positive and integers, numeric floats and exponential floats, plus a token that starts with a number and has letters.