Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Semantic tokenizer improvements: tests, simplification, syntax-in-syntax bugfix #497

Merged
merged 39 commits into from
Nov 8, 2024
Merged

Semantic tokenizer improvements: tests, simplification, syntax-in-syntax bugfix #497

merged 39 commits into from
Nov 8, 2024

Conversation

sungshik
Copy link
Contributor

@sungshik sungshik commented Nov 1, 2024 •
edited
Loading

sungshik and others added 28 commits October 30, 2024 09:55
@sungshik
Add first tests for the semantic highlighter
4cee25d
@sungshik
Remove unused imports
c588d7c
@sungshik
Sort tokens by location before comparing them with expected categories
42af05f
@sungshik
Improve test utilities for semantic highlighter
2552ca8
@sungshik
Add semantic tokenizer tests for nested categories
ee18e05
@sungshik
Improve test utilities for semantic tokenizer
52f6de9
@sungshik
Improve test utilities for semantic tokenizer
4ce36dc
@sungshik
Add semantic tokenizer tests for unicode (surrogate pair counting)
c52aff9
@sungshik
Add license
8b2ccf5
@sungshik
Improve testKeywordLastLine (semantic tokenizer)
32d5ff7
@sungshik
Merge pull request #494 from usethesource/semantic-tokenizer-fall2024…
a69585c 
…-tests

Add semantic tokenizer tests
@sungshik
Refactor class TokenCollector of the semantic tokenizer
6c2ed04
@sungshik
Simplify code to collect ambiguity semantic tokens
5b7b264
@sungshik
Make diff smaller
a7aa3f9
@sungshik
Make diff smaller
41cfe99
@sungshik
Add support for three token collector modes to the semantic tokenizer
f921a8f
@sungshik
Merge pull request #500 from usethesource/semantic-tokenizer-fall2024…
4931b7d 
…-refactor-collector

Simplify class `TokenCollector` to avoid two versions of maximal token logic
@sungshik
Merge branch 'semantic-tokenizer-fall2024' into semantic-tokenizer-fa…
cf2e8cc 
…ll2024-collector-modes
@sungshik
Fix identation
8d7da6b
@sungshik
Add comment to test
4266b26
@sungshik
Fix comments
9ce5e9e
@sungshik
Fix indentation
08cfe20
@sungshik
Fix typos
fac4f05
@sungshik
Add useLegacyHighlighting parameter to the LanguageService ADT (a…
0435743 
…nd make corresponding changes in the Java part)
@sungshik
Add support for useLegacyHighlighting in the semantic tokenizer
75e8b30
@sungshik
Update the semantic tokenizer tests for useLegacyHighlighting
161d638
@sungshik
Fix typos, shorten names
e3e12b9
@sungshik
Update Pico demo language server to not use legacy highlighting
29e222f
@sungshik sungshik mentioned this pull request Nov 6, 2024
Closed
@sungshik sungshik marked this pull request as ready for review November 6, 2024 15:43
@sungshik sungshik changed the title Semantic tokenizer 2024 (fall) Semantic tokenizer improvements: tests, simplification, syntax-in-syntax bugfix Nov 6, 2024
DavyLandman
DavyLandman requested changes Nov 6, 2024
View reviewed changes
Copy link
Member

@DavyLandman DavyLandman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, a few small things I noticed.

...-lsp/src/main/java/org/rascalmpl/vscode/lsp/parametric/LanguageContributionsMultiplexer.java Outdated Show resolved Hide resolved
rascal-lsp/src/main/java/org/rascalmpl/vscode/lsp/util/SemanticTokenizerTester.java Outdated Show resolved Hide resolved
rascal-lsp/src/main/rascal/lang/rascal/tests/semanticTokenizer/Util.rsc Show resolved Hide resolved
rascal-lsp/src/main/rascal/lang/rascal/tests/semanticTokenizer/Util.rsc Outdated Show resolved Hide resolved
rascal-lsp/src/main/rascal/lang/rascal/tests/semanticTokenizer/Util.rsc Show resolved Hide resolved
rascal-vscode-extension/src/lsp/ParameterizedLanguageServer.ts Outdated Show resolved Hide resolved
@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Nov 8, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
4.6% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

DavyLandman
DavyLandman approved these changes Nov 8, 2024
View reviewed changes
Copy link
Member

@DavyLandman DavyLandman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I also checked with @jurgenvinju and he also approves of the change to the parsing constructor.

@DavyLandman DavyLandman merged commit a23957c into main Nov 8, 2024
12 of 13 checks passed
@sungshik sungshik deleted the semantic-tokenizer-fall2024 branch November 8, 2024 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DavyLandman DavyLandman DavyLandman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Semantic tokenizer makes mistakes when a syntax tree (with category) has syntax children
2 participants
@sungshik @DavyLandman