Tokenization testing

.github/workflows/build.yaml

@@ -23,6 +23,14 @@ jobs:
           java-version: 11
           distribution: 'temurin'
           cache: 'maven'
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: 'npm'
+          cache-dependency-path: 'rascal-textmate-core/package-lock.json'
+      - name: install-tokenizer
+        working-directory: rascal-textmate-core
+        run: npm install vscode-tmgrammar-test
       - name: run-tests
         working-directory: rascal-textmate-core
         run: mvn test -B -Drascal.compile.skip -Drascal.tutor.skip

README.md

@@ -16,5 +16,18 @@ TextMate grammars, this project applies partial conversion. Alternatively, a
 previous [project](https://github.com/TarVK/syntax-highlighter) by
 [@TarVK](https://github.com/TarVK) applies total conversion.
 
-The [walkthrough](src/main/rascal/lang/textmate/conversiontests/Walkthrough.rsc)
+## Documentation
+
+The [walkthrough](rascal-textmate-core/src/main/rascal/lang/textmate/conversiontests/Walkthrough.rsc)
 explains the main ideas behind the conversion algorithm in this project.
+
+## Tests
+
+To test tokenization (as part of the conversion
+[tests](rascal-textmate-core/src/main/rascal/lang/textmate/conversiontests)),
+the [`vscode-tmgrammar-test`](https://github.com/PanAeon/vscode-tmgrammar-test)
+tool is used. Install it locally in directory `rascal-textmate-core` as follows:
+
+```
+npm install vscode-tmgrammar-test
+```

rascal-textmate-core/.gitignore

@@ -1 +1,2 @@
-target
+target
+node_modules