Restore updated categories (specifically for Rascal syntax highlighting)

[sungshik]

To generate a TextMate grammar for Rascal's own grammar, there are two alternatives for mapping Rascal categories to TextMate scope names:

• Use the current indirect mapping -- via semantic token types -- which is used by the semantic highlighter
• Use the previous direct mapping, which is not used anymore (replaced by the indirect mapping)

In principle, the direct mapping can be more precise as it doesn't need to "fit" the abstraction of semantic token types. Thus, in PR #1, we decided to use the direct mapping (discussion).

However, because the TextMate highlighter and the semantic highlighter are now based on different mappings, their results are inconsistent. Notably, string constants (which are assigned category Constant in Rascal's own grammar) are colored differently by the two highlighters (it's mapped to string using the indirect mapping, but to constant using the direct mapping).

For now, this PR restores the use of the indirect mapping instead of the direct mapping when generating a TextMate grammar for Rascal's own grammar. (Grammars for other languages, which presumably define their own mapping from categories to scope names, are unaffected by this PR.)

[DavyLandman]

Making all constant string is not helping. We should do something in the rascal highlighter to fix the underlying issue. Loosing precision here is not so nice.

[sungshik]

Making all constant string is not helping

The only productions with the @category("Constant") tag in Rascal's own grammar are string-related, so I don't think precision is actually lost here for Rascal's own grammar. (See the other comment for a full overview of the usage of @category tags.)

So, in the context of Rascal's own grammar, I can understand why the semantic highlighter maps Constant to string.

fix the underlying issue

I think the best way forward is to update the categories in Rascal's own grammar (which are "legacy token types") to directly use the intended TextMate scopes. For instance, each occurrence of @category("Constant") would then be replaced with @category("string") (or, for even more precision, @category("string.quoted.double") and @category("string.quoted.single"), depending on the production).

However, if it's logistically complicated to update Rascal's own grammar right now, and/or release such an update in the near future, we could also temporarily use a preprocessor to rewrite categories when generating a TextMate grammar -- only for Rascal's own grammar (other grammars unaffected). That's essentially this PR.

[DavyLandman]

Okay, I assumed numbers would also get a constant assigned? that is weird. We might have to introduce them (maybe via this rewrite logic to add a category for certain productions?)

[DavyLandman]

I hope we can find a different way to fix this. maybe let the semantic tokenizer do less.

[sungshik]

For reference, here's an overview of all non-terminals in Rascal's own grammar that have a category.

@category("Constant")

lexical PreStringChars
  = @category="Constant" [\"] StringCharacter* [\<] ;

lexical MidStringChars
  = @category="Constant" [\>] StringCharacter* [\<] ;

lexical PostStringChars
  = @category="Constant" [\>] StringCharacter* [\"] ;

lexical StringConstant
  = @category="Constant" "\"" StringCharacter* chars "\"" ;

lexical CaseInsensitiveStringConstant
  = @category="Constant" "\'" StringCharacter* chars "\'" ;

lexical Char
  = @category="Constant" "\\" [\  \" \' \- \< \> \[ \\ \] b f n r t] 
  | @category="Constant" ![\  \" \' \- \< \> \[ \\ \]] 
  | @category="Constant" UnicodeEscape ;

Note: There are only string/char constants; no number constants.

@category("Comment")

lexical Comment
  = @category="Comment" "/*" (![*] | [*] !>> [/])* "*/" 
  | @category="Comment" "//" ![\n]* !>> [\ \t\r \u00A0 \u1680 \u2000-\u200A \u202F \u205F \u3000] $ ;

syntax Tag
  = @Folded @category="Comment" \default   : "@" Name name TagString contents 
  | @Folded @category="Comment" empty     : "@" Name name 
  | @Folded @category="Comment" expression: "@" Name name "=" Expression expression !>> "@" ;

Other

lexical ConcretePart
  = @category="MetaSkipped" text   : ![`\<\>\\\n]+ !>> ![`\<\>\\\n]
  | newline: "\n" [\ \t \u00A0 \u1680 \u2000-\u200A \u202F \u205F \u3000]* "\'"
  | @category="MetaVariable" hole : ConcreteHole hole
  | @category="MetaSkipped" lt: "\\\<"
  | @category="MetaSkipped" gt: "\\\>"
  | @category="MetaSkipped" bq: "\\`"
  | @category="MetaSkipped" bs: "\\\\" ;

lexical Output   
  = @category="Result" resultOutput: "⇨" ![\n\r]* [\n] 
  | @category="StdOut" stdoutOutput: ^ "≫" ![\n\r]* [\n]
  | @category="StdErr" stderrOutput: ^ "⚠" ![\n\r]* [\n] ;

[DavyLandman]

I think we should dynamically add some categories to productions in the grammar. Just because the base grammar is harder to change. And then maybe when this is happening: usethesource/rascal#1928 improve the categories itself.

[sungshik]

I think we should dynamically add some categories to productions in the grammar. Just because the base grammar is harder to change. And then maybe when this is happening: usethesource/rascal#1928 improve the categories itself.

Agreed on this course of action; the latest commits bring in the additional hot-patching for numbers, strings, and locations.