Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple classes per token #1869

Closed
dhakehurst opened this issue Mar 13, 2020 · 7 comments
Closed

Support multiple classes per token #1869

dhakehurst opened this issue Mar 13, 2020 · 7 comments
Labels
feature-request Request for new features or functionality out-of-scope

Comments

@dhakehurst
Copy link

I am creating a custom tokenizer.
I want to add multiple 'classes' for each token.

I see the support for '.' in between individual token classes, but that implies a hierachy.

I want to simply add multiple tokens, without a hierarchy.
I.e. an array of scopes/token classes.

I spotted this issue [https://github.com//issues/929]

but it does not seem to solve the issue.

@alexdima alexdima added the feature-request Request for new features or functionality label Apr 2, 2020
@SimonCockx
Copy link

+1. This would also enable support for TextMate. See zikaari/monaco-editor-textmate#24

@SimonCockx
Copy link

SimonCockx commented Jan 13, 2023

@alexdima (and @svenefftinge ?) In #929 you state that the issue of accepting multiple scopes per token is resolved. That's not the case however; see this line. Having an array of scopes is necessary to fully support a tokenizer such as TextMate, which is why its current implementation is limited.

The workaround for picking the first scope that has a theme defined (which is how theia and monaco-textmate have done it) is only a partial fix. It doesn't allow you to do:

  1. theme merging. If multiple theme rules match scopes at different levels, their styles should be merged if they're non-conflicting. E.g.,
{
    "scope": "comment",
    "settings": {
        "fontStyle": "italic"
    }
},
{
    "scope": "comment.block.documentation",
    "settings": {
        "foreground": "#6A9955"
    }
}
  1. match multiple scopes at once. E.g.,
{
    "scope": "comment.block.documentation string.quoted",
    "settings": {
        "foreground": "#6A9955"
    }
}

-> this will only match tokens that have the scope string.quoted preceded by comment.block.documentation, which is impossible to express if you only pick a single scope.

These features are fully supported by VSCode, though.

@hediet
Copy link
Member

hediet commented Feb 21, 2023

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.

If you wonder what we are up to, please see our roadmap and issue reporting guidelines.

Thanks for your understanding, and happy coding!

@hediet hediet closed this as not planned Won't fix, can't repro, duplicate, stale Feb 21, 2023
@dhakehurst
Copy link
Author

yes I disagree :)

it is very important to me to be able to add multiple classes

@SimonCockx
Copy link

SimonCockx commented Mar 7, 2023

It's not only important to @dhakehurst though. :) It's very important if Monaco wants to fully support TextMate based tokenization, which is the most popular way of supporting syntax highlighting, simply because it is how VS Code's tokenization engine is powered.

See my comment above.

@hediet
Copy link
Member

hediet commented Mar 8, 2023

It's very important if Monaco wants to fully support TextMate based tokenization

But why is this not an issue in VS Code? It also uses textmate and the monaco editor is an extraction of VS Code.

@SimonCockx
Copy link

SimonCockx commented Mar 8, 2023

@hediet Good question. I've just been scrolling through VS Code's code and it seems that an array of scopes is represented as a string separated by spaces. So... I take back the statement that having an array of scopes is strictly necessary. In a sense this is already supported!

It makes me wonder. When performing the action "Inspect Tokens", does Monaco support a space-separated string of scopes, like VS Code does?

If the answer is yes, I wonder why repo's such as monaco-textmate reverts to ugly workarounds such as picking only a single scope. If the answer is no, I wonder where Monaco and VS Code diverge exactly.

Similarly, where does VS Code's support for theme merging and scope matching come from? Can the same mechanism be reused in Monaco?

Sorry, the realization that this issue is actually non-blocking for TextMate support in Monaco just gives me a load of different questions. :)

@github-actions github-actions bot locked and limited conversation to collaborators Apr 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature-request Request for new features or functionality out-of-scope
Projects
None yet
Development

No branches or pull requests

4 participants