what's wrong with this grammar? (newbie)

[VladimirAlexiev]

Hi! I'm a complete newbie to Langium and I'm trying to cobble a state transition grammar.
After my attempt, I found https://github.com/eclipse-langium/langium/tree/main/examples/statemachine but it's a bit wordy to my liking.
I want something like this (where the condition is not parsed for now):

rule drilling "Drilling transitions"

state normal "Normal operation"
  -> normal : -1 <= current <=1
  -> overSpeeding : 1 < current

state overSpeeding "Over-speeding, need to slow down"
  -> normal : -1 <= current <=1 

So I tried this grammar:

grammar Rules

entry Rule:
    'rule:' id=ID '"' name=NOQUOTE '"'
    states+=State;

State:
    'state' id=ID '"' name=NOQUOTE '"'
    transitions+=Transition;

Transition:
    '->' nextState=[State] ':' condition=NOQUOTE;

terminal ID: /[_a-zA-Z][\w_]*/;
terminal NOQUOTE: /[^"]+/;
hidden terminal WS: /\s+/;
hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

It causes the content (right) pane to hang with Loading...
I guess there's some conflict between terminals?

[msujew]

Hey @VladimirAlexiev,

So there are a few issues with the terminal NOQUOTE; In no particular order:

1. NOQUOTE does not have an end delimiter. Once the lexer starts tokenizing it, it will not stop until it encounters a " character (which isn't possible if you are in a Transition rule, so it consumes the rest of the input stream).
2. The langium token builder optimizes the order of terminals by moving terminals "to the front" (i.e. they are tried first when tokenizing), if they can match a single whitespace character. This is the case for NOQUOTE, which is why it shadows everything.
3. Something that could be a NOQUOTE terminal can also be lexed as an ID. I.e. they are ambiguous. This means that in cases where you would expect a NOQUOTE, you might receive an ID token.

In the end, your language isn't really a type 2 language (context-free language) that can be parsed using a separated tokenization/parsing phase, because tokenizing the NOQUOTE terminal requires context. This is in theory possible (type 1 parsers exist, and this can be technically achieved using a custom matcher function for the terminal), but is pretty unwieldy and complicated.

Some advice in general on the grammar: I would create a string token (i.e. terminal STRING: /"(\\.|[^"\\])*"|'(\\.|[^'\\])*'/;) and create an expression language for your condition property that properly encapsulates the semantics of an expression/condition language. That way, you don't need a "catch-all" terminal that is NOQUOTE.

[VladimirAlexiev]

Thanks a lot!