Question about parsing terminals in a text string

[mnaderhirn]

Good day.

I have this grammar:

grammar TitleGrammar

import 'common.tokens';

entry Title:
    'TITLE:' title=Text;

grammar CommonTokensGrammar

import 'release.tokens'

entry CommonTokens:
    token = INDENT | DEDENT | NAME | Text;


//hidden terminal NEWLINE: 'this_string_does_not_matter_newline#$%^&*((';

terminal NEWLINE: /[\r\n]+/;
terminal DEDENT: 'this_string_does_not_matter_dedent#$%^&*((';
terminal INDENT: 'this_string_does_not_matter_indent#$%^&*((';
//hidden terminal SPACES: ' ';
hidden terminal SPACES:/\s+/;

THE returns string: 'THE'|'The'|'the';
SHALL returns string: 'SHALL'|'Shall'|'shall';
terminal COMMA: /,/;
terminal DOT: /\./;

terminal TRIPLE_QUOTE: /"""/;
terminal COLON: /:/;
terminal QUOTE: /'/;


hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

terminal NAME: /[_a-zA-Z][\w_]*/;
Keyword returns string: THE | SHALL;
Text returns string: (Keyword | NAME)+;

I have implemented a Lexer to parse Indent and Dedent, which works just fine. Now i want to recognize sentences like
The bird is here. For this I have defined

• NAME as terminal to parse general words
• terminals like THE or SHALL should be included
• Text is the string i want to return
  if i start it in VS Code i get for this error from the language server:

Request initialize failed with message: Errors detected in definition of Lexer:
Token Type: ->NAME<- static 'GROUP' can only be Lexer.SKIPPED/Lexer.NA/A String

I really dont know what the issue is here. What be great if some could give me some hints

Thank you,
Michael

[msujew]

Hey @mnaderhirn,

I have implemented a Lexer to parse Indent and Dedent, which works just fine. Now i want to recognize sentences like

So you're saying you created a specialized TokenBuilder service? Can you share that? I have reason to believe a minor implementation detail might lead to the error.

[mnaderhirn]

Looks better now, thank you so far:
Now typing: The bird is here in VS Code results to

[{
	"resource": "/home/michael/projects/test_project/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "Expecting: one of these possible Token sequences:\n  1. [NAME]\n  2. [TITLE:]\n  3. [[RELEASE_DOCUMENT]]\nbut found: 'The'",
	"source": "easysafe",
	"startLineNumber": 1,
	"startColumn": 1,
	"endLineNumber": 1,
	"endColumn": 4
},{
	"resource": "/home/michael/projects/test_project/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "Expecting end of file but found `The`.",
	"source": "easysafe",
	"startLineNumber": 1,
	"startColumn": 1,
	"endLineNumber": 1,
	"endColumn": 4
},{
	"resource": "/home/michael/projects/test_project/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "unexpected character: -> <- at offset: 3, skipped 1 characters.",
	"source": "easysafe",
	"startLineNumber": 1,
	"startColumn": 4,
	"endLineNumber": 1,
	"endColumn": 5
},{
	"resource": "/home/michael/projects/test_project/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "unexpected character: -> <- at offset: 8, skipped 1 characters.",
	"source": "easysafe",
	"startLineNumber": 1,
	"startColumn": 9,
	"endLineNumber": 1,
	"endColumn": 10
},{
	"resource": "/home/michael/projects/test_project/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "unexpected character: -> <- at offset: 11, skipped 1 characters.",
	"source": "easysafe",
	"startLineNumber": 1,
	"startColumn": 12,
	"endLineNumber": 1,
	"endColumn": 13
}]

[msujew]

1. Why are you removing SPACES from the token list in your token builder? It should still be lexed, but hidden. Your code simply removes the token type, which prevents the lexer from reading whitespaces.
2. How does your grammar look like? Do you allow parsing of The as the first token?

[mnaderhirn]

@msujew -- Thank you for being so helpful
made the spaces not hidden as you can see in my grammar and that resolved the space erros, but now i am getting this errors for The
The bird is here --> see errors below the grammar. The is a keyword and should be tokenized since i use it for a formal language

grammar CommonTokensGrammar

import 'release.tokens'

entry CommonTokens:
    token = INDENT | DEDENT | NAME | Text;


//hidden terminal NEWLINE: 'this_string_does_not_matter_newline#$%^&*((';

terminal NEWLINE: /[\r\n]+/;
terminal DEDENT: 'this_string_does_not_matter_dedent#$%^&*((';
terminal INDENT: 'this_string_does_not_matter_indent#$%^&*((';
//hidden terminal SPACES: ' ';
terminal SPACES:/\s+/;

THE returns string: 'THE'|'The'|'the';
SHALL returns string: 'SHALL'|'Shall'|'shall';
terminal COMMA: /,/;
terminal DOT: /\./;

terminal TRIPLE_QUOTE: /"""/;
terminal COLON: /:/;
terminal QUOTE: /'/;


hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

terminal NAME: /[_a-zA-Z][\w_]*/;
Keyword returns string: THE | SHALL;
Textpart returns string: (Keyword | NAME);
Text: text+=Textpart SPACES* (text+=Textpart)*;

[{
	"resource": "/home/michael/projects/test_project/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "Expecting: one of these possible Token sequences:\n  1. [NAME]\n  2. [TITLE:]\n  3. [[RELEASE_DOCUMENT]]\nbut found: 'The'",
	"source": "easysafe",
	"startLineNumber": 1,
	"startColumn": 1,
	"endLineNumber": 1,
	"endColumn": 4
},{
	"resource": "/home/michael/projects/test_project/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "Expecting end of file but found `The`.",
	"source": "easysafe",
	"startLineNumber": 1,
	"startColumn": 1,
	"endLineNumber": 1,
	"endColumn": 4
}]

[msujew]

I've adjusted your grammar a bit - the Text rule didn't allow any spaces after the second word. See this playground grammar.

[mnaderhirn]

Yep now it is working. Thank you!
Cool project!

[msujew]

Thanks, you're welcome :)

[mnaderhirn]

Request initialize failed with message: Errors detected in definition of Lexer:
Token Type: ->NAME<- static 'GROUP' can only be Lexer.SKIPPED/Lexer.NA/A String

What excatly is causing this errors?

[msujew]

@mnaderhirn See this code in Chevrotain. The GROUP field of a token type cannot be set to undefined.

[mnaderhirn]

Thank you:

grammar DateGrammar

entry Datum:
'Date:' year=YEAR ('-'|'.') month=MONTH ('-'|'.') day=DAY;

terminal YEAR: /20[0-9]{2}/;
terminal MONTH: /1[0-2]|[1-9]/;
terminal DAY: /3[01]|[12][0-9]|[1-9]/;

see playground

Maybe a very primitive question:
How do i have to organize this, so that i can differentiate betweent Month and Day?

[msujew]

Hey @mnaderhirn,

Short answer: You can't, since the tokenization phase cannot take context into account.

Long answer: I would recommend using just a single INTEGER terminal and then using a validation to ensure that the values of year, month and day fall within the expected ranges. Not only does this solve the parsing issue, but also improve the comprehension of the error messages; While trying to solve this using tokenization leads to some pretty obscure and hard to read error messages in the editor, using validations allows to build specific error messages for your use case. The general design philosophy for building DSLs is usually: Be flexible in what you accept and create specific and useful error/warning messages for invalid inputs instead of rejecting them via the parser.

[mnaderhirn]

Understood, Thank you

[mnaderhirn]

Example

This grammar seems to work perfectly in the playground, but when i run a test case
it('should parse the version correctly', async () => {
const input = 'VERSION: v12.3.6';
const result = await parse(input);

    expect(result.parseResult).toBeDefined();
    const titleNode = result.parseResult.value;
    expect(titleNode).toBeDefined();
    expect(titleNode.major).toEqual(12);
    expect(titleNode.minor).toEqual(3);
    expect(titleNode.patch).toEqual(6);
});

, I ll get:
v12 to be of token type v12 which makes sense. how should i adopt the grammar in order to ensure it is working right?
E. g. allowing 'v' to be a NAME token and then writing a validation function which returns an error?

[msujew]

I can't really follow. What is the error? If you want to use v as a name, you can follow this guide.

Also, please consider opening separate discussions for separate questions. Makes my job a bit easier and allows for better indexing by search engines.