Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For Contexts, rely on token constants + Add MySQL 8.4 context #569

Closed
wants to merge 11 commits into from
Closed

For Contexts, rely on token constants + Add MySQL 8.4 context #569

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
593c240
Define context keywords relying on Token constants
Tithugues Aug 2, 2024
b64920a
Fix typos in README of ContextGenerator.
niconoe- Aug 6, 2024
de7500d
Generate context files with Token const instead of hard-coded numbers.
niconoe- Aug 6, 2024
3e8757e
Regenerate all contexts with Tokens instead of numeric values.
niconoe- Aug 6, 2024
3000982
Fix OFFSET reserved keyword from MariaDB 10.3 and add new missing res…
niconoe- Aug 6, 2024
0edf645
Add MySQL 8.4 context.
niconoe- Aug 6, 2024
7674df3
Fix linting, phpstan and psalm issues.
niconoe- Aug 6, 2024
14c4862
Fix unit test and fix psalm + phpstan baselines.
niconoe- Aug 6, 2024
2b95c81
Fix Psalm baseline.
niconoe- Aug 6, 2024
e535359
Fix Psalm, Phpstan and unit tests.
niconoe- Aug 6, 2024
fe27ccf
Enhance unit test to kill mutant.
niconoe- Aug 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions psalm-baseline.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,11 @@
<code><![CDATA[ContextMySql80300]]></code>
</UnusedClass>
</file>
<file src="src/Contexts/ContextMySql80400.php">
<UnusedClass>
<code><![CDATA[ContextMySql80400]]></code>
</UnusedClass>
</file>
<file src="src/Lexer.php">
<LoopInvalidation>
<code><![CDATA[$this->last]]></code>
Expand Down
1,157 changes: 881 additions & 276 deletions src/Contexts/ContextMariaDb100000.php
View file
Open in desktop

Large diffs are not rendered by default.

1,381 changes: 1,057 additions & 324 deletions src/Contexts/ContextMariaDb100100.php
View file
Open in desktop

Large diffs are not rendered by default.

1,384 changes: 1,059 additions & 325 deletions src/Contexts/ContextMariaDb100200.php
View file
Open in desktop

Large diffs are not rendered by default.

1,388 changes: 1,063 additions & 325 deletions src/Contexts/ContextMariaDb100300.php
View file
Open in desktop

Large diffs are not rendered by default.

1,389 changes: 1,064 additions & 325 deletions src/Contexts/ContextMariaDb100400.php
View file
Open in desktop

Large diffs are not rendered by default.

1,389 changes: 1,064 additions & 325 deletions src/Contexts/ContextMariaDb100500.php
View file
Open in desktop

Large diffs are not rendered by default.

1,389 changes: 1,064 additions & 325 deletions src/Contexts/ContextMariaDb100600.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb100700.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb100800.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb100900.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb101000.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb101100.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb110000.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb110100.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb110200.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb110300.php
View file
Open in desktop

Large diffs are not rendered by default.

1,390 changes: 1,065 additions & 325 deletions src/Contexts/ContextMariaDb110400.php
View file
Open in desktop

Large diffs are not rendered by default.

1,034 changes: 787 additions & 247 deletions src/Contexts/ContextMySql50000.php
View file
Open in desktop

Large diffs are not rendered by default.

1,139 changes: 869 additions & 270 deletions src/Contexts/ContextMySql50100.php
View file
Open in desktop

Large diffs are not rendered by default.

1,155 changes: 880 additions & 275 deletions src/Contexts/ContextMySql50500.php
View file
Open in desktop

Large diffs are not rendered by default.

1,284 changes: 983 additions & 301 deletions src/Contexts/ContextMySql50600.php
View file
Open in desktop

Large diffs are not rendered by default.

1,381 changes: 1,057 additions & 324 deletions src/Contexts/ContextMySql50700.php
View file
Open in desktop

Large diffs are not rendered by default.

1,389 changes: 1,064 additions & 325 deletions src/Contexts/ContextMySql80000.php
View file
Open in desktop

Large diffs are not rendered by default.

1,389 changes: 1,064 additions & 325 deletions src/Contexts/ContextMySql80100.php
View file
Open in desktop

Large diffs are not rendered by default.

1,389 changes: 1,064 additions & 325 deletions src/Contexts/ContextMySql80200.php
View file
Open in desktop

Large diffs are not rendered by default.

1,389 changes: 1,064 additions & 325 deletions src/Contexts/ContextMySql80300.php
View file
Open in desktop

Large diffs are not rendered by default.

1,084 changes: 1,084 additions & 0 deletions src/Contexts/ContextMySql80400.php
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/Token.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ class Token
// Flags that describe the tokens in more detail.
// All keywords must have flag 1 so `Context::isKeyword` method doesn't
// require strict comparison.
public const FLAG_KEYWORD = 1;
public const FLAG_KEYWORD_RESERVED = 2;
public const FLAG_KEYWORD_COMPOSED = 4;
public const FLAG_KEYWORD_DATA_TYPE = 8;
Expand Down
132 changes: 58 additions & 74 deletions src/Tools/ContextGenerator.php
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
<?php

Check failure on line 1 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2)

UnusedBaselineEntry 

src/Tools/ContextGenerator.php:0:0: UnusedBaselineEntry: Baseline for issue "RedundantCondition" has 2 extra entries. (see https://psalm.dev/316)

Check failure on line 1 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2) 

Ignored error pattern #^Parameter \#2 \.\.\.\$arrays of function array_merge expects array, array\<int, string\>\|false given\.$# in path /home/runner/work/sql-parser/sql-parser/src/Tools/ContextGenerator.php was not matched in reported errors.

declare(strict_types=1);

namespace PhpMyAdmin\SqlParser\Tools;

use PhpMyAdmin\SqlParser\Token;

use function array_map;
use function array_merge;
use function array_slice;
Expand All @@ -15,12 +17,10 @@
use function implode;
use function ksort;
use function preg_match;
use function round;
use function scandir;
use function sort;
use function sprintf;
use function str_contains;
use function str_repeat;
use function str_replace;
use function str_split;
use function strlen;
Expand All @@ -43,10 +43,10 @@
* @var array<string, int>
*/
public static array $labelsFlags = [
'(R)' => 2, // reserved
'(D)' => 8, // data type
'(K)' => 16, // keyword
'(F)' => 32, // function name
'(R)' => Token::FLAG_KEYWORD_RESERVED,
'(D)' => Token::FLAG_KEYWORD_DATA_TYPE,
'(K)' => Token::FLAG_KEYWORD_KEY,
'(F)' => Token::FLAG_KEYWORD_FUNCTION,
];

/**
Expand All @@ -64,6 +64,7 @@
'MySql80100' => 'https://dev.mysql.com/doc/refman/8.1/en/keywords.html',
'MySql80200' => 'https://dev.mysql.com/doc/refman/8.2/en/keywords.html',
'MySql80300' => 'https://dev.mysql.com/doc/refman/8.3/en/keywords.html',
'MySql80400' => 'https://dev.mysql.com/doc/refman/8.4/en/keywords.html',
'MariaDb100000' => 'https://mariadb.com/kb/en/reserved-words/',
'MariaDb100100' => 'https://mariadb.com/kb/en/reserved-words/',
'MariaDb100200' => 'https://mariadb.com/kb/en/reserved-words/',
Expand All @@ -83,6 +84,20 @@
'MariaDb110400' => 'https://mariadb.com/kb/en/reserved-words/',
];

/**
* Reversed const <=> int from {@see Token} class to write the constant name instead of its value.
*
* @var array<int, string>
*/
private static array $typesNumToConst = [
1 => 'Token::FLAG_KEYWORD',
2 => 'Token::FLAG_KEYWORD_RESERVED',
4 => 'Token::FLAG_KEYWORD_COMPOSED',
8 => 'Token::FLAG_KEYWORD_DATA_TYPE',
16 => 'Token::FLAG_KEYWORD_KEY',
32 => 'Token::FLAG_KEYWORD_FUNCTION',
];

/**
* The template of a context.
*
Expand Down Expand Up @@ -117,9 +132,7 @@
*
* The value associated to each keyword represents its flags.
*
* @see Token::FLAG_KEYWORD_RESERVED Token::FLAG_KEYWORD_COMPOSED
* Token::FLAG_KEYWORD_DATA_TYPE Token::FLAG_KEYWORD_KEY
* Token::FLAG_KEYWORD_FUNCTION
* @see Token
*
* @var array<string,int>
* @psalm-var non-empty-array<string,Token::FLAG_KEYWORD_*|int>
Expand All @@ -141,12 +154,9 @@
public static function sortWords(array &$arr): array
{
ksort($arr);
foreach ($arr as &$wordsByLen) {
ksort($wordsByLen);
foreach ($wordsByLen as &$words) {
sort($words, SORT_STRING);
}
}
foreach ($arr as &$words) {
sort($words, SORT_STRING);
} unset($words);

return $arr;
}
Expand All @@ -160,17 +170,21 @@
*/
public static function readWords(array $files): array
{
$words = [];
foreach ($files as $file) {
$words = array_merge($words, file($file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES));
}
$wordsByFile = array_map(
static fn(string $file): array => file($file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES),

Check failure on line 174 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2) 

Anonymous function should return array<int, string> but returns array<int, string>|false.
$files,
);
$words = array_merge(...$wordsByFile);

Check failure on line 177 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2)

NamedArgumentNotAllowed 

src/Tools/ContextGenerator.php:177:33: NamedArgumentNotAllowed: Method array_merge called with named unpacked array array<array-key, array<array-key, mixed>> (array with string keys) (see https://psalm.dev/268)

/** @var array<string, int> $types */
$types = [];

for ($i = 0, $count = count($words); $i !== $count; ++$i) {
$type = 1;
$value = trim($words[$i]);

Check failure on line 183 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2)

MixedArgument 

src/Tools/ContextGenerator.php:183:27: MixedArgument: Argument 1 of trim cannot be mixed, expecting string (see https://psalm.dev/030)
if ($value === '') {
continue;
}
$type = Token::FLAG_KEYWORD;

// Reserved, data types, keys, functions, etc. keywords.
foreach (static::$labelsFlags as $label => $flags) {
Expand All @@ -184,13 +198,8 @@

// Composed keyword.
if (str_contains($value, ' ')) {
$type |= 2; // Reserved keyword.
$type |= 4; // Composed keyword.
}

$len = strlen($words[$i]);
if ($len === 0) {
continue;
$type |= Token::FLAG_KEYWORD_RESERVED;
$type |= Token::FLAG_KEYWORD_COMPOSED;
}

$value = strtoupper($value);
Expand All @@ -201,71 +210,46 @@
}
}

// Prepare an array in a way to sort by type, then by word.
$ret = [];
foreach ($types as $word => $type) {
$len = strlen($word);
if (! isset($ret[$type])) {
$ret[$type] = [];
}

if (! isset($ret[$type][$len])) {
$ret[$type][$len] = [];
}

$ret[$type][$len][] = $word;
$ret[$type][] = $word;
}

return static::sortWords($ret);

Check failure on line 219 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2)

InvalidArgument 

src/Tools/ContextGenerator.php:219:34: InvalidArgument: Argument 1 of PhpMyAdmin\SqlParser\Tools\ContextGenerator::sortWords expects array<int, array<int, array<int, string>>>, but array<int, non-empty-list<string>> provided (see https://psalm.dev/004)
}

/**
* Prints an array of a words in PHP format.
*
* @param array<int, array<int, array<int, string>>> $words the list of words to be formatted
* @param int $spaces the number of spaces that starts every line
* @param int $line the length of a line
* @param array<int, list<string>> $words the list of words to be formatted
*/
public static function printWords(array $words, int $spaces = 8, int $line = 140): string
public static function printWords(array $words): string
{
$typesCount = count($words);
$ret = '';
$j = 0;

foreach ($words as $type => $wordsByType) {
foreach ($wordsByType as $len => $wordsByLen) {
$count = round(($line - $spaces) / ($len + 9)); // strlen("'' => 1, ") = 9
$i = 0;

foreach ($wordsByLen as $word) {
if ($i === 0) {
$ret .= str_repeat(' ', $spaces);
}

$ret .= sprintf('\'%s\' => %s, ', $word, $type);
if (++$i !== $count && ++$i <= $count) {
continue;
}

$ret .= "\n";
$i = 0;
}

if ($i === 0) {
continue;
}

$ret .= "\n";
foreach ($wordsByType as $word) {
$ret .= sprintf(" '%s' => %s,\n", $word, self::numTypeToConst($type));
}
}
return $ret;
}

if (++$j >= $typesCount) {
continue;
/**
* Convert a numeric value representing a set of const to a textual const value.
*
* @param int $type The numeric value.
* @return string The text to write considering the given numeric value.
*/
private static function numTypeToConst(int $type): string
{
$matchingFlags = [];
foreach (self::$typesNumToConst as $num => $value) {
if ($type & $num) {
$matchingFlags[] = $value;
}

$ret .= "\n";
}

// Trim trailing spaces and return.
return str_replace(" \n", "\n", $ret);
return implode(' | ', $matchingFlags);
}

/**
Expand All @@ -282,7 +266,7 @@
public static function generate(array $options): string
{
if (isset($options['keywords'])) {
$options['keywords'] = static::printWords($options['keywords']);

Check failure on line 269 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2)

ArgumentTypeCoercion 

src/Tools/ContextGenerator.php:269:55: ArgumentTypeCoercion: Argument 1 of PhpMyAdmin\SqlParser\Tools\ContextGenerator::printWords expects array<int, list<string>>, but parent type array<int, array<int, array<int, string>>> provided (see https://psalm.dev/193)

Check failure on line 269 in src/Tools/ContextGenerator.php

View workflow job for this annotation

GitHub Actions / analyse-php (8.2) 

Parameter #1 $words of static method PhpMyAdmin\SqlParser\Tools\ContextGenerator::printWords() expects array<int, array<int, string>>, array<int, array<int, array<int, string>>> given.
}

return sprintf(self::TEMPLATE, $options['name'], $options['class'], $options['link'], $options['keywords']);
Expand Down
2 changes: 1 addition & 1 deletion tools/contexts/MariaDb100600.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb100700.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb100800.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb100900.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb101000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb101100.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb110000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb110100.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
3 changes: 2 additions & 1 deletion tools/contexts/MariaDb110200.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ NULL (R)
NUMBER
NUMERIC (R)
NVARCHAR
OFFSET
OFFSET (R)
OLD_PASSWORD
ON (R)
ON COMPLETION NOT PRESERVE
Expand Down Expand Up @@ -466,6 +466,7 @@ ROUTINE
ROW
ROW_COUNT
ROW_FORMAT
ROW_NUMBER (R)
ROWS (R)
RTREE
SAVEPOINT
Expand Down
Loading
Loading