Scala tokenizer depends on the outdated library Scalariform

[markkuriekkinen]

The Scalariform library used in the Scala tokenizer has not been maintained since 2019.

In the commit 9e035e0, I updated the Scalariform library used to parse and tokenize the student's submitted Scala code. However, Scalariform has not been maintained since June 2019. I am not sure if the latest version fully supports parsing Scala 2.13 code. At least, Scalariform source code contains constants for Scala versions and the highest is Scala 2.11:
https://github.com/scala-ide/scalariform/blob/0.2.10/scalariform/src/main/scala/scalariform/ScalaVersions.scala#L38

Scalariform docs say that v0.2.10 supports Scala 2.13, but that might refer to compiling and running the library on Scala 2.13. Parsing input code is a different matter.

Scala 3 changes the syntax of the language. In order to tokenize Scala 3, we most definitely need a new solution since Scalariform will never support Scala 3.

Related to #30.

In Radar code, the Scala tokenizer is here:

• radar/tokenizer/scala.py

  Line 28 in 9e035e0

  parsed = run(("scala", "-cp", "tokenizer/scalariform/scalariform.jar", "ScalariformTokens"), source)

• https://github.com/apluslms/radar/tree/9e035e0f5a6597f56b4414b0df53fb227ec0cb15/tokenizer/scalariform