Planned extensions for Brent's master's thesis

[pdawyndt]

DSL extensions

• i18n support
  • identify DSL-attributes that can be translated
    • these attributes may use a !natural_language map for translations
      • mandatory if they can have other types of map-values (e.g. return)
      • optional if they cannot have other types of map-values (e.g. string attributes such as unit)
    • string attributes among those may also serve as template strings with placeholder variables
    • identified attributes:
      • unit or tab
      • statement
      • expression
      • description
      • return
      • arguments
      • stdin
      • stdout
      • stderr
      • files
      • file
  • support translated values (literal translations) (Support literal translations (map type !natural_language) #570)
    • make distinction between different map-values for return attribute
      • map as actual type of return value (already supported by TESTed)
      • return value with custom oracle: use !oracle tag (already supported by TESTed)
      • return value with map of natural language translations: use !natural_language tag
    • make distinction between different map-values for statement and expression
      • map of programming language translations: use !programming_language tag
      • map of natural language translations: use !natural_language tag
    • other attributes do not support maps (yet), so !natural_language tag may be optional here for a map containing natural language translations
  • support translation tables (inherited with merge strategy)
  • process strings as template strings with placeholder variables if translation table and/or data available (only named placeholders or also positional?)
• parameterized testing (data-driven testing)
  • testcase templates (simplified DSL)
    • default template
    • named templates
  • repeated execution with
    • oracle for individual execution
    • aggragation and stop criterium for executions
    • oracle across all executions (having access to aggregated results)
  • dynamic generation of test data
• consistent description of "file content" (stdin, stdout, stderr, input/output files)
  • inline content in DSL (short content)
  • link external file in DSL (long content)
  • heuristic for inlining / linking file content in feedback
• support testing of multiple generated files (now TESTed only supports testing of a single file per testcase)
• support additional file descriptors (now only stdin, stdout and stderr are supported)
• setup and teardown
  • language-agnostic setup and teardown (shell scripts)
  • specific support for environment variables (needed?)
• support for testing global scope (built-in + custom oracle)

generic language extensions

• line comments (Feat/add comments #555)
• indexing sequences
• indexing maps
• operators
• string conversion
• destructuring
• object identity checking
• object equivalence checking

new features

• TypeScript language module (Add TypeScript #553)
• pretty printing of nested data structures (in generated feedback)
• compute test coverage for test plan and sample solution
• static analysis
  • extract types from type hints (simplified DSL)
  • check explicitly if all required interfaces have been implemented (better feedback)
    • skip tests relying on interfaces that aren't implemented (faster feedback)
  • check implementation restrictions (e.g. recursive function implementation)
• support hidden units/test cases (visible to teachers but invisible to students; to avoid gaming — also known as programming to the test; Peveler et al., 2019)
• support for doctest: +REPR; i.e. implicit string conversion of return value as in REPL environment; could be covered by translation into explicit string conversion during conversion to TESTed or specific TESTed support (e.g. !repl next to !expression as a type for the return value in TESTed)
• two-fold execution of nested blocks (ACM/IEEE testing strategy explored in Python judge; low priority)