Suggestion: Meta-Interpreter

[r-stein]

Meta-Interpreter

summary

The meta-interpreter introduces a new way to call the expand-functions. Instead of directly calling the functions and processing the result, it will interpret a json-compatible description of the behavior. The advantages of this are:

• easier to write and add new languages / better readable language descriptions
• more error resistant description of new languages
• even the users can pass this descriptions as arguments and hence easily change existing languages or add own languages

motivation

In the current code base

• all expand functions are duck-typed

• and reused in the same way:

  result = expand(string, start, end)
  if result:
    return result

  Hence the calls could be done in a more generic way. In this pull request this is done with a meta-interpreter, which interprets a json-compatible python object. Hence it is possible to define languages easier and even on the user level possible.

meta-interpreter

expand-function: In the following the term "expand-function" is used for a function, which is currently used for every expansion. The signature is:

expand: (string: str, start: int, end: int) -> dict("start"->int, "end"->int, "string"->str, "type"->str)

This maps a selection limited by start and end in a string to a new selection or None.

The interpreter interprets a macro and returns the resulting object to expand. To do this there are two ways to register names in the interpreter:

1. register_command(name, function): registers a expand-function and binds it to a given name. This function will be called, when the name is interpreted.
2. create_macro(macro_name, macro): registers a macro (See Macro in syntax and semantics) and binds it to a given name. The macro will be interpreted if the name is interpreted. (This may create infinite recursions)

The user will be apple to call macros by names or directly write macros in the function call.

Developers can import the interpreter and register their functions or macros.
The interpreter must imported via a proxy and not directly, i.e. from ._minterp import interpreter instead of from .minterp import interpreter. This will import the interpreter module as a singleton.

syntax and semantics

This is an (informal) syntax definition with the corresponding semantical description.

• str is a python or JSON string ""
• List of is a python or JSON list [...]
• Dict is a python dict or a JSON object {"key": "value", ...}
• expand-function is a pointer to a expand-function (only python)

# the unit, which is interpreted
# the list of command will be searched until
# the first command with a result
Macro ::= List of Command | Command

# the possible ways to expand a selection to a result
# str: must be a valid name of a registered function
# expand-function: function pointer to a valid expand-function
#     (Not JSON compatible)
Command ::= str | expand-function | EnrichedCommand | ClosestCommand

# selects the closest command to the current selection
ClosestCommand ::= List of Command

# enriches the simple command with additional options
EnrichedCommand ::= Dict(
    # the executed command/macro
    command ::= Macro
    # (optional) restricts the string on which the command is executed
    [scope ::= Command]
    # (optional) will be forwarded to the expand function
    [args ::= Dict(str->object)]
    )

How to use:

• ClosestCommand: Just add a list of expansions and the closest one will be selected. If it is not definite, then the first one in the list will be selected, e.g. for parens and quotes in the text (Hello "W|or)ld".
• EnrichedCommand: Adds modifier to commands. Later additional useful modifiers might be added. Current modifiers:
  • scope: Limits the command on this scope, hence the string will limited to the result of the scope command.
  • args: Will be forwarded to the function. Therefore those args must arguments of the function, but the functions should define default arguments. A use-case could be the expand-regex function, which could accept a regex. E.g. {"command":"regex", "args": {"regex":"[\w']"}} could be used to expand to word characters and ' for match words like that's.

example

This creates the javascript language definition:

from . import interpreter
interpreter.create_macro("javascript", [
    "subword",
    "word",
    # search inside the quotes for parens/brackets
    {
        "scope": "quotes",
        "command": "symbol"
    },
    # search for the closest of line/quotes/symbol
    [
        "line",
        "quotes",
        "semantic_unit",
        "symbol"
    ]
])

impacts

I think it is a very easy and expandable way to define and add new language. Hence I would assume a lot of new languages and better maintainability of existing ones. In addition some user requests wishes for a better flexibility, which is provided by this pull request.
Since this is a major change I would change the Version number to v2.0.0.

[r-stein]

Since I anyway touch most of the files, I autoformated the indentation as suggested in pep8

[aronwoost]

For the next time: please do indention (and other "clean code" stuff) in a separate commit. This makes code review easier.

Also I found some commented-out code and print statements. Please remove them unless you have a very good reason.

Besides that: awesome idea! ⭐ I'm currently trying to understand the code/changes. 😉

[r-stein]

I agree on this and undid the indentation changes.

In general the code changes are

• the interpreter
• at the end of each file interpreter.register_command("whatever", expand_to_whatever)
• the language definitions
• the result in expand_region_handler.py is retrieved via result = interpreter.interp(language, string, start, end) instead of for example result = javascript.expand(string, start, end)

Most other changes could be (and maybe "should be" and "will be") done in an other PR.

For example I added an argument to symbols. With this one could run the command:

view.run_command("expand_region", { "language": {"command": "symbol", "args": {"symbols": "<>()"}}})

And it would expand to parenthesis () and angle brackets <>.

[aronwoost]

Thanks again for the work and sorry for the lack of feedback. I'll try to dive into the code again within the next days.

[r-stein]

What do you think about stoping ST2 support, because ST3 is out of the beta and it requires additional work in development and especially in testing?

[aronwoost]

@r-stein YES, excellent idea! It's time.

Does it make sense to have a separate PR (and release) only for removing ST2 support? With that we have a clear cut and it's easier to follow the code changes.

Also we should investigate how to provide the old version for ST2 users. I don't think we need to release a legacy version to Package Control (like you said; to much effort) but maybe we can provide instructions how to install via git.

[r-stein]

I would increase the major relase number with this PR (because it changes a lot of the API) and we could just stop ST2 support in that step. For PackageControl we can just edit the corresponding JSON file and the ST2 users won't get more updates, but keep the version they have (I can do that).

Install instructions would be easy:

git clone https://github.com/aronwoost/sublime-expand-region ExpandRegion
cd ExpandRegion
git checkout tags/v1.4.0

[aronwoost]

Cool! 🚀

[r-stein]

I think I may just create an beta-prerelease for this after releasing the bracket highlighter integration and make additional refactoring etc. in new PRs.