Remaining tasks from big-refactor

[slightknack]

Remaining tasks copied from #52.

This PR still has a long way to go, but is getting closer to completion. Here's what's left:

Phase 1

• Refactored the representation of syntax trees in the compiler, to reduce boilerplate
• Rewrote lexer to be token-based macro friendly
• Refactored parser to work with new lexer
  • Refactor the way spans are represented for richer errors
  • Take advantage of new error format in parser
• Reintroduce rest of compilation pipeline, ensure it builds
  • Lex
  • Read
  • Parse
  • Desugar (type definition mismatch oversight, working on it now)
  • Hoist
  • Gen
• Write tests for new pipeline
  • Lex
  • Read
  • Parse (only unit tests so far, need proptests)
  • Desugar
  • Hoist
  • Gen
• Test pipeline E2E
• Get snippet tests all working
• Clean up and document new code
• Remove overly generalized compilation pipeline
• Clean up library exports
  • Figure out what really needs to be in passerine_common.
• System Injection
  • Figure out Rust side of things
  • Proof of concept (in private pnx repo)
  • Writeup (WIP)
  • Remove magic (finally!)
  • Conversion for builtin types
  • Conversion proc derive macro
  • Add effect (eventually to be replaced with type)
  • Switch existing system code over to effects
• Look at mlua/hlua, rhai, and pen to determine what the high-level API to using Passerine from Rust should look like.
  • Calling Rust from Passerine
  • Calling Passerine from Rust
  • Passing data to and from Passerine/Rust
  • Sandboxing side effects
  • Inspecting and modifying the results of the top level (requires modules)
  • Binding handlers through system injection (implemented in pnx, needs to be transferred)

Phase 2

• Introduce a token-based macro system
  • In between lexing and parsing
  • That is hygenic (TODO: how to define hygine)
• Introduce prelude
  • standard library and core
  • reintroduce syntax-based macros in terms of token-based macros

Phase 3

• Finish implementing type ... and record
• Introduce rough modules
• Match expressions using macros and fibers
• Polish implementation write some projects to test for bugs
• Polish standard library and prelude a tad

Phase 4 (Maintenance stuff, post merge)

• Polish high-level API, write documentation and examples
• Go over README and correct anything that is inaccurate
• Move README Overview to the codex and link to it
• Update the code example in the repo and on the website
• Redo the website, add a 'try it' wasm-based playground