Merge pull request #12 from vrtbl/fledgling

integration testing snippets: Fledgling

CONTRIBUTING.md

@@ -4,7 +4,7 @@ If you're reading this, thanks for taking a look! If you would like to contribut
 ## Code of Conduct
 If you find any bugs or would like to request any features, please open an issue. Once the issue is open, feel free to put in a pull request to fix the issue. Small corrections, like fixing spelling errors, or updating broken urls are appreciated. All pull requests must have adequate testing and review to assure proper function before being accepted.
 
-We encourage an open, friendly, and supportive environment around the development of Passerine. If you disagree with someone for any reason, discuss the issue and express you opinions, don't attack the person. Discrimination of any kind against any person is not permitted. If you detract from this project's collaborative environment, you'll be prevented from participating in the future development of this project until you prove you can behave yourself adequately. Please provide arguments based on anecdotes and reasoning to support your suggestions - don't rely on arguments based on 'years of experience,' supposed skill, job title, etc. to get your points across.
+We encourage an open, friendly, and supportive environment around the development of Passerine. If you disagree with someone for any reason, discuss the issue and express you opinions, don't attack the person. Discrimination of any kind against any person is not permitted. If you detract from this project's collaborative environment, you'll be prevented from participating in the future development of this project until you prove you can behave yourself adequately. Please use sound reasoning to support your suggestions - don't rely on arguments based on 'years of experience,' supposed skill, job title, etc. to get your points across.
 
 # General Guidelines
 Readable code with clear behavior works better than illegible optimized code. For things that are very performance-oriented, annotations describing what, how, and why are essential.
@@ -30,4 +30,41 @@ Passerine strives to implement a modern compiler pipeline. Passerine is currentl
 - The command line interface and the package repository, [Aspen](https://github.com/vrtbl/aspen).
 
 > TODO: write about project structure
-> TODO: integration tests
+
+# Integration Tests
+If you notice any unsound behavior, like an internal compile error or an incorrect,
+
+1. Reduce the behavior to the minimum required amount of passerine code that causes that error
+2. Open an issue explaining what led to the error, what you expected to happen, and the minimum reproducible example.
+3. Optionally, add the snippet to `tests/snippets` and test that the test fails (by running `cargo test snippets`).
+
+## What is a test snippet?
+A test snippet is some Passerine code that tests a specific outcome. Here's a simple test snippet:
+
+```passerine
+-- action: run
+-- outcome: success
+-- expect: "Banana"
+
+print "Hello, World!"
+
+"Banana"
+```
+
+A test snippet starts with a series of comments, each one forming a key-value pair. The `action` is what the compiler should do with the snippet:
+
+- `lex`
+- `parse`
+- `desugar`
+- `compile`
+- `run`
+
+The outcome specifies the spefic result:
+
+- No errors are raised: `success`
+- A syntax error is raised: `syntax`
+- A runtime error is raised: `trace`
+
+Optionally, if the action is `run` an `outcome` may be specified. This treats the snippet like a function body, and compares the returned value with the expected value.
+
+Whenever you add a feature, add snippet tests that demonstrate how this feature should (and should not) work.

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "passerine"
-version = "0.8.0"
+version = "0.8.1"
 authors = [
     "Isaac Clayton (slightknack) <[email protected]>",
     "The Passerine Community",

src/compiler/gen.rs

@@ -139,26 +139,27 @@ impl Compiler {
     /// Tries to resolve a variable in enclosing scopes
     /// if resolution it successful, it captures the variable in the original scope
     /// then builds a chain of upvalues to hoist that upvalue where it's needed.
-    pub fn captured(&mut self, name: &str) -> Option<(Captured, bool)> {
+    pub fn captured(&mut self, name: &str) -> Option<Captured> {
         if let Some(index) = self.local(name) {
             let already = self.captures.contains(&index);
             if !already {
                 self.captures.push(index);
                 self.lambda.emit(Opcode::Capture);
                 self.lambda.emit_bytes(&mut split_number(index));
             }
-            return Some((Captured::Local(index), already));
+            return Some(Captured::Local(index));
         }
 
         if let Some(enclosing) = self.enclosing.as_mut() {
-            if let Some((captured, already)) = enclosing.captured(name) {
-                let upvalue = if !already {
+            if let Some(captured) = enclosing.captured(name) {
+                let included = self.lambda.captures.contains(&captured);
+                let upvalue = if !included {
                     self.lambda.captures.push(captured);
                     self.lambda.captures.len() - 1
                 } else {
                     self.lambda.captures.iter().position(|c| c == &captured).unwrap()
                 };
-                return Some((Captured::Nonlocal(upvalue), already));
+                return Some(Captured::Nonlocal(upvalue));
             }
         }
 
@@ -168,7 +169,7 @@ impl Compiler {
     /// returns the index of a captured non-local.
     pub fn captured_upvalue(&mut self, name: &str) -> Option<usize> {
         match self.captured(name) {
-            Some((Captured::Nonlocal(upvalue), _)) => Some(upvalue),
+            Some(Captured::Nonlocal(upvalue)) => Some(upvalue),
             _ => None,
         }
     }

src/vm/vm.rs

@@ -21,9 +21,9 @@ use crate::vm::{
 /// so more than one can be spawned if needed.
 #[derive(Debug)]
 pub struct VM {
-    closure: Closure,
-    stack:   Stack,
-    ip:      usize,
+    pub closure: Closure,
+    pub stack:   Stack,
+    pub ip:      usize,
 }
 
 // NOTE: use Opcode::same and Opcode.to_byte() rather than actual bytes

tests/fledgling.rs

@@ -0,0 +1,218 @@
+use std::{
+    fs,
+    path::PathBuf,
+    collections::HashMap,
+    rc::Rc,
+};
+
+use passerine::{
+    common::{
+        source::Source,
+        data::Data,
+        closure::Closure,
+    },
+    compiler::{
+        lex, parse, desugar, gen,
+        ast::AST,
+    },
+    vm::vm::VM,
+};
+
+/// Represents specific success/failure modes of a snippet test.
+#[derive(Debug, PartialEq, Eq)]
+pub enum Outcome {
+    Success,
+    Syntax,
+    Trace,
+}
+
+impl Outcome {
+    pub fn parse(outcome: &str) -> Outcome {
+        match outcome {
+            s if s == "success" => Outcome::Success,
+            s if s == "syntax"  => Outcome::Syntax,
+            t if t == "trace"   => Outcome::Trace,
+            invalid => {
+                println!("invalid: '{}'", invalid);
+                panic!("invalid outcome in strat heading");
+            },
+        }
+    }
+}
+
+/// Represents what part of the compiler a snippet tests.
+#[derive(Debug)]
+pub enum Action {
+    Lex,
+    Parse,
+    Desugar,
+    Gen,
+    Run,
+}
+
+impl Action {
+    pub fn parse(action: &str) -> Action {
+        match action {
+            l if l == "lex"     => Action::Lex,
+            p if p == "parse"   => Action::Parse,
+            d if d == "desugar" => Action::Desugar,
+            g if g == "gen"     => Action::Gen,
+            r if r == "run"     => Action::Run,
+            invalid => {
+                println!("invalid: '{}'", invalid);
+                panic!("invalid action in strat heading");
+            },
+        }
+    }
+}
+
+/// Represents a test strategy for executing a snippet,
+/// Found at the top of each file.
+#[derive(Debug)]
+pub struct TestStrat {
+    /// How to run the test.
+    action:  Action,
+    /// The expected outcome.
+    outcome: Outcome,
+    /// Optional data to check against.
+    /// Should only be used with Action::Run
+    expect:  Option<Data>
+}
+
+impl TestStrat {
+    /// Uses a heading to construct a test strat
+    pub fn heading(heading: HashMap<String, String>) -> TestStrat {
+        let mut outcome = None;
+        let mut action = None;
+        let mut expect = None;
+
+        for (strat, result) in heading.iter() {
+            match strat {
+                o if o == "outcome" => outcome = Some(Outcome::parse(result)),
+                a if a == "action"  => action  = Some(Action::parse(result)),
+                e if e == "expect"  => expect  = {
+                    let tokens = lex(Source::source(result)).expect("Could not lex expectation");
+                    let ast    = parse(tokens).expect("Could not parse expectation");
+
+                    if let AST::Block(b) = ast.item {
+                        if let AST::Data(d) = &b[0].item {
+                            Some(d.clone())
+                        } else { panic!("expected data in block") }
+                    } else { panic!("expected block in ast") }
+                },
+                invalid => {
+                    println!("invalid: '{}'", invalid);
+                    panic!("invalid strat in strat heading");
+                },
+            }
+        }
+
+        TestStrat {
+            outcome: outcome.expect("no outcome provided"),
+            action: action.expect("no action provided"),
+            expect,
+        }
+    }
+
+    /// Parses the Test Strat from a given snippet.
+    pub fn snippet(source: &Rc<Source>) -> TestStrat {
+        let mut heading = HashMap::new();
+        let lines = source.contents.lines();
+
+        // build up a list of key-value pairs
+        for line in lines {
+            if line.len() <= 2 || &line[0..2] != "--" { break };
+
+            let spliced = line[2..].trim().split(":").collect::<Vec<&str>>();
+            if spliced.len() <= 1 { panic!("Missing colon in test strat heading") }
+
+            let strat = spliced[0];
+            let result = spliced[1..].join(":");
+            if heading.insert(strat.trim().to_string(), result.trim().to_string()).is_some() {
+                panic!("Key present twice in test strat heading");
+            }
+        }
+
+        return TestStrat::heading(heading);
+    }
+}
+
+fn test_snippet(source: Rc<Source>, strat: TestStrat) {
+    let actual_outcome: Outcome = match strat.action {
+        Action::Lex => if lex(source)
+            .is_ok() { Outcome::Success } else { Outcome::Syntax },
+
+        Action::Parse => if lex(source)
+            .and_then(parse)
+            .is_ok() { Outcome::Success } else { Outcome::Syntax },
+
+        Action::Desugar => if lex(source)
+            .and_then(parse)
+            .and_then(desugar)
+            .is_ok() { Outcome::Success } else { Outcome::Syntax },
+
+        Action::Gen => if lex(source)
+            .and_then(parse)
+            .and_then(desugar)
+            .and_then(gen)
+            .is_ok() { Outcome::Success } else { Outcome::Syntax },
+
+        Action::Run => {
+            if let Ok(lambda) = lex(source)
+                .and_then(parse)
+                .and_then(desugar)
+                .and_then(gen)
+            {
+                let mut vm = VM::init();
+
+                match vm.run(Closure::wrap(lambda)) {
+                    Ok(()) => {
+                        if let Some(expected) = &strat.expect {
+                            let top = vm.stack.pop_data();
+                            if expected != &top {
+                                println!("Top: {}", top);
+                                println!("Expected: {}", expected);
+                                panic!("Top stack data does not match")
+                            }
+                        }
+                        Outcome::Success
+                    },
+                    Err(_) => Outcome::Trace
+                }
+            } else {
+                Outcome::Syntax
+            }
+        }
+    };
+
+    if actual_outcome != strat.outcome {
+        println!("expected outcome {:?}", strat.outcome);
+        println!("actual outcome {:?}", actual_outcome);
+        panic!("test failed, outcomes are not the same");
+    }
+}
+
+#[test]
+fn test_snippets() {
+    let paths = fs::read_dir("./tests/snippets")
+        .expect("You must be in the base passerine directory, snippets in ./tests/snippets");
+
+    let mut to_run: Vec<PathBuf> = vec![];
+    for path in paths { to_run.push(path.expect("Could not read path").path()) }
+
+    let mut counter = 0;
+    println!("\nRunning {} snippet test(s)...", to_run.len());
+
+    // TODO: subdirectories of tests
+    while let Some(path) = to_run.pop() {
+        println!("test {}: {}...", counter, path.display());
+
+        let source = Source::path(path).expect("Could not get snippet source");
+        let test_strat = TestStrat::snippet(&source);
+
+        test_snippet(source, test_strat);
+        counter += 1;
+    }
+
+    println!("All tests passed!\n");
+}

tests/snippets/ambiguious.pn

@@ -0,0 +1,9 @@
+-- action: desugar
+-- outcome: syntax
+
+syntax a 'where b { 0.0 }
+
+a = 1.0
+b = 2.0
+
+a where b where a

tests/snippets/call.pn

@@ -0,0 +1,13 @@
+-- action: run
+-- outcome: success
+-- expect: "Nice"
+
+a = x -> x "Nice"
+b = x -> a x
+c = x -> b x
+d = x -> c x
+e = x -> d x
+
+i = x -> x
+
+e i

tests/snippets/chaining.pn

@@ -0,0 +1,8 @@
+-- action:  run
+-- outcome: success
+-- expect:  "Bye"
+
+i = x -> x
+bye = x -> "Bye"
+
+"Hello" |> i |> bye |> i

tests/snippets/closure.pn

@@ -0,0 +1,12 @@
+-- action: run
+-- outcome: success
+-- expect: 3.14
+
+pi = 2.7
+
+update_a = () -> () -> { pi = 3.14 }
+
+updater = update_a ()
+updater ()
+
+pi

tests/snippets/constant_macro.pn

@@ -0,0 +1,7 @@
+-- action: run
+-- outcome: success
+-- expect: false
+
+syntax 'not_true { x -> false }
+
+true |> not_true

tests/snippets/double_capture.pn

@@ -0,0 +1,7 @@
+-- action: gen
+-- outcome: success
+
+a = "Heck"
+
+() -> a
+() -> a