diff --git a/src/pages/docs/features.mdx b/src/pages/docs/0001-features.mdx
similarity index 91%
rename from src/pages/docs/features.mdx
rename to src/pages/docs/0001-features.mdx
index 5789b15..4c987a6 100644
--- a/src/pages/docs/features.mdx
+++ b/src/pages/docs/0001-features.mdx
@@ -4,6 +4,7 @@ layout: '../../layouts/DocsLayout.astro'
---
import Status from '../../components/Status.astro';
+import Note from '../../components/Note.astro';
# Language Features
@@ -61,11 +62,11 @@ String literals are automatically converted to `sr` or `str`.
* Use `libs.strings.array.del_str_array()` to delete a string array.
* Supports `+` operator to join two `str` values.
-; Data Type - `str`
-
-; Internally this is a binary string.
-
-; `sds` library (part of runtime lib) takes care of the heavy lifting.
+
+* Data Type - `str`
+* Internally this is a binary string.
+* `sds` library (part of runtime lib) takes care of the heavy lifting.
+
```yaksha
a: str = "hello world"
@@ -79,9 +80,11 @@ println(a)
### Standard integers
-*Default integer is a 32bit signed integer.
-*This is compiled to `int32_t` on all platforms.
-;Data Type - `int` or `i32`
+* Default integer is a 32bit signed integer.
+* This is compiled to `int32_t` on all platforms.
+
+Data Type - `int` or `i32`
+
```yaksha
a: int = 4 # datatype ✅ | literal ✅
print("Four =")
@@ -121,7 +124,8 @@ j: u64 = 5u64 # datatype ✅ | literal ✅
* `f32` or `float` - single precision floats.
* `f64` - double precision floats.
-;Data Type - `f32`, `float` and `f64`
+
+Data Type - `f32`, `float` and `f64`
```yaksha
a: f32 = 1.0f # datatype ✅ | literal ✅
@@ -141,7 +145,7 @@ c: float = 3.0f # datatype ✅ | literal ✅
* If you want to assign to a variable, it needs to be created.
* If no value is provided default value for data type is used.
-; Default value for `str` is an empty string.
+Default value for `str` is an empty string.
```yaksha
def main() -> int:
@@ -187,7 +191,7 @@ def get_arg(n: int) -> str:
def get_global_arg(n: int) -> str:
ccode "yk__sdsdup(global_args[yy__n])"
```
-;If `ccode` is there instead of an argument, then it is used as the message body.
+If `ccode` is there instead of an argument, then it is used as the message body.Click to see output C code
@@ -261,14 +265,14 @@ def new(count: int, s: str) -> Array[str]:
---
-
+
### Template functions
* Return type can also be a template-arg if that template-arg is used in parameters.
* String passed inside `@template(` should be single upper case characters separated by commas.
-;This means it is limited to 26 template arguments max.
+This means it is limited to 26 template arguments max.
```yaksha
@native("yk__arrput")
@@ -312,8 +316,11 @@ def calculate(n: int) -> int:
* This behaviour is different from what you see in go-lang.
* Before end of `if` body or end of `else` body.
-;`defer` works as a stack. \
-;That means deferred expressions are executed in last deferred first executed order.
+
+* `defer` works as a stack.
+* That means deferred expressions are executed in last deferred first executed order.
+* Please note that this is not compatible with how `go` programming language `defer` works.
+
```yaksha
def onexit() -> int:
@@ -351,7 +358,7 @@ def main() -> int:
println(a[0])
return 0
```
-; Compiles to `free` or other runtime functions.
+Compiles to `free` or other runtime functions.
---
@@ -423,9 +430,9 @@ def main() -> int:
class Something:
something_id: int
-# Use @dotaccess for purely stack allocated structs
+# Use @onstack for purely stack allocated structs
@nativedefine("Color")
-@dotaccess
+@onstack
class Color:
r: int
g: int
@@ -462,7 +469,7 @@ def main() -> int:
return 0
```
-;Name mangling takes place for this.
+Name mangling takes place for this.
---
@@ -552,9 +559,10 @@ def main() -> int:
return 0
```
-;Note that Yaksha allows omitting `return`, assuming last expression matches the return data type.
-
-;This is why `add` function does not have a return statement.
+
+* Note that Yaksha allows omitting `return`, assuming last expression matches the return data type.
+* This is why `add` function does not have a return statement.
+
---
@@ -593,7 +601,7 @@ def main() -> int:
* ✅ `make("T") -> Ptr[T]` - Allocate a single object.
* ✅ `inlinec("T", "code") -> T` - Inline `C` code resulting in `T` data type. Example - `inlinec("int", "sizeof(char)")`
-; Builtin functions may call different implementations based on input.
+Builtin functions may call different implementations based on input.
---
@@ -618,7 +626,7 @@ def main() -> int:
return 0
```
-; Must ensure array elements are freed. `int` need not be deleted as they are primitive types.
+Must ensure array elements are freed. `int` need not be deleted as they are primitive types.
---
@@ -626,8 +634,8 @@ def main() -> int:
### String Hash Map
-HashMap with str keys and given data type values.
-Values need to be deleted when no longer needed.
+* HashMap with str keys and given data type values.
+* Values need to be deleted when no longer needed.
```yaksha
def main() -> int:
@@ -640,9 +648,10 @@ def main() -> int:
return r
```
-; `Array[SMEntry[?]]` keys are deleted automatically when `del` is invoked.
-
-; `len` will give the total number of elements of the String Hash Map.
+
+* `Array[SMEntry[?]]` keys are deleted automatically when `del` is invoked.
+* `len` will give the total number of elements of the String Hash Map.
+
---
@@ -652,9 +661,10 @@ def main() -> int:
Simple single key single value hashmaps.
-;Data type `Array[MEntry[K,T]]`.
-
-;key and value both need to be deleted.
+
+* Data type `Array[MEntry[K,T]]`.
+* key and value both need to be deleted.
+
---
@@ -663,13 +673,14 @@ Simple single key single value hashmaps.
YakshaLisp macros are one of the most important features of Yaksha. You can think of it as an interpreted language that lives in Yaksha compiler. \
-; Because that is what it is!
+Because that is what it is!
It has it's own built in functions, can use `import`, read/write files and even use `metamacro` directive to create quoted input functions(similar to `defun`, except input args are not evaluated and returns quoted output that is immediately evaluated). Has multiple data types (list, map, callable, string, expression). Support's q-expressions `{1 2 3}` (inspired by build-your-own-lisp's lisp dialect), and special forms. A simple mark and sweep garbage collector is used. Provides a mediocre REPL and ability to execute standalone lisp code using a command. Not only that, it also support reflection using builtin callables such as `this` and `parent` (which returns a mutable current or parent scope as a map).
-; Yo dog!, I heard you like macros, so I added meta-macro support in your macro processor.
-
-; So you can meta-program while you meta-program.
+
+* Yo dog!, I heard you like macros, so I added meta-macro support in your macro processor.
+* So you can meta-program while you meta-program.
+
---
@@ -677,7 +688,7 @@ It has it's own built in functions, can use `import`, read/write files and even
YakshaLisp provide the ability to write token-list to token-list conversion macros. One can use `my_macro!{t t t}` style expansion to achieve this. So why YakshaLisp? because it needs to process a list of tokens.
Additionally, Yaksha and YakshaLisp are polar opposites of each other, therefore I think they can compliment each other nicely.
-; Think Yin-Yang!
+Think Yin-Yang!
---
diff --git a/src/pages/docs/runtime-library.mdx b/src/pages/docs/0002-runtime-library.mdx
similarity index 85%
rename from src/pages/docs/runtime-library.mdx
rename to src/pages/docs/0002-runtime-library.mdx
index 7366d88..d1b0cc4 100644
--- a/src/pages/docs/runtime-library.mdx
+++ b/src/pages/docs/0002-runtime-library.mdx
@@ -4,6 +4,7 @@ layout: '../../layouts/DocsLayout.astro'
---
import Status from '../../components/Status.astro';
+import Note from '../../components/Note.astro';
# Yaksha Library
@@ -13,7 +14,7 @@ Core parts of runtime library functionality starts with `yk__` or `YK__`.
Yaksha programming language preserves anything that starts with `yk__` or `YK__` while prefixing any other name with `yy__`.
-; This avoids collisions with C standard library functions or keywords.
+This avoids collisions with C standard library functions or keywords.
---
@@ -73,4 +74,4 @@ def main() -> int:
cl.run(dvc, my_program)
```
-; Placeholder code, API is not designed yet.
+Placeholder code, API is not designed yet.
diff --git a/src/pages/docs/grammar.mdx b/src/pages/docs/0003-grammar.mdx
similarity index 85%
rename from src/pages/docs/grammar.mdx
rename to src/pages/docs/0003-grammar.mdx
index 0e48181..d8c9074 100644
--- a/src/pages/docs/grammar.mdx
+++ b/src/pages/docs/0003-grammar.mdx
@@ -5,8 +5,9 @@ layout: '../../layouts/DocsLayout.astro'
import Warning from '../../components/Warning.astro';
import Status from '../../components/Status.astro';
+import Note from '../../components/Note.astro';
-this section needs rewriting
+This section needs rewriting
This describes Yaksha language grammar.
Lot learned from `Crafting Interpreters` book and Python grammar page.
@@ -74,9 +75,11 @@ c_code_statement: "ccode" (string | triple_quote_string) st_end
block: ":" (newline ba_indent standard_statement+ ba_dedent | simple_statement)
```
-;`defer` statement is special, it can take either a `del` statement or any expression.
-
-;This allows you to defer a `del`.
+
+* `defer` statement is special, it can take either a `del` statement or any expression.
+* This allows you to defer a `del`.
+* Example - `defer del my_element`
+
---
@@ -91,7 +94,7 @@ primitive_data_type: "int" | "i8" | "i16" | "i32" | "i64" |
"f32" | "f64"
```
-;Data type describes, primitives, custom data types and non primitives.
+Data type describes, primitives, custom data types and non primitives.
---
@@ -116,7 +119,7 @@ primary: "True" | "False" | "None" | number | string | triple_quote_string | nam
"(" expr ")"
```
-; Based on [lox grammar](https://craftinginterpreters.com/appendix-i.html) and [Python operator precedence](https://www.mathcs.emory.edu/~valerie/courses/fall10/155/resources/op_precedence.html).
+Based on [lox grammar](https://craftinginterpreters.com/appendix-i.html) and [Python operator precedence](https://www.mathcs.emory.edu/~valerie/courses/fall10/155/resources/op_precedence.html).
---
@@ -135,5 +138,5 @@ eof: "end of file"
ba_indent: "increase in indentation"
ba_dedent: "decrease is indentation"
```
-; `ba_indent` and `ba_dedent` gets calculated before parsing in `block_analyzer` phase.
+`ba_indent` and `ba_dedent` gets calculated before parsing in `block_analyzer` phase.
diff --git a/src/pages/docs/0004-compiler-internals.mdx b/src/pages/docs/0004-compiler-internals.mdx
new file mode 100644
index 0000000..fac8d04
--- /dev/null
+++ b/src/pages/docs/0004-compiler-internals.mdx
@@ -0,0 +1,247 @@
+---
+title: Compiler Internals
+layout: '../../layouts/DocsLayout.astro'
+---
+
+import Note from '../../components/Note.astro';
+import Status from '../../components/Status.astro';
+import Warning from '../../components/Warning.astro';
+
+# Compiler Internals
+
+This section describes how compiler works.
+
+Total of `12+` passes are planned for the compiler.
+
+## What happens when you compile a `.yaka` file?
+### Let's look at a sample
+
+#### Input code
+
+This section needs updating
+
+```yaksha
+def factorial(x: int) -> int:
+ if x <= 0:
+ return 1
+ return x * factorial(x - 1)
+
+
+def main() -> int:
+ a: int = 10
+ b: str = "b"
+ while a > 0:
+ print(factorial(a))
+ print("\n")
+ a = a - 1
+ b = "a" + b
+ print(b)
+ print("\n")
+ return 0
+```
+
+#### Output C code
+
+```c
+// --yaksha header section--
+#include "yk__lib.h"
+int32_t yy__factorial(int32_t);
+int32_t yy__main();
+// --yaksha body section--
+int32_t yy__factorial(int32_t yy__x) {
+ if ((yy__x <= 0)) { return 1; }
+ return (yy__x * yy__factorial((yy__x - 1)));
+}
+int32_t yy__main() {
+ int32_t yy__a = 10;
+ yk__sds t__0 = yk__sdsnew("b");
+ yk__sds yy__b = yk__sdsdup(t__0);
+ while (1) {
+ if (!((yy__a > 0))) { break; }
+ {
+ printf("%d", (yy__factorial(yy__a)));
+ yk__sds t__1 = yk__sdsnew("\n");
+ printf("%s", (t__1));
+ yy__a = (yy__a - 1);
+ yk__sds t__2 = yk__sdsnew("a");
+ yk__sds t__3 = yk__sdscatsds(yk__sdsdup(t__2), yy__b);
+ yk__sdsfree(yy__b);
+ yy__b = yk__sdsdup(t__3);
+ yk__sdsfree(t__3);
+ yk__sdsfree(t__2);
+ yk__sdsfree(t__1);
+ }
+ }
+ printf("%s", (yy__b));
+ yk__sds t__4 = yk__sdsnew("\n");
+ printf("%s", (t__4));
+ yk__sdsfree(t__4);
+ yk__sdsfree(t__0);
+ yk__sdsfree(yy__b);
+ return 0;
+}
+// --yaksha footer section--
+int main(void) { return (int) yy__main(); }
+```
+
+* Notice `yk__sdsfree` is automatically generated for simple string uses.
+* This makes strings immutable. However, it is doing a lot of unnecessary processing. 😓
+* Therefore `sr` is useful to avoid unnecessary memory allocations.
+
+
+## Phases of the compiler
+
+
+### Tokenizer
+
+
+* Tokenizer breaks down input to individual tokens and Identifies Keywords.
+* Parse numbers and strings and check if they are valid according to the grammar.
+
+If any error is detected at this point we still continue up to parser so we can identify maximum number of errors.
+
+
+### Block analyzer
+
+
+* Convert indentation to `ba_indent`, `ba_dedent` type tokens.
+* Remove comments.
+* Remove extra new lines.
+
+
+* Only 2-spaces, 4-spaces or tab based indents are supported.
+* Yaksha will try to guess indentation type.
+* Still continue to parser even after errors.
+
+
+
+### Parser
+
+
+* Parses tokens returned by block analyzer an AST.
+* AST is represented as a `std::vector` of `stmt*`.
+
+Any errors from previous stages and parsing stage is printed here and program will exit.
+
+
+### Import analyzer
+
+
+
+
+* Analyzes imports.
+* Parses imported files.
+* This step will use more instances of Tokenizer, Parser and Compiler objects.
+
+### Def-Struct-Const Visitor
+
+
+* Visit `def` statements and collect functions to a map.
+* Visit `class` statements and collect structures to a map.
+* Visit global constants.
+
+
+### Return path analyzer
+
+
+* Analyzes return paths.
+* Ensure all functions return.
+
+
+### Type checker
+
+
+* Type checker visits AST and check for invalid types.
+* Checks for undefined variables.
+* Checks for undefined functions.
+* Checks for undefined structures.
+* Check all return types are same as that of the encapsulating functions.
+
+
+### Template Compiler
+
+
+* Rewrite @template to be normal functions based on what's passed to them.
+* Rewrite `fncall` expressions to use newly created functions.
+
+
+### Optimizer
+
+
+* Remove dead code.
+* Basic constant folding.
+
+
+### To-CL-Compiler
+
+
+* Convert `@device` code to OpenCL program code.
+* Copy necessary structures.
+* Check validity - no generics, no str, no allocations.
+
+
+### To-C-Compiler
+
+
+
+* Writes `C` code from AST.
+* Do any simple optimizations.
+* Handle `defer` statements.
+* Handle `str` deletions.
+* Create `struct` and function declarations.
+
+
+### C-To-C Compiler
+
+
+* We generate code to a single `C` file which can then be further optimized.
+* Parse and optimize subset of generated `C` code.
+
+## How does the Yaksha-lang library get packaged?
+
+
+
+
+* Multiple sources are packaged into a single header file.
+* Library functionality is exposed by prefixing with `yk__` or `YK__`.
+
+
+### Packer components
+
+* `packer.py` - Run packer DSL scripts and create packaged single header libraries.
+* `inctree.py` - Topological sort `#include` DAG to determine best order for combining headers.
+
+DAG - Directed Acyclic Graph.
+
+* `cids.exe` - Use `stb_c_lexer.h` library to parse C code and extract identifiers.
+* `single_header_packer.py` - ApoorvaJ's single header C code packager. [Repo](https://github.com/ApoorvaJ/libs/blob/master/scripts/single_header_packer.py)
+* `python-patch` - techtonik's patch script. [Repo](https://github.com:techtonik/python-patch)
+* `fcpp`- Frexx C Preprocessor by Daniel Stenberg. (Patch for Windows compilation was needed, failed to compile with MSVC, works with MingW with patch).
+
+### Third Party Libraries
+
+If you stack few giants, you can stand very tall on top of them.
+
+* `sds` - Salvatore Sanfilippo's string library. (Needed a patch to support MSVC 2019)
+* `stb` - Single header libraries by Sean Barrett.
+* `utf8proc` - UTF-8 library - Jan Behrens, Public Software Group and Julia developers.
+
+Note - currently only sds and stb_ds is used. This selection of libraries may change.
+
+### Packer DSL
+
+```python
+import re
+use_source("libs")
+for lib in ["ini", "thread", "http"]:
+ ids = extract_ids(lib + ".h")
+ P = [x for x in ids if x.startswith(lib)]
+ PU = [x for x in ids if x.startswith(lib.upper())]
+ prefix(lib + ".h", PREFIX, P)
+ prefix(lib + ".h", PREFIX_U, PU)
+ rename(lib + ".h", [[re.escape(r'yk__http://'), 'http://'],
+ ["YK__THREAD_PRIORITY_HIGHEST", "THREAD_PRIORITY_HIGHEST"]])
+ copy_file(lib + ".h", PREFIX + lib + ".h", is_temp=False)
+ clang_format(PREFIX + lib + ".h", is_temp=False)
+```
+It is just Python 3.x with extra functions added and evaluated.🤫
+ )
+ }
+
+* Analyzes imports.
+* Parses imported files.
+* This step will use more instances of Tokenizer, Parser and Compiler objects.
+
+
+