From e3aa4086ff74bc7435a3ccc75dec974c73e3580d Mon Sep 17 00:00:00 2001 From: JaDogg Date: Wed, 13 Dec 2023 12:10:28 +0000 Subject: [PATCH] rebuild documents --- docs/documentation.html | 90 +++---- docs/library-docs.html | 6 +- docs/tutorials.html | 2 +- docs/yama.html | 235 ++++++++++-------- yaksha_proposals/0001_c_enums.md | 14 +- yaksha_proposals/0002_structs.md | 10 +- .../0003_syntax_sugar_for_loop.md | 16 +- yaksha_proposals/0004_iterator.md | 8 +- yaksha_proposals/0005_dsl_macros.md | 14 +- yaksha_proposals/0006_c2c.md | 8 +- yaksha_proposals/0007_structs_revisited.md | 10 +- yaksha_proposals/0008_sugar_methods.md | 6 +- yaksha_proposals/0009_strings_revisited.md | 6 +- yaksha_proposals/0011_http.md | 4 +- 14 files changed, 230 insertions(+), 199 deletions(-) diff --git a/docs/documentation.html b/docs/documentation.html index 71b74db..8a6e14a 100644 --- a/docs/documentation.html +++ b/docs/documentation.html @@ -1,4 +1,4 @@ -Yaksha Programming Language

1 Language Features

Created 2022-08-09, Last Updated 2023-09-03

1.1 Data Types

1.1.1 Wrapped String References

done

Wrapped string literals sr wraps multiple source strings in an uniform structure. There is no reason to delete a sr unless you want to de-allocate, referred data of the string.

You can add two sr using + operator. However, resulting value will be of type str, as memory allocation is required. It is recommended to use a string buffer for concatenating a string.

a: sr = "A"
+Yaksha Programming Language

1 Language Features

Created 2023-04-07, Last Updated 2023-09-03

1.1 Data Types

1.1.1 Wrapped String References

done

Wrapped string literals sr wraps multiple source strings in an uniform structure. There is no reason to delete a sr unless you want to de-allocate, referred data of the string.

You can add two sr using + operator. However, resulting value will be of type str, as memory allocation is required. It is recommended to use a string buffer for concatenating a string.

a: sr = "A"
 b: sr = "B"
 c = a + b
 # Type of c would be str
@@ -40,11 +40,11 @@
 @native
 def get_global_arg(n: int) -> str:
     ccode "yk__sdsdup(global_args[yy__n])"
-
Click to see output C code
yk__sds yy__get_arg(int32_t nn__n) { return getarg(nn__n); }
-yk__sds yy__get_global_arg(int32_t nn__n)
-{
-    yk__sdsdup(global_args[yy__n]);
-}
+
Click to see output C code
yk__sds yy__get_arg(int32_t nn__n) { return getarg(nn__n); }
+yk__sds yy__get_global_arg(int32_t nn__n)
+{
+    yk__sdsdup(global_args[yy__n]);
+}
 

If ccode is there instead of an argument, then it is used as the message body.

1.2.2.2.2 @nativemacro - macros with arguments
@nativemacro
 def min_int(a: int, b:int) -> int:
     ccode "((nn__a < nn__b) ? nn__a : nn__b)"
@@ -244,7 +244,7 @@
 def main() -> int:
     println(fizzbuzz!{})
     return 0
-


2 Yaksha library

Created 2022-08-18, Last Updated 2023-07-01

Yaksha library uses multiple 3rd party libraries, set of python scripts and few other nuts and bolts to create library sources.

Core parts of runtime library functionality starts with yk__ or YK__.

Yaksha programming language preserves anything that starts with yk__ or YK__ while prefixing any other name with yy__.

This avoids collisions with C standard library functions or keywords.

2.1 Standard Library Components

2.1.1 Core Library

in progress

Small core library with useful features.

import libs
+


2 Yaksha library

Created 2023-04-07, Last Updated 2023-07-01

Yaksha library uses multiple 3rd party libraries, set of python scripts and few other nuts and bolts to create library sources.

Core parts of runtime library functionality starts with yk__ or YK__.

Yaksha programming language preserves anything that starts with yk__ or YK__ while prefixing any other name with yy__.

This avoids collisions with C standard library functions or keywords.

2.1 Standard Library Components

2.1.1 Core Library

in progress

Small core library with useful features.

import libs
 

2.1.2 UI Library

in progress

Use raygui with raylib.

import raylib.gui
 

2.1.3 raylib - Graphics/Game/Audio Library

in progress

Use raylib to build games and such

import raylib
 

2.1.4 CL Library

not started

Access to OpenCL features and parallel programming.

import cl
@@ -256,7 +256,7 @@
 def main() -> int:
     dvc: cl.device = cl.get_device(0)
     cl.run(dvc, my_program)
-

Placeholder code, API is not designed yet.



3 Language Grammar

Created 2022-08-09, Last Updated 2023-08-05

Warning! this section needs rewriting

This describes Yaksha language grammar.

Lot learned from Crafting Interpreters book and Python grammar page.

in progress

3.1 Statements

This section describes statements and program structure. Program is made from one or more program statements. Standard statements are what goes inside functions.

program: import_statement* program_declaration*
+

Placeholder code, API is not designed yet.



3 Language Grammar

Created 2023-04-07, Last Updated 2023-08-05

Warning! this section needs rewriting

This describes Yaksha language grammar.

Lot learned from Crafting Interpreters book and Python grammar page.

in progress

3.1 Statements

This section describes statements and program structure. Program is made from one or more program statements. Standard statements are what goes inside functions.

program: import_statement* program_declaration*
 program_declarations:
     | const_statement
     | struct_statement
@@ -341,7 +341,7 @@
 eof: "end of file"
 ba_indent: "increase in indentation"
 ba_dedent: "decrease is indentation"
-

ba_indent and ba_dedent gets calculated before parsing in block_analyzer phase.



4 Compiler Internals

Created 2022-08-18, Last Updated 2023-08-13

This section describes how compiler works.

Total of 12+ passes are planned for the compiler.

4.1 What happens when you compile a .yaka file?

4.1.1 Let's look at a sample

4.1.1.1 Input code
def factorial(x: int) -> int:
+

ba_indent and ba_dedent gets calculated before parsing in block_analyzer phase.



4 Compiler Internals

Created 2023-04-07, Last Updated 2023-08-13

This section describes how compiler works.

Total of 12+ passes are planned for the compiler.

4.1 What happens when you compile a .yaka file?

4.1.1 Let's look at a sample

4.1.1.1 Input code
def factorial(x: int) -> int:
     if x <= 0:
         return 1
     return x * factorial(x - 1)
@@ -359,44 +359,44 @@
     print("\n")
     return 0
 
4.1.1.2 Output C code
// --yaksha header section--
-#include "yk__lib.h"
-int32_t yy__factorial(int32_t);
-int32_t yy__main();
+#include "yk__lib.h"
+int32_t yy__factorial(int32_t);
+int32_t yy__main();
 // --yaksha body section--
-int32_t yy__factorial(int32_t yy__x) {
-  if ((yy__x <= 0)) { return 1; }
-  return (yy__x * yy__factorial((yy__x - 1)));
-}
-int32_t yy__main() {
-  int32_t yy__a = 10;
-  yk__sds t__0 = yk__sdsnew("b");
-  yk__sds yy__b = yk__sdsdup(t__0);
-  while (1) {
-    if (!((yy__a > 0))) { break; }
-    {
-      printf("%d", (yy__factorial(yy__a)));
-      yk__sds t__1 = yk__sdsnew("\n");
-      printf("%s", (t__1));
-      yy__a = (yy__a - 1);
-      yk__sds t__2 = yk__sdsnew("a");
-      yk__sds t__3 = yk__sdscatsds(yk__sdsdup(t__2), yy__b);
-      yk__sdsfree(yy__b);
-      yy__b = yk__sdsdup(t__3);
-      yk__sdsfree(t__3);
-      yk__sdsfree(t__2);
-      yk__sdsfree(t__1);
-    }
-  }
-  printf("%s", (yy__b));
-  yk__sds t__4 = yk__sdsnew("\n");
-  printf("%s", (t__4));
-  yk__sdsfree(t__4);
-  yk__sdsfree(t__0);
-  yk__sdsfree(yy__b);
-  return 0;
-}
+int32_t yy__factorial(int32_t yy__x) {
+  if ((yy__x <= 0)) { return 1; }
+  return (yy__x * yy__factorial((yy__x - 1)));
+}
+int32_t yy__main() {
+  int32_t yy__a = 10;
+  yk__sds t__0 = yk__sdsnew("b");
+  yk__sds yy__b = yk__sdsdup(t__0);
+  while (1) {
+    if (!((yy__a > 0))) { break; }
+    {
+      printf("%d", (yy__factorial(yy__a)));
+      yk__sds t__1 = yk__sdsnew("\n");
+      printf("%s", (t__1));
+      yy__a = (yy__a - 1);
+      yk__sds t__2 = yk__sdsnew("a");
+      yk__sds t__3 = yk__sdscatsds(yk__sdsdup(t__2), yy__b);
+      yk__sdsfree(yy__b);
+      yy__b = yk__sdsdup(t__3);
+      yk__sdsfree(t__3);
+      yk__sdsfree(t__2);
+      yk__sdsfree(t__1);
+    }
+  }
+  printf("%s", (yy__b));
+  yk__sds t__4 = yk__sdsnew("\n");
+  printf("%s", (t__4));
+  yk__sdsfree(t__4);
+  yk__sdsfree(t__0);
+  yk__sdsfree(yy__b);
+  return 0;
+}
 // --yaksha footer section--
-int main(void) { return (int) yy__main(); }
+int main(void) { return (int) yy__main(); }
 

Notice yk__sdsfree is automatically generated for simple string uses. This makes strings immutable. Currently, it is doing a lot of unnecessary processing. 😓

4.2 Phases of the compiler

4.2.1 Tokenizer

in progress

  • Tokenizer breaks down input to individual tokens and Identifies Keywords.

  • Parse numbers and strings and check if they are valid according to the grammar.

If any error is detected at this point we still continue up to parser so we can identify maximum number of errors.

4.2.2 Block analyzer

in progress

  • Convert indentation to ba_indent, ba_dedent type tokens.

  • Remove comments.

  • Remove extra new lines.

Only 2-spaces, 4-spaces or tab based indents are supported. Will try to guess indentation type.

Still continue to parser even after errors.

4.2.3 Parser

in progress

  • Parses tokens returned by block analyzer an AST.

  • AST is represented as a std::vector of stmt*.

Any errors from previous stages and parsing stage is printed here and program will exit.

4.2.4 Import analyzer

in progress

Import Analyzer
  • Analyzes imports.

  • Parses imported files.

  • This step will use more instances of Tokenizer, Parser and Compiler objects.

4.2.5 Def-Struct-Const Visitor

in progress

  • Visit def statements and collect functions to a map.

  • Visit class statements and collect structures to a map.

  • Visit global constants.

4.2.6 Return path analyzer

in progress

  • Analyzes return paths.

  • Ensure all functions return.

4.2.7 Type checker

in progress

  • Type checker visits AST and check for invalid types.

  • Checks for undefined variables.

  • Checks for undefined functions.

  • Checks for undefined structures.

  • Check all return types are same as that of the encapsulating functions.

4.2.8 Template Compiler

not started

  • Rewrite @template to be normal functions based on what's passed to them.

  • Rewrite fncall expressions to use newly created functions.

4.2.9 Optimizer

not started

  • Remove dead code.

  • Basic constant folding.

4.2.10 To-CL-Compiler

not startedplanned for v0.5

  • Convert @device code to OpenCL program code.

  • Copy necessary structures.

  • Check validity - no generics, no str, no allocations.

4.2.11 To-C-Compiler

in progress

  • Writes C code from AST.

  • Do any simple optimizations.

  • Handle defer statements.

  • Handle str deletions.

  • Create struct and function declarations.

4.2.12 C-To-C Compiler

not started

  • We generate code to a single C file which can then be further optimized.

  • Parse and optimize subset of generated C code.

4.3 How does the Yaksha-lang library get packaged?

Multiple sources are packaged into a single header file.

Library functionality is exposed by prefixing with yk__ or YK__.

4.3.1 Packer components

  • packer.py - Run packer DSL scripts and create packaged single header libraries.

  • inctree.py - Topological sort #include DAG to determine best order for combining headers.

  • cids.exe - Use stb_c_lexer.h library to parse C code and extract identifiers.

  • single_header_packer.py - ApoorvaJ's single header C code packager. Repo

  • python-patch - techtonik's patch script. Repo

  • fcpp- Frexx C Preprocessor by Daniel Stenberg. (Patch for Windows compilation was needed, failed to compile with MSVC, works with MingW with patch).

DAG - Directed Acyclic Graph.

4.3.2 Third Party Libraries

  • sds - Salvatore Sanfilippo's string library. (Needed a patch to support MSVC 2019)

  • stb - Single header libraries by Sean Barrett.

  • libs - Mattias Gustavsson's single header C libraries.

  • utf8proc - UTF-8 library - Jan Behrens, Public Software Group and Julia developers.

If you stack few giants, you can stand very tall on top of them.


Note - currently only sds and stb_ds is used. This selection of libraries may change.

4.3.3 Packer DSL

import re
 use_source("libs")
 for lib in ["ini", "thread", "http"]:
diff --git a/docs/library-docs.html b/docs/library-docs.html
index 796831d..cf37b77 100644
--- a/docs/library-docs.html
+++ b/docs/library-docs.html
@@ -1,4 +1,4 @@
-Yaksha Programming Language

1 Core Library

Created 2022-08-09, Last Updated 2023-03-19

Core library can be accessed by importing libs.

This import name is subject to change.

1.1 libs

def version() -> str
+Yaksha Programming Language

1 Core Library

Created 2023-04-07, Last Updated 2023-04-07

Core library can be accessed by importing libs.

This import name is subject to change.

1.1 libs

def version() -> str
 # This returns standard library version
 # (Note: this is hardcoded)
 

1.2 libs.argparse

ARGPARSE_DEFAULT: Const[int]
@@ -555,7 +555,7 @@
 def lessser(a: TimeSpec, b: TimeSpec) -> bool
 # is a < b ?
 def now() -> TimeSpec
-


2 Raylib

Created 2022-08-14, Last Updated 2023-04-07

Raylib can be accessed by importing raylib.

raylib is created by Ramon Santamaria and contributors.

Yaksha wraps raylib using a generator script.

2.1 raylib

BLEND_ADDITIVE: Const[int]
+


2 Raylib

Created 2023-04-07, Last Updated 2023-04-07

Raylib can be accessed by importing raylib.

raylib is created by Ramon Santamaria and contributors.

Yaksha wraps raylib using a generator script.

2.1 raylib

BLEND_ADDITIVE: Const[int]
 # Blend textures adding colors
 BLEND_ADD_COLORS: Const[int]
 # Blend textures adding colors (alternative)
@@ -3535,7 +3535,7 @@
 def tan_deg(x: float) -> float
 def todeg(radians: float) -> float
 def torad(degrees: float) -> float
-


3 WASM4 Support Library

Created 2022-10-21, Last Updated 2023-03-19

Support for WASM4 fantasy console.

Additionally following function in libs work.

  • libs.random.random_u64

  • libs.random.set_seed

wasm4 is created by Bruno Garcia and contributors.

Yaksha wraps wasm4.h

3.1 w4

BLIT_1BPP: Const[u32]
+


3 WASM4 Support Library

Created 2023-04-07, Last Updated 2023-04-07

Support for WASM4 fantasy console.

Additionally following function in libs work.

  • libs.random.random_u64

  • libs.random.set_seed

wasm4 is created by Bruno Garcia and contributors.

Yaksha wraps wasm4.h

3.1 w4

BLIT_1BPP: Const[u32]
 BLIT_2BPP: Const[u32]
 BLIT_FLIP_X: Const[u32]
 BLIT_FLIP_Y: Const[u32]
diff --git a/docs/tutorials.html b/docs/tutorials.html
index 2afb46d..c9ba387 100644
--- a/docs/tutorials.html
+++ b/docs/tutorials.html
@@ -1,7 +1,7 @@
 Yaksha Programming Language

1 Getting started

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera

1.1 Setting things up

1.1.1 Downloading

  • Head to releases

  • Pick an archive format you like .7z or .zip (both has same content and .7z is almost always likely to be smaller in size)

  • Since this is a new language we recommend you always download latest version. Please expect things to not work.

1.1.2 Installing

  • After download is completed you can extract the archive. Here are few ways:

  • GNU/Linux: use 7za or unzip depending on archive format.
  • Now you can add bin directory to PATH.

  • If you are in GNU/Linux ensure that binaries in the bin directory have +x (executable) permission assigned. chmod +x *

1.1.3 Basics of Yaksha Programming Language

1.1.3.1 Hello World

Steps:

First save below code to a file named "hello.yaka"

def main() -> int:
     println("Hello World!")
     return 0
-

In Yaksha we use .yaka extension for source files. Also, we use -> to denote return type of a function.

Additionally entry point of a program is main function. Main function must always return an int value. In this case we return 0 to indicate that program has exited successfully. Additionally we use println builtin function to print a line to standard output.

Now you can compile it using yaksha command.

yaksha build -R hello.yaka
+

In Yaksha we use .yaka extension for source files. Also, we use -> to denote return type of a function.

Additionally entry point of a program is main function. Main function must always return an int value. In this case we return 0 to indicate that program has exited successfully. Additionally we use println builtin function to print a line to standard output.

Now you can compile it using yaksha command.

yaksha build -R hello.yaka
 

Internally yaksha build command will invoke carpntr binary included in the distribution. carpntr is the builder tool for Yaksha. It will compile the source file and generate a binary named hello in the current directory. Additionally it will generate a hello.c file which is the C code generated by the compiler. You can inspect this file to see how the compiler works. -R option for carpntr (or yaksha build) will execute the binary after compilation.

carpntr (or yaksha build) has many options. You can see them by running yaksha build --help or carpntr --help. Under the hood carpntr uses bundled zig compiler (with zig cc) and compile c code to a binary.

1.1.3.2 Comments

Yaksha programming language use # as an indicator for starting a comment.

# This is a comment
 def main() -> int:
     a = "Hello # This is not a comment"
diff --git a/docs/yama.html b/docs/yama.html
index 710188b..06a4ebe 100644
--- a/docs/yama.html
+++ b/docs/yama.html
@@ -1,36 +1,36 @@
-Yaksha Programming Language

YAMA 0001 - Exposing C enums/#defines/consts

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : ✅

Problem

In C we can define enums such as below.

// Simple
-enum something1 {
-  SOMETHING1,
-  ANOTHER1
-};
+Yaksha Programming Language

YAMA 0001 - Exposing C enums/#defines/consts

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : ✅

Problem

In C we can define enums such as below.

// Simple
+enum something1 {
+  SOMETHING1,
+  ANOTHER1
+};
 
 // Separate typedef
-enum something2 {
-  SOMETHING2,
-  ANOTHER2
-};
-typedef enum something2 something2_t;
+enum something2 {
+  SOMETHING2,
+  ANOTHER2
+};
+typedef enum something2 something2_t;
 
 // Combined
-typedef enum
-{
-  SOMETHING3,
-  ANOTHER3
-} something3_t;
-

Currently we can use something like

SOMETHING: Const[int] = 0
-ANOTHER: Const[int] = 1
-

This however make it annoying to maintain, as we now have a copy of the enum value.

We can use below as well.

@nativemacro("SOMETHING")
+typedef enum
+{
+  SOMETHING3,
+  ANOTHER3
+} something3_t;
+

Currently we can use something like

SOMETHING: Const[int] = 0
+ANOTHER: Const[int] = 1
+

This however make it annoying to maintain, as we now have a copy of the enum value.

We can use below as well.

@nativemacro("SOMETHING")
 def something() -> int:
     pass
 

This does not copy the value, but make it annoying to use the API as now we need to access this as something().

Suggestions

Suggestion 1

# Expose enum value / #define value
-nativexp "SOMETHING" as SOMETHING: Const[int]
+nativexp "SOMETHING" as SOMETHING: Const[int]
 
 # Some simple constant calculations
-nativexp """1 + 1""" as TWO: Const[int]
-
  • Pro:
  • Clearly define that this is a native expression
  • Con:
  • Looks different

Suggestion 2

TWO: Const[int] = ccode """1 + 1"""
-
  • Pro:
  • Consistent syntax
  • Clearly define that this is a native expression (ccode)
  • Con:
  • None

Suggestion 3

TWO: Const[int] <- nativexp """1 + 1"""
-
  • Pro:
  • Clearly define that this is a native expression
  • Con:
  • Looks different

Suggestion 4

nativexp """1 + 1""" -> TWO: Const[int]
-
  • Pro:
  • None
  • Con:
  • Looks different

Suggestion 5

TWO: Const[int] = nativexp """1 + 1"""
+nativexp """1 + 1""" as TWO: Const[int]
+
  • Pro:
  • Clearly define that this is a native expression
  • Con:
  • Looks different

Suggestion 2

TWO: Const[int] = ccode """1 + 1"""
+
  • Pro:
  • Consistent syntax
  • Clearly define that this is a native expression (ccode)
  • Con:
  • None

Suggestion 3

TWO: Const[int] <- nativexp """1 + 1"""
+
  • Pro:
  • Clearly define that this is a native expression
  • Con:
  • Looks different

Suggestion 4

nativexp """1 + 1""" -> TWO: Const[int]
+
  • Pro:
  • None
  • Con:
  • Looks different

Suggestion 5

TWO: Const[int] = nativexp """1 + 1"""
 
  • Pro:
  • Consistent syntax
  • Clearly define that this is a native expression (ccode)
  • Con:
  • New keyword is introduced

This would compile to

#define yk__TWO (1 + 1)
 

Conclusion

I'm going with suggestion 2 as it uses already existing keywords and also at the same time looks consistent with current constants.



YAMA 0002 - Adding support for stack allocated structs

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : ✅

Problem

class Enemy:
     x: int
@@ -38,7 +38,7 @@
     hit_points: int
 
 def main() -> int:
-    enemies: Array[Enemy] = arrnew("Enemy", 2)
+    enemies: Array[Enemy] = arrnew("Enemy", 2)
     enemies[0] = Enemy()
     enemies[1] = Enemy()
     return 0
@@ -66,7 +66,7 @@
 # Rename @dotaccess to @onstack
 # And use that for this purpose
 Enemy() # This is disallowed. Just create a variable and you can use it.
-
  • Pro:
  • Defines that this is different from a normal class
  • Clearly visible that this get allocated on stack
  • Con:
  • Require changes to how @dotaccess works currently

Conclusion

I am thinking suggestion 4 is the best approach.

Update

Additionally Yaksha now supports struct keyword. That act as a syntax sugar for @onstack class.



YAMA 0003 - For loop / loop

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : ✅

Problem

Yaksha programming language does not offer a for syntax sugar. This is annoying for someone coming from Python or other languages that has this feature.

We have foreach builtin and while loop, which can be used at the moment. for can make certain things easy.

Use case 1 - for each element in array ✅

How it works now

items: Array[int] = get_it()
+
  • Pro:
  • Defines that this is different from a normal class
  • Clearly visible that this get allocated on stack
  • Con:
  • Require changes to how @dotaccess works currently

Conclusion

I am thinking suggestion 4 is the best approach.

Update

Additionally Yaksha now supports struct keyword. That act as a syntax sugar for @onstack class.



YAMA 0003 - For loop / loop

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : ✅

Problem

Yaksha programming language does not offer a for syntax sugar. This is annoying for someone coming from Python or other languages that has this feature.

We have foreach builtin and while loop, which can be used at the moment. for can make certain things easy.

Use case 1 - for each element in array ✅

How it works now

items: Array[int] = get_it()
 
 c: int = 0
 length: int = len(items)
@@ -75,24 +75,24 @@
     if items[c] == 2:
         c += 1
         continue
-    println(items[c])
+    println(items[c])
     c += 1
 

Syntax sugar (we will keep compulsory data types for now)

for item: int in get_it(): # get_it should return Array[T]
     if item == 2:
         continue
-    println(item)
-

Desugared

yy__1t: Array[int] = get_it()
+    println(item)
+

Desugared

yy__1t: Array[int] = get_it()
 yy__2t: int = 0
 yy__3t: int = len(yk__1t)
 
 while yy__2t < yy__3t:
-    ccode """#define yy__item yy__1t[yy__2t]\n// desugar begin"""
+    ccode """#define yy__item yy__1t[yy__2t]\n// desugar begin"""
     expose_native item: int      # To expose what we put in above C code
     if item == 2:
         yy__2t += 1
         continue                 # Continue must increase the counter
-    println(item)
-    ccode """#undef yy__item\n// desugar end"""
+    println(item)
+    ccode """#undef yy__item\n// desugar end"""
     yy__2t += 1               # Increase count at the very end
 

Use case 2 - endless loops ✅

Syntax sugar

for:
     game_step()
@@ -112,17 +112,17 @@
     for element: Book in it:
         print_book(element)
 

Use case 4 - range for loops ⚠️(not required a we have c-like for loops)

Syntax sugar

# Example 1
-r: Range[int] = range(1, 5, 2)
+r: Range[int] = range(1, 5, 2)
 for i: int in r:
     if i == 1: # do not print 1
        continue
-    println(i)
+    println(i)
 
 # Example 2
-for i: int in range(1, 5, 2):
+for i: int in range(1, 5, 2):
     if i == 1: # do not print 1
        continue
-    println(i)
+    println(i)
 

Desugared

r: Tuple[int, int, int]
 r[0] = 1
 r[1] = 5
@@ -134,12 +134,12 @@
     if i == 1:
         hidden__c += r[2]
         continue
-    println(i)
+    println(i)
     hidden__c += r[2]
-

Conclusion

Iterators/Ranges are deferred for now. It would be better to come up with a generic Iterator approach for these.

At the moment I think endless loops and simple for-each style loops should be implemented.



YAMA 0004 - Iterators

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : Draft

Problem

We would like to introduce support for iterators, such that it allows Yaksha to simply iterate arbitrary content.

Use case 1 - custom iterators / iterables

Syntax sugar

def next_book(x: Ptr[Books]) -> Book:
+

Conclusion

Iterators/Ranges are deferred for now. It would be better to come up with a generic Iterator approach for these.

At the moment I think endless loops and simple for-each style loops should be implemented.



YAMA 0004 - Iterators

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : Draft

Problem

We would like to introduce support for iterators, such that it allows Yaksha to simply iterate arbitrary content.

Use case 1 - custom iterators / iterables

Syntax sugar

def next_book(x: Ptr[Books]) -> Book:
     pass
 
-def has_next_book(x: Ptr[Books]) -> bool:
+def has_next_book(x: Ptr[Books]) -> bool:
     pass
 
 def process() -> None:
@@ -150,10 +150,10 @@
 
     for element: Book in it:
         print_book(element)
-

Desugared

def next_book(x: Ptr[Books]) -> Book:
+

Desugared

def next_book(x: Ptr[Books]) -> Book:
     pass
 
-def has_next_book(x: Ptr[Books]) -> bool:
+def has_next_book(x: Ptr[Books]) -> bool:
     pass
 
 def process() -> None:
@@ -164,21 +164,21 @@
     it[1] = next_book
     it[2] = has_next_book
 
-    while (it[1])(getref(it[0])):
-        element: Book = (it[2])(getref(it[0]))
+    while (it[1])(getref(it[0])):
+        element: Book = (it[2])(getref(it[0]))
         print_book(element)
 

Use case 2 - range for loops

Syntax sugar

# Example 1
-r: Iterator[int] = range(1, 5, 2)
+r: Iterator[int] = range(1, 5, 2)
 for i: int in r:
     if i == 1: # do not print 1
        continue
-    println(i)
+    println(i)
 
 # Example 2 (would desugar similarly)
-for i: int in range(1, 5, 2):
+for i: int in range(1, 5, 2):
     if i == 1: # do not print 1
        continue
-    println(i)
+    println(i)
 

Desugared

r: Tuple[Tuple[int, int, int, int], .., ..]
 yy__1t: Tuple[int, int, int, int]
 yy__1t[0] = 1       # start
@@ -189,33 +189,33 @@
 r[1] = yk__next_range
 r[2] = yk__has_range
 
-while (r[1])(getref(r[0])):
-    i: int = (r[2])(getref(r[0]))
+while (r[1])(getref(r[0])):
+    i: int = (r[2])(getref(r[0]))
     if i == 1:
         continue
-    println(i)
-

Conclusion

Iterators make a fine addition to foreach loop.



YAMA 0005 - DSL macros for Yaksha

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : In Progress

Problem

Macros allow for compile time code generation. Yaksha already supports multiple ways of exposing and defining C macros. However, C macros are not easy to use. Other than for exposing things written in C one should avoid using C macros in Yaksha (@nativexxx).

Proposal

YakshaLisp would be a language of it's own. Capable of executing at compile time of Yaksha (and also as a command line interpreter).

WHY?

  • Lisp is a great language for meta programming.

  • Lisp can be used to manipulate lists. (List processing 😄)

  • Token list to token list transformation is a good fit for lisp.

  • Simplest form of syntax to learn.

  • Yak-shaving is fun. (Did you see what I did there? 😄)

  • Feels nicer and user friendly than C macros and @nativexxx in Yaksha.

Command line interpreter

Use YakshaLisp as it's own programming language.

  • yaksha lisp - Run YakshaLisp interpreter REPL (Just YakshaLisp code, no macros!{} required)

  • yaksha lisp <file> - Execute just YakshaLisp code. (Don't use macros!{} here)

DSL macros

DSL macros get a list of tokens and outputs a list of tokens. This allows for more complex logic to be written in YakshaLisp macros.

Any DSL macro inside {} will be also expanded.

Most nested macro invocation will be expanded first. These macros can call other macros as you call a function and support iteration and recursion. These macros will only be expanded once.

Builtin functionality of YakshaLisp
  • Printing to console ✅

  • Reading from console ✅ - Requires enable_print to be called, before calling print and println, console output of yaksha compile maybe altered using this. Therefore, this is disabled by default.

  • Reading/writing text files ✅

  • Strings, decimal Integers ✅

  • List manipulation ✅

  • Maps (string -> value only) ✅

  • Map literals, and manipulation ✅

  • Working with Yaksha tokens ✅

  • Iteration ✅

  • Conditions - if, cond ✅

  • Few functional built ins - map, reduce, range ✅

  • Q-Expressions inspired by Build Your Own Lisp lisp dialect. ✅ (This is somewhat similar to '(+ 1 2 3))

  • Eval and parse ✅

  • Token generation for Yaksha ✅

  • Executing commands ❌ (not started, planned) - Will be disabled by default.

  • Macros for YakshaLisp ✅

  • Imports ✅

  • Hygienic macros - gensym ✅ / metagensym 🟡

  • Garbage collection ✅

Item 1 - Initial DSL macro support ✅

# ╔═╗┌─┐┌┬┐┌─┐┬┬  ┌─┐  ╔╦╗┬┌┬┐┌─┐
+    println(i)
+

Conclusion

Iterators make a fine addition to foreach loop.



YAMA 0005 - DSL macros for Yaksha

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : In Progress

Problem

Macros allow for compile time code generation. Yaksha already supports multiple ways of exposing and defining C macros. However, C macros are not easy to use. Other than for exposing things written in C one should avoid using C macros in Yaksha (@nativexxx).

Proposal

YakshaLisp would be a language of its own. Capable of executing at compile time of Yaksha (and also as a command line interpreter).

WHY?

  • Lisp is a great language for meta programming.

  • Lisp can be used to manipulate lists. (List processing 😄)

  • Token list to token list transformation is a good fit for lisp.

  • Simplest form of syntax to learn.

  • Yak-shaving is fun. (Did you see what I did there? 😄)

  • Feels nicer and user-friendly than C macros and @nativexxx in Yaksha.

Command line interpreter

Use YakshaLisp as its own programming language.

  • yaksha lisp - Run YakshaLisp interpreter REPL (Just YakshaLisp code, no macros!{} required)

  • yaksha lisp <file> - Execute just YakshaLisp code. (Don't use macros!{} here)

DSL macros

DSL macros get a list of tokens and outputs a list of tokens. This allows for more complex logic to be written in YakshaLisp macros.

Any DSL macro inside {} will be also expanded.

Most nested macro invocation will be expanded first. These macros can call other macros as you call a function and support iteration and recursion. These macros will only be expanded once.

Builtin functionality of YakshaLisp
  • Printing to console ✅

  • Reading from console ✅ - Requires enable_print to be called, before calling print and println, console output of yaksha compile maybe altered using this. Therefore, this is disabled by default.

  • Reading/writing text files ✅

  • Strings, decimal Integers ✅

  • List manipulation ✅

  • Maps (string -> value only) ✅

  • Map literals, and manipulation ✅

  • Working with Yaksha tokens ✅

  • Iteration ✅

  • Conditions - if, cond ✅

  • Few functional built ins - map, reduce, range ✅

  • Q-Expressions inspired by Build Your Own Lisp lisp dialect. ✅ (This is somewhat similar to '(+ 1 2 3))

  • Eval and parse ✅

  • Token generation for Yaksha ✅

  • Executing commands ❌ (not started, planned) - Will be disabled by default.

  • Macros for YakshaLisp ✅

  • Imports ✅

  • Hygienic macros - gensym ✅ / metagensym 🟡

  • Garbage collection ✅

Item 1 - Initial DSL macro support ✅

# ╔═╗┌─┐┌┬┐┌─┐┬┬  ┌─┐  ╔╦╗┬┌┬┐┌─┐
 # ║  │ ││││├─┘││  ├┤    ║ ││││├┤
 # ╚═╝└─┘┴ ┴┴  ┴┴─┘└─┘   ╩ ┴┴ ┴└─┘
 # ╔═╗┬┌─┐┌─┐  ╔╗ ┬ ┬┌─┐┌─┐
 # ╠╣ │┌─┘┌─┘  ╠╩╗│ │┌─┘┌─┘
 # ╚  ┴└─┘└─┘  ╚═╝└─┘└─┘└─┘
-macros!{
+macros!{
     (defun to_fb (n) (+ (if (== n 1) "" " ") (cond
         ((== 0 (modulo n 15)) "FizzBuzz")
         ((== 0 (modulo n 3)) "Fizz")
         ((== 0 (modulo n 5)) "Buzz")
         (true (to_string n))
         )))
-    (defun fizzbuzz () (list (yk_create_token YK_TOKEN_STRING (reduce + (map to_fb (range 1 101))))))
+    (defun fizzbuzz () (list (yk_create_token YK_TOKEN_STRING (reduce + (map to_fb (range 1 101))))))
     (yk_register {dsl fizzbuzz fizzbuzz})
 }
 
 def main() -> int:
-    println(fizzbuzz!{})
+    println(fizzbuzz!{})
     return 0
-

Item 2 - Hygienic DSL macros using gensym ✅ and metagensym 🟡

  • gensym - This is not a function unlike metagensym, simply return a name such as $a (any valid Yaksha IDENTIFIER prefixed with $) and it will be replaced with a unique symbol. ✅

  • metagensym - Generates a unique symbol to be used in YakshaLisp code.

Item 3 - Macros for YakshaLisp via metamacro

Did you ever think that your meta programming might benefit from meta-meta programming?

Macros get all parameters non-evaluated. The output from macro will be evaluated (in calling scope) and needs to be a list of expressions.

Other than that it is similar to a defun. Metamacros are executed similarly to functions and can also call other metamacros or defuns.

If you consider just YakshaLisp as it's own language, this allows you to meta program in YakshaLisp during runtime of YakshaLisp.

  • (is_callable my_meta) --> false, this is true only for builtins, def, lambda.

macros!{
-    (metamacro twice (x) (+ (list x) (list x)))
+

Item 2 - Hygienic DSL macros using gensym ✅ and metagensym 🟡

  • gensym - This is not a function unlike metagensym, simply return a name such as $a (any valid Yaksha IDENTIFIER prefixed with $) and it will be replaced with a unique symbol. ✅

  • metagensym - Generates a unique symbol to be used in YakshaLisp code.

Item 3 - Macros for YakshaLisp via metamacro

Did you ever think that your meta programming might benefit from meta-meta programming?

Macros get all parameters non-evaluated. The output from macro will be evaluated (in calling scope) and needs to be a list of expressions.

Other than that it is similar to a defun. Metamacros are executed similarly to functions and can also call other metamacros or defuns.

If you consider just YakshaLisp as it's own language, this allows you to meta program in YakshaLisp during runtime of YakshaLisp.

  • (is_callable my_meta) --> false, this is true only for builtins, def, lambda.

macros!{
+    (metamacro twice (x) (+ (list x) (list x)))
     # since x is an expression (non evaluated), it can be converted to {}
     # by simply calling list.
     # x ----> (= value (+ value 1))) # as an expr
@@ -233,11 +233,11 @@
 
 def main() -> int:
     return 0
-
(twice (println "Hello")) ----> (eval {(println "Hello") (println "Hello")})
+
(twice (println "Hello")) ----> (eval {(println "Hello") (println "Hello")})
 

Item 4 - Import macros from other files ✅

import libs.macro as m
-macros!{
+macros!{
     (enable_print)
-    (m.twice (println "Hello"))
+    (m.twice (println "Hello"))
 }
 
 def main() -> int:
@@ -245,17 +245,17 @@
 

Item 5 - Execute commands (not started, planned) ❌

Will be disabled by default. Can be enabled by a setting a flag in yaksha.toml or setting an environment variable YAKSHA_SERIOUSLY_ENABLE_SHELL_EXEC to True.

import libs.macro.shell as m
 
 def main() -> int:
-    m.execute!{"echo Hello World"}
-    println(m.readout!{"echo Hello World", "Failed"})
+    m.execute!{"echo Hello World"}
+    println(m.readout!{"echo Hello World", "Failed"})
     return 0
-

Item 6 - Tracing macro expansion and debugging (not started, planned) ❌

Allow Yaksha compiler to output macro expansion steps. This will be useful for debugging macros.

Design TBD.

Some tracing features are supported by YakshaLisp if DEBUG is enabled during compilation of Yaksha, but this does not look nice and only useful for Yaksha/YakshaLisp language development. Perhaps these DEBUG features can be made to output more readable output and enable/disable with a flag instead of #ifdef DEBUG.

Item 7 - Garbage collection ✅

YakshaLisp will have a simple mark and sweep garbage collector. This will be used to manage memory allocated by YakshaLisp.



YAMA 0006 - C 2 C Compiler - Intermediate C99 to C99 optimizing compiler

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : Draft

Problem

Yaksha uses C as a final compilation target. This is to introduce a way to have an intermediate C as a IR that can then be compiled to C

Naming

  • iC99 - Intermediate C 99

Language

  • Looks like C99.

  • Loops -> for, while, do while

  • structs, constants, unions, #define

  • We have built-ins such as yk__sdsnew, etc.

#define yy__raylib_Music Music
+

Item 6 - Tracing macro expansion and debugging (not started, planned) ❌

Allow Yaksha compiler to output macro expansion steps. This will be useful for debugging macros.

Design TBD.

Some tracing features are supported by YakshaLisp if DEBUG is enabled during compilation of Yaksha, but this does not look nice and only useful for Yaksha/YakshaLisp language development. Perhaps these DEBUG features can be made to output more readable output and enable/disable with a flag instead of #ifdef DEBUG.

Item 7 - Garbage collection ✅

YakshaLisp will have a simple mark and sweep garbage collector. This will be used to manage memory allocated by YakshaLisp.



YAMA 0006 - C 2 C Compiler - Intermediate C99 to C99 optimizing compiler

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : Draft

Problem

Yaksha uses C as a final compilation target. This is to introduce a way to have an intermediate C as a IR that can then be compiled to C

Naming

  • iC99 - Intermediate C 99

Language

  • Looks like C99.

  • Loops -> for, while, do while

  • structs, constants, unions, #define

  • We have built-ins such as yk__sdsnew, etc.

#define yy__raylib_Music Music
 
-i32 yy__main() {
-  yy__raylib_Music x;
+i32 yy__main() {
+  yy__raylib_Music x;
 
-  return 0;
-}
-

Architecture

  • Tokenize

  • Level 1 -> no keyword recognition

  • Recognizes normal operators, identifiers, #?? and comments

  • Discard comments

  • Parse all with pre processor statements

  • Remove comments

  • Apply #defines -> until there are no changes present

  • AST based

  • Only works with 1 file

  • Entry points yy__main yy__game_step, yy__init_state

Optimizations

Opt 01 - apply all #defines starting with yy__

  • Implement a simple pre processor, however only capable of applying #defines that start with yy__

  • Opt 01.01 - Support non function like macros

  • Opt 01.02 - Support for function like macros

Opt 02 - data types

  • Opt 02.01 - Push down support of integer data types from Yaksha compiler to iC99.

  • Instead of having extra complexity at the compiler level for i32, etc like data types, we push it to iC99.

Opt 03 - clean up

  • Opt 03.01 - Remove any method that cannot be reached from yy__main()

  • Opt 03.02 - Remove any operation in same block after return

Opt 04 - constant folding

  • Opt 04.01 - Apply -

  • Opt 04.02 - Remove excess ( ) parentheses

  • Opt 04.03 - Do basic math

  • Opt 04.04 - Apply basic string built-ins

  • Opt 04.05 - Apply basic array built-ins

  • Opt 04.06 - Dead code elimination

Opt 05 - apply constant function calls

  • Everything is a constant expression?

  • For each function things with simple math, assignments and built ins are considered pure

  • Any of these pure functions can be invoked during compile time and folded

Opt 06 - small function in-lining

  • Inline small functions < 20 lines?

Opt 07 - multiple rounds of previous optimizations

while changed:

  • Opt 03

  • Opt 04

  • Opt 05

  • Opt 06



YAMA 0007 - Additional features for structs

Created 2023-07-30, Last Updated 2023-08-05
  • Author(s): Bhathiya Perera
  • Status : In Progress

Structures are at the moment can be created as follows

@onstack
+  return 0;
+}
+

Architecture

  • Tokenize

  • Level 1 -> no keyword recognition

  • Recognizes normal operators, identifiers, #?? and comments

  • Discard comments

  • Parse all with pre-processor statements

  • Remove comments

  • Apply #defines -> until there are no changes present

  • AST based

  • Only works with 1 file

  • Entry points yy__main yy__game_step, yy__init_state

Optimizations

Opt 01 - apply all #defines starting with yy__

  • Implement a simple pre-processor, however only capable of applying #defines that start with yy__

  • Opt 01.01 - Support non function like macros

  • Opt 01.02 - Support for function like macros

Opt 02 - data types

  • Opt 02.01 - Push down support of integer data types from Yaksha compiler to iC99.

  • Instead of having extra complexity at the compiler level for i32 ike data types, we push it to iC99.

Opt 03 - clean up

  • Opt 03.01 - Remove any method that cannot be reached from yy__main()

  • Opt 03.02 - Remove any operation in same block after return

Opt 04 - constant folding

  • Opt 04.01 - Apply -

  • Opt 04.02 - Remove excess ( ) parentheses

  • Opt 04.03 - Do basic math

  • Opt 04.04 - Apply basic string built-ins

  • Opt 04.05 - Apply basic array built-ins

  • Opt 04.06 - Dead code elimination

Opt 05 - apply constant function calls

  • Everything is a constant expression?

  • For each function things with simple math, assignments and built ins are considered pure

  • Any of these pure functions can be invoked during compile time and folded

Opt 06 - small function in-lining

  • Inline small functions < 20 lines?

Opt 07 - multiple rounds of previous optimizations

while changed:

  • Opt 03

  • Opt 04

  • Opt 05

  • Opt 06



YAMA 0007 - Additional features for structs

Created 2023-07-30, Last Updated 2023-08-05
  • Author(s): Bhathiya Perera
  • Status : In Progress

Structures are at the moment can be created as follows

@onstack
 class Banana:
     color: int
     origin: int
@@ -264,25 +264,25 @@
 class Orange:
     color: int
     origin: int
-

Item 1 - Allocate single object on heap

a: Ptr[Banana] = make("Banana")
+

Item 1 - Allocate single object on heap

a: Ptr[Banana] = make("Banana")
 
 b: Orange = make("Orange")
 

Item 2 - Create structures or classes ✅

a: Banana = Banana {color: YELLOW, origin: SRI_LANKA}
-
struct Banana a = (struct Banana){.color = YELLOW, .origin = SRI_LANKA};
+
struct Banana a = (struct Banana){.color = YELLOW, .origin = SRI_LANKA};
 
b: Orange = Orange {color: ORANGE, origin: SOUTH_AFRICA}
-
struct Orange* _temp = calloc(1, sizeof(struct Orange));
-_temp->color  = ORANGE;
-_temp->origin = SOUTH_AFRICA;
+
struct Orange* _temp = calloc(1, sizeof(struct Orange));
+_temp->color  = ORANGE;
+_temp->origin = SOUTH_AFRICA;
 
-struct Orange* b = _temp;
-

Item 3 - Introduce struct keyword, desugar to @onstack class

struct Banana:
+struct Orange* b = _temp;
+

Item 3 - Introduce struct keyword, desugar to @onstack class

struct Banana:
     color: int
     origin: int
-


YAMA 0008 - Sugar methods

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : Draft

Idea 01 - Arrow

struct Banana:
+


YAMA 0008 - Sugar methods

Created 2023-07-30, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : Draft

Idea 01 - Arrow

struct Banana:
     color: int
     origin: int
 
-def display(b: Ptr[Banana]) -> None:
+def display(b: Ptr[Banana]) -> None:
     return
 
 def main() -> int:
@@ -294,23 +294,23 @@
     color: int
     origin: int
 
-def display(b: Ptr[Banana]) -> None:
+def display(b: Ptr[Banana]) -> None:
     return
 
 def main() -> int:
     b: Banana = ...
     b.display()
     return 0
-

If @onstack or struct then we need to use Ptr[DataType] otherwise DataType can be used.

Going with . as it is already valid AST. Additionally, we can find the method during type checking and desugar it.

display(getref(b))
+

If @onstack or struct then we need to use Ptr[DataType] otherwise DataType can be used.

Going with . as it is already valid AST. Additionally, we can find the method during type checking and desugar it.

display(getref(b))
 


YAMA 0009 - Strings revisited

Created 2023-08-13, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : Draft

Strings in programming languages are tough to design. You can make things simple by using it like a value type (Copying and deleting it). However, value like strings come at a cost as it does large number of alloc/dealloc resulting in fragmented memory. In WASM4 or embedded (potential future scenario) cases, this kind of alloc/dealloc is not acceptable.

This is an all or nothing proposal. Need to support all scenarios explained below. After this is completed, libs/runtime needs to rewritten to use new sr data type instead of str data type.

There is also an invisible string data type literal that is only available at compile time (a: literal = "hi" is invalid).

This type will be auto-magically converted to sr.

This is considered to avoid wasting time doing - runtime concatenation and runtime comparison of literals.

  • Breaking changes** - a = "hello" results in a's data type to be sr instead of str and sr is now a builtin data type.

Few areas to explore

String reference

  • String reference does not own the underlying heap allocated memory of a string.

  • No need to delete or duplicate.

  • But it is dangerous to store it in a structure as it might be invalid later on.

Static storage strings

  • Immutable static strings that lives in the static storage, lasts for the whole life time of the program.

  • No need to de-allocate.

  • Can store as a reference anywhere you like.

  • Not copied when passing.

String buffers

  • Mutable string that owns memory.

  • Not copied when passing to a function.

  • Can use same underlying yk__sds as data structure.

Semi managed value like strings

  • What str represents in Yaksha.

  • What is semi-managed? you need to manually de-allocate when a copy is added to a container.

String literals

  • This is a "Hello World" like string literal in Yaksha.

  • This is currently automatically converted to str.

  • We can consider string literals to be static storage strings.

  • Length is known at compile time.

Library/runtime use

  • Runtime functionality will not need to de-allocate strings. Once we switch to use of sr data type.

Implementation

Underlying data structure

struct yk__bstr { // Yaksha Base String
-    union {
+    union {
         yk__sds h;              // reference to a heap allocated string
         const char * s;         // reference to a statically allocated string
         char* c;                // reference to c.CStr that is heap allocated
-    };
-    size_t l;
-    enum yk__bstr_type { yk__bstr_str, yk__bstr_static, yk__bstr_cstr } t;
-};
+    };
+    size_t l;
+    enum yk__bstr_type { yk__bstr_str, yk__bstr_static, yk__bstr_cstr } t;
+};
 

Yaksha

First we assume we have something like below defined, that takes the new data type sr as an argument.

literal is a hidden data type -- this cannot be used by end users directly. (Internally represented as :s:)

Legend
  • ✅ - Completed

  • N - No allocation should happen here (except in Windows due to need to convert to utf16).

  • A - Alloc/Dealloc happens

  • C - Compile time

  • ! - Type-checking / compile error

def do_something(s: sr) -> int: # N
     print("Printing sr: ")      # N
     println(s)                  # N
@@ -366,32 +366,32 @@
     do_something(d)              # A
     return 0
 
Comparison of literals ("a" == "a") --> bool
def main() -> int:
-    println("a" == "a")          # AC
-    println("a" == "b")          # AC
-    println("a" != "a")          # AC
-    println("a" != "b")          # AC
+    println("a" == "a")          # AC
+    println("a" == "b")          # AC
+    println("a" != "a")          # AC
+    println("a" != "b")          # AC
     return 0
 
Comparison of mixed str, sr and literals --> bool
def main() -> int:
     a: str = "a"                 # N
     b: sr = "oi"                 # A
-    println(b == b)              # A
-    println(a == a)              # A
-    println((a == b) or (a != b))# AA
-    println("a" == a)            # AA
-    println(b != a)              # AA
-    println(b != "x")            # AA
+    println(b == b)              # A
+    println(a == a)              # A
+    println((a == b) or (a != b))# AA
+    println("a" == a)            # AA
+    println(b != a)              # AA
+    println(b != "x")            # AA
     return 0
 
Wrap/Unwrap c.CStr, Const[Ptr[Const[c.Char]]] as sr
  • Create a simple wrapper function in libs.c to wrap c.CStr --> getref_cstr

  • Create a simple wrapper function in libs.c to unwrap c.CStr --> unref_cstr

  • Create a simple wrapper function in libs.c to wrap Const[Ptr[Const[c.Char]]] --> getref_ccstr

  • Create a simple wrapper function in libs.c to unwrap Const[Ptr[Const[c.Char]]] --> unref_ccstr

  • See do_something and take_str

Wrap Array[u8] as sr
  • Create a simple wrapper function in libs.strings to wrap getref_u8a

Wrap Ptr[u8] as sr
  • Create a simple wrapper function in libs.strings to wrap getref_u8p

Compare with None ✅
def main() -> int:
     a: sr = "a"                  # N
     b: str = "b"                 # A
-    println(a == None)           # N
-    println(b == None)           # N
-    println("a" != None)         # NC
-    println("a" == None)         # NC
-    println(None != a)           # N
-    println(None != b)           # N
-    println(None == "a")         # NC
-    println(None != "a")         # NC
+    println(a == None)           # N
+    println(b == None)           # N
+    println("a" != None)         # NC
+    println("a" == None)         # NC
+    println(None != a)           # N
+    println(None != b)           # N
+    println(None == "a")         # NC
+    println(None != "a")         # NC
     return 0
 

Additional tasks

  • Update documentation explaining sr data type.



YAMA 0010 - Numbers auto casting when widening

Created 2023-08-13, Last Updated 2023-09-07
  • Author(s): Bhathiya Perera
  • Status : ✅

Following casting is suggested for basic binary mathematical operations.

Assignment would follow same casting style, except lhs needs to be wider.

Bitwise operations would require manual casting to same data type.

    .-------------.
     |    bool     |
@@ -426,4 +426,35 @@
 .'-------------V-V-.
 |       i16        |
 '------------------'
-
\ No newline at end of file +


YAMA 0011 - Add minimal support for a HTTP client in Yaksha

Created 2023-11-12, Last Updated 2023-11-25
  • Author(s): Bhathiya Perera
  • Status : 🟡

Summary

All programming languages provide a way to make HTTP requests. However, this is currently not available to Yaksha. This proposal aims to add minimal support for HTTP requests to Yaksha.

Design

We plan on using mongoose to implement this feature. Mongoose is a lightweight HTTP server written in C. It is easy to integrate. However, it's license is GPL and users of http library will have to be aware of this.

Client

Simple HTTP client will use following structure. This will be placed in a separate module called http/client.yaka.

# Mongoose mgr data structure
+@nativedefine("struct mg_mgr")
+stuct MongooseMgr:
+    pass
+
+# Mongoose connection data structure
+@nativedefine("struct mg_connection")
+struct MongooseCon:
+    pass
+
+# HTTP client data structure using above 2 structs
+struct HttpClient:
+    mgr: MongooseMgr
+    nc: Ptr[MongooseCon]
+    success: bool
+
+def http_client(url: sr) -> HttpClient:
+    pass  # placeholder
+
+def request(method: sr, url: sr, data: sr, callback: Function[..]) -> HttpRequest:
+    pass  # placeholder
+

Usage of the client will be as follows:

import http.client as client
+
+def callback(...): # placeholder
+    pass
+
+def main() -> int:
+    cl = client.http_client("http://localhost:8080")
+    cl.request("GET", "/api/v1/users", "", callback)
+    defer free_http_client(cl)
+    return 0
+

Required changes to Yaksha ecosystem

  • Implement ability to support fixed sized arrays in Yaksha (parser, compiler, etc).

  • Implement ability in YakshaLisp to execute programs and access output.

  • Implement ability in carpntr to use YakshaLisp to build programs. (Either use program execution ability or use make/cmake to build programs)

  • Implement ability in YakshaLisp to detect platform and use appropriate build system.

  • CI - Add ability to build Yaksha release on supported platforms using github actions. (Instead of cross compiling)

  • Remove hammer and just use cmake to build Yaksha.

  • Update build and other scripts accordingly.

Third party libraries to be included

  • mongoose - for making HTTP requests

  • mbedtls - for TLS support

  • reproc - for executing programs

\ No newline at end of file diff --git a/yaksha_proposals/0001_c_enums.md b/yaksha_proposals/0001_c_enums.md index d54da1c..3fd401f 100644 --- a/yaksha_proposals/0001_c_enums.md +++ b/yaksha_proposals/0001_c_enums.md @@ -33,7 +33,7 @@ Currently we can use something like -```python +```yaksha SOMETHING: Const[int] = 0 ANOTHER: Const[int] = 1 ``` @@ -42,7 +42,7 @@ This however make it annoying to maintain, as we now have a copy of the enum val We can use below as well. -```python +```yaksha @nativemacro("SOMETHING") def something() -> int: pass @@ -54,7 +54,7 @@ This does not copy the value, but make it annoying to use the API as now we need ### Suggestion 1 -```python +```yaksha # Expose enum value / #define value nativexp "SOMETHING" as SOMETHING: Const[int] @@ -69,7 +69,7 @@ nativexp """1 + 1""" as TWO: Const[int] ### Suggestion 2 -```python +```yaksha TWO: Const[int] = ccode """1 + 1""" ``` @@ -81,7 +81,7 @@ TWO: Const[int] = ccode """1 + 1""" ### Suggestion 3 -```python +```yaksha TWO: Const[int] <- nativexp """1 + 1""" ``` @@ -92,7 +92,7 @@ TWO: Const[int] <- nativexp """1 + 1""" ### Suggestion 4 -```python +```yaksha nativexp """1 + 1""" -> TWO: Const[int] ``` @@ -103,7 +103,7 @@ nativexp """1 + 1""" -> TWO: Const[int] ### Suggestion 5 -```python +```yaksha TWO: Const[int] = nativexp """1 + 1""" ``` diff --git a/yaksha_proposals/0002_structs.md b/yaksha_proposals/0002_structs.md index aeb9830..75887ba 100644 --- a/yaksha_proposals/0002_structs.md +++ b/yaksha_proposals/0002_structs.md @@ -5,7 +5,7 @@ ## Problem -```python +```yaksha class Enemy: x: int y: int @@ -26,7 +26,7 @@ Keep in mind that the Tuple's in Yaksha are mutable (Mutable `Tuple`s are what w ### Suggestion 1 -```python +```yaksha @namedtuple class Enemy: x: int @@ -42,7 +42,7 @@ class Enemy: ### Suggestion 2 -```python +```yaksha @dataclass class Enemy: x: int @@ -60,7 +60,7 @@ class Enemy: ### Suggestion 3 -```python +```yaksha class Enemy: x: int y: int @@ -79,7 +79,7 @@ new Enemy() # << allocate on heap ### Suggestion 4 ✅ -```python +```yaksha @onstack class Enemy: x: int diff --git a/yaksha_proposals/0003_syntax_sugar_for_loop.md b/yaksha_proposals/0003_syntax_sugar_for_loop.md index 435ce94..d33887b 100644 --- a/yaksha_proposals/0003_syntax_sugar_for_loop.md +++ b/yaksha_proposals/0003_syntax_sugar_for_loop.md @@ -14,7 +14,7 @@ We have `foreach` builtin and `while` loop, which can be used at the moment. `fo How it works now -```python +```yaksha items: Array[int] = get_it() c: int = 0 @@ -30,7 +30,7 @@ while c < length: Syntax sugar (we will keep compulsory data types for now) -```python +```yaksha for item: int in get_it(): # get_it should return Array[T] if item == 2: continue @@ -39,7 +39,7 @@ for item: int in get_it(): # get_it should return Array[T] Desugared -```python +```yaksha yy__1t: Array[int] = get_it() yy__2t: int = 0 yy__3t: int = len(yk__1t) @@ -59,7 +59,7 @@ while yy__2t < yy__3t: Syntax sugar -```python +```yaksha for: game_step() log_stuff() @@ -67,7 +67,7 @@ for: Desugared -```python +```yaksha while True: game_step() log_stuff() @@ -77,7 +77,7 @@ while True: Syntax sugar -```python +```yaksha def next_book(x: Books) -> Book: pass @@ -95,7 +95,7 @@ def process() -> None: Syntax sugar -```python +```yaksha # Example 1 r: Range[int] = range(1, 5, 2) for i: int in r: @@ -112,7 +112,7 @@ for i: int in range(1, 5, 2): Desugared -```python +```yaksha r: Tuple[int, int, int] r[0] = 1 r[1] = 5 diff --git a/yaksha_proposals/0004_iterator.md b/yaksha_proposals/0004_iterator.md index 5f71dd6..82d8385 100644 --- a/yaksha_proposals/0004_iterator.md +++ b/yaksha_proposals/0004_iterator.md @@ -13,7 +13,7 @@ We would like to introduce support for iterators, such that it allows Yaksha to Syntax sugar -```python +```yaksha def next_book(x: Ptr[Books]) -> Book: pass @@ -32,7 +32,7 @@ def process() -> None: Desugared -```python +```yaksha def next_book(x: Ptr[Books]) -> Book: pass @@ -56,7 +56,7 @@ def process() -> None: Syntax sugar -```python +```yaksha # Example 1 r: Iterator[int] = range(1, 5, 2) for i: int in r: @@ -73,7 +73,7 @@ for i: int in range(1, 5, 2): Desugared -```python +```yaksha r: Tuple[Tuple[int, int, int, int], .., ..] yy__1t: Tuple[int, int, int, int] yy__1t[0] = 1 # start diff --git a/yaksha_proposals/0005_dsl_macros.md b/yaksha_proposals/0005_dsl_macros.md index eeb403e..669d3c9 100644 --- a/yaksha_proposals/0005_dsl_macros.md +++ b/yaksha_proposals/0005_dsl_macros.md @@ -11,7 +11,7 @@ Macros allow for compile time code generation. Yaksha already supports multiple ## Proposal -YakshaLisp would be a language of it's own. Capable of executing at compile time of Yaksha (and also as a command line interpreter). +YakshaLisp would be a language of its own. Capable of executing at compile time of Yaksha (and also as a command line interpreter). ### WHY? @@ -20,11 +20,11 @@ YakshaLisp would be a language of it's own. Capable of executing at compile time * Token list to token list transformation is a good fit for lisp. * Simplest form of syntax to learn. * Yak-shaving is fun. (Did you see what I did there? 😄) -* Feels nicer and user friendly than `C` macros and `@nativexxx` in Yaksha. +* Feels nicer and user-friendly than `C` macros and `@nativexxx` in Yaksha. ### Command line interpreter -Use YakshaLisp as it's own programming language. +Use YakshaLisp as its own programming language. * `yaksha lisp` - Run YakshaLisp interpreter REPL (Just YakshaLisp code, no macros!{} required) * `yaksha lisp ` - Execute just YakshaLisp code. (Don't use macros!{} here) @@ -61,7 +61,7 @@ Most nested macro invocation will be expanded first. These macros can call other ## Item 1 - Initial DSL macro support ✅ -```python +```yaksha # ╔═╗┌─┐┌┬┐┌─┐┬┬ ┌─┐ ╔╦╗┬┌┬┐┌─┐ # ║ │ ││││├─┘││ ├┤ ║ ││││├┤ # ╚═╝└─┘┴ ┴┴ ┴┴─┘└─┘ ╩ ┴┴ ┴└─┘ @@ -100,7 +100,7 @@ If you consider just YakshaLisp as it's own language, this allows you to meta pr * `(is_callable my_meta)` --> `false`, this is `true` only for `builtins`, `def`, `lambda`. -```python +```yaksha macros!{ (metamacro twice (x) (+ (list x) (list x))) # since x is an expression (non evaluated), it can be converted to {} @@ -128,7 +128,7 @@ def main() -> int: ## Item 4 - Import macros from other files ✅ -```python +```yaksha import libs.macro as m macros!{ (enable_print) @@ -143,7 +143,7 @@ def main() -> int: Will be disabled by default. Can be enabled by a setting a flag in `yaksha.toml` or setting an environment variable `YAKSHA_SERIOUSLY_ENABLE_SHELL_EXEC` to `True`. -```python +```yaksha import libs.macro.shell as m def main() -> int: diff --git a/yaksha_proposals/0006_c2c.md b/yaksha_proposals/0006_c2c.md index 75252d2..9fd1ba2 100644 --- a/yaksha_proposals/0006_c2c.md +++ b/yaksha_proposals/0006_c2c.md @@ -20,7 +20,7 @@ Yaksha uses `C` as a final compilation target. This is to introduce a way to hav * structs, constants, unions, #define * We have built-ins such as `yk__sdsnew`, etc. -``` +```c #define yy__raylib_Music Music i32 yy__main() { @@ -36,7 +36,7 @@ i32 yy__main() { * Level 1 -> no keyword recognition * Recognizes normal operators, identifiers, #?? and comments * Discard comments -* Parse all with pre processor statements +* Parse all with pre-processor statements * Remove comments * Apply #defines -> until there are no changes present * AST based @@ -47,14 +47,14 @@ i32 yy__main() { ### Opt 01 - apply all #defines starting with `yy__` -* Implement a simple pre processor, however only capable of applying #defines that start with `yy__` +* Implement a simple pre-processor, however only capable of applying #defines that start with `yy__` * Opt 01.01 - Support non function like macros * Opt 01.02 - Support for function like macros ### Opt 02 - data types * Opt 02.01 - Push down support of integer data types from Yaksha compiler to iC99. -* Instead of having extra complexity at the compiler level for i32, etc like data types, we push it to iC99. +* Instead of having extra complexity at the compiler level for i32 ike data types, we push it to iC99. ### Opt 03 - clean up diff --git a/yaksha_proposals/0007_structs_revisited.md b/yaksha_proposals/0007_structs_revisited.md index 1d47e06..9ed06d8 100644 --- a/yaksha_proposals/0007_structs_revisited.md +++ b/yaksha_proposals/0007_structs_revisited.md @@ -5,7 +5,7 @@ Structures are at the moment can be created as follows -```python +```yaksha @onstack class Banana: color: int @@ -19,7 +19,7 @@ class Orange: ## Item 1 - Allocate single object on heap -```python +```yaksha a: Ptr[Banana] = make("Banana") b: Orange = make("Orange") @@ -27,7 +27,7 @@ b: Orange = make("Orange") ## Item 2 - Create structures or classes ✅ -```python +```yaksha a: Banana = Banana {color: YELLOW, origin: SRI_LANKA} ``` @@ -35,7 +35,7 @@ a: Banana = Banana {color: YELLOW, origin: SRI_LANKA} struct Banana a = (struct Banana){.color = YELLOW, .origin = SRI_LANKA}; ``` -```python +```yaksha b: Orange = Orange {color: ORANGE, origin: SOUTH_AFRICA} ``` @@ -49,7 +49,7 @@ struct Orange* b = _temp; ## Item 3 - Introduce `struct` keyword, desugar to `@onstack class` ✅ -```python +```yaksha struct Banana: color: int origin: int diff --git a/yaksha_proposals/0008_sugar_methods.md b/yaksha_proposals/0008_sugar_methods.md index c1b4011..fa9677d 100644 --- a/yaksha_proposals/0008_sugar_methods.md +++ b/yaksha_proposals/0008_sugar_methods.md @@ -5,7 +5,7 @@ ## Idea 01 - Arrow -```python +```yaksha struct Banana: color: int origin: int @@ -21,7 +21,7 @@ def main() -> int: ## Idea 02 - Dot ✅ -```python +```yaksha @onstack class Banana: color: int @@ -39,6 +39,6 @@ def main() -> int: If `@onstack` or `struct` then we need to use `Ptr[DataType]` otherwise `DataType` can be used. Going with `.` as it is already valid AST. Additionally, we can find the method during `type checking` and desugar it. -```python +```yaksha display(getref(b)) ``` \ No newline at end of file diff --git a/yaksha_proposals/0009_strings_revisited.md b/yaksha_proposals/0009_strings_revisited.md index 65d0822..81f5c33 100644 --- a/yaksha_proposals/0009_strings_revisited.md +++ b/yaksha_proposals/0009_strings_revisited.md @@ -184,7 +184,7 @@ def main() -> int: #### Comparison of `literal`s (`"a" == "a"`) --> `bool` ✅ -``` +```yaksha def main() -> int: println("a" == "a") # AC println("a" == "b") # AC @@ -195,7 +195,7 @@ def main() -> int: #### Comparison of mixed `str`, `sr` and `literal`s --> `bool` ✅ -``` +```yaksha def main() -> int: a: str = "a" # N b: sr = "oi" # A @@ -229,7 +229,7 @@ def main() -> int: #### Compare with None ✅ -``` +```yaksha def main() -> int: a: sr = "a" # N b: str = "b" # A diff --git a/yaksha_proposals/0011_http.md b/yaksha_proposals/0011_http.md index 463d7a3..f5a6161 100644 --- a/yaksha_proposals/0011_http.md +++ b/yaksha_proposals/0011_http.md @@ -15,7 +15,7 @@ We plan on using mongoose to implement this feature. Mongoose is a lightweight H Simple HTTP client will use following structure. This will be placed in a separate module called `http/client.yaka`. -```Yaksha +```yaksha # Mongoose mgr data structure @nativedefine("struct mg_mgr") stuct MongooseMgr: @@ -41,7 +41,7 @@ def request(method: sr, url: sr, data: sr, callback: Function[..]) -> HttpReques Usage of the client will be as follows: -```Yaksha +```yaksha import http.client as client def callback(...): # placeholder