Merge branch 'main' into DaphneDSL-Frame/Mat-Extension

.github/workflows/main.yml

@@ -45,9 +45,7 @@ jobs:
       shell: bash
 
     - name: Testing
-      run: |
-        mkdir --parents src/api/python/tmp
-        LD_LIBRARY_PATH=$PWD/lib:$LD_LIBRARY_PATH PATH=$PWD/bin:/usr/lib/llvm-10/bin:$PATH PYTHONPATH="$PYTHONPATH:$PWD/src/:/usr/lib/llvm-10/build/utils/lit/" bin/run_tests
+      run: LD_LIBRARY_PATH=$PWD/lib:$LD_LIBRARY_PATH PATH=$PWD/bin:/usr/lib/llvm-10/bin:$PATH DAPHNELIB_DIR_PATH=$PWD/lib PYTHONPATH="$PYTHONPATH:$PWD/src/api/python/:/usr/lib/llvm-10/build/utils/lit/" bin/run_tests
 
     - name: "List generated files"
       run: |

.gitignore

@@ -29,10 +29,10 @@ __pycache__/
 .clion.source.upload.marker
 
 # local test/dev scripts
-tmpdaphne.daphne
-*.daphne
-*.mlir
-*.log
+/tmpdaphne.daphne
+/*.daphne
+/*.mlir
+/*.log
 
 # cuda compiler
 compiler-debug-cuda.txt
@@ -61,7 +61,3 @@ profiler/
 precompiled-dependencies/
 /cmake*/
 /data
-
-# Allow .daphne and .mlir files in test/
-!test/**/*.mlir
-!test/**/*.daphne

CMakeLists.txt

@@ -182,6 +182,7 @@ add_subdirectory(src/compiler/inference)
 add_subdirectory(src/compiler/lowering)
 add_subdirectory(src/compiler/utils)
 add_subdirectory(src/parser)
+add_subdirectory(src/parser/catalog)
 add_subdirectory(src/parser/config)
 add_subdirectory(src/parser/metadata)
 add_subdirectory(src/runtime/distributed/proto)

UserConfig.json

@@ -1,4 +1,11 @@
 {
+    "matmul_vec_size_bits": 0,
+    "matmul_tile": false,
+    "matmul_use_fixed_tile_sizes": true,
+    "matmul_fixed_tile_sizes": [4, 4, 4, 4, 4],
+    "matmul_unroll_factor": 1,
+    "matmul_unroll_jam_factor": 4,
+    "matmul_num_vec_registers": 16,
     "use_cuda": false,
     "use_vectorized_exec": false,
     "use_obj_ref_mgnt": true,
@@ -19,7 +26,7 @@
     "taskPartitioningScheme": "STATIC",
     "numberOfThreads": -1,
     "minimumTaskSize": 1,
-    "library_paths": [],
+    "libdir": "{exedir}/../lib",
     "daphnedsl_import_paths": {},
     "logging": [
         { "log-level-limit": "ERROR" },

containers/README.md

@@ -60,7 +60,7 @@ The images mentioned above come with different tags optionally support accelerat
 * **< date >_< flavour >_< optional-version-string >_< OS-Version >**: This is the format of tags that indicate a daily/nightly
 build. E.g., <br /> ```2023-05-12_CUDA_12.1.1-cudnn8-runtime-ubuntu20.04``` identifies a **CUDA** image of CUDA version
 12.1.1 with *CUDNN 8** support that is based on Ubuntu 20.04 and was generated on the 12th of May 2023. 
-* For the currently imminent release 0.2 we will have a v0.2 tag. 
+* There are dedicated tags for releases (e.g., `v0.2`).
 * All images we provide at the moment are Ubuntu 20.04 based and run on the amd64 platform only. Developers frequently
 build for other OS and platform versions themselves. Contributions on the GitHub issue tracker are welcome.
 
@@ -131,16 +131,16 @@ work inside the container with remote development features of several popular ID
 **2. The password** of a running daphneeu/daphne-dev container can be retrieved (if it's been forgotten and scrolled out of sight)
 by searching the log output of the container.
    * First retrieve the container id:
-    ``` bash
-    $ docker ps
-    CONTAINER ID   IMAGE                             COMMAND                  CREATED       STATUS       PORTS     NAMES
-    99a5b6c85bbb   daphneeu/daphne-dev:latest_BASE   "/entrypoint-interac…"   2 hours ago   Up 2 hours   22/tcp    tender_mcclintock
-    ```
-    *  The password can subsequently be retrieved with 
-    ``` bash
-    $ docker logs 99a5b6c85bbb | grep password
-    Use docker-username with password Docker!4556 for SSH login
-    ```
+     ``` bash
+     $ docker ps
+     CONTAINER ID   IMAGE                             COMMAND                  CREATED       STATUS       PORTS     NAMES
+     99a5b6c85bbb   daphneeu/daphne-dev:latest_BASE   "/entrypoint-interac…"   2 hours ago   Up 2 hours   22/tcp    tender_mcclintock
+     ```
+   * The password can subsequently be retrieved with:
+     ``` bash
+     $ docker logs 99a5b6c85bbb | grep password
+     Use docker-username with password Docker!4556 for SSH login
+     ```
 
 
 

containers/build-containers.sh

@@ -152,4 +152,5 @@ FINAL_BASE_IMAGE=nvidia/cuda:$CUDA_TAG
 DAPHNE_BUILD_FLAGS="--mpi --cuda"
 build_daphne
 $USE_SUDO docker tag $IMAGE_REPO:$DAPHNE_TAG daphneeu/daphne:latest_${ARCH}_CUDA
+
 set +e

doc/DaphneDSL/Builtins.md

@@ -80,9 +80,8 @@ DaphneDSL's built-in functions can be categorized as follows:
 
 - **`seq`**`(from:scalar, to:scalar, inc:scalar)`
 
-    Generates a column *(n x 1)* matrix containing an arithmetic sequence of values starting at `from`, going through `to`, in increments of `inc`.
+    Generates a column matrix containing an arithmetic sequence of values starting at `from`, going through `to`, in increments of `inc`.
     Note that `from` may be greater than `to`, and `inc` may be negative.
-    If `inc` does not lead from `from` to `to`, the result will be an empty column matrix; however, `inc` must not be zero or NaN.
 
 ## Matrix/frame dimensions
 
@@ -145,7 +144,7 @@ The following built-in functions all follow the same scheme:
 ## Elementwise binary
 
 DaphneDSL supports various elementwise binary operations.
-Some of those can be used through *operators in infix notation*, e.g., `+`; and some through *built-in functions*.
+Some of those can be used through *operators in infix notation*, e.g., `+`; and some through *built-in functions*, e.g., `log()`.
 Some operations even support both, e.g., `pow(a, b)` and `a^b` have the same semantics.
 
 The built-in functions all follow the same scheme:
@@ -551,9 +550,10 @@ These must be provided in a separate [`.meta`-file](/doc/FileMetaDataFormat.md).
     The *(1 x m)* row-matrix `info` specifies the details (in the following, *d[j]* is short for `info[0, j]`):
 
     - If *d[j]* == -1, then the *j*-th column of `arg` will remain as it is.
-    - If *d[j]* >= 0, then the *j*-th column of `arg` will be encoded.
+    - If *d[j]* == 0, then the *j*-th column of `arg` will be omitted in the output.
+    - If *d[j]* > 0, then the *j*-th column of `arg` will be encoded to a vector of length *d[j]*.
 
-        More precisely, the *j*-th column of `arg` must contain only integral values in the range *[0, d[j] - 1]*, and will be replaced by *d[j]* columns containing only zeros and ones.
+        More precisely, if *d[j]* > 0 the *j*-th column of `arg` must contain only integral values in the range *[0, d[j] - 1]*, and will be replaced by *d[j]* columns containing only zeros and ones.
         For each row *i* in `arg`, the value in the `as.scalar(arg[i, j])`-th of those columns is set to 1, while all others are set to 0.
 
 - **`recode`**`(arg:matrix, orderPreserving:bool)`

doc/DaphneDSL/LanguageRef.md

@@ -107,9 +107,10 @@ Literals represent hard-coded values and can be of various data and value types:
 
 ##### Scalar literals
 
-**Integer literals** are specified in decimal notation and have the type `si64`.
+**Integer literals** are specified in decimal notation.
+By default, they have the type `si64`, but if the optional suffix `u` is appended, the type is `ui64`.
 
-*Examples*: `0`, `123`, `-456`
+*Examples*: `0`, `123`, `-456`, `18446744073709551615u`
 
 **Floating-point literals** are specified in decimal notation and have the type `f64`.
 Furthermore, the following literals stand for special floating-point values: `nan`, `inf`, `-inf`.
@@ -789,6 +790,27 @@ At call sites, a value of any type, or any value type, can be passed to an untyp
 As a consequence, an untyped function is compiled and specialized on demand according to the types at a call site.
 Consistently, the types of untyped return values are infered from the parameter types and operations.
 
+## Compiler Hints
+
+One of DAPHNE's strengths is its (WIP) ability to make various decisions on its own, e.g., regarding physical data representation (such as dense/sparse), physical operators (kernels), and data/operator placement (such as local/distributed, CPU/GPU/FPGA, computational storage).
+However, expert users may optionally provide hints to influence compiler decisions.
+This feature is useful for experimentation and in the context of DAPHNE's extensibility.
+For instance, a user could force the use of a certain custom kernel at a certain point in a larger DaphneDSL script to measure the impact of that custom kernel, even if the DAPHNE compiler would normally not choose that kernel in that situation.
+
+*The support for compiler hints is still experimental and it is currently not guaranteed that the DAPHNE compiler respects these hints.*
+
+### Kernel Hints
+
+Users can provide hints on the physical kernel that should be used for a specific occurrence of a DaphneDSL operation.
+So far, kernel hints are only supported for DaphneDSL built-in functions.
+Here, the name of the pre-compiled kernel function can optionally be attached to the name of the built-in function, separated by `::`.
+
+*Examples:*
+
+```r
+res = sum::my_custom_sum_kernel(X);
+```
+
 ## Example Scripts
 
 A few example DaphneDSL scripts can be found in:

doc/DaphneLib/APIRef.md

@@ -31,8 +31,11 @@ However, as the methods largely map to DaphneDSL built-in functions, you can fin
 
 **Importing data from other Python libraries:**
 
-- **`from_numpy`**`(mat: np.array, shared_memory=True) -> Matrix`
-- **`from_pandas`**`(df: pd.DataFrame) -> Frame`
+- **`from_numpy`**`(mat: np.array, shared_memory=True, verbose=False) -> Matrix`
+- **`from_pandas`**`(df: pd.DataFrame, shared_memory=True, verbose=False, keepIndex=False) -> Frame`
+- **`from_tensorflow`**`(tensor: tf.Tensor, shared_memory=True, verbose=False, return_shape=False) -> Matrix`
+- **`from_pytorch`**`(tensor: torch.Tensor, shared_memory=True, verbose=False, return_shape=False) -> Matrix`
+
 
 **Generating data in DAPHNE:**
 
@@ -48,6 +51,10 @@ However, as the methods largely map to DaphneDSL built-in functions, you can fin
 - **`readMatrix`**`(file:str) -> Matrix`
 - **`readFrame`**`(file:str) -> Frame`
 
+**Extended relational algebra:**
+
+- **`sql`**`(query) -> Frame`
+
 ## Building Complex Computations
 
 Complex computations can be built using Python operators (see [DaphneLib](/doc/DaphneLib/Overview.md)) and using DAPHNE matrix/frame/scalar methods.
@@ -159,6 +166,11 @@ In the following, we describe only the latter.
 - **`ncol`**`()`
 - **`ncell`**`()`
 
+**Frame label manipulation:**
+
+- **`setColLabels`**`(labels)`
+- **`setColLabelsPrefix`**`(prefix)`
+
 **Reorganization:**
 
 - **`cbind`**`(other)`
@@ -167,13 +179,19 @@ In the following, we describe only the latter.
 
 **Extended relational algebra:**
 
+- **`registerView`**`(table_name: str)`
 - **`cartesian`**`(other)`
+- **`innerJoin`**`(right_frame, left_on, right_on)`
 
 **Input/output:**
 
 - **`print`**`()`
 - **`write`**`(file: str)`
 
+**Conversions, casts, and copying:**
+
+- **`toMatrix`**`(value_type="f64") -> Matrix`
+
 ### `Scalar` API Reference
 
 **Elementwise unary:**