Replace make + Makefile with Python Invoke library + tasks.py ((also …

…fix installation under path with whitespaces) (#1239)

.github/workflows/codeql.yml → .github/workflows/codeql.yaml

@@ -55,6 +55,9 @@ jobs:
         # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
         # queries: security-extended,security-and-quality
 
+    # We need to install pip invoke library.
+    - name: pip3 install invoke
+      run: pip3 install invoke
 
     # Autobuild attempts to build any compiled languages  (C/C++, C#, Go, or Java).
     # If this step fails, then you should remove it and run the build manually (see below)

.github/workflows/mediasoup-worker-prebuild.yaml

@@ -54,6 +54,11 @@ jobs:
       - name: npm ci
         run: npm ci --ignore-scripts --omit=dev
 
+      # However we also need to install pip invoke manually (since
+      # `--ignore-scripts` prevented invoke from being installed).
+      - name: pip3 install invoke
+        run: pip3 install invoke
+
       - name: npm run worker:build
         run: npm run worker:build
 

.github/workflows/mediasoup-worker.yaml

@@ -64,25 +64,25 @@ jobs:
           restore-keys: |
             ${{ matrix.build.os }}-node-${{matrix.build.cc}}-
 
-      # We need to install some NPM production deps for npm-scripts.mjs to work.
-      - name: npm ci
-        run: npm ci --ignore-scripts --omit=dev
+      # We need to install pip invoke manually.
+      - name: pip3 install invoke
+        run: pip3 install invoke
 
-      # We need to install deps on worker/scripts/package.json.
-      - name: npm run install-worker-dev-tools
-        run: npm run install-worker-dev-tools
+      # We need to install npm deps of worker/scripts/package.json.
+      - name: npm ci --prefix worker/scripts
+        run: npm ci --prefix worker/scripts
         # TODO: Maybe fix this one day.
         if: runner.os != 'Windows'
 
-      - name: npm run lint:worker
-        run: npm run lint:worker
+      - name: invoke -r worker lint
+        run: invoke -r worker lint
         # TODO: Maybe fix this one day.
         if: runner.os != 'Windows'
 
-      - name: npm run worker:build
-        run: npm run worker:build
+      - name: invoke -r worker mediasoup-worker
+        run: invoke -r worker mediasoup-worker
 
-      - name: npm run test:worker
-        run: npm run test:worker
+      - name: invoke -r worker test
+        run: invoke -r worker test
         # TODO: Maybe fix this one day.
         if: runner.os != 'Windows'

.gitignore

@@ -20,10 +20,11 @@
 # Flatc generated files.
 /worker/include/FBS
 /worker/prebuild
-# Vistual Studio generated Stuff.
+# Python invoke.
+/worker/pip_invoke
+# Build artifacts.
 /worker/**/Debug
 /worker/**/Release
-/worker/.vs
 # clang-fuzzer stuff is too big.
 /worker/deps/clang-fuzzer
 # Ignore all fuzzer generated test inputs.

CHANGELOG.md

@@ -1,6 +1,11 @@
 # Changelog
 
 
+### NEXT
+
+* Replace make + Makefile with Python Invoke library + tasks.py (also fix installation under path with whitespaces) ([PR #1239](https://github.com/versatica/mediasoup/pull/1239)).
+
+
 ### 3.13.5
 
 * Fix RTCP SDES packet size calculation ([PR #1236](https://github.com/versatica/mediasoup/pull/1236) based on PR [PR #1234](https://github.com/versatica/mediasoup/pull/1234) by @ybybwdwd).

doc/Building.md

@@ -16,11 +16,11 @@ Compiles mediasoup TypeScript code (`lib` folder) JavaScript, places it into the
 
 ### `npm run worker:build`
 
-Builds the `mediasoup-worker` binary. It invokes `make`below.
+Builds the `mediasoup-worker` binary. It invokes `invoke`below.
 
 ### `npm run worker:prebuild`
 
-Creates a prebuilt of `mediasoup-worker` in the `worker/prebuild` folder.
+Creates a prebuilt of `mediasoup-worker` binary in the `worker/prebuild` folder.
 
 ### `npm run lint`
 
@@ -32,11 +32,23 @@ Validates mediasoup JavaScript files using [ESLint](https://eslint.org).
 
 ### `npm run lint:worker`
 
-Validates mediasoup-worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html). It invokes `make lint` below.
+Validates mediasoup worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html). It invokes `invoke lint` below.
 
 ### `npm run format:worker`
 
-Rewrites mediasoup-worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html). It invokes `make format` below.
+Rewrites mediasoup worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html). It invokes `invoke format` below.
+
+### `npm run flatc`
+
+Runs both `npm run flatc:node` and `npm run flatc:worker`.
+
+### `npm run flatc:node`
+
+Compiles [FlatBuffers](https://github.com/google/flatbuffers) `.fbs` files in `worker/fbs` to TypeScript code.
+
+### `npm run flatc:worker`
+
+Compiles [FlatBuffers](https://github.com/google/flatbuffers) `.fbs` files in `worker/fbs` to C++ code.
 
 ### `npm run test`
 
@@ -48,129 +60,136 @@ Runs [Jest](https://jestjs.io) test units located at `test/` folder.
 
 ### `npm run test:worker`
 
-Runs [Catch2](https://github.com/catchorg/Catch2) test units located at `worker/test/` folder. It invokes `make test` below.
+Runs [Catch2](https://github.com/catchorg/Catch2) test units located at `worker/test/` folder. It invokes `invoke test` below.
 
 ### `npm run coverage:node`
 
 Same as `npm run test:node` but it also opens a browser window with JavaScript coverage results.
 
-### `npm run install-deps:node`
+### `npm run release:check`
 
-Installs NPM dependencies and updates `package-lock.json`.
-
-### `npm run install-worker-dev-tools`
-
-Installs worker NPM packages needed for local development.
+Runs linters and tests in Node and C++ code.
 
 ## Rust
 
-The only special feature in Rust case is special environment variable `"KEEP_BUILD_ARTIFACTS", that when set to `1` will allow incremental recompilation of changed C++ sources during hacking on mediasoup.
+The only special feature in Rust case is special environment variable "KEEP_BUILD_ARTIFACTS", that when set to "1" will allow incremental recompilation of changed C++ sources during hacking on mediasoup.
+
 It is not necessary for normal usage of mediasoup as a dependency.
 
-## Makefile
+## Python Invoke and `tasks.py` file
+
+mediasoup uses Python [Invoke](https://www.pyinvoke.org/) library for managing and organizing tasks in the `worker` folder (mediasoup worker C++ subproject). `Invoke` is basically a replacemente of `make` + `Makefile` written in Python. mediasoup automatically installs `Invoke` in a local custom path during the installation process (in both Node and Rust) so the user doesn't need to worry about it.
 
-The `worker` folder contains a `Makefile` for the mediasoup-worker C++ subproject. It includes the following tasks:
+Tasks are defined in `worker/tasks.py`. For development purposes, developers or contributors can install `Invoke` using `pip3 install invoke` and run tasks below within the `worker` folder.
 
-### `make` or `make mediasoup-worker`
+See all the tasks by running `invoke --list` within the `worker` folder.
 
-Alias of ``make mediasoup-worker` below.
+*NOTE:* For some of these tasks to work, npm dependencies of `worker/scripts/package.json` must be installed:
 
-### `make meson-ninja`
+```bash
+npm ci --prefix worker/scripts
+```
+
+### `invoke` (default task)
 
-Installs `meson` and `ninja`.
+Alias of `invoke mediasoup-worker` task below.
 
-### `make clean`
+### `invoke meson-ninja`
+
+Installs `meson` and `ninja` into a local custom path.
+
+### `invoke clean`
 
 Cleans built objects and binaries.
 
-### `make clean-build`
+### `invoke clean-build`
 
 Cleans built objects and other artifacts, but keeps `mediasoup-worker` binary in place.
 
-### `make clean-pip`
+### `invoke clean-pip`
 
 Cleans `meson` and `ninja` installed in local prefix with pip.
 
-### `make clean-subprojects`
+### `invoke clean-subprojects`
 
 Cleans subprojects downloaded with Meson.
 
-### `make clean-all`
+### `invoke clean-all`
 
 Cleans built objects and binaries, `meson` and `ninja` installed in local prefix with pip and all subprojects downloaded with Meson.
 
-### `make update-wrap-file`
+### `invoke update-wrap-file [subproject]`
 
-Update the wrap file of a subproject with Meson. Usage example:
+Updates the wrap file of a subproject (those in `worker/subprojects` folder) with Meson. Usage example:
 
 ```bash
 cd worker
-make update-wrap-file SUBPROJECT=openssl
+invoke update-wrap-file openssl
 ```
 
-### `make mediasoup-worker`
+### `invoke mediasoup-worker`
 
 Builds the `mediasoup-worker` binary at `worker/out/Release/`.
 
 If the "MEDIASOUP_MAX_CORES" environment variable is set, the build process will use that number of CPU cores. Otherwise it will auto-detect the number of cores in the machine.
 
-"MEDIASOUP_BUILDTYPE" environment variable controls build types, `Release` and `Debug` are presets optimized for those use cases.
-Other build types are possible too, but they are not presets and will require "MESON_ARGS" use to customize build configuration.
+"MEDIASOUP_BUILDTYPE" environment variable controls build types, "Release" and "Debug" are presets optimized for those use cases. Other build types are possible too, but they are not presets and will require "MESON_ARGS" use to customize build configuration.
+
 Check the meaning of useful macros in the `worker/include/Logger.hpp` header file if you want to enable tracing or other debug information.
 
 Binary is built at `worker/out/MEDIASOUP_BUILDTYPE/build`.
 
-In order to instruct the mediasoup Node.js module to use the `Debug` mediasoup-worker binary, an environment variable must be set before running the Node.js application:
+In order to instruct the mediasoup Node.js module to use the "Debug"` `mediasoup-worker` binary, an environment variable must be set before running the Node.js application:
 
 ```bash
 MEDIASOUP_BUILDTYPE=Debug node myapp.js
 ```
 
-If the "MEDIASOUP_WORKER_BIN" environment variable is set (it must be an absolute file path), mediasoup will use the it as mediasoup-worker binary and **won't** compile the binary:
+If the "MEDIASOUP_WORKER_BIN" environment variable is set (it must be an absolute file path), mediasoup will use the it as `mediasoup-worker` binary and **won't** compile the binary:
 
 ```bash
 MEDIASOUP_WORKER_BIN="/home/xxx/src/foo/mediasoup-worker" node myapp.js
 ```
 
-### `make libmediasoup-worker`
+### `invoke libmediasoup-worker`
 
 Builds the `libmediasoup-worker` static library at `worker/out/Release/`.
 
 "MEDIASOUP_MAX_CORES"` and "MEDIASOUP_BUILDTYPE" environment variables from above still apply for static library build.
 
-### `make xcode`
+### `invoke xcode`
 
-Builds a Xcode project for the mediasoup-worker subproject.
+Builds a Xcode project for the mediasoup worker subproject.
 
-### `make lint`
+### `invoke lint`
 
-Validates mediasoup-worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html) and rules in `worker/.clang-format`.
+Validates mediasoup worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html) and rules in `worker/.clang-format`.
 
-### `make format`
+### `invoke format`
 
-Rewrites mediasoup-worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html).
+Rewrites mediasoup worker C++ files using [clang-format](https://clang.llvm.org/docs/ClangFormat.html).
 
-### `make test`
+### `invoke test`
 
 Builds and runs the `mediasoup-worker-test` binary at `worker/out/Release/` (or at `worker/out/Debug/` if the "MEDIASOUP_BUILDTYPE" environment variable is set to "Debug"), which uses [Catch2](https://github.com/catchorg/Catch2) to run test units located at `worker/test/` folder.
 
-### `make test-asan`
+### `invoke test-asan`
 
 Run test with Address Sanitizer.
 
-### `make tidy`
+### `invoke tidy`
 
 Runs [clang-tidy](http://clang.llvm.org/extra/clang-tidy/) and performs C++ code checks following `worker/.clang-tidy` rules.
 
 **Requirements:**
 
-* `make clean` and `make` must have been called first.
+* `invoke clean` and `invoke mediasoup-worker` must have been called first.
 * [PyYAML](https://pyyaml.org/) is required.
   - In OSX install it with `brew install libyaml` and `sudo easy_install-X.Y pyyaml`.
 
 "MEDIASOUP_TIDY_CHECKS" environment variable with a comma separated list of checks overrides the checks defined in `.clang-tidy` file.
 
-### `make fuzzer`
+### `invoke fuzzer`
 
 Builds the `mediasoup-worker-fuzzer` binary (which uses [libFuzzer](http://llvm.org/docs/LibFuzzer.html)) at `worker/out/Release/` (or at `worker/out/Debug/` if the "MEDIASOUP_BUILDTYPE" environment variable is set to "Debug").
 
@@ -182,33 +201,45 @@ Builds the `mediasoup-worker-fuzzer` binary (which uses [libFuzzer](http://llvm.
 
 Read the [Fuzzer](Fuzzer.md) documentation for detailed information.
 
-### `make fuzzer-run-all`
+### `invoke fuzzer-run-all`
 
 Runs all fuzzer cases.
 
-### `make docker`
+### `invoke docker`
 
 Builds a Linux Ubuntu Docker image with fuzzer capable clang++ and all dependencies to run mediasoup.
 
 **NOTE:** Before running this command, a specific version of Linux clang must be downloaded. To get it, run:
 
 ```bash
 cd worker
-./scripts/get-dep.sh clang-fuzzer
+scripts/get-dep.sh clang-fuzzer
 ```
 
-### `make docker-run`
+### `invoke docker-run`
 
-Runs a container of the Ubuntu Docker image created with `make docker`. It automatically executes a `bash` session in the `/mediasoup` directory, which is a Docker volume that points to the real `mediasoup` directory.
+Runs a container of the Ubuntu Docker image created with `invoke docker`. It automatically executes a `bash` session in the `/mediasoup` directory, which is a Docker volume that points to the real `mediasoup` directory.
 
-**NOTE:** To install and run mediasoup in the container, previous installation (if any) must be properly cleaned by entering the `worker` directory and running `make clean-all`.
+**NOTE:** To install and run mediasoup in the container, previous installation (if any) must be properly cleaned by entering the `worker` directory and running `invoke clean-all`.
 
-### `make docker-alpine`
+### `invoke docker-alpine`
 
 Builds a Linux Alpine Docker image with all dependencies to run mediasoup.
 
-### `make docker-alpine-run`
+### `invoke docker-alpine-run`
+
+Runs a container of the Alpine Docker image created with `invoke docker-alpine`. It automatically executes an `ash` session in the `/mediasoup` directory, which is a Docker volume that points to the real `mediasoup` directory.
+
+**NOTE:** To install and run mediasoup in the container, previous installation (if any) must be properly cleaned by entering the `worker` directory and running `invoke clean-all`.
+
+## Makefile
+
+The `worker` folder contains a `Makefile` file for the mediasoup worker C++ subproject. It acts as a proxy to the `Invoke` tasks defined in `tasks.py`. The `Makefile` file exists to help developers or contributors that prefer keep using `make` commands.
 
-Runs a container of the Alpine Docker image created with `make docker-alpine`. It automatically executes an `ash` session in the `/mediasoup` directory, which is a Docker volume that points to the real `mediasoup` directory.
+All tasks defined in `tasks.py` (see above) are available in `Makefile`. There is only one exception:
 
-**NOTE:** To install and run mediasoup in the container, previous installation (if any) must be properly cleaned by entering the `worker` directory and running `make clean-all`.
+- The `update-wrap-file` needs a "SUBPROJECT" environment variable indicating the subproject to update. Usage example:
+  ```bash
+  cd worker
+  make update-wrap-file SUBPROJECT=openssl
+  ```