Add script to install Craig locally (#197)

.gitignore

@@ -33,6 +33,7 @@ cook/*.macosx
 apps/download/downloads/
 
 # Database
+*.rdb
 *.db
 *.db-shm
 *.db-wal
@@ -51,3 +52,5 @@ nohup.out
 config.json
 apps/*/config/default.js
 .patreon-credentials.json
+install.config
+

README.md

@@ -1,89 +1,7 @@
 # What is Craig?
-Craig is a multi-track voice recorder for Discord. Craig is divided into three
-components: The actual recording bot, the web page (for downloading
-recordings), and the audio processing scripts.
-
-The recording bot is craig.js. It is a Node.js application written with
-the Eris Discord library. For historical reasons it uses the sharding manager
-from discord.js. Its dependencies can be installed with `npm install`. In
-principle the recording bot works fine without the other components, but the
-audio files it produce aren't really usable without processing.
-
-The web page is in PHP (yuck), but otherwise has essentially no dependencies.
-The "other formats" feature, which performs audio processing in the browser,
-requires ffmpeg.js. The patch for ffmpeg.js to build it with the required
-modules is in ffmpeg-js-craig.diff. ffmpeg.js should be built to
-ffmpeg-worker-craig.js.
-
-The audio processing scripts require the following command-line tools:
-
-- ffmpeg ( http://ffmpeg.org/ ) compiled with libopus support
-- flac ( https://xiph.org/flac/ )
-- oggenc ( https://xiph.org/vorbis/ )
-- opusenc ( http://opus-codec.org/ )
-- fdkaac ( https://github.com/nu774/fdkaac )
-- zip and unzip ( http://infozip.org/ )
-
-Various components of the audio processing scripts require compilation or
-translation:
-```sh
-cd cook
-for i in *.c; do gcc -O3 -o ${i%.c} $i; done
-for i in *.svg; do inkscape -o ${i%.svg}.png $i; done
-```
-
-The self extractors require compilation if you want them:
-```sh
-cd windows; make
-cd macosx; make
-```
-
-Finally, the avatar stuff requires ImageMagick.  
-
-If you want EnnuiCastr (browser-based recording) support, you must set  
-config.ennuicastr to the web address, which must be on the same host as Craig.  
-EnnuiCastr requires no further modification to work with Craig.  
-
-If you want Ennuizel (browser-based processing) support, you must make it  
-available at the location linked from the downloader (dl.php), and run ces.sh  
-in ennuicastr/server.  
-
-More information on Craig can be found at  
-https://craig.chat/  
+Craig is a multi-track voice recorder for Discord.
 
-
-Craig can integrate with other tools (largely also by Yahweasel) to provide  
-improved functionality.  
-
-EnnuiCastr ( https://github.com/Yahweasel/ennuicastr/ ) is a browser-based  
-audio capture tool which records more reliably than Discord, and can be used to  
-get higher-quality audio if config.ennuicastr is set to an appropriate URL.  
-
-Ennuizel ( https://github.com/Yahweasel/ennuizel/ ) is a browser-based audio  
-editor which can provide audio processing features too heavy to put on the  
-server. The 'ennuizel' directory here is the associated client plugin for  
-Ennuizel as well as the server component, only required when combining Ennuizel  
-with Craig.  
-
-
-No documentation is provided for running your own instances of Craig. A user  
-has kindly added an installation guide to the wiki on github at  
-https://github.com/Yahweasel/craig/wiki , but note that this is not maintained  
-by the author of Craig.  
-
-RUNNING CRAIG IS NOT LIKE RUNNING ANY OTHER BOT. MOST OF THE COMPLEXITY OF  
-CRAIG IS NOT IN THE BOT ITSELF, BUT IN THE AUDIO PROCESSING.  
-
-IF YOU ARE NOT FAMILIAR WITH C AND UNIX, DO NOT WASTE YOUR TIME OR MINE. IF YOU  
-DELUDE YOURSELF INTO THINKING YOU WILL HOST CRAIG ON WINDOWS, DO NOT WASTE YOUR  
-TIME OR MINE. CRAIG IS NOT MERELY BUILT FOR UNIX, CRAIG IS BUILT ON THE UNIX  
-PHILOSOPHY.  
-
-"THIS ONE TIME I RAN A LINUX VM" IS NOT SUFFICIENT. "I CAN USE PUTTY" WON'T CUT  
-IT. IF THE WORD "WINDOWS" DOESN'T MAKE YOUR SKIN CRAWL, YOU ARE NOT THE KIND OF  
-PERSON WHO WILL SUCCEED IN RUNNING CRAIG.  
-
-If you're familiar with C and Unix, it shouldn't be too difficult to get your  
-own Craig working. Yahweasel is generally happy to help if you have the  
-requisite background.  
-
+More information can be found here: https://craig.chat/  
+
+# Self-hosting/installation
+Detailed instructions on how to self-host Craig are located here: [SELFHOST.md](SELFHOST.md)

SELFHOST.md

@@ -0,0 +1,177 @@
+# Self-hosting/installation
+Craig can be installed and ran locally. Some use cases for this are:
+- Contributing to Craig's source code
+- Creating multiple instances of Craig
+
+## Pre-requisites/dependencies
+Craig can only be ran on a Linux machine. The installation has been tested on a fresh install of **Ubuntu 22.04**.
+
+The following `apt` packages will be automatically installed by the install script [install.sh](install.sh):
+
+```
+make inkscape ffmpeg flac fdkaac vorbis-tools opus-tools zip unzip lsb-release curl gpg redis postgresql sed coreutils build-essential 
+```
+
+
+## 1. Clone source code
+
+```
+git clone --recurse-submodules https://github.com/CraigChat/craig.git
+```
+
+## 2. Create a Discord Bot application
+We need to create a Discord Bot account for the Craig instance. 
+
+### Required steps
+
+1. Log into [Discord](https://discord.com/)
+2. Navigate to the [developer application page](https://discord.com/developers/applications)
+3. Click on `New Application` in the top-right
+4. Give your new Craig instance a name (e.g. *LocalCraig* or *MyCraigBot*)
+5. Accept terms of service and click `Create`
+6. Under `SETTINGS->General Information` take note and copy down the following:
+   - **APPLICATION ID** (corresponds to `DISCORD_APP_ID`)
+7. Under `SETTINGS->Bot` take note and copy down the following:
+   - **TOKEN** (corresponds to `DISCORD_BOT_TOKEN`)
+     - you might have to click `Reset Token` if this is hidden
+8. Under `SETTINGS->OAuth2->General` take note and copy down the following:
+   - **CLIENT ID** (corresponds to `CLIENT_ID`)
+   - **CLIENT SECRET** (correpsonds to `CLIENT_SECRET`)
+     - you might have to click `Reset Secret` if this is hidden
+9. Under `SETTINGS->OAuth2->General` click on `Add Redirect` and paste in the following URI: http://localhost:3000/api/login
+10. Save changes
+
+
+### Optional steps
+
+#### Upload an app icon for Craig
+
+1. Under `SETTINGS->General Information->APP ICON` click on the photo icon
+2. You can use the default one, which is located in the git repo: [apps/download/page/public/craig.png](apps/download/page/public/craig.png). I recommend changing it slightly so as not to get confused with the actual Craig bot
+
+#### Create a development guild (server) for Craig
+
+After installation, your self-hosted instance of Craig can be invited to any Discord server. There is an optional environment variable in [install.config.example](install.config.example) which sets the development server for synchronized slash commands: `DEVELOPMENT_GUILD_ID`. Running `yarn run sync:dev` in the main git repo will then only synchronize commands to that development sever, as opposed to all the servers your Craig bot has been invited to. This gives you a sandbox in which to test experimental slash commands.
+
+1. Follow [this guide](https://support.discord.com/hc/en-us/articles/204849977) to create a Discord server
+2. Follow [this guide](https://support.discord.com/hc/en-us/articles/206346498) to obtain your new server's ID
+
+## 3. Edit configuration files
+
+### Required configuration
+
+Copy the example config file in the main directory and rename it
+
+```sh
+cp ./install.config.example ./install.config
+```
+
+Edit the following environment variables in the newly created [install.config](install.config) file with values you obtained from the previous step:
+
+```
+DISCORD_BOT_TOKEN
+DISCORD_APP_ID
+CLIENT_ID
+CLIENT_SECRET
+```
+and optionally:
+
+```
+DEVELOPMENT_GUILD_ID
+```
+
+### Optional and advanced configuration
+
+There are additional configuration variables in [install.config.example](install.config.example), such as Craig's PostgreSQL database username and password, Patreon ID, etc.
+
+There are even more configuration options located in: [apps/bot/config/_default.js](apps/bot/config/_default.js) and [apps/tasks/config/_default.js](apps/tasks/config/_default.js).
+
+Changing some of these variables from their default values will break Craig, so be careful.
+
+## 4. Run install script
+
+Go into the main directory and run the following:
+
+```sh
+./install.sh
+```
+
+Note that the script will prompt for `sudo` privileges, in order to automatically install dependencies and configure PostgreSQL.
+
+The installation should take a while. Please make note of any errors or warnings. The install generates an output log located in the main directory: [install.log](install.log).
+
+## 5. Invite Craig to a server
+
+You can invite Craig to any Discord server by going to the following URL in your browser. Just make sure to replace `CLIENT_ID` in the URL with the actual value of your Discord Bot's client ID.
+
+https://discord.com/oauth2/authorize?client_id=CLIENT_ID&permissions=68176896&scope=bot%20applications.commands
+
+## 6. Try recording
+
+If everything went smoothly up to now, there should be a running instance of Craig on your local machine. Congratulations!
+
+You can try recording and using Craig's slash commands.
+
+## Important information
+
+### Main dashboard
+
+Located at: http://localhost:3000/login
+
+
+### HTTPS and localhost - can't go to download page
+
+Craig automatically serves the download pages with the `https://` protocol. For example, if you try to go to the download page after recording, the link will be of the form:
+
+https://localhost:5029/rec/RECORDING_ID
+
+Most if not all browsers won't serve this because `localhost` doesn't have a signed certificate for `https://`. You can simply change the protocol to `http://` by removing the `s` and then you will be able to access your recording download:
+
+http://localhost:5029/rec/RECORDING_ID
+
+### Restarting Craig after reboot
+
+By default, Craig will not automatically restart if you reboot your computer. You can re-run a modified version of [install.sh](install.sh) script so that the following steps are performed:
+
+- Make sure the Redis server is online
+- Make sure the PostgreSQL database is online
+- Start the application
+
+To speed up the restart (and avoid rebuilding Craig), you should comment out some unnecessary steps (function calls) in the install script:
+
+```sh
+# install_apt_packages
+# install_node
+start_redis
+start_postgresql
+# config_env
+# config_react
+# config_yarn
+# config_cook
+start_app
+```
+
+### Monitoring, starting, and stopping Craig
+
+Here are some useful commands for debugging the installation and for monitoring the Craig application. Make sure that you have sourced `nvm` and you are using the installed version of `node` in order to run `pm2` commands:
+
+```sh
+source ~/.nvm/nvm.sh
+nvm use node
+```
+
+#### Monitor processes
+
+`pm2 monit`
+
+#### Output process logs to a file
+
+`pm2 logs > pm2.log`
+
+#### Restart all processes
+
+`pm2 restart all`
+
+#### Kill all processes
+
+`pm2 stop all`

apps/download/api/src/api.ts

@@ -120,7 +120,7 @@ export async function start(): Promise<void> {
     return;
   });
 
-  const port = parseInt(process.env.API_PORT, 10) || 3000;
+  const port = parseInt(process.env.API_PORT, 10) || 5029;
   const host = process.env.API_HOST || 'localhost';
   const addr = await server.listen({ host, port });
   console.info(`Running webhook on ${addr}, env: ${process.env.NODE_ENV || 'development'}`);

install.config.example

@@ -0,0 +1,59 @@
+###################################################
+# Discord bot
+###################################################
+
+DISCORD_BOT_TOKEN=
+DISCORD_APP_ID=
+CLIENT_ID=
+CLIENT_SECRET=
+DEVELOPMENT_GUILD_ID=
+
+##################################################
+# Dashboard app environment
+##################################################
+
+PATREON_CLIENT_ID=test
+PATREON_CLIENT_SECRET=test
+PATRON_TIER_MAP={\"00001\":2,\"0002\":3}
+PATREON_WEBHOOK_SECRET=test
+GOOGLE_CLIENT_ID=test
+GOOGLE_CLIENT_SECRET=test
+MICROSOFT_CLIENT_ID=test
+MICROSOFT_CLIENT_SECRET=test
+APP_URI=http://localhost:3000
+JWT_SECRET=1234
+
+##################################################
+# Download app environment
+##################################################
+
+API_PORT=5029
+API_HOST=127.0.0.1
+API_HOMEPAGE=https://craig.chat/
+ENNUIZEL_BASE=https://ez.craig.horse/
+TRUST_PROXY=
+SENTRY_DSN=
+SENTRY_HOST=
+SENTRY_SAMPLE_RATE=1.0
+SENTRY_DSN_API=
+SENTRY_ENV=
+SENTRY_SAMPLE_RATE_API=1.0
+INFLUX_URL=
+INFLUX_TOKEN=
+INFLUX_ORG=
+INFLUX_BUCKET=
+SERVER_NAME=dev
+REDIS_HOST=
+REDIS_PORT=
+
+##################################################
+# Other
+##################################################
+
+NODE_VERSION="18.18.2"
+DATABASE_NAME="craig"
+POSTGRESQL_USER="$(whoami)"
+POSTGRESQL_PASSWORD="craig"
+POSTGRESQL_START_TIMEOUT_S=10
+REDIS_START_TIMEOUT_S=10
+DATABASE_URL=\"postgresql://$POSTGRESQL_USER:$POSTGRESQL_PASSWORD@localhost:5432/$DATABASE_NAME?schema=public\"