Merge pull request #19 from shayaantx/feature/matrix

Feature/matrix - Add matrix chat client support

This PR contains many fixes/changes:

    Addition of matrix client support to botdarr, i.e., https://github.com/vector-im/element-web & https://github.com/matrix-org/synapse
    Removed cancel/lookup torrents commands (they were implemented partially early on - so they are buggy)
    All commands run in their own thread pool (10 threads)
    All api requests to radarr/sonarr/lidarr have timeouts of 5 seconds (which should be more than enough)
    Can blacklist content from showing up from apis, using new property "existing-item-paths-blacklist"
    Changed default command prefix from / to !
    I changed above prefix because I keep finding inherent functionality with / in many of the chat clients
    Changed default notification time for downloads from 1 hour to 5 minute

Matrix client details:

    The new client supports all the existing commands (for radarr, sonarr, lidarr)
    Its over HTTP only so make sure you don't use this specific impl over the internet (if there is a big need for https, let me know and I'll add it sooner than later)
    We use the preview url api to show images instead of relying on matrix client/servers to preview display the url for us. From my own testing, url previews by itself doesn't work well or consistently (for whatever reason). So we use the preview url api to get mxc urls instead (i.e., basically uploading the image to synapse and getting back a safe mxc url). This works on the mobile/desktop clients I've tried.
    I only tested against element web (and desktop) client variants. On ios and android
    The matrix client expects a valid room id in the config and will join it on start.
    We use the /sync api + long polling to get messages and process them as commands.

README.md

@@ -3,7 +3,7 @@
 
 # Summary
 
-Made this simple slack/discord/telegram bot so I could access radarr, sonarr, and lidarr all from a multiple slack/discord/telegram channels without a UI/server.
+Made this simple slack/discord/telegram/matrix bot so I could access radarr, sonarr, and lidarr all from a multiple slack/discord/telegram/matrix channels without a UI/server.
 
 <br/>
 
@@ -20,6 +20,7 @@ Made this simple slack/discord/telegram bot so I could access radarr, sonarr, an
 - [x] Discord
 - [x] Slack
 - [x] Telegram
+- [x] Matrix
 
 ## Currently Supported Feature's
 
@@ -34,11 +35,12 @@ Made this simple slack/discord/telegram bot so I could access radarr, sonarr, an
 - [x] Configurable value for url base for radarr, sonarr, and lidarr
 - [x] Lookup torrents for movies and force download
 - [x] (discord/slack only) Thumbs up reaction will add search results 
-- [x] User requests audited to local database
+- [x] User requests audited to local database\
+- [x] Blacklist content by paths from showing up in searches
 - [ ] Cancel/blacklist existing downloads
 - [ ] Episode/season search
 - [ ] Album/song search
-- [ ] Run bot in mode where all 3 chat clients work in same process (right now you would need 3 separate processes/containers)
+- [ ] Run bot in mode where all 4 chat clients work in same process (right now you would need 4 separate processes/containers)
 
 ## Discord Bot Installation
 
@@ -54,6 +56,10 @@ See https://github.com/shayaantx/botdarr/wiki/Install-Slack-Bot
 
 See https://github.com/shayaantx/botdarr/wiki/Install-Telegram-Bot
 
+## Matrix bot installation
+
+See https://github.com/shayaantx/botdarr/wiki/Install-Matrix-Bot
+
 ## Jar installation/Configuration
 
 1. Get latest copy of botdarr botdarr-release.jar
@@ -69,26 +75,35 @@ Radarr seems to return a 200 http code, not actually add the movie, and return j
 the api url with your radarr url base. So MAKE SURE you account for this in your config/setup.
 ```
 # your discord bot token
-discord-token=
+#discord-token=
 # the discord channel(s) you want the bot installed on
-discord-channels=
+#discord-channels=
 
 # Your slack bot oauth authentication token
-slack-bot-token=
+#slack-bot-token=
 # Your slack user oauth authentication token
-slack-user-token=
+#slack-user-token=
 # the slack channel(s) you want the bot installed on
-slack-channels=
+#slack-channels=
 
 # Your telegram bot token
-telegram-token=
+#telegram-token=
 # Your actual telegram channels your bot can respond in
 # this should be a list containing the name and id of the channel, i.e., CHANNEL_NAME:CHANNEL_ID
 # to get the channel id, right click any post in private channel and copy post link
 # you should see something like this, https://t.me/c/1408146664/63
 # the id is between c/<id>/<postId>
-# example: plex-channel1:id1,plex-channel2:id2
-telegram-private-channels=
+# example: channel1:id1,channel2:id2
+#telegram-private-channels=
+
+# Your matrix bot user
+#matrix-username=
+# Your matrix bot password
+#matrix-password=
+# The room your matrix bot will send messages in
+#matrix-room=
+# The url of your homeserver
+#matrix-home-server-url=
 
 # your radarr url (i.e., http://SOME-IP:SOME-PORT)
 radarr-url=
@@ -145,9 +160,13 @@ lidarr-url-base=
 # If you set this to any value less than 0, the bot won't startup
 #max-results-to-show=20
 
-# The command prefix (default is /)
+# The command prefix (default is !)
 # Any prefix is allowed (but I haven't tested every single prefix in every client)
-command-prefix=/
+command-prefix=!
+
+# If you want content to NOT appear in searches against your library, you can list blacklisted paths here
+# in comma delimited form, and they will be ignored when building up responses
+#existing-item-paths-blacklist=
 ```
 
 1. Run the jar using java
@@ -187,20 +206,18 @@ botdarr:
 
 ## Usage
 
-* Type /help in your configured chat client to get information about commands and what is supported
+* Type !help in your configured chat client to get information about commands and what is supported
 * Notifications will appear indicating the current downloads (based on your configuration for max downloads), their status, and their time remaining.
-* When you search for content (i.e., /movie title add History of Dumb People) if too many results are returned you will be presented with multiple results. You can either use the thumbs up reaction (in discord or slack) or copy the add command (which will be embedded in the result) into the chat client.
-
-![](https://raw.githubusercontent.com/shayaantx/botdarr/development/images/search-results.png)
-* The success of the bot depends a lot on how diverse your trackers you use in radarr, sonarr, lidarr and your quality profiles. If you have a trackers with little content or very restrictive quality profiles, a lot of content will never actually get added. The bot can't do anything about this.
+* When you search for content (i.e., !movie title add History of Dumb People) if too many results are returned you will be presented with multiple results. You can either use the thumbs up reaction (in discord or slack) or copy the add command (which will be embedded in the result) into the chat client.
 * Example commands:
-  * /movie title add Lion Fling
-  * /show title add One Fliece
-  * /movie find new zombies
-  * /artist find new Linkin Flarp
-  * /movie downloads
-  * /show downloads
-  * /help
-  * /shows help
-  * /movies help
+  * !movie title add Lion Fling
+  * !show title add One Fliece
+  * !movie find new zombies
+  * !artist find new Linkin Flarp
+  * !movie downloads
+  * !show downloads
+  * !help
+  * !shows help
+  * !movies help
+* The default command prefix is !. I chose ! because / (original command prefix) is commonly used by many chat clients and has existing functionality with it that leads to some commands not working nicely.
 <br/>

sample.properties

@@ -1,24 +1,33 @@
 # Your bot token goes here (don't share)
-discord-token=
+#discord-token=
 # The actual discord channel(s) the bot lives in
-discord-channels=
+#discord-channels=
 
 # Your slack bot oauth authentication token
-slack-bot-token=
+#slack-bot-token=
 # Your slack user oauth authentication token
-slack-user-token=
+#slack-user-token=
 # The actual slack channel(s) you want to post slack messages to
-slack-channels=
+#slack-channels=
 
 # Your telegram bot token
-telegram-token=
+#telegram-token=
 # Your actual telegram channels your bot can respond in
 # this should be a list containing the name and id of the channel, i.e., CHANNEL_NAME:CHANNEL_ID
 # to get the channel id, right click any post in private channel and copy post link
 # you should see something like this, https://t.me/c/1408146664/63
 # the id is between c/<id>/<postId>
 # example: plex-channel1:id1,plex-channel2:id2
-telegram-private-channels=
+#telegram-private-channels=
+
+# Your matrix bot user
+#matrix-username=
+# Your matrix bot password
+#matrix-password=
+# The room your matrix bot will send messages in
+#matrix-room=
+# The url of your homeserver
+#matrix-home-server-url=
 
 # Your various media tool urls/keys go here
 radarr-url=
@@ -67,6 +76,10 @@ max-downloads-to-show=20
 # If you set this to any value less than 0, the bot won't startup
 max-results-to-show=20
 
-# The command prefix (default is /)
+# The command prefix (default is !)
 # Any prefix is allowed (but I haven't tested every single prefix in every client)
-command-prefix=/
+command-prefix=!
+
+# If you want content to NOT appear in searches against your library, you can list blacklisted paths here
+# in comma delimited form, and they will be ignored when building up responses
+existing-item-paths-blacklist=

src/main/java/com/botdarr/Config.java

@@ -7,9 +7,7 @@
 
 import java.io.FileInputStream;
 import java.io.InputStream;
-import java.util.Arrays;
-import java.util.Comparator;
-import java.util.Properties;
+import java.util.*;
 import java.util.stream.Collectors;
 
 public class Config {
@@ -84,6 +82,12 @@ private Config() {
       if (!Strings.isEmpty(configuredPrefix)  && configuredPrefix.length() > 1) {
         throw new RuntimeException("Command prefix must be a single character");
       }
+      if (chatClientType == ChatClientType.SLACK && configuredPrefix.equals("/")) {
+        throw new RuntimeException("Cannot use / command prefix in slack since /help command was deprecated by slack");
+      }
+      if (chatClientType == ChatClientType.MATRIX && configuredPrefix.equals("/")) {
+        throw new RuntimeException("Cannot use / command prefix in matrix since /help command is used by element by default");
+      }
     } catch (Exception ex) {
       LOGGER.error("Error loading properties file", ex);
       throw new RuntimeException(ex);
@@ -110,7 +114,35 @@ public static ChatClientType getChatClientType() {
     return getConfig().chatClientType;
   }
 
+  public static List<String> getExistingItemBlacklistPaths() {
+    String paths = getProperty(Constants.EXISTING_ITEMS_PATHS_BLACKLIST);
+    if (paths != null && paths.contains(",")) {
+      return Arrays.asList(paths.split(","));
+    }
+    return new ArrayList<String>() {{add(paths);}};
+  }
+
   public static final class Constants {
+    /**
+     * The matrix bot username
+     */
+    public static final String MATRIX_USERNAME = "matrix-username";
+
+    /**
+     * The matrix bot password
+     */
+    public static final String MATRIX_PASSWORD = "matrix-password";
+
+    /**
+     * The matrix room for the bot
+     */
+    public static final String MATRIX_ROOM = "matrix-room";
+
+    /**
+     * The matrix home server for the bot
+     */
+    public static final String MATRIX_HOME_SERVER = "matrix-home-server-url";
+
     /**
      * The telegram auth token
      */
@@ -267,6 +299,11 @@ public static final class Constants {
      * The prefix for all commands
      */
     public static final String COMMAND_PREFIX = "command-prefix";
+
+    /**
+     * The paths of items to blacklist from searches
+     */
+    public static final String EXISTING_ITEMS_PATHS_BLACKLIST = "existing-item-paths-blacklist";
   }
 
   private static String propertiesPath = "config/properties";

src/main/java/com/botdarr/api/CacheContentStrategy.java

@@ -21,7 +21,7 @@ public CacheContentStrategy(Api api, String url) {
 
   public void cacheData() {
     List<Z> itemsAddedUpdated = new ArrayList<>();
-    ConnectionHelper.makeGetRequest(this.api, this.url, new ConnectionHelper.SimpleEntityResponseHandler<T>() {
+    ConnectionHelper.makeGetRequest(this.api, this.url, new ConnectionHelper.SimpleEntityResponseHandler<List<T>>() {
       @Override
       public List<T> onSuccess(String response) throws Exception {
         JsonParser parser = new JsonParser();

src/main/java/com/botdarr/api/LookupStrategy.java

@@ -19,14 +19,22 @@ public LookupStrategy(ChatClientResponseBuilder<? extends ChatClientResponse> ch
   public abstract T lookupExistingItem(T lookupItem);
   public abstract List<T> lookup(String searchTerm) throws Exception;
   public abstract ChatClientResponse getNewOrExistingItem(T lookupItem, T existingItem, boolean findNew);
+  public abstract boolean isPathBlacklisted(T item);
 
   public List<ChatClientResponse> lookup(String search, boolean findNew) {
     try {
       List<ChatClientResponse> responses = new ArrayList<>();
       List<T> lookupItems = lookup(search);
+      if (lookupItems == null) {
+        return Arrays.asList(chatClientResponseBuilder.createErrorMessage("Something failed during lookup for search term=" + search));
+      }
       for (T lookupItem : lookupItems) {
         T existingItem = lookupExistingItem(lookupItem);
         boolean isExistingItem = existingItem != null;
+        if (isExistingItem && isPathBlacklisted(existingItem)) {
+          //skip any items that have blacklisted paths
+          continue;
+        }
         boolean skip = findNew ? isExistingItem : !isExistingItem;
         if (skip) {
           continue;

src/main/java/com/botdarr/api/lidarr/LidarrApi.java

@@ -2,6 +2,7 @@
 
 import com.botdarr.Config;
 import com.botdarr.api.*;
+import com.botdarr.api.sonarr.SonarrShow;
 import com.botdarr.api.sonarr.SonarrUrls;
 import com.botdarr.clients.ChatClient;
 import com.botdarr.clients.ChatClientResponse;
@@ -65,7 +66,7 @@ public void deleteFromCache(List<String> profilesAddUpdated) {
 
       @Override
       public List<LidarrQualityProfile> getProfiles() {
-        return ConnectionHelper.makeGetRequest(LidarrApi.this, LidarrUrls.PROFILE, new ConnectionHelper.SimpleEntityResponseHandler<LidarrQualityProfile>() {
+        return ConnectionHelper.makeGetRequest(LidarrApi.this, LidarrUrls.PROFILE, new ConnectionHelper.SimpleEntityResponseHandler<List<LidarrQualityProfile>>() {
           @Override
           public List<LidarrQualityProfile> onSuccess(String response) {
             List<LidarrQualityProfile> lidarrQualityProfiles = new ArrayList<>();
@@ -95,7 +96,7 @@ public void deleteFromCache(List<String> profilesAddUpdated) {
 
       @Override
       public List<LidarrMetadataProfile> getProfiles() {
-        return ConnectionHelper.makeGetRequest(LidarrApi.this, LidarrUrls.METADATA_PROFILE, new ConnectionHelper.SimpleEntityResponseHandler<LidarrMetadataProfile>() {
+        return ConnectionHelper.makeGetRequest(LidarrApi.this, LidarrUrls.METADATA_PROFILE, new ConnectionHelper.SimpleEntityResponseHandler<List<LidarrMetadataProfile>>() {
           @Override
           public List<LidarrMetadataProfile> onSuccess(String response) {
             List<LidarrMetadataProfile> lidarrMetadataProfiles = new ArrayList<>();
@@ -164,6 +165,16 @@ public List<LidarrArtist> lookup(String searchTerm) throws Exception {
       public ChatClientResponse getNewOrExistingItem(LidarrArtist lookupItem, LidarrArtist existingItem, boolean findNew) {
         return chatClientResponseBuilder.getNewOrExistingArtist(lookupItem, existingItem, findNew);
       }
+
+      @Override
+      public boolean isPathBlacklisted(LidarrArtist item) {
+        for (String path : Config.getExistingItemBlacklistPaths()) {
+          if (item.getPath() != null && item.getPath().startsWith(path)) {
+            return true;
+          }
+        }
+        return false;
+      }
     }.lookup(search, findNew);
   }
 
@@ -226,7 +237,7 @@ public List<ChatClientResponse> onSuccess(String response) {
 
   private List<LidarrArtist> lookupArtists(String search) throws Exception {
     return ConnectionHelper.makeGetRequest(this, LidarrUrls.LOOKUP_ARTISTS, "&term=" + URLEncoder.encode(search, "UTF-8"),
-      new ConnectionHelper.SimpleEntityResponseHandler<LidarrArtist>() {
+      new ConnectionHelper.SimpleEntityResponseHandler<List<LidarrArtist>>() {
         @Override
         public List<LidarrArtist> onSuccess(String response) {
           List<LidarrArtist> artists = new ArrayList<>();