implement partial library scan (filescanner)

[whatdoineed2do]

Fixes #382, workaround to #1173

Extend library interface to provide a rescan path entry point and expose on json api. Accepts a path for scan - currently implemented in filescanner and allows for partial local library (re)scanning.

Could be future extended for RSS lib and spotify lib scanners, with RSS potentially forcing refresh of items under a server (ie a path being https://bbc.co.uk/rss/ to force rescan of these hosted RSS).

Uses existing filescanner.c's bulk_scan() function but extended to take a requested path otherwise use library directories as existing functionality.

[chme]

The last time I looked into implementing this, I ran into problems with the purge_cruft(starttime) deleting to many items. I may have missed something, but I don't see how you make sure, that only items that are under the rescan path are deleted. Additionally I remember symlinks to be a potential problem.

[whatdoineed2do]

Thanks for that - I was so tunnel visioned on the artwork piece i missed that rather large problem!

I've given this a wider stress test today and it appears to resolve the problem on deleting too many items by doing a negative ping on files, playlists and the directories (ie all items that are not the path being scanned are marked as good) and also the inotify table is not cleared down but only the path in question being cleared out.

[whatdoineed2do]

@ejurgensen @chme
Is there any comments for this PR?

I've had this on my own branch for a while and its been fine - I use it most on a slow (secondary) RPi 1 that has its (largish) media library on a remote NFS and configured not to rescan on startup - changes on NFS arent picked up by inotify and thus when I modify existing albums I can force a rescan of that specific path rather than force full libraary rescan.

[ejurgensen]

Sorry haven't looked at it. I will try to get around it.

[whatdoineed2do]

@chme - i rebased this against the recent changes in master for the partial scanning basd on scna type. with that work fresh in your memory are you able to give any comments on this PR?

[whatdoineed2do]

@ejurgensen - any thoughts on this PR?

[ejurgensen]

I'm a bit unsure about the intended use of this, could you share a bit on that? I mean, I get that it means that you can trigger a partial rescan, but how is the user supposed to use it? Is it the goal that advanced users can integrate in library update scripts via the api, or is it the goal that users in the web UI can navigate to the directory they want rescanned? Both seem a bit clunky to me, but maybe there isn't a better alternative.

[whatdoineed2do]

In my own branch, I've built the UI extensions to trigger rescan which i've kept out of this PR

But the use case is the latter you described and echo'd on #382 - large library mounted on NFS, I add a new album under the artist directory like above (/media/music is on NFS), because its NFS, inotify doesnt pick up the files dropped onto the remote NFS server and thus doesn't appear on the owntown server using NFS. With this, patch, I'd trigger a rescan on the artist directory which will find the new album i've added instead of scanning the full remote NFS library

This is a little pain for my ancient RPI1 - scanning my ~10k files takes about 2.5 mins rather than the few seconds it would for scanning the specific directory.

[ejurgensen]

Thanks for the elaboration, I will try to get around to looking at the change.

Just out of curiousity, have you tried something like this https://github.com/mhallin/notify-forwarder? That particular project seems like it is dead, but that way of solving the problem might have some merit.

[benediktkr]

i have successfully rebased the file-scan-dir-path from your whatdoineed2do/forked-daapd repo, rebased it off the current HEAD of he main branch first. works great and i can use the api to trigger rescans for a given path, thank you :)

In my own branch, I've built the UI extensions to trigger rescan which i've kept out of this PR

which branch is that? would be great to have it in the UI like i can see in your screenshot.

[whatdoineed2do]

@benediktkr - thanks for testing. It should work as intended; i've been using it since the original submission although there were concerns raised by others about accuracy of the rescans impacting existing items in db (i've not observed though)

In my own branch, I've built the UI extensions to trigger rescan which i've kept out of this PR

which branch is that? would be great to have it in the UI like i can see in your screenshot.

This might be messy as my tree (my master etc) have fixes that work for me but not been accepted into the upstream, that incls web work. If you can do a bit of manual cherry-picking and to rebuild the UI I can point you at the files of interest:

• dialog box for "update/rescan"- https://github.com/whatdoineed2do/forked-daapd/blob/6a1b8035402b93a4c87dfa33d75b95c9a3b6536f/web-src.v3/src/components/ModalDialogDirectory.vue#L65-L68
• interface to backend - https://github.com/whatdoineed2do/forked-daapd/blob/6a1b8035402b93a4c87dfa33d75b95c9a3b6536f/web-src.v3/src/webapi/index.js#L54-L60

[benediktkr]

I actually like the web ui changes youve made :) so im fine with using them.

slightly related, but ive been trying to get a dark theme working, using a css file that the Dark Reader addon for Firefox has generated. I havent gotten it to work though (but it works great in my browser using the addon itself).

if youre interested you can find it in my own builds of OwnTone at git.sudo.is/ben/builds/owntone.

[Bockiii]

In case you are still following this up @whatdoineed2do or @benediktkr : I would very much love something like this, even though my use case is different.

I have the issue that my downloader seems to still have "a finger" on the file when it puts it into the download folder (if I understand the code correctly it downloads the content into the mp3 and then updates the id3 tags etc, but by that time, the folder watcher has already retrieved the non-existing info). So I have to do a "rescan all with metadata" every time I add something, because it wont update the metadata for just that one album or so. My collection is 100k+ files so its getting a little stupid in regards to runtime and also file reads (nothing to wear out a drive than to metadata scan 100k files for no good reason once in a while).

So if you could get this into owntone, it would be awesome.

As for the usecase: I would argue that most people that run owntone are advanced users in some manner, so that adding an additional scan option like this wont make a difference.

Plex lets you do it too ;)

[ejurgensen]

@Bockiii that sounds like something isn't working as it should. Would you mind opening an issue with debug logging of when the downloader adds a track?

Btw if the downloader is a script you might be able to add a 'touch' command to avoid a full rescan.

[Bockiii]

The downloader is deemix and its development has been abandoned a while ago (but its still working until they fix the loophole). So I cant open a PR with the deemix dev :)

I will open another issue for this

[whatdoineed2do]

In case you are still following this up @whatdoineed2do or @benediktkr : I would very much love something like this, even though my use case is different.

I have the issue that my downloader seems to still have "a finger" on the file when it puts it into the download folder (if I understand the code correctly it downloads the content into the mp3 and then updates the id3 tags etc, but by that time, the folder watcher has already retrieved the non-existing info). So I have to do a "rescan all with metadata" every time I add something, because it wont update the metadata for just that one album or so. My collection is 100k+ files so its getting a little stupid in regards to runtime and also file reads (nothing to wear out a drive than to metadata scan 100k files for no good reason once in a while).

So if you could get this into owntone, it would be awesome.

As for the usecase: I would argue that most people that run owntone are advanced users in some manner, so that adding an additional scan option like this wont make a difference.

Plex lets you do it too ;)

@Bockiii
I've been using this feature in my own tree for a number of years now and I think its ready (its never destroyed my DB :) but this is awaiting PR feedback - I don't own the merge decision

However, its less of a push for me since, as I say, I have my own tree with own features I wanted that were not accepted into the upstream.