Updater fails to complete (step Delete old files) / Could not rmdir
#158
Comments
|
I have the same problem. Update progress stucks on OpenSUSE Tumbleweed with Apache php7.2 and MariaDB. I also tried to add some debugging lines to the I am wondering about why the updater.log file looks like this afterwards: Is it possible, that the unlink file fails, because there is no path given to the function and therefore the files are not deleted? This would furthermore lead to an error in deleting the directory l10n an the folder activity. So if I change to I get a bit further. The error is now: I have to add, that I didn't really configure my apache2. So maybe it is just a misconfiguration? |
|
I just did further investigations: The "LICENSE"-files which are in the subfolder of 3drparty are not deleted. This is why the directory deletion fails. If I delete these files manually, the updater proceeds a bit... The "LICENSE"-files are not recognized as a folder nor as a file! Is this the normal behavior? |
|
I think I have two "errors" here:
This let me use the web based updater. Maybe this problem occurs because I installed nextcloud via I now changed the installation method of nextcloud to the setup-nextcloud.php. With this method the problem has vanished on my system. |
|
I don't think that the LICENSE files are the reason for the issue, at least in my case. I rather think that on your part there is an issue with files without a file extension. Looking through the files the LICENSE files might be the only files without a file extension! Nevertheless, I tried your patches, but they did not solve the issue on my side. I then how ever again invested several hours to trace down the issue. Line 853 in 6609d9a Line 939 in 6609d9a respectively This lead to the following updater log: This clearly proves that the folder is not empty and therefore can not be @MorrisJobke and @LukasReschke can you please have a look on it? This is a really nasty bug! Side notes:
|
|
Could you log the Could you also check what is in those two folders after the updater runs? Is there anything left in those folders? |
|
What I did to install nextcloud: File structure before the update: After the update process has stopped working with the error: updater.log It seems, that symlinks cause the problem because the result for Line 840 in 6609d9a |
|
I don't know why this does not work for symlinks, but I think I know now, where the symlinks come from: |
|
@ggeorgg I do not see why there should be an issue with symlinks! Can you try to replace those symlinks with the real data and try again? |
|
@MorrisJobke See below my filesystem lists and the updater log: core/doc/admin/_images before updatesettings/l10n before updateUpdater log:core/doc/admin/_images after updatesettings/l10n after updateIn my nextcloud installation there seem not to be any symlinks (which anyway would be rater unusual since it also must install on a windows system). I can't see any pattern between the deleted and the not deleted files. They seem to be randomly selected. |
Also using a package to install nextcloud and then use the built-in updater feel wrong for me, because you then overwrite files that were installed by the package manager. I would advise against this approach and rather install via packages and stick to the packages or install manually and then use the built-in updater. Nevertheless the symlinks are weird. Maybe the iterator does not list symlinks and only lists files. Thanks for this hint 👍
Still weird, because the very same mechanism is used by the deletion code 🤔 @rullzer @icewind1991 Do you have an idea why the directory iterator somehow only lists part of the files? |
|
I think I might have found the culprit (or one of them)... First of all, the The first error always occurs on the folder Line 937 in 6609d9a After that fix, the deletion process runs much much longer without an issue... ... until it bangs its head on the path Now, looking at the lines after Line 917 in 6609d9a Line 899 in 6609d9a In case it contains something that triggers a match, that path gets skipped, meaning it and its children never gets deleted. And here we have a false positive! The mentioned path contains a folder 'data', and 'data' also is in the exclude list: What I do not understand is why this issue only fires very few systems. Line 924 in 6609d9a Line 928 in 6609d9a updater/index.php already got replaced by the new version.Is there any way I can enforce to use a modified tar.gz file? |
This sounds weird, because this folder is there also on other systems. Also the exclude list should only be checked against for the root folder. :/ This sounds super suspicious. |
|
And thanks for the detailed analysis. Could you create a PR with the before mentioned change to properly delete the subfolder so we could have a proper review of the code in context? Thanks |
Ah, indeed! Only the most top folder gets checked, I misinterpreted that. That's why it didn't make fully sense.
Yes, I will, but it might not be next week as I will be away. If you could answer the following questions that would help:
|
|
@MorrisJobke and @LukasReschke, do you have any tips for the questions in #158 (comment)? |
|
@MorrisJobke and @LukasReschke, I will not be able to investigate this further without some support from your side! |
|
I will have a look at the pending PR the next days. Sorry for the delay ... was busy with other stuff. |
|
OS: CentOS Linux release 7.5.1804 (Core) I'm running NC on CentOS and I had the same problem. The Updater complained about deleting README from Themes, AUTHORS(/var/www/html/nextcloud/updater/../AUTHORS), then /var/www/html/nextcloud/updater/../themes/example ...then a nonexistent symlink /var/www/html/nextcloud/updater/../lib/private/App/PlatformRepository.php ....for me kind of weird. I have on the machine SELinux active, so I checked the audit.log:
As you can see, SELinux was my problem. Simple solution it is to disable it for the update. |
|
@caco3 @MorrisJobke the preliminary findings of the above analysis never materialized in any code changes, did they? I still see the same behavior with 20.x. |
|
@marcelstoer I wanted to investigate it further but needed the support of the devs - which never happened (see above comments). It looks like as if it is an issue with certain server configuration and mainly (or only) occurres on a hostpoint.ch shared webspace. #!/bin/bash
cd /home/<MY USERNAME>/www
php -f nextcloud/occ app:list > nextcloud/apps.txt
wget -c https://download.nextcloud.com/server/releases/latest.tar.bz2
mv -f nextcloud nextcloud.old
echo "Extracting new version..."
tar -xvf latest.tar.bz2 nextcloud
cp -rf nextcloud.old/config nextcloud
cd nextcloud
php -f occ upgrade # => If "Command 'upgrade' is not defined". => check if config.php is missing, else load using webbrowser
cd ..
rm -f latest.tar.bz2
php -f nextcloud/occ app:list > nextcloud/apps.txt
diff nextcloud.old/apps.txt nextcloud/apps.txt # Manually compare apps versions
# manually remove old installation if everything is fine |
Same here. That's an interesting script, thanks! It's OT for this discussion but I wouldn't know how else to ask: the new installation won't contain any of the auxiliary apps not shipped with the server that you installed at some point, right? At what point do you (re-)install them? The comment states you keep |
I run the script when ever there is an update for the core system. Simple app updates I do through the web interface. That mostly works but sporadically fails ( |
|
@caco3 Amazing script! I went ahead and modified it a bit to be more adaptable to other type of setups and upgrade paths. Also it enables maintenance mode before upgrading: #!/bin/bash
# Inspired by https://github.com/nextcloud/updater/issues/158#issuecomment-829592036
# ATTENTION: This scripts assumes that you have your data directory outside the installation directory!
set -e
if [[ "$1" == '' ]]; then
echo "Please specify a version to update to!"
exit -1
fi
CNCU_DOWNLOAD_ARCHIVE_NAME="nextcloud-$1.tar.bz2"
CNCU_BASE_DIR="/home/<server>/www/"
CNCU_PUBLIC_DIR_NAME="nextcloud"
cd "${CNCU_BASE_DIR}"
php -f "${CNCU_PUBLIC_DIR_NAME}/occ" app:list > "${CNCU_PUBLIC_DIR_NAME}/apps.txt"
wget -c "https://download.nextcloud.com/server/releases/${CNCU_DOWNLOAD_ARCHIVE_NAME}"
php -f "${CNCU_PUBLIC_DIR_NAME}/occ" maintenance:mode --on
mv -f "${CNCU_PUBLIC_DIR_NAME}" "${CNCU_PUBLIC_DIR_NAME}.old"
echo "Extracting new version"
tar -xvf ${CNCU_DOWNLOAD_ARCHIVE_NAME}
cp -rf "${CNCU_PUBLIC_DIR_NAME}.old/config" nextcloud
if [[ "${CNCU_PUBLIC_DIR_NAME}" != "nextcloud" ]]; then
mv nextcloud "${CNCU_PUBLIC_DIR_NAME}"
fi
cd "${CNCU_PUBLIC_DIR_NAME}"
php -f occ upgrade
cd ..
rm -f "${CNCU_DOWNLOAD_ARCHIVE_NAME}"
php -f "${CNCU_PUBLIC_DIR_NAME}/occ" app:list > "${CNCU_PUBLIC_DIR_NAME}/apps.txt"
diff "${CNCU_PUBLIC_DIR_NAME}.old/apps.txt" "${CNCU_PUBLIC_DIR_NAME}/apps.txt" |
|
Hi @caco3 , thank you for your efforts.
Unfortunately, I still encounter the issue when updating from 27.1.2 to 27.1.3:
Files are owned by |
|
@EddyLB It seems that on your system it uses the phar file which is not used on my system. I do not know why, but I tried to pack a new phar file. |
The iterator gets gets messed up when modifying the directory. Thus we copy() instead of rename() and remove the whole folders/files afterwards in a separate iteration. See nextcloud#509 and nextcloud#158 for details. Signed-off-by: CaCO3 <[email protected]>
The iterator gets gets messed up when modifying the directory. Thus we copy() instead of rename() and remove the whole folders/files afterwards in a separate iteration. See nextcloud#509 and nextcloud#158 for details. Signed-off-by: CaCO3 <[email protected]>
The iterator gets gets messed up when modifying the directory. Thus we copy() instead of rename() and remove the whole folders/files afterwards in a separate iteration. See nextcloud#509 and nextcloud#158 for details. Signed-off-by: CaCO3 <[email protected]>
The iterator gets gets messed up when modifying the directory. Thus we copy() instead of rename() and remove the whole folders/files afterwards in a separate iteration. See nextcloud#509 and nextcloud#158 for details. Signed-off-by: CaCO3 <[email protected]>
|
Dear all As mentioned before, I created a fix which to my understanding solves the issue. So please support me in letting him know that I am not the only one wanting this fixed. CC @stjosh |
The iterator gets gets messed up when modifying the directory. Thus we copy() instead of rename() and remove the whole folders/files afterwards in a separate iteration. See nextcloud#509 and nextcloud#158 for details. Signed-off-by: CaCO3 <[email protected]>
The iterator gets gets messed up when modifying the directory. Thus we copy() instead of rename() and remove the whole folders/files afterwards in a separate iteration. See #509 and #158 for details. Signed-off-by: CaCO3 <[email protected]>
|
Solved with #515 |
|
I have a prototype that addresses #507 and #484 (and #397 when I adapt it). It prunes the excluded folders from the iterator entirely so there's no longer any performance impact from traversing these folders that are going to be excluded anyhow. It still needs some refinement / polish by me before submitting for review and more extensive testing. When/if what I'm working on gets merged, any excluded folders will truly no longer be included in the iterator (right now we drop excluded elements within the iterator rather than preventing them from getting into the iterator entirely - and this is the reason for the unpredictable memory use across environments from doing something like was proposed in #515). This is relevant to this issue here because this change means that, for example, Memory requirements would be fixed to that required to hold an array of the entire current installation folder excluding both the data folder (if located in the install folder) and all configured apps folders. This would presumably overcome the regression in #517 introduced by #515 (before it was reverted) as long as the current installation folder file contents (w/o data/apps) can be held in somewhere between 256M-512M of memory when stored as an array. Still more than the like ~2M required when using the iterator w/o an array conversion, but possibly acceptable. (Side note: I guess if we really were concerned about this additional memory consumption we could hide it behind a compatibility mode switch or something so that overall memory requirements don't increase for the like >=80% of environments where this isn't a problem). |
|
Thanks for picking this up. I am happy to test it, just let me know when you are ready. |
|
Every update for me hangs at "delete old file" very long. Stracing the process shows it does stuff for images in the It goes on for 30+ Minutes but finishes in the end since I run the update from CLI without a timeout. On every update. Right now on the update to 29. I guess this is the same problem? Or is this something different and I should open a new issue? |
joshtrichards
changed the title
Could not rmdir
|
When updating (Ubuntu VM, LAMP, CLI) from v29.0.0-23-g6a5f9ec to 30.0.1 i got the same error: [ ] Move new files in place ...PHP Warning: rmdir(/mnt/ncdata/data/updater-ocdff7clyqk1/downloads/nextcloud/core/doc/user/talk): Directory not empty in phar:///var/www/nextcloud/updater/updater.phar/lib/Updater.php on line 984 I checked that data directory (which is mounted on SMB server), and it is EMPTY! When i retry, it now tries to update from 30.0.1 (non-functioning) to 30.0.2. |
@expressrussian That's what the manual update process is for: https://docs.nextcloud.com/server/latest/admin_manual/maintenance/manual_upgrade.html
This is likely part of the problem here. That's a bit non-standard. |
|
I believe issue nextcloud/server#2737 is not properly solved yet in NC13.0.0.
See discussions in above issue.
Summary:
To trace it further down, I added the lines
after
in the function
recursiveDeletein updater/index.php.This gives the following updater.log (shortened) below.
The
recursiveDeleteseems to work fine for many many times, but then suddenly fails with the messageAgainst my initial thought, this seems not to be caused by a
rmdirfailure!updater.log
Server configuration detail
Operating system: FreeBSD 10.3-RELEASE-p24 FreeBSD 10.3-RELEASE-p24 #14 r326012M: Mon Nov 20 12:42:55 UTC 2017 [email protected]:/usr/obj/usr/src/sys/HOSTPOINT amd64
Webserver: Apache/2.4 (cgi-fcgi)
Database: mysql 10.1.26
PHP version: 5.6.33
Modules loaded: Core, date, ereg, libxml, pcre, Reflection, SPL, hash, session, cgi-fcgi, standard, recode, apcu, bcmath, bz2, calendar, ctype, curl, dba, dbase, dom, mbstring, fileinfo, filter, ftp, gd, geoip, gettext, gmp, gnupg, mysqlnd, iconv, imagick, imap, intl, jsmin, json, ldap, exif, mcrypt, mssql, mysql, mysqli, ncurses, odbc, openssl, pcntl, PDO, pgsql, posix, pspell, readline, SimpleXML, snmp, soap, sockets, sqlite3, ssh2, sysvmsg, sysvsem, sysvshm, tidy, tokenizer, xml, xmlwriter, yaml, zip, zlib, mailparse, mongodb, PDFlib, pdo_dblib, pdo_mysql, pdo_pgsql, pdo_sqlite, Phar, scrypt, syck, wddx, xmlreader, xmlrpc, xsl, mhash, apc, Zend OPcache
Nextcloud version: 13.0.0 - 13.0..10
Updated from an older Nextcloud/ownCloud or fresh install:
Where did you install Nextcloud from: updater
Signing status
Array
List of activated apps
Configuration (config/config.php)
The text was updated successfully, but these errors were encountered: