Iterator work overview

[joshtrichards]

Bugs

• Updater fails to complete (step Delete old files) / Could not rmdir #158
  • Only impacts certain environments.

Suspected to be same root cause:

The below are likely related to #158 (which can be confirmed once a solution is developed).

• core/shipped.json disappearing from prior updates leading to failures on the next update
  • Update failed with core/shipped.json is not available #131
  • NC updater 15.0.2 still fails on Delete old files (core/shipped.json is not available) #222
  • [Bug]: Updater fails deleting old files as core/shipped.json couldn't be deleted #434
• Other previously mysterious deleteOldFiles() situations
  • Error updater - Delete old files #249
• Likely some of the previously mysterious createBackup() situations
  • https://github.com/nextcloud/updater/issues?q=is%3Aopen+is%3Aissue+label%3A%22feature%3A+backups%22

Fixes/PRs:

• Prevent directory modifications when iterating #515 targeting Updater fails to complete (step Delete old files) / Could not rmdir #158
  • Note: reverted in Revert "Prevent directory modifications when iterating" #518 due to Out of memory with new directory iterator code #517
  • a new solution is required 😞

Enhancements

• Updater recurses into datadirectory even though it's later excluded in createBackup() #507
• Updater's should stop iterating through ALL files in deleteOldFiles() #397

The above are performance matters that impact all environments. The solution required for both of the above will in all likelihood be the same.

Fixes/PRs:

• Improve iterators  #516 (WIP/draft)

Cc: @caco3, @Altahrim, @blizzz, @kesselb, @come-nc

[joshtrichards]

Another suspected to be same root cause:

• Some moveNewVersionInPlace() situations:
  • Move new files in place: Could not rename ... #276
  • Possibly some in Update 12 -> 13 ; Stuck in step 10 : Move new files in place : Parsing response failed #156

[joshtrichards]

A preliminary update: #158 (comment)

[marcelstoer]

Could you merge the two "Fixes/PRs:" sections in the original description? Should the open bugs be closed in favor of referencing this issue?