[Bug]: Very slow access to apps.nextcloud.com/api/v1/apps.json for checking app updates

[hammeractual]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Often to always failed when checking for Apps Update caused by very slow access to https://apps.nextcloud.com/api/v1/apps.json
Connection speed on the Client and on the Server seems normal, but when accessing https://apps.nextcloud.com/api/v1/apps.json either from the server or from the client is pretty slow (mostly failed to load) for json file to be loaded completely.

Steps to reproduce

1. Login
2. Go to Admin Page
3. Overview Menu
4. Never finish "Checking apps for compatible versions"

Expected behavior

Checking update for apps completed successfully.

Installation method

Community Manual installation with Archive

Nextcloud Server version

28

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 22.1 to 22.2)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json
	GET /ocs/v2.php/apps/updatenotification/api/v1/applist/28.0.5.1
	from x.x.x.x by e2cd37b6-6ebf-103d-84c2-addc165358a8 at 3 Mei 2024, 09.09.50

Additional info

No response

[bigcat88]

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

[hammeractual]

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

Could you please tell the infrastructure division/department to check/troubleshoot the issue?

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

[bigcat88]

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today.
there was no peak load in the appstore today, there shouldn't have been any slowdowns

[hammeractual]

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today. there was no peak load in the appstore today, there shouldn't have been any slowdowns

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

[joshtrichards]

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$ 

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

#40082 has a manual approach for doing so. Also it's tracking the feature request.

[hammeractual]

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$ 

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

#40082 has a manual approach for doing so. Also it's tracking the feature request.

Thanks for the info

Here is mine.

Meanwhile, the ping result is pretty good to that address.

[bigcat88]

Here is mine.

I was wrong.
It seems that this is not a problem with the appstore code itself, since during high loads the appstore simply does not serve the file at all, but if it has already started serving, it does it quickly, because the file itself is given by NGINX(reverse proxy).

[joshtrichards]

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

[hammeractual]

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

Here is mine

Quite fast while curl to another server/address.

[hammeractual]

Any updates guys? :'(

I've tried to increase timeout value in Fetcher.php to 1800 seconds, but no success.

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 1800000 milliseconds with 3661824 out of 6421307 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json GET /settings/apps from x.x.x.x by e2cd37b6-6ebf-103d-84c2-addc165358a8 at 7 Mei 2024, 17.14.29

[joshtrichards]

Any updates guys? :'(

I think the challenge is we're having a hard time reproducing this and there doesn't seem be an explanation for why your environment is experiencing this.

It's tough to fix without a clear (or even suspected possible) cause. :-)

[joshtrichards]

P.S. Is your Server @ Hetzner?

You may have missed this question. I asked because your ping times are ~1-2 ms.

At the time I was taking a shot in the dark that maybe there is something unusual about Hetzner's internal client-to-client connectivity. Today I ran across a note in the Portainer Knowledge Base while trying to help out a user on the forums that suggests Hetzner's private networking uses an MTU of 1450. Got me thinking about this again.

[the-djmaze]

Not only that. Try to load https://apps.nextcloud.com/apps/mail
It takes 1 minute and 13 seconds for TTFB!

Ok, it is hosted at the terrible Hetzner, but this is not only Hetzner at fault.

[briank-git]

Same issue with our Nextcloud instance ver. 30.0.2. It's located on the west coast of Canada in a datacenter with 1 gigabit connection, not sure if that would make a difference.

It seems to have gotten worse over the last two weeks as I was able to open the app store after several tries before, now it times out consistently.

Here's the output of the curl command:

curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15.4M    0 15.4M    0     0  61761      0 --:--:--  0:04:23 --:--:-- 57628

ping:

PING apps.nextcloud.com (65.21.231.50) 56(84) bytes of data.
64 bytes from s16.nextcloud.com (65.21.231.50): icmp_seq=1 ttl=46 time=172 ms
64 bytes from s16.nextcloud.com (65.21.231.50): icmp_seq=2 ttl=46 time=172 ms
64 bytes from s16.nextcloud.com (65.21.231.50): icmp_seq=3 ttl=46 time=173 ms
64 bytes from s16.nextcloud.com (65.21.231.50): icmp_seq=4 ttl=46 time=174 ms
64 bytes from s16.nextcloud.com (65.21.231.50): icmp_seq=5 ttl=46 time=172 ms
64 bytes from s16.nextcloud.com (65.21.231.50): icmp_seq=6 ttl=46 time=172 ms

[rwv37]

I just installed Nextcloud a few days ago, and this (or something like this) is happening to me too. Over five minutes for under 16 MB.

$ curl https://apps.nextcloud.com/api/v1/apps.json > fetchapps
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed 
100 15.7M    0 15.7M    0     0  51756      0 --:--:--  0:05:19 --:--:-- 29259
$

This causes the app store page to be completely unusable, due to always timing out. From nextcloud.log:

 ConnectException cURL error 28: Operation timed out after 60004 milliseconds with 3133106 out of 7155704 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json
Failed to connect to the app store 

I understand I can (or at least can try) to increase the time limit, but even so, at five minutes per load it won't be terribly usable anyway.

It seems like this is obviously not a problem with my Nextcloud configuration, as the curl command is unrelated to it. Also, I can get much larger files (from other websites) much faster, both via curl and via a web browser. So, I think it should be noted that in general nextcloud.com is very very slow for me; for example opening up https://help.nextcloud.com/ in my browser just took me something like 15 to 20 seconds.

I should also note that I strongly believe this is not some temporary condition:

(1) The app store has been unusable for me ever since I installed Nextcloud, days ago. Up to tonight I've been troubleshooting other issues, and just started investigating this one, but I was checking the app store page every once in a while, and it always failed. Looking at the logs now, it was always due to timing out on the download.

(2) I've been vaguely interested in Nextcloud for years, and my memory is that the site itself has always seemed extremely slow whenever I looked at it.

(3) Several years ago, I installed Nextcloud, but eventually gave up on it. When I installed it this time (a few days ago), I had forgotten why I gave up, but now I believe I remember it was also because the app store page was unusable, timing out consistently.

[rwv37]

Yeah, just tried bumping the timeout (by manually modifying Fetcher.php) from one minute to ten minutes, and now the app page loads. Very, very slowly, of course, but it loads. Don't know if related problems are going to occur when I try doing something besides just load the page, but hey, it's a start, I guess.