ListUnspent() times out or stalls after 10k+ requests?

[Ekliptor]

I'm using Fulcrum server for prompt.cash with this Go client https://github.com/checksum0/go-electrum

We have a loop to update all wallet balances (multiple threads concurrent):

for all accounts in DB:
   start, end := GetXpubRange(account)
   addressList := GenerateAddressesByXpub(start, end)
   foreach addr in addressList:
      unspent, err := electrum.ListUnspent(addr)

We configured a timeout of 3sec for ListUnspent(). When there are 10k+ requests this timeout gets hit repeatedly as if Fulcrum has throttled/blacklisted our IP. Fulcrum continues to answer requests to a 2nd client on another IP within seconds (and the server load is ~0).
Without this timeout + retry Fulcrum just stops responding (no error).

We run our own Fulcrum server on localhost (see Fulcum debug log below for when this happens). There are no errors other than the repeated reconnects which are done by our Go implementation when we don't get the expected response from Fulcrum)

Questions:

1. Is there a rate limiting config we overlooked? We played with subnets_to_exclude_from_per_ip_limits unsuccessfully (localhost allowed by default)
2. could there be some semaphore/mutex in that's throttling us (per client IP)?

[2023-06-19 02:43:26.439] <TcpSrv 0.0.0.0:60001> (Debug) killClient (id: 1988)
[2023-06-19 02:43:26.439] <TcpSrv 0.0.0.0:60001> (Debug) do_disconnect (abort) 1988
[2023-06-19 02:43:26.439] <TcpSrv 0.0.0.0:60001> (Debug) Client 1988 destructing
[2023-06-19 02:43:26.439] <TcpSrv 0.0.0.0:60001> (Debug) Got connection from: 127.0.0.1:45654
[2023-06-19 02:43:26.439] <TcpSrv 0.0.0.0:60001> (Debug) on_connected 1998
[2023-06-19 02:43:26.439] <TcpSrv 0.0.0.0:60001> New TCP Client.1998 127.0.0.1:45654, 2 clients total
[2023-06-19 02:43:31.486] <TcpSrv 0.0.0.0:60001> (Debug) TCP Client.1998 (id: 1998) 127.0.0.1:45654 socket disconnected
[2023-06-19 02:43:31.486] <TcpSrv 0.0.0.0:60001> (Debug) TCP Client.1998 (id: 1998) 127.0.0.1:45654 lost connection
[2023-06-19 02:43:31.486] <TcpSrv 0.0.0.0:60001> (Debug) killClient (id: 1998)
[2023-06-19 02:43:31.486] <TcpSrv 0.0.0.0:60001> (Debug) do_disconnect (abort) 1998
[2023-06-19 02:43:31.487] <TcpSrv 0.0.0.0:60001> (Debug) Client 1998 destructing
[2023-06-19 02:43:31.487] <TcpSrv 0.0.0.0:60001> (Debug) Got connection from: 127.0.0.1:45656
[2023-06-19 02:43:31.487] <TcpSrv 0.0.0.0:60001> (Debug) on_connected 2005
[2023-06-19 02:43:31.487] <TcpSrv 0.0.0.0:60001> New TCP Client.2005 127.0.0.1:45656, 2 clients total
[2023-06-19 02:43:36.533] <TcpSrv 0.0.0.0:60001> (Debug) TCP Client.2005 (id: 2005) 127.0.0.1:45656 socket disconnected
[2023-06-19 02:43:36.534] <TcpSrv 0.0.0.0:60001> (Debug) TCP Client.2005 (id: 2005) 127.0.0.1:45656 lost connection
[2023-06-19 02:43:36.534] <TcpSrv 0.0.0.0:60001> (Debug) killClient (id: 2005)
[2023-06-19 02:43:36.534] <TcpSrv 0.0.0.0:60001> (Debug) do_disconnect (abort) 2005
[2023-06-19 02:43:36.534] <TcpSrv 0.0.0.0:60001> (Debug) Client 2005 destructing
[2023-06-19 02:43:36.534] <TcpSrv 0.0.0.0:60001> (Debug) Got connection from: 127.0.0.1:45658
[2023-06-19 02:43:36.534] <TcpSrv 0.0.0.0:60001> (Debug) on_connected 2015
[2023-06-19 02:43:36.534] <TcpSrv 0.0.0.0:60001> New TCP Client.2015 127.0.0.1:45658, 2 clients total
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> (Debug) TCP Client.2015 (id: 2015) 127.0.0.1:45658 socket disconnected
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> (Debug) TCP Client.2015 (id: 2015) 127.0.0.1:45658 lost connection
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> (Debug) killClient (id: 2015)
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> (Debug) do_disconnect (abort) 2015
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> (Debug) Client 2015 destructing
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> (Debug) Got connection from: 127.0.0.1:45660
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> (Debug) on_connected 2022
[2023-06-19 02:43:41.581] <TcpSrv 0.0.0.0:60001> New TCP Client.2022 127.0.0.1:45660, 2 clients total

[cculianu]

Hmm. This is strange. Sorry for the late reply. Is this still happening?

If it is still happening - Any chance you can send me (via email maybe?) the address in question?

[Ekliptor]

Yes it's still happening with latest Fulcrum 1.9.7. You can just use any xPub and generate addresses (even with empty balances) in a loop and Fulcrum will timeout in ListUnspent() after around 10k requests unless we re-connect.