rpc timeouts/"private.api.ratelimit is too big"

[imerr]

Hi

Switching over from geth since this looks very promising

We're doing a bunch of data collecting via the http rpc interface and with geth we were able to "max out" the server running the geth node
With erigon (latest alpha, eth mainnet, archive node) we're getting a bunch of timeouts though and the server is practially idle (barely any io use on the local datacenter pcie 4.0 nvme ssds and cpu load is low too)

Currently running the built-in rpc daemon (should we be switching this to a seperate locally run one for better performance?), the documentation says crank up the --private.api.ratelimit limit, but it already seems to be at the max value it can be as default and is hard limited to 32000-128 from what I can tell
The message comes from here: https://github.com/ledgerwatch/erigon/blob/f560f3788f9fff593037aeaae73d7c8c618ba8b6/turbo/cli/flags.go#L393-L397
And kv.ReadersLimit seems to be hard coded to 32k: https://github.com/ledgerwatch/erigon-lib/blob/95e94b2eb5350b3194d03a9a731dccd6d2901236/kv/kv_interface.go#L25-L27

Any advice?

[AskAlexSharov]

Show your cli flags

[imerr]

/opt/erigon/erigon --datadir=/opt/erigon/data --http --ws --http.api="net,eth,web3" --http.addr "0.0.0.0" --private.api.ratelimit 63744

[AskAlexSharov]

"private.api" - it's a low level grpc API to which used when RPCDaemon is deployed on another machine. If you using internal RPCDaemon or external with parameter --datadir - then private.api not really used.

[AskAlexSharov]

if you need to tune something, you may try --db.read.concurrency and --rpc.batch.concurrency

[imerr]

Right, thank you. The docs weren't really clear on if it applied to the built-in rpc serveras well.
Will try the other paramters you mentioned and report back 👍

Would running the rpc daemon externally via --datadir provide a performance boost or should that not make a difference?

[imerr]

That seems to have helped slightly (tried read concurrency up to 256 with no significant change in performance and batch concurrency up to 64), not seeing timeouts anymore, cpu use is hovering around 50% (less than one thread/core) though

[imerr]

Running the rpcdaemon as its own process instead of using the built-in one seems to allow it to scale better

[avinashbo]

@imerr Could you share the --db.read.concurrency and --rpc.batch.concurrency values you are using when running rpcdaemon as a separate process?

[imerr]

Same values I tried on the main process, currently using --db.read.concurrency 32 --rpc.batch.concurrency 16, but I have not tried to tweak those as we're now running into the limits of our hardware (either cpu or io depending on load)

[AskAlexSharov]

there was an issue - built-in rpcdaemon didn't use --db.read.concurrency flag. fixed by #5540