8.22.0-RCx crashes after receiving large number of transactions #164
Comments
|
CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. forked testnet chain: 2754930 After 185,000 blocks we need to update the seeder GetRequireHeight to 2,760,000 |
|
Thanks for discovering this, and testing like this. Definitely worth checking out. My initial hunch is this might be an issue specific to testnet if all the nodes on our current testnet have not been updated with many of the recent fixes we made. Could be competing forks causing issues here and DB corruption.
Lines 1366 to 1383 in e8247b0 Did you compile the node you are getting these logs from with the debug flag? For more verbose logs?
Also have you tried something like LLDB (macOS debugger) to debug this? Or GDB on linux |
|
Let me clarify what "large number of transactions" means. I am talking about 3000 or more. On the one node, I think I counted just over 3000 coinbase txs that had gone to that node when it crashed. My other node was probably similar. @JaredTate As for the debug flag, no. The version that I am running with the maturity fixes was given to me by @Jongjan88. The other nodes are just running RC2. I do see there are still at least a few testnet nodes running on the forked chain. To be honest, I don't even know what LLDB and GDB are, so no, I have not tried those. I imagine in the next 12 hours or so, I should know if this same thing happens in 7.17.3. |
|
Thanks to some help from @Jongjan88, we were able to flood my 7.17.3 core wallet with transactions. It did crash, just like the v8 wallet. |
|
2 transactions per second. Crash after 1869 transactions. |
Mining rewards from mining 3 algos were also going to it for approximately 12 hours before we flooded it with more txs. |
|
We are talking about the QT desktop core client crashing here correct? aka the GUI? I have seen the client crash in the past while flooding a desktop GUI client with 1000s of tx's. In both cases, I switched to digibyte-cli on a server via the command line and it handled any use case I threw at it. In those two cases, it was an exchange and a mining pool. Actually I had the same wallet.dat on a desktop GUI and a server so I could watch it. The desktop crashed but digibyte-cli kept up just fine. Also similar problem when de-dusting 1000s of artifact UTXOs on a mining pool. It memory crashed the desktop GUI but cli was fine. So my question is how often will a normal end user send 1000s of txs in a few seconds to a desktop GUI? And at what point is it appropriate to use the command line digibyte-cli on a server. Now if this same bug is persistent on a digibyte-cli is definitely something to look much deeper at. |
|
You are correct @JaredTate, it is the GUI. I had the same thought that this might not be a practical real-world scenario. It’s not just 1000’s in a few seconds though. In most of my cases, it was over 1 to 3 days. But again, maybe not a scenario that people would really run into. |
I would be very curious to see if the same problem persists with digibyte-cli on a server. I doubt it will. My hunch is this is a memory leak error in the QT GUI related to some other crashes we have seen in the past. I have spent some time trying to isolate memory leaks in the QT, which we have a few of them for sure, but its not an easy task to isolate. I think we have some pointers in a few places not properly being closed. |
|
This time i tested 4000 transactions. 2 per second. Digibyte QT remained stable. I am mining now for more then a month on Testnet. Sometimes with 5 miners at the same time. And send all the coins to one QT wallet. I never had a crash. System 1: DigiByte QT Dev branch 2024-01-31 ( Sender )
Small memory leaks could be the cause. But don't forget the rc2 nodes without the coinbase maturity fix. |
|
Now that I look at my NodeB and NodeC again, they are running a version with PR157, so the coin maturity fix was in there. I am going to put the newest version in place and will see if I get the same crash after a day or so. |
ghost
mentioned this issue
|
Our testnet chain is corrupt atm. So this crash will probably not stop till 2024-03-01. I think the best thing we can do with this pr is to leave it open. After 2024-03-01 we can ban/block nodes lower than block 2,754,931 on the testnet chain. |
|
I updated the node versions being used in the issue description. None of my nodes were RC2, they all had at least PR157. #167 has not affected this issue at all. I still get the same errors and crash. We are going to wait until next month when the testnet chain is longer than the forked chain to look at this more. |
ghost
mentioned this issue
ghost
mentioned this issue
|
@beggsdl are you using Descriptor Wallet or Legacy Wallet?
|
It would be legacy because I have never checked the Descriptor checkbox when creating a new wallet. |
|
Was this fixed with RC4? Thanks for your feedback |
|
@beggsdl Are you able to reproduce the error on RC5? |
Yes. It just crashed on me about 30 minutes ago. It probably is not an issue for a regular user though. I have the QT wallet open all the time and it is receiving all my testnet coinbase transactions. After several million DGB, so several thousand coinbase TXs, it crashes. I would think anyone having this many TXs on mainnet would be running the cli wallet. I have not tested to see if the same thing happens to the cli wallet, but I doubt that it does. |
Setup (solo cpu mining against each of these testnet nodes):
NodeA:
NodeB:
NodeC:
NodeD:
Solo mining against testnet core NodeA with a wallet (walletA) setup. This core node is configured for mining scrypt.
I have 2 other testnet nodes currently configured for mining sha256 and quibit. All coinbase transactions are directed to walletA.
This wallet (walletA) has several addresses for it to receive coinbase txs from solo mining scrypt, sha256, and qubit (some from odo and skein also).
With walletA, after some period of time (typically about once per day, with coinbase txs from 3 algos going to this wallet), the node will crash.
The following messages are visible in debug.log:
Changing the algo mining against NodeA does not impact the result. I still get the same result of NodeA crashing about 1 time per day.
I then changed the sha256 coinbase txs (mining against NodeB) to be directed to walletB on NodeB to see if NodeB would also crash at some point. NodeB did crash. It took a bit longer to do so, maybe because there were coinbase txs from only one algo going to it.
The log is basically the same as above, except some different messages at the very end:
My next test was to direct all coinbase txs to walletD on NodeD instead of to a wallet on a node being mined against.
NodeD was NOT being mined against. It was only receiving coinbase txs from the three solo miners.
I moved the wallet.dat file from the NodeA folder to the NodeD folder, so the wallet receiving all the coinbase txs had the same addresses and all the previous coinbase txs. NodeA was still being mined against, but now did not have a wallet configured.
After running like this for about 24 hours, NodeD crashed:
My conclusion is a core wallet that is open/running and receiving lots of transactions will eventually crash. If this core wallet is being mined against, mining will obviously stop when it crashes. The crashing from receiving lots of transactions does not seem to be impacted at all by mining against that core node or not.
The only test I have not yet completed is to direct all coinbase txs to the 7.17.3 node to see if it will also crash. I am currently doing this test and update with the result.
The text was updated successfully, but these errors were encountered: