[BUG] Wallet height: 247520 Sync status: Not synced - Everything was fine then !!! Bam stuck wallet

[room101-dev]

Describe the bug
Wallet height: 247520
Sync status: Not synced

To Reproduce
Not sure how, nobody else is reporting.
I have deleted and recreated the /maininet/wallet/db many times, always stops at this height

Installed the chaingreen weeks ago, it was fine, and even found CGC coin. Then the mainnet database started getting corrupted, fixed that by running the blockchain in a sandbox, but then this stuck problem, wallet process seems to be cpu bound.

Expected behavior

Well expect it to work.

Screenshots
If applicable, add screenshots to help explain your problem.

Seems to be saying that 'peers' refuse to respond on post 247519, I'm running all the peers that I know of.

2021-07-01T16:31:13.505 full_node full_node_server : WARNING Trying to ban localhost for 10, but will not ban
2021-07-01T16:31:13.505 full_node full_node_server : INFO Connection closed: 127.0.0.1, node id: 44f55daf76217f68c577e5c8bdafb0b49d5e34e724a83e427a0ba00858da46fa
2021-07-01T16:31:13.505 full_node chaingreen.full_node.full_node: INFO peer disconnected {'host': '127.0.0.1', 'port': 8749}
2021-07-01T16:31:13.505 wallet wallet_server : INFO Connection closed: 127.0.0.1, node id: eaca68048f18e2db1ab3ebfe4c6c300163fd7514995d5941d93d70a4012e7844
2021-07-01T16:31:13.506 wallet chaingreen.wallet.wallet_node: ERROR Error while trying to fetch from peer:Peer returned no response Traceback (most recent call last):
File "/media/master/SSD-Temp/Green-VM/chaingreen-blockchain/chaingreen/wallet/wallet_node.py", line 599, in _sync
peer, uint32(start_height), uint32(end_height), None if advanced_peak else fork_height
File "/media/master/SSD-Temp/Green-VM/chaingreen-blockchain/chaingreen/wallet/wallet_node.py", line 637, in fetch_blocks_and_validate
raise ValueError("Peer returned no response")
ValueError: Peer returned no response

2021-07-01T16:31:13.506 wallet chaingreen.wallet.wallet_node: ERROR Loop exception in sync Was not able to add blocks 247519-247551. Traceback (most recent call last):
File "/media/master/SSD-Temp/Green-VM/chaingreen-blockchain/chaingreen/wallet/wallet_node.py", line 511, in sync_job
await self._sync()
File "/media/master/SSD-Temp/Green-VM/chaingreen-blockchain/chaingreen/wallet/wallet_node.py", line 608, in _sync
raise RuntimeError(f"Was not able to add blocks {start_height}-{end_height}")
RuntimeError: Was not able to add blocks 247519-247551

2021-07-01T16:31:13.506 wallet chaingreen.wallet.wallet_node: INFO Loop end in sync job
2021-07-01T16:31:13.507 wallet chaingreen.wallet.wallet_node: INFO Loop start in sync job
2021-07-01T16:31:13.902 wallet chaingreen.wallet.wallet_node: INFO self.sync_event.set()
2021-07-01T16:31:13.902 wallet chaingreen.wallet.wallet_node: INFO self.sync_event.set()
2021-07-01T16:31:13.902 wallet chaingreen.wallet.wallet_node: INFO Have collected 17 potential peaks
2021-07-01T16:31:13.902 wallet chaingreen.wallet.wallet_node: INFO No peers to sync to
2021-07-01T16:31:13.902 wallet chaingreen.wallet.wallet_node: INFO Loop end in sync job
2021-07-01T16:31:13.903 wallet chaingreen.wallet.wallet_node: INFO Loop start in sync job

Desktop (please complete the following information):

• OS: [e.g. Linux]
• OS Version/Flavor: [e.g. CentOS 7.2]
• CPU: [e.g. Intel Xeon 8175M]

Additional context
Add any other context about the problem here.

[room101-dev]

Yes, I have test the sql database they are good both mainnet & wallet. It's seems to be add to be banned right at this number, while prior to this number the wallet db loads fine, then bam it stops and all peers refuse to provide additional wallet data.

how to test your database, to make sure they're not corrupt ...
test sql
blockchain_wallet_v1_mainnet_xxx.sqlite
sqlite3 blockchain_wallet_v1_mainnet_xxx.sqlite "PRAGMA integrity_check

returns 'ok' or else
"
...

wallet_node.py, it certainly appears that NO PEERS have the wallet blocks past this address, maybe u need to have 'magic peers'?? Bails at line 608, where in the list of peers, no peers had these wallet blocks,
It's odd, it gets this far, like I have said, I have now seen this problem for 3 days,
If anybody has wallet-block beyond this height (247519), then they should post their peer list,

591 for i in range(max(0, fork_height - 1), peak_height, batch_size):
592 start_height = i
593 end_height = min(peak_height, start_height + batch_size)
594 peers = self.server.get_full_node_connections()
595 added = False
596 for peer in peers:
597 try:
598 added, advanced_peak = await self.fetch_blocks_and_validate(
599 peer, uint32(start_height), uint32(end_height), None if advanced_peak else fork_height
600 )
601 if added:
602 break
603 except Exception as e:
604 await peer.close()
605 exc = traceback.format_exc()
606 self.log.error(f"Error while trying to fetch from peer:{e} {exc}")
607 if not added:
608 raise RuntimeError(f"Was not able to add blocks {start_height}-{end_height}")
609
610 peak = self.wallet_state_manager.blockchain.get_peak()
611 assert peak is not None
612 self.wallet_state_manager.blockchain.clean_block_record(
613 min(
614 end_height - self.constants.BLOCKS_CACHE_SIZE,
615 peak.height - self.constants.BLOCKS_CACHE_SIZE,
616 )
617 )
618

[djjaeger82]

Having the exact same issue, stuck at wallet 258917. Driving me nuts, farming says its synced but haven't won any coin in 4+ days because of this (and eta is every 12hr with my 1700 plots). SPARE, CHIA, FLAX all working fine, only chaingreen keeps having these issues with syncing.(wallet specifically)

[volklala]

I have the same issue only with chaingreen. It stops at 170528 and nothing helps