Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix race on sizeEstimation in Flushable #76

Merged
merged 1 commit into from
Dec 15, 2023
Merged

Fix race on sizeEstimation in Flushable #76

merged 1 commit into from
Dec 15, 2023

Conversation

thaarok
Copy link
Contributor

@thaarok thaarok commented Dec 12, 2023 •
edited
Loading

When running Opera with -race parameter, we observe following race conditions:

==================
WARNING: DATA RACE
Write at 0x00c029064498 by goroutine 2517:
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*Flushable).put()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/flushable.go:82 +0x152
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*Flushable).Put()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/flushable.go:76 +0x113
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*closeDropWrapped).Put()
      <autogenerated>:1 +0xac
  github.com/Fantom-foundation/go-opera/utils/dbutil/asyncflushproducer.(*store).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/lachesis-base/kvdb/cachedproducer.(*StoreWithFn).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/lachesis-base/kvdb/multidb.(*closableTable).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/lachesis-base/kvdb/skipkeys.(*Store).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/go-opera/utils/dbutil/threads.(*countedStore).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/go-opera/utils/rlpstore.(*Helper).Set()
      /home/jand/go-opera-norma/utils/rlpstore/rlpstore.go:20 +0x1f0
  github.com/Fantom-foundation/go-opera/gossip.(*Store).SetEvent()
      /home/jand/go-opera-norma/gossip/store_event.go:37 +0x184
  github.com/Fantom-foundation/go-opera/gossip.(*Service).saveAndProcessEvent()
      /home/jand/go-opera-norma/gossip/c_event_callbacks.go:96 +0x91
  github.com/Fantom-foundation/go-opera/gossip.(*Service).processEvent()
      /home/jand/go-opera-norma/gossip/c_event_callbacks.go:263 +0x5fa
  github.com/Fantom-foundation/go-opera/gossip.newService.func2()
      /home/jand/go-opera-norma/gossip/service.go:259 +0xa4
  github.com/Fantom-foundation/go-opera/gossip.(*handler).makeDagProcessor.func3()
      /home/jand/go-opera-norma/gossip/handler.go:480 +0x16e
  github.com/Fantom-foundation/lachesis-base/gossip/dagordering.(*EventsBuffer).processCompleteEvent()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagordering/event_buffer.go:145 +0x128
  github.com/Fantom-foundation/lachesis-base/gossip/dagordering.(*EventsBuffer).pushEvent()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagordering/event_buffer.go:88 +0x152
  github.com/Fantom-foundation/lachesis-base/gossip/dagordering.(*EventsBuffer).PushEvent()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagordering/event_buffer.go:66 +0x324
  github.com/Fantom-foundation/lachesis-base/gossip/dagprocessor.(*Processor).process()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagprocessor/processor.go:181 +0x115
  github.com/Fantom-foundation/lachesis-base/gossip/dagprocessor.(*Processor).Enqueue.func2()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagprocessor/processor.go:146 +0x5b4
  github.com/Fantom-foundation/lachesis-base/utils/workers.worker()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:66 +0xf2
  github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start.func1()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:31 +0x90

Previous read at 0x00c029064498 by goroutine 2211:
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*Flushable).NotFlushedSizeEst()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/flushable.go:184 +0x1b4
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*SyncedPool).NotFlushedSizeEst()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/synced_pool.go:224 +0x118
  github.com/Fantom-foundation/go-opera/utils/dbutil/asyncflushproducer.(*Producer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/lachesis-base/kvdb/cachedproducer.(*AllDBProducer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/lachesis-base/kvdb/multidb.(*Producer).NotFlushedSizeEst()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/multidb/producer.go:163 +0xb6
  github.com/Fantom-foundation/lachesis-base/kvdb/skipkeys.(*AllDBProducer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/go-opera/utils/dbutil/threads.(*countedFullDbProducer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/go-opera/gossip.(*Store).isCommitNeeded()
      /home/jand/go-opera-norma/gossip/store.go:183 +0x133
  github.com/Fantom-foundation/go-opera/gossip.newService.(*Service).makePeriodicFlusher.func4()
      /home/jand/go-opera-norma/gossip/service.go:316 +0x55
  github.com/Fantom-foundation/go-opera/gossip.(*PeriodicFlusher).loop()
      /home/jand/go-opera-norma/gossip/tflusher.go:30 +0x22c
  github.com/Fantom-foundation/go-opera/gossip.(*PeriodicFlusher).Start.func1()
      /home/jand/go-opera-norma/gossip/tflusher.go:41 +0x33

Goroutine 2517 (running) created at:
  github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:29 +0x32
  github.com/Fantom-foundation/lachesis-base/gossip/dagprocessor.(*Processor).Start()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagprocessor/processor.go:80 +0x44
  github.com/Fantom-foundation/go-opera/gossip.(*handler).Start()
      /home/jand/go-opera-norma/gossip/handler.go:694 +0x9bc
  github.com/Fantom-foundation/go-opera/gossip.(*Service).Start()
      /home/jand/go-opera-norma/gossip/service.go:451 +0x504

Goroutine 2211 (running) created at:
  github.com/Fantom-foundation/go-opera/gossip.(*PeriodicFlusher).Start()
      /home/jand/go-opera-norma/gossip/tflusher.go:41 +0xa5
  github.com/Fantom-foundation/go-opera/gossip.(*Service).Start()
      /home/jand/go-opera-norma/gossip/service.go:432 +0x1be
==================
==================
WARNING: DATA RACE
Read at 0x00c029065188 by goroutine 2517:
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*Flushable).NotFlushedSizeEst()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/flushable.go:184 +0x1b4
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*SyncedPool).NotFlushedSizeEst()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/synced_pool.go:224 +0x118
  github.com/Fantom-foundation/go-opera/utils/dbutil/asyncflushproducer.(*Producer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/lachesis-base/kvdb/cachedproducer.(*AllDBProducer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/lachesis-base/kvdb/multidb.(*Producer).NotFlushedSizeEst()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/multidb/producer.go:163 +0xb6
  github.com/Fantom-foundation/lachesis-base/kvdb/skipkeys.(*AllDBProducer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/go-opera/utils/dbutil/threads.(*countedFullDbProducer).NotFlushedSizeEst()
      <autogenerated>:1 +0x43
  github.com/Fantom-foundation/go-opera/gossip.(*Store).isCommitNeeded()
      /home/jand/go-opera-norma/gossip/store.go:183 +0x133
  github.com/Fantom-foundation/go-opera/gossip.(*Store).IsCommitNeeded()
      /home/jand/go-opera-norma/gossip/store.go:176 +0xb0
  github.com/Fantom-foundation/go-opera/gossip.(*Service).mayCommit()
      /home/jand/go-opera-norma/gossip/c_event_callbacks.go:306 +0x55
  github.com/Fantom-foundation/go-opera/gossip.(*Service).processEvent()
      /home/jand/go-opera-norma/gossip/c_event_callbacks.go:280 +0xa04
  github.com/Fantom-foundation/go-opera/gossip.newService.func2()
      /home/jand/go-opera-norma/gossip/service.go:259 +0xa4
  github.com/Fantom-foundation/go-opera/gossip.(*handler).makeDagProcessor.func3()
      /home/jand/go-opera-norma/gossip/handler.go:480 +0x16e
  github.com/Fantom-foundation/lachesis-base/gossip/dagordering.(*EventsBuffer).processCompleteEvent()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagordering/event_buffer.go:145 +0x128
  github.com/Fantom-foundation/lachesis-base/gossip/dagordering.(*EventsBuffer).pushEvent()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagordering/event_buffer.go:88 +0x152
  github.com/Fantom-foundation/lachesis-base/gossip/dagordering.(*EventsBuffer).PushEvent()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagordering/event_buffer.go:66 +0x324
  github.com/Fantom-foundation/lachesis-base/gossip/dagprocessor.(*Processor).process()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagprocessor/processor.go:181 +0x115
  github.com/Fantom-foundation/lachesis-base/gossip/dagprocessor.(*Processor).Enqueue.func2()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagprocessor/processor.go:146 +0x5b4
  github.com/Fantom-foundation/lachesis-base/utils/workers.worker()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:66 +0xf2
  github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start.func1()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:31 +0x90
  ...

Previous write at 0x00c029065188 by goroutine 2212:
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*Flushable).put()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/flushable.go:82 +0x152
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*Flushable).Put()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/flushable/flushable.go:76 +0x113
  github.com/Fantom-foundation/lachesis-base/kvdb/flushable.(*closeDropWrapped).Put()
      <autogenerated>:1 +0xac
  github.com/Fantom-foundation/go-opera/utils/dbutil/asyncflushproducer.(*store).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/lachesis-base/kvdb/cachedproducer.(*StoreWithFn).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/lachesis-base/kvdb/table.(*Table).Put()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/kvdb/table/table.go:63 +0x30f
  github.com/Fantom-foundation/lachesis-base/kvdb/multidb.(*closableTable).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/lachesis-base/kvdb/skipkeys.(*Store).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/go-opera/utils/dbutil/threads.(*countedStore).Put()
      <autogenerated>:1 +0x96
  github.com/Fantom-foundation/go-opera/gossip.(*Store).SetBlockIndex()
      /home/jand/go-opera-norma/gossip/store_block.go:101 +0x2ce
  github.com/Fantom-foundation/go-opera/cmd/opera/launcher.makeNode.(*Service).GetConsensusCallbacks.consensusCallbackBeginBlockFn.func5.5.4()
      /home/jand/go-opera-norma/gossip/c_block_callbacks.go:402 +0x1a45
  github.com/Fantom-foundation/go-opera/cmd/opera/launcher.makeNode.(*Service).GetConsensusCallbacks.consensusCallbackBeginBlockFn.func5.5.5()
      /home/jand/go-opera-norma/gossip/c_block_callbacks.go:465 +0xdd
  github.com/Fantom-foundation/lachesis-base/utils/workers.worker()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:66 +0xf2
  github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start.func1()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:31 +0x90

Goroutine 2517 (running) created at:
  github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:29 +0x32
  github.com/Fantom-foundation/lachesis-base/gossip/dagprocessor.(*Processor).Start()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/gossip/dagprocessor/processor.go:80 +0x44
  github.com/Fantom-foundation/go-opera/gossip.(*handler).Start()
      /home/jand/go-opera-norma/gossip/handler.go:694 +0x9bc
  github.com/Fantom-foundation/go-opera/gossip.(*Service).Start()
      /home/jand/go-opera-norma/gossip/service.go:451 +0x504
  github.com/ethereum/go-ethereum/node.(*Node).Start()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:184 +0x3ca

Goroutine 2212 (running) created at:
  github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/utils/workers/workers.go:29 +0x32
  github.com/Fantom-foundation/go-opera/gossip.(*Service).Start()
      /home/jand/go-opera-norma/gossip/service.go:447 +0x40f
  github.com/ethereum/go-ethereum/node.(*Node).Start()
      /home/jand/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:184 +0x3ca
      ...
==================

This is caused by concurrent access to Flushable.sizeEstimation.
This PR fixes also similar issue on LazyFlushable.underlying variable, which is being written during Flush of LazyFlushable.

@thaarok thaarok requested a review from andrecronje as a code owner December 12, 2023 11:10
@uprendis uprendis merged commit 2653b30 into Fantom-foundation:develop Dec 15, 2023
6 checks passed
@thaarok thaarok deleted the jkalina/race2-fix branch December 15, 2023 17:23
@thaarok thaarok mentioned this pull request Dec 15, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrecronje andrecronje Awaiting requested review from andrecronje andrecronje is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@thaarok @uprendis