Node crash #984
Description
Hi, my node (v1.0.0-beta.9) crashed a while back. I reported it on Discord, but it seems you guys might have missed it while you are busy with the other project. So I reupload the relevant part of the log here, so it's not forgotten. I will keep the full log, so if you need more than that just let me know.
iota-core3 | 2024-08-15T05:10:40Z INFO Protocol.Engine0.NotarizationManager Committing (commitment=Commitment{
iota-core3 | Index: 811013
iota-core3 | PrevID: CommitmentID(0x3730d3371df3e07c3460f46bdfd9a3425d293e0fd5d9cd35de0d57ddaa1cbfb404600c00:811012)
iota-core3 | RootsID: 0x9411a79a12361e48421c1170c8c71dc70972d55268bf2d5814a43ccdd7972d35
iota-core3 | CumulativeWeight: 12353162
iota-core3 | RMC: 1
iota-core3 | } roots =Roots(0x9411a79a12361e48421c1170c8c71dc70972d55268bf2d5814a43ccdd7972d35): TangleRoot: 0xedd67bdc18e422786170fd7a6fc3fb2c7b7663eeffe546d73889a28a3e7ed43c, StateMutationRoot: 0x0000000000000000000000000000000000000000000000000000000000000000, StateRoot: 0xf16257cc3e68be8d62579145e35a9f9d4a3966b34af167a3d109ea6d09af08a6, AccountRoot: 0x33c2760f68962373699dd3393ba2e59373ef482e5b9550f9efb9ca43249732db, AttestationsRoot: 0xcaeef8999e4ff2cc5783df893d4a56af38c78ec80fdda30a54654246b18b88b7, CommitteeRoot: 0xd26640160b6baf1d572b83339dc40badcd1dd20796774881950e380164f3686e, RewardsRoot: 0xf68e0698babbcaf615bf53314b8c14a12059303024c5d7afc4c9b556c726387e, ProtocolParametersHash: 0x9786d0ead851be3d740e0956a2c8e2dd7eec59a2fb3e608834ca8a57ee1eede8)
iota-core3 | 2024-08-15T05:10:49Z ERROR Protocol.Engine0 engine error (err=blockRetainer: failed to store on BlockDropped in retainer: cannot update block metadata for block BlockID(0xcfc5b34c34bb352f1ea617f20dcc6590a5770b3320986199ad0cd26e173a4cde00600c00:811008) with state dropped as block is already committed)
iota-core3 | 2024-08-15T05:10:49Z ERROR Protocol.Engine0 engine error (err=blockRetainer: failed to store on BlockDropped in retainer: cannot update block metadata for block BlockID(0x427964fa4a08419eaa6991fc5003073b55e32292505d459b59555e1d5994736900600c00:811008) with state dropped as block is already committed)
iota-core3 | 2024-08-15T05:10:49Z ERROR Protocol.Engine0 engine error (err=blockRetainer: failed to store on BlockDropped in retainer: cannot update block metadata for block BlockID(0xc1839567acd200d6d85abd9fcc79d6c9643622c5b936ea36d50711fca94813d700600c00:811008) with state dropped as block is already committed)
...
iota-core3 | 2024-08-15T07:55:40Z ERROR Protocol.Engine0 engine error (err=blockRetainer: failed to store on BlockDropped in retainer: cannot update block metadata for block BlockID(0x388a8256af1969b71702150dde7ed84a719f57a10a63872e0b3a42a8a1423e15dd630c00:811997) with state dropped as block is already committed)
iota-core3 | 2024-08-15T07:55:41Z ERROR Protocol.Engine0 engine error (err=blockRetainer: failed to store on BlockDropped in retainer: cannot update block metadata for block BlockID(0x5ae46c0a711327ca54197906faa911aff6ec89948516ba097ab43d0653a47b54dd630c00:811997) with state dropped as block is already committed)
iota-core3 | 2024-08-15T07:55:41Z ERROR Protocol.Engine0 engine error (err=blockRetainer: failed to store on BlockDropped in retainer: cannot update block metadata for block BlockID(0x7bff94e9b97113c1c77ff53174698b07794b56fe595f445c4855ecfefdeb8aaadd630c00:811997) with state dropped as block is already committed)
iota-core3 | panic: heap tail and oldest submitted block do not exist. Trying to remove tail of an empty queue?
iota-core3 |
iota-core3 | goroutine 202 [running]:
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/congestioncontrol/scheduler/drr.(*IssuerQueue).RemoveTail(0xc05e908f00)
iota-core3 | /scratch/pkg/protocol/engine/congestioncontrol/scheduler/drr/issuerqueue.go:189 +0x18e
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/congestioncontrol/scheduler/drr.(*BasicBuffer).dropTail(0xc00027c280, 0xc0010d88d8, 0x3e8)
iota-core3 | /scratch/pkg/protocol/engine/congestioncontrol/scheduler/drr/basicbuffer.go:304 +0xbf
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/congestioncontrol/scheduler/drr.(*BasicBuffer).Submit(0xc00027c280, 0xaa4fe16d726980c7?, 0xa3184d3260c9c76b?, 0xc0010d88d8, 0x3e8)
iota-core3 | /scratch/pkg/protocol/engine/congestioncontrol/scheduler/drr/basicbuffer.go:177 +0x46
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/congestioncontrol/scheduler/drr.(*Scheduler).enqueueBasicBlock(0xc0000c8210, 0xc0c18eda40)
iota-core3 | /scratch/pkg/protocol/engine/congestioncontrol/scheduler/drr/scheduler.go:307 +0x17e
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/congestioncontrol/scheduler/drr.(*Scheduler).AddBlock(0xc0000c8210, 0xc0c18eda40)
iota-core3 | /scratch/pkg/protocol/engine/congestioncontrol/scheduler/drr/scheduler.go:275 +0xf8
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol.NewDefaultOptions.NewProvider.func17.1.5(0xc000387a40?)
iota-core3 | /scratch/pkg/protocol/engine/congestioncontrol/scheduler/drr/scheduler.go:109 +0x25
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc000156870?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0xc0010d8a28, 0xc0010d8a20?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0x0?, 0x0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc0005fa270?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0x2d39700, 0xc0010d8b20?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0x2d3e0b8?, 0xc0c2271a80?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/booker/inmemorybooker.(*Booker).book(0xc0005d2660, 0xc0c18eda40)
iota-core3 | /scratch/pkg/protocol/engine/booker/inmemorybooker/booker.go:217 +0x3e5
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/booker/inmemorybooker.(*Booker).setupBlock.func3()
iota-core3 | /scratch/pkg/protocol/engine/booker/inmemorybooker/booker.go:186 +0x29
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*event).OnTrigger.func1(0x60?, 0x8e?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/event_impl.go:31 +0x13
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*readableVariable[...]).OnUpdate(0x2d42a60, 0xc0c2c79940, {0x0, 0x0, 0x1})
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/variable_impl.go:200 +0x179
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*event).OnTrigger(0xc0c2c79330, 0xc0c2591170)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/event_impl.go:30 +0x71
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/booker/inmemorybooker.(*Booker).setupBlock(0xc0005d2660, 0xc0c18eda40, {0x0?, 0x0?})
iota-core3 | /scratch/pkg/protocol/engine/booker/inmemorybooker/booker.go:185 +0x2f8
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/booker/inmemorybooker.(*Booker).Queue(0xc0005d2660, 0xc0c18eda40)
iota-core3 | /scratch/pkg/protocol/engine/booker/inmemorybooker/booker.go:100 +0x1e7
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol.NewDefaultOptions.NewProvider.func6.1.1(0xf7369ce400000063?)
iota-core3 | /scratch/pkg/protocol/engine/booker/inmemorybooker/booker.go:45 +0x25
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc0005fa210?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0xc055c22ac8, 0xc0010d8ff8?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0x0?, 0x485634?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc0005f14a0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0x6b824b85a89c52c6, 0xc0010d90f8?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0xaa4fe16d726980c7?, 0xa3184d3260c9c76b?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/sybilprotection/sybilprotectionv1.New.NewProvider.func2.1.1(0xc0c18eda40)
iota-core3 | /scratch/pkg/protocol/sybilprotection/seatmanager/topstakers/topstakers.go:59 +0x186
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc0005f13e0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0x7481a43e14f80575, 0xc0010d9290?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0xf47cccf233cd88a4?, 0xc2899b27538963bd?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc0005fa0f0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0x57b19bd544ec6, 0xc0010d9390?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0x20?, 0x20?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/filter/postsolidfilter/postsolidblockfilter.(*PostSolidBlockFilter).ProcessSolidBlock(0xc0001bfe00, 0xc0c18eda40)
iota-core3 | /scratch/pkg/protocol/engine/filter/postsolidfilter/postsolidblockfilter/post_solid_block_filter.go:214 +0x104c
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc0005fa150?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0xc0010d98e8, 0xc0010d98e0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0x0?, 0x0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1(0xc0005f1bc0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:88 +0x177
iota-core3 | github.com/iotaledger/hive.go/ds/orderedmap.(*OrderedMap[...]).ForEach(0xc0c3084660, 0xc0010d99e0?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/orderedmap/orderedmap.go:117 +0xc3
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger(0x4c9860?, 0xc0c3084690?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:70 +0x71
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/blockdag/inmemoryblockdag.(*BlockDAG).setupBlock.func1.1(0x30?, 0x9a?)
iota-core3 | /scratch/pkg/protocol/engine/blockdag/inmemoryblockdag/blockdag.go:148 +0x68
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*readableVariable[...]).OnUpdateOnce.func2()
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/variable_impl.go:234 +0x4f
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*readableVariable[...]).OnUpdateOnce.(*event).OnTrigger.func3(0x40?)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/event_impl.go:31 +0x13
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*readableVariable[...]).OnUpdate(0x2d42a60, 0xc0c2c79520, {0x0, 0x0, 0xc0010d9b01})
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/variable_impl.go:200 +0x179
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*event).OnTrigger(...)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/event_impl.go:30
iota-core3 | github.com/iotaledger/hive.go/ds/reactive.(*readableVariable[...]).OnUpdateOnce(0x2d42a60, 0xc0c2271040, {0x0, 0x0, 0x0})
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/ds@v0.0.0-20240520064018-c635e5900894/reactive/variable_impl.go:231 +0x243
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/blockdag/inmemoryblockdag.(*BlockDAG).setupBlock.func1({{0x0, 0x2a, 0xde, 0x55, 0x98, 0x37, 0xc1, 0xbe, 0x5, 0xfa, ...}, ...})
iota-core3 | /scratch/pkg/protocol/engine/blockdag/inmemoryblockdag/blockdag.go:145 +0x113
iota-core3 | github.com/iotaledger/iota.go/v4.(*Block).ForEachParent(0xc0c1b83cc0?, 0xc0010d9d30)
iota-core3 | /go/pkg/mod/github.com/iotaledger/iota.go/v4@v4.0.0-20240503105040-c86882e71808/block.go:245 +0x65
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/blocks.(*Block).ForEachParent(...)
iota-core3 | /scratch/pkg/protocol/engine/blocks/block.go:169
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/blockdag/inmemoryblockdag.(*BlockDAG).setupBlock(0xc000119900, 0xc0c18eda40)
iota-core3 | /scratch/pkg/protocol/engine/blockdag/inmemoryblockdag/blockdag.go:137 +0x94
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol/engine/blockdag/inmemoryblockdag.(*BlockDAG).Append(0xc000119900, 0xffffffffffffffff?)
iota-core3 | /scratch/pkg/protocol/engine/blockdag/inmemoryblockdag/blockdag.go:110 +0x388
iota-core3 | github.com/iotaledger/iota-core/pkg/protocol.NewDefaultOptions.NewProvider.func3.1.1(0xc0c37feaa0)
iota-core3 | /scratch/pkg/protocol/engine/blockdag/inmemoryblockdag/blockdag.go:49 +0x36
iota-core3 | github.com/iotaledger/hive.go/runtime/event.(*Event1[...]).Trigger.func1.1()
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/event/events.go:86 +0x1f
iota-core3 | github.com/iotaledger/hive.go/runtime/workerpool.(*Task).run(0xc0c2ee9d10)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/workerpool/task.go:48 +0x72
iota-core3 | github.com/iotaledger/hive.go/runtime/workerpool.(*WorkerPool).workerReadLoop(0xc0003879d0)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/workerpool/workerpool.go:226 +0x25
iota-core3 | github.com/iotaledger/hive.go/runtime/workerpool.(*WorkerPool).worker(0xc0003879d0)
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/workerpool/workerpool.go:206 +0x50
iota-core3 | created by github.com/iotaledger/hive.go/runtime/workerpool.(*WorkerPool).startWorkers in goroutine 178
iota-core3 | /go/pkg/mod/github.com/iotaledger/hive.go/runtime@v0.0.0-20240520064018-c635e5900894/workerpool/workerpool.go:198 +0x26