Lite node crashed, database corrupted afterwards

[chrizpower]

Bug Report

Bug: SEGV in aleth

#0  std::__find_if<unsigned char const*, __gnu_cxx::__ops::_Iter_pred<dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}> >(unsigned char const*, unsigned char const*, __gnu_cxx::__ops::_Iter_pred<dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}>, std::random_access_iterator_tag) (__last=0x224 <error: Cannot access memory at address 0x224>, __first=<optimized out>, __pred=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_algobase.h:2107
#1  std::__find_if<unsigned char const*, __gnu_cxx::__ops::_Iter_pred<dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}> >(unsigned char const*, unsigned char const*, __gnu_cxx::__ops::_Iter_pred<dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}>) (__first=0x210 <error: Cannot access memory at address 0x210>, __last=0x224 <error: Cannot access memory at address 0x224>, __pred=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_algobase.h:2152
#2  std::find_if<unsigned char const*, dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}>(unsigned char const*, unsigned char const*, dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}) (
    __first=0x210 <error: Cannot access memory at address 0x210>, __last=0x224 <error: Cannot access memory at address 0x224>, __pred=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_algo.h:3876
#3  std::none_of<unsigned char const*, dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}>(unsigned char const*, unsigned char const*, dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}) (
    __first=0x210 <error: Cannot access memory at address 0x210>, __last=0x224 <error: Cannot access memory at address 0x224>, __pred=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_algo.h:429
#4  std::any_of<unsigned char const*, dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}>(unsigned char const*, unsigned char const*, dev::FixedHash<20u>::operator bool() const::{lambda(unsigned char)#1}) (
    __first=0x210 <error: Cannot access memory at address 0x210>, __last=0x224 <error: Cannot access memory at address 0x224>, __pred=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_algo.h:448
#5  dev::FixedHash<20u>::operator bool (this=0x210) at /tmp/tmp2efjn7js/taraxa-node/libraries/aleth/libdevcore/../libdevcore/FixedHash.h:111
#6  taraxa::DagBlock::getSender (this=0x0) at /tmp/tmp2efjn7js/taraxa-node/libraries/types/dag_block/src/dag_block.cpp:152
#7  0x0000560c8111dc50 in taraxa::DagBlockProposer::selectDagBlockTips (this=this@entry=0x560ca6998b10, frontier_tips=..., gas_limit=2099475000) at /tmp/tmp2efjn7js/taraxa-node/libraries/core_libs/consensus/src/dag/dag_block_proposer.cpp:319
#8  0x0000560c8111d2f4 in taraxa::DagBlockProposer::createDagBlock (this=this@entry=0x560ca6998b10, frontier=..., level=60299071, trxs=..., estimations=..., vdf=...) at /tmp/tmp2efjn7js/taraxa-node/libraries/core_libs/consensus/src/dag/dag_block_proposer.cpp:361
#9  0x0000560c8111b6ce in taraxa::DagBlockProposer::proposeDagBlock (this=this@entry=0x560ca6998b10) at /tmp/tmp2efjn7js/taraxa-node/libraries/core_libs/consensus/src/dag/dag_block_proposer.cpp:181
#10 0x0000560c8111e516 in taraxa::DagBlockProposer::start()::$_0::operator()() const (this=<optimized out>) at /tmp/tmp2efjn7js/taraxa-node/libraries/core_libs/consensus/src/dag/dag_block_proposer.cpp:219
#11 std::__invoke_impl<void, taraxa::DagBlockProposer::start()::$_0>(std::__invoke_other, taraxa::DagBlockProposer::start()::$_0&&) (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61
#12 std::__invoke<taraxa::DagBlockProposer::start()::$_0>(taraxa::DagBlockProposer::start()::$_0&&) (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96
#13 std::thread::_Invoker<std::tuple<taraxa::DagBlockProposer::start()::$_0> >::_M_invoke<0ul> (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:301
#14 std::thread::_Invoker<std::tuple<taraxa::DagBlockProposer::start()::$_0> >::operator() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:308
#15 std::thread::_State_impl<std::thread::_Invoker<std::tuple<taraxa::DagBlockProposer::start()::$_0> > >::_M_run (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:253
#16 0x0000560c824a6f54 in execute_native_thread_routine ()
#17 0x00007f217f6cdb7b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x00007f217f74b7b8 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

After this crash, the database seems corrupted and the node kept crashing with:

#0  runtime.raise () at /usr/lib/go-1.24/src/runtime/sys_linux_amd64.s:154
#1  0x00005587dca01fa5 in runtime.dieFromSignal (sig=6) at /usr/lib/go-1.24/src/runtime/signal_unix.go:967
#2  0x00005587dc9ea913 in runtime.crash () at /usr/lib/go-1.24/src/runtime/signal_unix.go:1056
#3  runtime.fatalpanic (msgs=<optimized out>) at /usr/lib/go-1.24/src/runtime/panic.go:1328
#4  0x00005587dca1daa8 in runtime.gopanic (e=...) at /usr/lib/go-1.24/src/runtime/panic.go:811
#5  0x00005587dcb97089 in github.com/Taraxa-project/taraxa-evm/taraxa/util/asserts.Holds (condition=<optimized out>, msg=..., ret=<optimized out>) at /tmp/tmp2efjn7js/taraxa-node/submodules/taraxa-evm/taraxa/util/asserts/index.go:18
#6  github.com/Taraxa-project/taraxa-evm/taraxa/trie.Reader.resolve (self=..., db_tx=..., hash=0xc00942e260, key_prefix=..., ret=..., ret_bytes=...) at /tmp/tmp2efjn7js/taraxa-node/submodules/taraxa-evm/taraxa/trie/reader.go:79
#7  0x00005587dcb96bf5 in github.com/Taraxa-project/taraxa-evm/taraxa/trie.Reader.for_each_node_hash (self=..., db_tx=..., n=..., cb={void (github.com/Taraxa-project/taraxa-evm/common.Hash *, []uint8)} 0xc000c35cf0, prefix=...)
    at /tmp/tmp2efjn7js/taraxa-node/submodules/taraxa-evm/taraxa/trie/reader.go:27
#8  0x00005587dcb9cf71 in github.com/Taraxa-project/taraxa-evm/taraxa/trie.Reader.ForEachNodeHash (self=..., db_tx=..., root_hash=0xc00942e260, cb={void (github.com/Taraxa-project/taraxa-evm/common.Hash *, []uint8)} 0xc000c35d38)
    at /tmp/tmp2efjn7js/taraxa-node/submodules/taraxa-evm/taraxa/trie/reader.go:21
#9  github.com/Taraxa-project/taraxa-evm/taraxa/state/state_db.ExtendedReader.ForEachMainNodeHashByRoot (self=..., storage_root=0xc00942e260, f={void (github.com/Taraxa-project/taraxa-evm/common.Hash *, []uint8)} 0xc000c35d38)
    at /tmp/tmp2efjn7js/taraxa-node/submodules/taraxa-evm/taraxa/state/state_db/block_reader.go:73
#10 0x00005587dcba9f52 in github.com/Taraxa-project/taraxa-evm/taraxa/state/state_db_rocksdb.(*DB).recreateMainTrie (self=0xc000243cc0, state_root_to_keep=0xc000d183d8, blk_num=<optimized out>)
    at /tmp/tmp2efjn7js/taraxa-node/submodules/taraxa-evm/taraxa/state/state_db_rocksdb/db.go:195
#11 0x00005587dcbab2af in github.com/Taraxa-project/taraxa-evm/taraxa/state/state_db_rocksdb.(*DB).Prune.func2 () at /tmp/tmp2efjn7js/taraxa-node/submodules/taraxa-evm/taraxa/state/state_db_rocksdb/db.go:295
#12 0x00005587dca24ec1 in runtime.goexit () at /usr/lib/go-1.24/src/runtime/asm_amd64.s:1700
#13 0x0000000000000000 in ?? ()

Description

Lite node crashed today. The database didn't survive and

Steps to Reproduce

N/A

Expected Result

N/A

Actual Results

N/A

Environment

Node version

v1.13.2

Operating System details.

Debian 13

CPU, memory, disk details.

CPU: Intel(R) Xeon(R) CPU E3-1275 v6 @ 3.80GHz
Disk: NVMe 256GB
Memory: 65GB