Pruning not working on v3.3.1

[chillyvee]

 ERR CONSENSUS FAILURE!!! err="cannot delete latest saved version (190)" module=consensus
stack="goroutine 679 [running]:
runtime/debug.Stack()
        /home/dig/golang/1.19.3/go/src/runtime/debug/stack.go:24 +0x65
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:724 +0x4c
panic({0x2218e20, 0xc005351368})
        /home/dig/golang/1.19.3/go/src/runtime/panic.go:884 +0x212
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc000fb2630)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/chillyvee/cosmos-sdk@v0.46.6-dig.2/store/rootmulti/store.go:441 +0x334
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000e4e540)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/chillyvee/cosmos-sdk@v0.46.6-dig.2/baseapp/abci.go:313 +0x150
github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc00117d080)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/abci/client/local_client.go:264 +0xb6
github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc0027664c0?)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/proxy/app_conn.go:93 +0x22
github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc001486fc0, 0x7}}, {0xc001486fc7, 0x5}, 0x1, 0x43580a, {{0xc002dad2a0, ...}, ...}, ...}, ...)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/state/execution.go:228 +0x269
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc001486fc0, 0x7}}, {0xc001486fc7, 0x5}, 0x1, 0x43580a, {{0xc002dad2a0, ...}, ...}, ...}, ...)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/state/execution.go:180 +0x6ee
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc000db2380, 0x43580a)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:1654 +0xafd
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc000db2380, 0x43580a)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:1563 +0x2ff
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:1498 +0x94
github.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc000db2380, 0x43580a, 0x0)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:1536 +0xccf
github.com/tendermint/tendermint/consensus.(*State).addVote(0xc000db2380, 0xc0049aee60, {0xc000fc57d0, 0x28})
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:2150 +0x18dc
github.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc000db2380, 0xc0049aee60, {0xc000fc57d0?, 0xc004a82100?})
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:1948 +0x2c
github.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000db2380, {{0x2f86700?, 0xc001d27fe0?}, {0xc000fc57d0?, 0x0?}})
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:853 +0x170
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000db2380, 0x0)
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:760 +0x3f9
created by github.com/tendermint/tendermint/consensus.(*State).OnStart
        /home/dig/golang/1.19.3/packages/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/state.go:379 +0x12d

[chillyvee]

Location of panic
https://github.com/chillyvee/cosmos-sdk/blob/v0.46.6-dig.2/store/rootmulti/store.go#L440

Error likely from:
https://github.com/chillyvee/cosmos-sdk/blob/v0.46.6-dig.2/store/rootmulti/store.go#L592

[chillyvee]

Related code in cosmos/iavl

iavl/mutable_tree.go
989: return fmt.Errorf("cannot delete latest saved version (%d)", version)

iavl/nodedb.go
532: return fmt.Errorf("cannot delete latest saved version (%d)", latest)

In this case iavl/nodedb.go is the issue.

func (ndb *nodeDB) DeleteVersionsRange(fromVersion, toVersion int64) error {
...
        latest, err := ndb.getLatestVersion()
...
        if latest < toVersion {
                return errors.Errorf("cannot delete latest saved version (%d)", latest) <--
        }                       
}

Our panic says cannot delete latest saved version (190) which is misleading.

The actual situation is the store we prune only has version 190 (why?) but we are trying to prune up to current height 4413380.

Is getLatestVersion() correct?

If we disable the error, we find that latest = 990, 1000, 1010 and so forth.

This is probably why rollback is broken as well since the latest height is not the blockheight.

Which store keeps heights that are not related to the current height?

Pruning store: key=TransientStoreKey{0xc0012dad60, transient_params}
Pruning store: key=KVStoreKey{0xc0012dac60, staking}
Pruning store: key=KVStoreKey{0xc0012dad30, icahost}
*nodedb.goCannot delete latest=1080 toVersion=(4413301)
*nodedb.goCannot delete latest=1080 toVersion=(4413351)
*nodedb.goCannot delete latest=1080 toVersion=(4413401)
*nodedb.goCannot delete latest=1080 toVersion=(4414240)
Pruning store: key=KVStoreKey{0xc0012dad00, transfer}
Pruning store: key=KVStoreKey{0xc0012dacd0, upgrade}

Turns out it is icahost haunting us!

[chillyvee]

Is there a chance that icahost is stuck on a low height and can't "jump" to the current height due to iavl version restrictions?

Maybe when icahost was introduced, we were supposed to force (SetInitialVersion) the current height to avoid a mismatch?

Currently icahost starts from height 0 probably
rootmulti loadVersion key= KVStoreKey{0xc0015d7990, icahost}<<<<<<< Found previous version 2127

even if the other store heights are much higher.

Is it better to:
A) remove all trace of icahost from snapshots/pruning since we don't use it?
B) manually force a height for icahost?