Description
What happened?
After comparing the PublicKeyToMsaId values with msa_getKeysByMsaId rpc. It appears that we got some inconsistency between them and some of the keys are not mapped to any MsaId in offchain worker.
It is not evident what might've caused this but my leading theory is that this is most probably due to forking. We had an assumption that if a fork happens the last event would win meaning that if fork happened any previous events would eventually get overridden with correct ones and since we only apply those events in offchain DB when we are sure a block is finalized we are going to store the correct keys and values but this assumption might be wrong and it might be cases that the winning branch might have be written with a losing one and that would cause anomalies on the offchain side.
- We need to fix the root cause which means we need to find a way to only apply the changes came from canonical branch
- We need to have a way to fix (re-index) these keys that are not indexed correctly . A runtime or rpc might be useful for this until we might be able to find the root cause.
Some keys that are not correctly mapped in mainnet:
MsaID: 1414115
key: 0x56da6ee0934baeaa36dcece2b91d44807ab7b616fc79f0a131199365599bee18
MsaID: 1548906
key: 0x3858cb5715f4ac1cce83a8b5ae5119dd31c552592861ff76db965283254eb765
Some keys that are attached to a msaId in reverse-index but not included in PublicKeyToMsaId
MsaId: 1329603
key: f6YfqZ75qHVHfFVaTXt9Fo9Lw8syfFpTV9vjU1i6xfPeBTRmy
Relevant log output
1329606a1329607
> 1329603,0x32587886e31f110cb7a92bbeb72028215d08dbf193522f48c2f2263b2a34fc0b
1394380a1394382
> 1394377,0x3a9cfcec7b3a9a8032d985bd268f0cb804dd31f75cab9679f402cd2b6e9e7467
1414118d1414119
< 1414115,0x56da6ee0934baeaa36dcece2b91d44807ab7b616fc79f0a131199365599bee18
1428900a1428902
> 1428898,0x469207aad5cb4dcc958b5b48d114847b30080e821a264a1a2b8cc2723b278e69
1428901a1428904
> 1428899,0x84f020fbacf3629893fcf235583536d593930840e516490c3c6187878df89042
1477970a1477974
> 1477968,0x26dc70b3e7c0cee91c9eeec43054f10388a9abaee0db09c2ed601ff8579fb473
1548909d1548912
< 1548906,0x3858cb5715f4ac1cce83a8b5ae5119dd31c552592861ff76db965283254eb765
1551524a1551528
> 1551521,0x8007de296e6145c0b2c649465aa596bab630e76fe0d646559c3a26a5d5d0d522
1588703a1588708
> 1588701,0x0c63daffd49eb5e7c8cccb1bda350d246024d8b3fd79dbfc5c3c52af13b2880f
1614352a1614358
> 1614349,0x48b6c20f4073216afdd946c42dc0cbab64da7723d78bdde59f3c8bd097b5be7d
1648942a1648949
> 1648939,0xecd4d0dc82fe6f831333719699853b9b98742037aabb6a3d8d4873c3064d9863
1651062c1651069
< 1651059,0xb873312b7009b576b94a3ff17fe82ff23cffe17a0829e762ff136e68133d6e63
---
> 1651059,0x54588afe3ab43301972485a461d4693fa04c4e5225d6abb780ccbfe322962732
1661697a1661705
> 1661695,0x9273549bd806e9b252bcc9a4aae593c893968ca9ae1e8f33ebfbc310a1e9bf17
1666004a1666013
> 1666002,0x5a956272fe5513badfe6f733b7a7c822396ba886581cf40d277a3792a44cfb0f
1666006c1666015
< 1666003,0x5a956272fe5513badfe6f733b7a7c822396ba886581cf40d277a3792a44cfb0f
---
> 1666003,0xc23218afacd868a064af2403a50b1140fa81f0eca706916cef5d23b844574f09
1667994a1668004
> 1667991,0xcaed2902447150f719b2aa37b64d1b0be4661e537b7c8cb8f6036c981ded9c31
1668161a1668172
> 1668159,0x9e21becafc027aba8dee05c639bd6763a8caf0509f7e494efddeb0e68abd2108
1674978a1674990
> 1674976,0xd0c085962a121852b49434438e46134abe52d51861ba368f83a5672fc7f52031
1674984a1674997
> 1674982,0x70f27b380ae1e69d29fcc47e1be76438110e132fe715603db181b62d4cdb9720