HMY prone to more "nonce too low" and "transaction underpriced" issues

[mindstyle85]

Recent hmy versions seem to have more issues with nonces getting stuck. We have had a few experiences with that in p-ops group, even myself in my testnet validator (no scripts or high frequency tx sending).

Seb also confirmed this is getting more frequent, and today even Li started experiencing it with his sending script.

Will post any screenshots if this happens to me again, but i think this can easily be reproduced; cc @sebastianj

[mindstyle85]

adding to it, it seems to become a frequent bug with onewallet and metamask also; adding "transaction not confirmed after 40 seconds" to this list too as i believe they all originate from the same core bug

[rlan35]

quote RockTheBlockchain validator:

Hi Rongjian. Thanks for checking in on this for me.

what happened before the first failed txn on your address? Did you send two txns fairly closely in time? Can you give me more context?
+++ Yes. I was adding keys back-to-back fairly quickly from the same node. The first five were successful, while the 6th generated the error above and all additional attempts, including sending ONE, had failed. This was just before epoch change this morning (US) and I had probably entered that last one moments before the change, which I could imagine might contribute to the problem.
To resolve the issue, I had to send and receive ONE with a high gas price to and from the validator wallet. After doing that a few times, I waited about two hours and it seems the "edit-validator" commands are working now.
After sending or receiving ONE from the wallet, it appeared like a previous --remove-bls-key had unstuck itself. I still had to wait a couple hours but now it seems to be functioning.

In case you're still interested, and if it helps, one of the previous failure examples is:

root@SF-HMY-1:~# ./hmy --node="https://api.s0.t.hmny.io/" staking edit-validator --validator-addr one1p2rmvndevvw682qynqu08hyvx24hh4runsw6pz --remove-bls-key 748b82cd23........cec02d7bcbb14 --passphrase

Enter wallet keystore passphrase: .....

[EditValidator] existing transaction price was not bumped enough: replacement transaction underpriced -- 2021-03-29 21:55:48 +0000 UTC
{"transaction-hash":"0x3de62dfc46b040f5555403d02ad103463ecde376624b074474051f0e46c84eba"}
commit: v403-fe5160a, error: staking transaction error

Before that example, I had a different error:

root@SF-HMY-1:~# ./hmy --node="https://api.s0.t.hmny.io/" staking edit-validator --validator-addr one1p2rmvndevvw682qynqu08hyvx24hh4runsw6pz --remove-bls-key 618555d20bb.........25b1420d1790 --passphrase

Enter wallet keystore passphrase:

Try increasing the timeout or look for the transaction receipt with hmy blockchain transaction-receipt
{"transaction-hash":"0xfa44213f23d926fbaf608d07c337760fd8d4112e15688bdc8cfa5b8a2ea0af51"}
commit: v403-fe5160a, error: could not confirm 0xfa44213f23d926fbaf608d07c337760fd8d4112e15688bdc8cfa5b8a2ea0af51 even after 40 seconds

[lijiang2087]

i got this issue when using the payment script

./hmy-csv.py ~/Downloads/file.csv --fast --node https://api.s0.t.hmny.io/

[mindstyle85]

i got this issue when using the payment script

./hmy-csv.py ~/Downloads/file.csv --fast --node https://api.s0.t.hmny.io/

Hmm this just looks like youre passing nonce that was already used, you would need to check on explorer which is your latest nonce, or use ./hmy blockchain current-nonce <address> --node="https://api.s0.t.hmny.io"

[JackyWYX]

1. Observed from log

A typical worker proposing transaction log message from leader node.

Observed facts:

1. Transaction pool have a lot of pending transactions, but only a few of them can be executed correctly for each block.
2. Normal transactions execution errors are not observed. All failed transactions are staking transactions.
3. These failed staking transactions will stay in transaction pool for a long time until they are removed after a long time or use promote another transaction with high gas price.
4. Almost all errored staking transactions has the same error: Staking delegation does not have enough balance.
5. One account has a "slot key to add already exists" error, and followed by trailing "nonce too high" error. This might results from user sending the same edit-validator transaction multiple times before transaction verified.
6. The trailing "Nonce too high" error comes from failed execution of the previous "duplicate slot key" transaction.

2. Analysis

2.1 Why staking transaction will stay in transaction pool for a long time? What's the further impact?

Because when the leader node failed to execute the staking transaction, there is no "receipt" from the block produced to inform transaction pool that the staking transaction has been processed but failed. So the staking transaction is staying in leader node's txPool and being executed for each block proposal.

These staking transactions staying the transaction pool will block future transactions for a long time since they are taking low nonces, which might be the potential cause of transaction block issue.

2.2 Staking transaction validation

Every staking need to go through validateTx (core/tx_pool.go:677) before adding to the transaction pool. In validateTx, balances and validator information are also checked to ensure the staking transaction will be executed correctly. So ideally, all transactions added to txPool.pending shall be executed correctly. However there are two exceptions:

1. Validation for delegation staking is not complete because of redelegation. Since delegation amount computation requires existing undelegation information, txn.Cost is not checked for delegation. On the other hand, txPool.validateStakingTx only check balance without gas fees, which causes some delegation transactions slipped through validation but cannot be executed. (Some new users might not be aware of the transaction fee for staking transaction)
2. The statedb in txPool is static (does not change value as new transactions coming in). So if a user send multiple transactions at a time, the validation might not correctly validation the transactions after the first one. This is the cause of "slot key to add already exists" error in the log. The user send multiple staking transactions with the same edit-validator request, which causes the first one to success, but blocked at the second one.

2.3 Reproduction

The problem is reproduced in local net with sending delegation transaction with the same amount of balance.

3. Proposed fixes

3.1 Temporary fix

1. Add balance validation for delegation with gas ([txPool] added gas balance check when validating delegate transaction harmony#3648).
2. Add a validation before broadcasting staking transaction in RPC endpoint (need to be discuss).
3. For staking dashboard, max value set to current balance - 1 to avoid user neglect the gas price setting.

3.2 Systematic fix

For each failed staking transaction, add it to block with a failed receipt, so that there is a confirmation of whether the staking transaction is failed or not. A fork is required.

[JackyWYX]

@rlan35 @LeoHChen Please take a look. 🖕