[Bug]: Atom Archival Node v22.1.0 Not syncing to peers

[mayank-daga]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Hi ,
I am running atom archival node v22.1.0 but getting following error logs:

6:27AM INF service start impl=MConn{202.61.240.62:14956} module=p2p msg="Starting MConnection service" peer=[email protected]:14956

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 57.180.212.166:26656: i/o timeout" module=p2p tries=4

6:27AM INF service start impl="Peer{MConn{65.109.157.115:26656} 870bf4d4f3c39b183155187d5cdef327202d0ac9 out}" module=p2p msg="Starting Peer service" peer=[email protected]:26656

6:27AM INF service start impl=MConn{65.109.157.115:26656} module=p2p msg="Starting MConnection service" peer=[email protected]:26656

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:34574->66.165.248.66:26656: i/o timeout" module=p2p tries=3

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:2000 err="dial tcp 185.119.118.109:2000: i/o timeout" module=p2p tries=4

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=4

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=4

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:52392->65.21.47.174:14956: i/o timeout" module=p2p tries=3

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:52942->65.109.63.97:26656: i/o timeout" module=p2p tries=3

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:34536->148.113.16.99:14956: i/o timeout" module=p2p tries=3

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:44564->5.9.95.101:14956: i/o timeout" module=p2p tries=3

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 128.199.128.15:26090: i/o timeout" module=p2p tries=5

6:27AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 168.119.138.215:26656: connect: connection refused" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:29656 err="dial tcp 103.241.50.31:29656: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 135.181.174.106:26656: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 178.128.238.183:26090: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:28156 err="dial tcp 65.108.98.235:28156: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:2000 err="dial tcp 185.119.118.109:2000: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 57.180.212.166:26656: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 168.119.138.215:26656: connect: connection refused" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:34578->66.165.248.66:26656: i/o timeout" module=p2p tries=4

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:33962->65.21.47.174:14956: i/o timeout" module=p2p tries=4

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 128.199.128.15:26090: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:45494->148.113.16.99:14956: i/o timeout" module=p2p tries=4

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:38294->5.9.95.101:14956: i/o timeout" module=p2p tries=4

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:34028->65.109.63.97:26656: i/o timeout" module=p2p tries=4

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 135.181.174.106:26656: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:29656 err="dial tcp 103.241.50.31:29656: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 178.128.238.183:26090: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:28156 err="dial tcp 65.108.98.235:28156: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 57.180.212.166:26656: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 168.119.138.215:26656: connect: connection refused" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:2000 err="dial tcp 185.119.118.109:2000: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:57222->66.165.248.66:26656: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 128.199.128.15:26090: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:59720->65.21.47.174:14956: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 178.128.238.183:26090: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 135.181.174.106:26656: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:54020->148.113.16.99:14956: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:28156 err="dial tcp 65.108.98.235:28156: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:52818->5.9.95.101:14956: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:59716->65.109.63.97:26656: i/o timeout" module=p2p tries=5

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:2000 err="dial tcp 185.119.118.109:2000: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:29656 err="dial tcp 103.241.50.31:29656: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 168.119.138.215:26656: connect: connection refused" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 57.180.212.166:26656: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 128.199.128.15:26090: i/o timeout" module=p2p tries=8

6:28AM INF Ensure peers module=pex numDialing=2 numInPeers=0 numOutPeers=8 numToDial=0

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 178.128.238.183:26090: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 135.181.174.106:26656: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:57788->66.165.248.66:26656: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:28156 err="dial tcp 65.108.98.235:28156: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:2000 err="dial tcp 185.119.118.109:2000: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:54034->148.113.16.99:14956: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 168.119.138.215:26656: connect: connection refused" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:33990->65.21.47.174:14956: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:29656 err="dial tcp 103.241.50.31:29656: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:35282->65.109.63.97:26656: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:55748->5.9.95.101:14956: i/o timeout" module=p2p tries=6

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 57.180.212.166:26656: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=8

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 128.199.128.15:26090: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 135.181.174.106:26656: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:2000 err="dial tcp 185.119.118.109:2000: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 178.128.238.183:26090: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 168.119.138.215:26656: connect: connection refused" module=p2p tries=10

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:28156 err="dial tcp 65.108.98.235:28156: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:29656 err="dial tcp 103.241.50.31:29656: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:49060->66.165.248.66:26656: i/o timeout" module=p2p tries=7

6:28AM ERR Error in validation err="wrong Block.Header.AppHash. Expected 014D6642D091F0E4FB3F30CBEDB9D8C3649BE7E73E6EB54D2BD44330FBF756C8, got 56D225F2A3E7BCDF515EA0722CEFAE569A3361EC345B1426441EA70933D8542C" module=blocksync

6:28AM ERR Stopping peer for error err="reactor validation error: wrong Block.Header.AppHash. Expected 014D6642D091F0E4FB3F30CBEDB9D8C3649BE7E73E6EB54D2BD44330FBF756C8, got 56D225F2A3E7BCDF515EA0722CEFAE569A3361EC345B1426441EA70933D8542C" module=p2p peer="Peer{MConn{116.202.85.179:26656} 6681cee74de13aaac561442bcbc420bdb025aacc out}"

6:28AM INF service stop impl=MConn{116.202.85.179:26656} module=p2p msg="Stopping MConnection service" peer=[email protected]:26656

6:28AM ERR Stopping peer for error err="read tcp 172.17.0.3:52812->116.202.85.179:26656: use of closed network connection" module=p2p peer="Peer{MConn{116.202.85.179:26656} 6681cee74de13aaac561442bcbc420bdb025aacc out}"

6:28AM ERR error while stopping peer error="already stopped" module=p2p

6:28AM INF service stop impl="Peer{MConn{116.202.85.179:26656} 6681cee74de13aaac561442bcbc420bdb025aacc out}" module=p2p msg="Stopping Peer service" peer=[email protected]:26656

6:28AM INF Reconnecting to peer addr=[email protected]:26656 module=p2p

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:46220->148.113.16.99:14956: i/o timeout" module=p2p tries=7

6:28AM INF service start impl="Peer{MConn{116.202.85.179:26656} 6681cee74de13aaac561442bcbc420bdb025aacc out}" module=p2p msg="Starting Peer service" peer=[email protected]:26656

6:28AM INF service start impl=MConn{116.202.85.179:26656} module=p2p msg="Starting MConnection service" peer=[email protected]:26656

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:34000->65.21.47.174:14956: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 57.180.212.166:26656: i/o timeout" module=p2p tries=9

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26090 err="dial tcp 128.199.128.15:26090: i/o timeout" module=p2p tries=10

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="dial tcp 135.181.174.106:26656: i/o timeout" module=p2p tries=10

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.3:46254->5.9.95.101:14956: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.3:43648->65.109.63.97:26656: i/o timeout" module=p2p tries=7

6:28AM INF Error reconnecting to peer. Trying again addr=[email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=9

Gaia Version

v22.1.0

How to reproduce?

No response

[Eric-Warehime]

Did this start occurring around the time of the upgrade this morning? It's possible that some of your peers didn't cleanly upgrade and reconnects will fail to those peers.

[mayank-daga]

Hi @Eric-Warehime no it was happening before I upgraded. So i thought an upgrade might resolve the issue but it didn’t work

[mayank-daga]

Hi @Eric-Warehime just following up I tried resertting with new peers and running but still same issue. Any help would be appreciated

[mayank-daga]

HI just following up if anyone can help

[mayank-daga]

@mpoke @MSalopek