Corruption in some packets due to payload being partially overwritten by path bytes

[warrenjmcdonald]

A high amount of packet corruption was being detected from several pyMC repeaters using E22P 915 modules. An investigation was started to determine where the corruption was happening which revealed a corruption pattern that seems deterministic.

In advertisements and group text messages, which were chosen for practical evidence collection, it is evident that the path bytes are being written over a portion of the payload up till the last 2 bytes. The corruption is seen in both letsmesh and error messages when these packet are attempted to be ingested into Corescope.

The hashes are different and the signatures are invalid. Sometimes the corruption is happening on retransmit and some nearby observers pass the packets intact to letsmesh. Sometimes the observers on pyMC-repeater send the corrupt packets to MQTT.

It is unclear what preconditions trigger the corruption.

Some example advert packets from letsmesh.

[dkmr27]

I would like to add one a further example.

Original packet 15 02 a6 49 11 72 e8 5b 81 2e 7a ba 26 7a d7 be ba 63 bb 0a 82 f0 b4 8c 11 67 be 27 78 19 f6 56 7f 79 08 83 fa 7b 1f 76 13 ca d1 73 86 81 9d 28 ac fc c7 d6 c7 5a 77 46 aa b2 1d 76 33 b0 57 f3 90 87 04 15 27 9c 63 95 ae 93 00 28 58 de 91 6c d6 6a 0b 49 76 94 5a 95 f1 d8 70 6a ff 29 24 15 94 59 a2 ec 75 16 39 94 64 90 60 68 93 dc e3 b3 f9 df b8 49 5a 33 24 d4 6c 0d f4 ee a3 74 a2 8b 62 59 5f 9e 62 40 94
This is the packet as heard by the pyMC repeater and sent via mqtt

Corrupted packet 15 03 a6 49 08 11 72 e8 5b 81 2e 7a ba 26 7a d7 be ba 63 bb 0a 82 f0 b4 8c 11 67 be 27 78 19 f6 56 7f 79 08 83 fa 7b 1f 76 13 ca d1 73 86 81 9d 28 ac fc c7 d6 c7 5a 77 46 aa b2 1d 76 33 b0 57 f3 90 87 04 15 27 9c 63 95 ae 93 00 28 58 de 91 6c d6 6a 0b 49 76 94 5a 95 f1 d8 70 6a ff 29 24 15 94 59 a2 ec 75 16 39 94 64 90 60 68 93 dc e3 b3 f9 df b8 49 5a 33 24 d4 6c 0d f4 ee a3 74 a2 15 03 a6 49 94 11 72 e8
as transmitted

End of the payload
Good: 74 A2 8B 62 59 5F 9E 62 40 94
Bad: 74 a2 15 03 a6 49 94 11 72 e8
Note the path a6 49 94 then the beginning of the payload 11 72 e2. The 94 is significant as it's not in the corrupted packet as received/transmitted, but the 94 was seen in paths by other observers, could a second reception of the same hash could be a contributing factor?

[warrenjmcdonald]

The devices in use are Luckfox Pico Ultra with Zindello Industries Luckfox Ultrapeater board https://zindello.com.au/ultrapeater/ with E22P 915M modules. There are many in use around Sydney, but so far we have identified 4 unit exhibiting this behaviour. We can't rule out the others as we are not sure of how they are deployed. The ones identified so far are in high volume sites.

[zindello]

This is potentially caused by pyMC-dev/pyMC_core#78

Waiting on @rightup to review as this is much more his area than mine.

[wx4cb]

not sure if im seeing the same thing here using the mishsmith board, but if you can show me how to get to the packets that have the crc errors, im sure i can get them to you as well.

This is what vscode/claude pulled out of the repeater log - hope this helps

can you look through the repeater log and give me some of the payloads for the CRC packets
2:41 PM


Yep. Here are some of the recent CRC packet payloads from repeater.log, all from today:

#199  13:56:47  RSSI=-97  SNR=-1.8  Len=71
1541bda3813f30b6926c3956770d403a28a30a673a3a49f06f9712a2c4c9dbd51db412188c6f92a9cd2bc2d905b196f80c54e17a6f4e16492a1a513801bdbd8777efb9eb47ed88

#200  13:59:00  RSSI=-75  SNR=-8.5  Len=69
1540812ba005402143af38ef8c5e5e5583c05f7a1af06f9712a2c4c9dbd511d21418bc5f9289e0357efc44fbc3dc6b0f7376fcddd1a32e01eec4ec287748b5519436d2c878

#201  14:00:43  RSSI=-98  SNR=-7.2  Len=75
1543e816243a0a329bdb20970e00b210bb26647b8d7e2b2e3e21aff06f9712a2c4c9dbd511d21418bc5f925fa53f8464a952e21df017fa1184b9f23b362067db8c161b013e05bce65ad1bb

#202  14:01:16  RSSI=-98  SNR=-7.0  Len=75
1543e886bda30a83810b8d7400e4966b345c075c20bc8aa3ab9d18f06c9722a2c4c9dbb274f11f18bc5f92103dbae808a5c48a8c0a55dad67fce66492a1a513801bdbd8777efb9eb47ed88
A few earlier today:

#196  10:52:42  RSSI=-67  SNR=+5.8  Len=69
154381f6dd3f0f4e1b189a7e2d6e83c85835714270f06f9712a2c4c9dbd511d21418bc5f92920e666de3773a10c73621b6f6c0b0d1baf5907a2c0ae9b797822f72b4ed4fa2

#197  10:53:17  RSSI=-98  SNR=-5.8  Len=73
1542e8860a8381d1a284ffb9907e2d9b452cc7f0e6995b0802a06c9712a2c4c9dbd511d21418bc5f925b68b2fc734aecf369e7dce89c02b81f28a2bf2c3f5250695c1b8662d56e7571

#198  10:56:34  RSSI=-76  SNR=-8.2  Len=69
1540818b6abe1df557434f459c7123c76368672603f06f9712a2c4c9dbd511d21418bc5f9276051048657cab804c1955fc609a4a18a32e0177cdb92c7748b55b525682fb7b
The recent ones mostly look like normal MeshCore-looking frames that failed CRC late, not random garbage. Several start with 1541, 1542, or 1543, which lines up with flood/group text style packets with different path lengths.

[rightup]

Some changes have been made to tighten the locking process between tx/rx please pull latest dev branch and see if this improves.