Skip to content

fix(kaos): Make a failed install cost only what it overwrote - #115

Merged
jbenedictocenteno merged 9 commits into
developfrom
fix/install-recovery
Nov 13, 2026
Merged

jbenedictocenteno merged 9 commits into
developfrom
fix/install-recovery

Conversation

@jbenedictocenteno

Copy link
Copy Markdown
Contributor

Description of Change

An install that needs room takes apps out of the directory before the first byte arrives. Whatever then went wrong, a dropped link, a bad file, a reset, cost every one of them, though the region erases 64 KB as the data gets there and most of that flash was never touched.

Giving back what was not reached

  • A failed install puts back every victim its write did not reach, in one directory commit. The log says which were kept and which were overwritten.
  • The same after a reset: the table of victims goes to NVS (kappmgr/undo) before the eviction commit, each with a CRC-32 of its extent. A start-up that finds one takes it out of NVS first, recomputes each CRC 512 KB a step, and puts back the victims whose bytes are as they were.
  • A file on the card counts as an app's copy only at the image's length. Any /sdcard/<project>.bin, an older version included, used to let the app be evicted with "its copy is on the card".

Judging an image before hashing it

  • esp_image_verify(), and esp_ota_end() through it, hash every segment by the length its header declares and compare the total with the partition only afterwards. Over an image cut short those lengths are read from stale flash. The region and the OS slot now walk the segment chain first, as the mailbox already did, and refuse such an image unhashed.
  • A raw image cut inside its first segment is refused before anything is placed: the header says how long the segment is and the source how long the file is.

Saying what happened

  • install and app_run answer AO_AppMgr's verdict, not >ok once the request is posted: >err code=4 msg=busy, or >err code=5 msg=missing|empty|refused. install src="pending" with an empty mailbox says so and drops the request's flags.
  • The write card and the LED follow the progress reports alone, and an https install made none until DNS, TLS and the headers were done. AO_AppMgr reports 0 % when the install is accepted and again every second while it waits on the network.
  • A terminal install over BLE is dropped after 15 s without a chunk, not 5 s (the cable keeps 5 s), and the launch after run=on waits for the console's queued bytes to leave the link, so the >ok arrives before the reset.

Cost: taking a victim's CRC adds about 70 ms per MB to the step that evicts it (0.44 s for 6.2 MB), in the order of the verify at the end.

Test Scenarios

  • tools/host_tests.sh and tools/check_layers.sh, green at this branch; tests/target built on v6.1 in its five variants with no warnings on the whole series.
  • Two DVT boards, kodeOS built from this series, real apps from the store (2.9 to 4.2 MB):
What breaks What the board does
Wi-Fi cut at 40 % of a download install failed; the apps it had not reached are back, the one overwritten is named
Reset at 20 to 25 % with two victims Boots, checks in 1.2 s, gives one back and names the other
16 installs reset at random points, 0 to 99 % Boots every time with no watchdog; loses only the apps it names
Cable transfer stopped half way Gives up after 5 s: the host is presumed gone
BLE transfer: client killed, or link dropped at 30 % The same after 15 s, and advertises again
A second install while one runs >err code=4 msg=busy; the first ends well
A file cut inside its first segment Refused before anything is evicted
A file cut further on, or an image that claims more than it holds Refused at the end in under 1 s, unhashed
An OS image cut short, as an update Refused at the first chunk; the running slot does not change
  • An earlier form of the reset recovery verified each victim with esp_image_verify() and put a board in a boot loop on the bench (the watchdog, at every boot, with the table still in NVS). The recovery in this change does not parse the image, and removes the table before it checks anything.
  • Not tested: an install that reaches a board asleep on battery.

Related links

install src="pending" with nothing parked answered ok and said nothing,
anywhere. It also left the request's destination and run flag set, so
the next adoption of the same boot took them: an image uploaded after it
was installed as that request asked, not as its own upload did.

An install that was asked for logs that the mailbox is empty, and the
request's flags are dropped. The boot's own look stays quiet.
An install took its victims out of the directory before the first byte
arrived, so a CRC mismatch, a watchdog or a dropped link cost every app
it would have replaced. The region erases 64 KB as the data gets there,
so most of that flash was never touched. A failed install puts back every
victim its write did not reach, in one directory commit. The log says
which ones it kept and which were overwritten.
The console said `>ok` as soon as it had posted the request, even when
AppMgr was busy, the file was missing or the mailbox was empty. AppMgr
now posts the asker one verdict: taken, busy, missing, empty or refused.
The console holds the line for it and closes with `>ok`,
`>err code=4 msg=busy` or `>err code=5 msg=<word>`. Requests dropped
while AppMgr is busy log a warning that names them.
The write card and the LED follow APP_FLASH_PROGRESS_SIG alone, and an
https install made none until DNS, TLS and the headers were done, up to
30 s. A data fetch made none at all. AppMgr reports 0 % when an https
install is accepted and when a fetch begins, and says the percent again
every second while it waits on the network. A percent the event pool had
no room for is not counted as reported.
A terminal install over BLE was dropped after 5 s without a chunk, while
the phone app waits 10 s for an ack before it retries. Over BLE the bound
is 15 s; the cable keeps 5 s. The launch after `run=on` also waits, up to
2 s, for the console's queued bytes to leave the BLE link, so the `>ok`
reaches the phone before the board resets.
The table of an install's victims lived in RAM, so a reset in the middle
of a download cost every app it had evicted, though most of their flash
was never overwritten. AppMgr now writes the table to NVS before the
eviction commit, each victim with a CRC-32 of its extent, and erases it
when the install ends. A start-up that finds one takes it out of NVS,
recomputes each CRC 512 KB a step and puts back, in one commit, the
victims whose bytes are as they were and that no entry lies over. The
image is not parsed: over an extent a reset left half written,
esp_image_verify() hashes by lengths read from stale flash and holds
the step past the task watchdog.
esp_image_verify(), and esp_ota_end() through it, hash every segment by
the length its header declares and compare the total with the partition
only afterwards. Over a file cut short those lengths are read from stale
flash and the hash can run past the task watchdog. The region and the OS
slot now walk the chain first, the way the mailbox already did, and
refuse such an image unhashed.
A file cut short was written over the apps it displaced and refused
only by the verify at the end, when they were gone. The image header
says how long the first segment is and the source how long the file is,
so the decoder now refuses such a file where it refuses another chip's
image: before anything is placed. A cut further on is still the end's.
The keep round counted any /sdcard/<project>.bin as the victim's copy.
An older version of the app, kept by an earlier eviction, or another file
of that name, then let the app be evicted with "its copy is on the card"
when the card held something else. A file is the copy now only at the
image's length; any other is said in the log and the app is copied over
it. The Make room card's per-victim answer follows the same rule.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant