How to "compile in" code for a product?

[todbot]

CircuitPython is old enough now that people (including myself) are selling products using it.
But installing the firmware for these products is pretty onerous for the end user:

• Trigger UF2 bootloader
• Install CircuitPython
• Wait for CircuitPython to present CIRCUITPY drive
• Copy over needed libraries to CIRCUITPY
• Copy over settings.toml and code.py
• Reset
• Be sure to watch REPL in case you didn't copy the library files correctly!

In the case of RP2040/RP2350, we can at least use picotool to create a UF2 of a working system and distribute that.

But is there a way to do this for the general case? Those who make ESP32-based products would really benefit I think.

As a stopgap, I created #10073 to let us freeze-in some functionality, maybe a glorified "hello world" that shows the unit works. But it would be really neat if I could tell the CircuitPython build system "here's a directory, make it the CIRCUITPY drive". I realize this may need some help from the platform-specific UF2 bootloaders, but maybe do-able?

[dhalbert]

esptool.py also has dump/load options that can do all of flash. So you could use it like you use picotool.

[todbot]

Agreed, but requiring the user to install esptool.py for product updates is also asking a lot.

[dhalbert]

Agreed, but requiring the user to install esptool.py for product updates is also asking a lot.

You can point the user to https://adafruit.github.io/Adafruit_WebSerial_ESPTool/ or similar browser tools. Especially on Espressif chips, since the UF2 bootloader is not protected, they may need to be familiar with it anyway.

[tannewt]

Agreed, but requiring the user to install esptool.py for product updates is also asking a lot.

Do we really need CircuitPython to support this? Would it be enough to have a folder -> fat filesystem -> uf2 converter? This would be orthogonal to CircuitPython except for placing the filesystem in the same place.

I don't really want to bundle with CP itself because it'll overwrite the user's file system.

[tannewt]

Also, I think many other UF2 bootloaders besides the RP2 version have a current.uf2 file that images the whole flash including the filesystem.

[todbot]

Do we really need CircuitPython to support this? Would it be enough to have a folder -> fat filesystem -> uf2 converter? This would be orthogonal to CircuitPython except for placing the filesystem in the same place.

This would be ideal. Is there a way to pull out of a CircuitPython UF2 all the information needed to append the CIRCUITPY filesystem? It seemed to me this hypothetical tool would need to know some internals of particular CircuitPython builds in order to do that. And/or would need a modified UF2 bootloader for those systems with internal+external flash.

I don't really want to bundle with CP itself because it'll overwrite the user's file system.

If we could do it outside of CirPy, that'd be awesome. The current stop-gap PR works just like the "make a default code.py when no filesystem exists" so there's no worries about filesystem overriding. But point taken: CircuitPython programmers are used to the filesystem staying intact when upgrading CircuitPython version.

Also, I think many other UF2 bootloaders besides the RP2 version have a current.uf2 file that images the whole flash including the filesystem.

afaik, only the SAMD21 bootloader can do this.

[jepler]

I wrote this a long time ago, thinking it might be useful for shipping prepared CIRCUITPY filesystems, but it was never used and so I archived it: https://github.com/jepler/mkfatimg

[tannewt]

Do we really need CircuitPython to support this? Would it be enough to have a folder -> fat filesystem -> uf2 converter? This would be orthogonal to CircuitPython except for placing the filesystem in the same place.

This would be ideal. Is there a way to pull out of a CircuitPython UF2 all the information needed to append the CIRCUITPY filesystem? It seemed to me this hypothetical tool would need to know some internals of particular CircuitPython builds in order to do that. And/or would need a modified UF2 bootloader for those systems with internal+external flash.

The filesystem isn't "appended" necessarily. It wouldn't be easy to get the location out of a UF2. You might be able to get it out of an elf with debug info. The location doesn't vary build to build because we don't want to destroy user file systems. The location may only change on major version boundary.

What platform are you targeting specifically? The implementation will depend on that.

The broadcom port builds its own SD card filesystem image here:

circuitpython/ports/broadcom/Makefile

Lines 150 to 156 in 457edc3

	$(Q)dd if=/dev/zero of=$(BUILD)/circuitpython-disk.img bs=1 count=0 seek=256M
	$(Q)parted -s $(BUILD)/circuitpython-disk.img mktable msdos
	$(Q)parted -s $(BUILD)/circuitpython-disk.img mkpart primary fat32 0% 100%
	$(Q)mkfs.fat -F 32 -n BOOT --offset=2048 $(BUILD)/circuitpython-disk.img
	$(Q)mcopy -i $(BUILD)/circuitpython-disk.img@@1M config.txt firmware/bootcode.bin firmware/fixup* firmware/start* firmware/*.dtb ::
	$(Q)mcopy -i $(BUILD)/circuitpython-disk.img@@1M $(BUILD)/kernel$(SUFFIX).img ::

[todbot]

The filesystem isn't "appended" necessarily.

Ah okay. I phrased it that way because at some point I recall having success literally concatenating UF2 files to append the CIRCUITPY filesystem on rp2. Looks like there's things like uf2tool now to be little less cavalier about it.

What platform are you targeting specifically? The implementation will depend on that.

For architectures, I think rp2 and esp32 would cover most systems people are concerned about. (The lack of something like picotool to produce full CirPy+filesystem for esp32 has definitely steered me away that using that chip in a product these last few years)

[RetiredWizard]

The Blues.io circuitpython-fs-builder might be of interest to you, https://github.com/blues/circuitpython-fs-builder

I stumbled across it while playing with the Swan microcontroller a couple years ago. It's really meant for a different use case, but I thought it might be useful for the use case I think you're talking about here. I'm easily distracted though and didn't really follow up. There's a pretty good video explanation starting at around 50:20 of this video: https://youtu.be/fUUOmpEwpCE

Edit: Perhaps this tool is only applicable to STM boards though?

[tannewt]

For architectures, I think rp2 and esp32 would cover most systems people are concerned about.

rp2 should be possible already. The bootloader allows you to overwrite any portion of flash.

ESP32 doesn't allow a uf2 to write outside of the ota_0 partition. (tinyuf2 README So, we'd need to add the ability to write to the data partition.

Alternatively, idf.py could be used to flash directly. It can include both TinyUF2, CircuitPython and the filesystem.

[bill88t]

Regarding #10073:
I have an idea. A whole fs into an oneliner. I'm doing the generation script right now.
If I just disable auto-reload, import os, and blast things in code.py, eventually overwriting code.py itself, it's gonna work perfectly.
Since zlib exists I can just encode the payload and decode it.