Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to "compile in" code for a product? #10074

Open
todbot opened this issue Feb 16, 2025 · 11 comments
Open

How to "compile in" code for a product? #10074

todbot opened this issue Feb 16, 2025 · 11 comments
Milestone

Comments

@todbot
Copy link

todbot commented Feb 16, 2025

CircuitPython is old enough now that people (including myself) are selling products using it.
But installing the firmware for these products is pretty onerous for the end user:

  • Trigger UF2 bootloader
  • Install CircuitPython
  • Wait for CircuitPython to present CIRCUITPY drive
  • Copy over needed libraries to CIRCUITPY
  • Copy over settings.toml and code.py
  • Reset
  • Be sure to watch REPL in case you didn't copy the library files correctly!

In the case of RP2040/RP2350, we can at least use picotool to create a UF2 of a working system and distribute that.

But is there a way to do this for the general case? Those who make ESP32-based products would really benefit I think.

As a stopgap, I created #10073 to let us freeze-in some functionality, maybe a glorified "hello world" that shows the unit works. But it would be really neat if I could tell the CircuitPython build system "here's a directory, make it the CIRCUITPY drive". I realize this may need some help from the platform-specific UF2 bootloaders, but maybe do-able?

@dhalbert
Copy link
Collaborator

esptool.py also has dump/load options that can do all of flash. So you could use it like you use picotool.

@dhalbert dhalbert added this to the Long term milestone Feb 16, 2025
@todbot
Copy link
Author

todbot commented Feb 16, 2025

Agreed, but requiring the user to install esptool.py for product updates is also asking a lot.

@dhalbert
Copy link
Collaborator

Agreed, but requiring the user to install esptool.py for product updates is also asking a lot.

You can point the user to https://adafruit.github.io/Adafruit_WebSerial_ESPTool/ or similar browser tools. Especially on Espressif chips, since the UF2 bootloader is not protected, they may need to be familiar with it anyway.

@tannewt
Copy link
Member

tannewt commented Feb 18, 2025

Agreed, but requiring the user to install esptool.py for product updates is also asking a lot.

Do we really need CircuitPython to support this? Would it be enough to have a folder -> fat filesystem -> uf2 converter? This would be orthogonal to CircuitPython except for placing the filesystem in the same place.

I don't really want to bundle with CP itself because it'll overwrite the user's file system.

@tannewt
Copy link
Member

tannewt commented Feb 18, 2025

Also, I think many other UF2 bootloaders besides the RP2 version have a current.uf2 file that images the whole flash including the filesystem.

@todbot
Copy link
Author

todbot commented Feb 18, 2025

Do we really need CircuitPython to support this? Would it be enough to have a folder -> fat filesystem -> uf2 converter? This would be orthogonal to CircuitPython except for placing the filesystem in the same place.

This would be ideal. Is there a way to pull out of a CircuitPython UF2 all the information needed to append the CIRCUITPY filesystem? It seemed to me this hypothetical tool would need to know some internals of particular CircuitPython builds in order to do that. And/or would need a modified UF2 bootloader for those systems with internal+external flash.

I don't really want to bundle with CP itself because it'll overwrite the user's file system.

If we could do it outside of CirPy, that'd be awesome. The current stop-gap PR works just like the "make a default code.py when no filesystem exists" so there's no worries about filesystem overriding. But point taken: CircuitPython programmers are used to the filesystem staying intact when upgrading CircuitPython version.

Also, I think many other UF2 bootloaders besides the RP2 version have a current.uf2 file that images the whole flash including the filesystem.

afaik, only the SAMD21 bootloader can do this.

@jepler
Copy link
Member

jepler commented Feb 18, 2025

I wrote this a long time ago, thinking it might be useful for shipping prepared CIRCUITPY filesystems, but it was never used and so I archived it: https://github.com/jepler/mkfatimg

@tannewt
Copy link
Member

tannewt commented Feb 18, 2025

Do we really need CircuitPython to support this? Would it be enough to have a folder -> fat filesystem -> uf2 converter? This would be orthogonal to CircuitPython except for placing the filesystem in the same place.

This would be ideal. Is there a way to pull out of a CircuitPython UF2 all the information needed to append the CIRCUITPY filesystem? It seemed to me this hypothetical tool would need to know some internals of particular CircuitPython builds in order to do that. And/or would need a modified UF2 bootloader for those systems with internal+external flash.

The filesystem isn't "appended" necessarily. It wouldn't be easy to get the location out of a UF2. You might be able to get it out of an elf with debug info. The location doesn't vary build to build because we don't want to destroy user file systems. The location may only change on major version boundary.

What platform are you targeting specifically? The implementation will depend on that.

The broadcom port builds its own SD card filesystem image here:

$(Q)dd if=/dev/zero of=$(BUILD)/circuitpython-disk.img bs=1 count=0 seek=256M
$(Q)parted -s $(BUILD)/circuitpython-disk.img mktable msdos
$(Q)parted -s $(BUILD)/circuitpython-disk.img mkpart primary fat32 0% 100%
$(Q)mkfs.fat -F 32 -n BOOT --offset=2048 $(BUILD)/circuitpython-disk.img
$(Q)mcopy -i $(BUILD)/circuitpython-disk.img@@1M config.txt firmware/bootcode.bin firmware/fixup* firmware/start* firmware/*.dtb ::
$(Q)mcopy -i $(BUILD)/circuitpython-disk.img@@1M $(BUILD)/kernel$(SUFFIX).img ::

@todbot
Copy link
Author

todbot commented Feb 18, 2025

The filesystem isn't "appended" necessarily.

Ah okay. I phrased it that way because at some point I recall having success literally concatenating UF2 files to append the CIRCUITPY filesystem on rp2. Looks like there's things like uf2tool now to be little less cavalier about it.

What platform are you targeting specifically? The implementation will depend on that.

For architectures, I think rp2 and esp32 would cover most systems people are concerned about. (The lack of something like picotool to produce full CirPy+filesystem for esp32 has definitely steered me away that using that chip in a product these last few years)

@RetiredWizard
Copy link

RetiredWizard commented Feb 19, 2025

The Blues.io circuitpython-fs-builder might be of interest to you, https://github.com/blues/circuitpython-fs-builder

I stumbled across it while playing with the Swan microcontroller a couple years ago. It's really meant for a different use case, but I thought it might be useful for the use case I think you're talking about here. I'm easily distracted though and didn't really follow up. There's a pretty good video explanation starting at around 50:20 of this video: https://youtu.be/fUUOmpEwpCE

Edit: Perhaps this tool is only applicable to STM boards though?

@tannewt
Copy link
Member

tannewt commented Feb 19, 2025

For architectures, I think rp2 and esp32 would cover most systems people are concerned about.

rp2 should be possible already. The bootloader allows you to overwrite any portion of flash.

ESP32 doesn't allow a uf2 to write outside of the ota_0 partition. (tinyuf2 README So, we'd need to add the ability to write to the data partition.

Alternatively, idf.py could be used to flash directly. It can include both TinyUF2, CircuitPython and the filesystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants