Crochet-built FreeBSD 11.1-stable r329320 aarch64 RPI3 hangs on boot

[skaarj0]

As we already know, FreeBSD 12-Current has some very strange issues with UFS2+j+softupdates filesystem, which may or may not be caused by growfs: when a certain number of occupied inodes is reached, system gets stuck with cylinder group checksum failures - slowing it down - and not even the all-mighty aarch64-built fsck is able to correct this. However, fsck + mounting the system on a traditional architecture (x86, x64) shows no problems.

So I decided to switch to FreeBSD 11 and to try booting my RPI3. Now here's what I've got so far:

0. System setup:

• one Raspberry PI 3, one physical amd64 machine;
• one 32GB Samsung microSD, one 16GB A-Data microSD, one Kingston 16GB MicroSD, one Sandisk 16GB MicroSD, one Western Digital Mechanical external HDD (40GB), one 128GB SSD built by Kingston;
• I am running both FreeBSD11.1-stable-amd64 and FreeBSD12.0-CURRENT-amd64 in Virtualbox, as guests on the FreeBSD11.0-stable-amd64 physical system;
• On both VMs, Crochet fails to download its required files from github due to failed cerificates, so in /root/.cshrc I added << setenv SSL_NO_VERIFY_PEER 1 >>;
• FreeBSD 12.0-CURRENT binutils cause some crochet compilation trouble so I had to downgrade to a lower version (I read that somewhere around here);
• FreeBSD 11.1-stable binutils work perfectly;
• I built two images: FreeBSD 11.1-stable-aarch64-GENERIC-rpi3.img and FreeBSD12.0-CURRENT-aarch64-GENERIC-NODEBUG-rpi3.img without any compilation failures/errors/issues. First one hangs on boot, second one is able to boot;
• On all the above-mentioned storage devices, FreeBSD 12.0-CURRENT-aarch64-GENERIC-NODEBUG-RPI3 shows filesystem checksum failures in cylinder groups - on all MicroSDs, on the <> HDD and also on the SSD - after compilng some /usr/ports/www/firefox to make sure there are A LOT of small files on the storage device, so hardware-related issues are excluded.

Here's what I got so far:

1. Booting the crochet-resulted FreeBSD 11.1-stable r329320 aarch64 rpi3 image:
   At startup it loads u-boot, starts loader.efi, it is able to read the filesystem, to list files and load the kernel. After kernel is loaded I have 10 seconds to press enter to boot or space to pause. At boot it clears the screen, it hangs and turns off the monitor. The last message I am able to see (before clearing the screen) is "using dtb provided by EFI at 0x8004000";

2. Booting the crochet-resulted FreeBSD12.0-CURRENT-aarch64-rpi3 image
   It boots normal, as it is expected to boot. However it causes the above mentioned filesystem issues after portsnap fetch extract + building some heavy stuff such as firefox.

3. Booting the crochet-resulted <> FreeBSD 11.1-stable r329320 aarch64 rpi3 image using the 12.0-current kernel+modules:
   I replaced just the kernel and modules: 11.1-stable image containing 11.1-stable world and 12.0-current kernel+modules in /boot/kernel. No modifications were made on the fat partition or anywhere else.
   The resulting image causes the system to boot normally, but with the same filesystem issues (cg group checksum failed after portsnap fetch extract + compiling some heavy stuff such as firefox).

4. Booting the crochet-resulted <> FreeBSD 12.0-current aarch64 rp3 image using the 11.1-stable kernel
   Exactly like step3, I replaced just the kernel + modules: 12.0-current image containing 12.0-current world and 11.1-stable kernel+modules in /boot/kernel.
   It is performing exactly as in step 1.

From these experiments I reached the conclusion that there's something wrong going on during 11.1-stable kernel compilation process for aarch64-rpi3.

There are no display information even when full debug is enabled in the 11.1-aarch64-rpi3 kernel configuration, so I am unable to determine any further steps to investigate.

Please feel free to re-run these experiments to confirm.
For weeks I have been searching every developer mailing lists I could find to get more information about any working FreeBSD 11.x kernel on RPI3. So far there's a strange silence on this subject.

In the mean time I am stuck - please advise.

[skaarj0]

Temporary issue solved.

I do not have knowledge about issue closing procedures but I may have a workaround for this.

Here is a fast desperate workaround explaining how you can get a working FreeBSD for RPI3 without storage corruption due to error in inode number counting (df -i showing all ok) and cg checksum failures (caused probably by growfs).
Warning: this will probably give you a working FreeBSD-RPI3 image BUT it does not give a proper explanation of WHY bsd12 is corrupting the storage media.

Use it if you really really need it working as soon as possible and you have no time to waste.
Security and stability are not guaranteed.

Ingredients:

• Virtual Box;
• One ISO image of FreeBSD-11.1-stable;
• One ISO image of FreeBSD 12.0-CURRENT;
• Two virtual machines running the above-mentioned systems - 2 GB RAM and 32 GB virtual disk image for each vm;
• FreeBSD 12.0-CURRENT with GENERIC-NODEBUG kernel compiled and installed on the corresponding VM (to get rid of WITNESS and other debuging stuff which slow the system);
• crochet installed on both VMs;
• customize RaspberryPi3/config files on both VMs to adjust config.txt for your monitor settings;
• customize config.sh on both VMs to specify board = RaspberryPi3 and to avoid certificate errors when fetching boot-related files from github: SSL_NO_VERIFY_PEER=1;
• customize config.sh on both VMs to specify a 4 GBytes SDCARD; (too lazy to try 2 GB image size);
• don't specify anything about swap partitions/files in any of config.sh's;
• launch crochet to build image for FreeBSD-11.1-stable-rpi3-GENERIC.img;
• launch crochet to build image for FreeBSD 12.0-CURRENT-rpi3-GENERIC-NODEBUG.img;
• ftp mput both resulting images from their corresponding guest VMs to the host system running vbox;
• shutdown both VMs because you don't need them anymore;
• mount ${11.1-STABLE} image to ${mnt_dir_bsd11}, rename ${mnt_dir_bsd11}/boot/kernel to ${mnt_dir_bsd11}/boot/kernel.bsd11; # (man mdconfig)
• mount ${12.0-CURRENT} image to ${mnt_dir_bsd12}; # (man mdconfig)
• copy ${mnt_dir_bsd12}/boot/kernel to ${mnt_dir_bsd11}/boot/kernel;
• modify ${mnt_dir_bsd11}/etc entries in order to obtain global environment variable UNAME_r = 11.1-stable;
• umount both images, mdconfig detach units;
• dd if=FreeBSD-11.1-stable-rpi3-GENERIC.img of=${YOUR_FAVOURITE_SDCARD} bs=1m; (win32diskimager works faster);
• boot resulting physical ${YOUR_FAVOURITE_SDCARD} on RPI3;

---

so far RPI3 should read ${YOUR_FAVOURITE_SDCARD}, load the required files from msdos partition, read the ${rootdir}/boot/loader.conf and load the freebsd12-current kernel.

---

• install following packages: nano, wget, findutils, xorg-server, xorg-fonts, xorg-apps, xorg-drivers, xf86-video-scfb, windowmaker (or your favorite window manager) and whatever other apps you require;
• mv /etc/rc.conf /etc/rc.conf.rpi3;
• cp /usr/share/examples/etc/defaults/rc.conf /etc/rc.conf;
• cat /etc/rc.conf.rpi3 >> /etc/rc.conf;
• customize your rc.conf according to your desires;
• nano /root/.xinitrc to instruct it to run your favourite window manager: exec wmaker
• startx - customize your window manager, save settings, exit X windows;
• portsnap fetch extract;
• cd /usr/ports/misc/mc && make config -> instruct it to use curses instead of slang; make install clean;
• prepare an external (USB) hard drive with bsdinstall partedit: 100MB boot partition - fstype = !12, root partition, swap partition = 2 x RAM just to be sure;
• newfs_msdos ${external_hdd_boot_partition};
• mount ${external_hdd_boot_partition} /mnt; cp -Rv /boot/efi /mnt; umount /mnt;
• newfs -t -O2 -U -j -f 512 -b 4096 ${external_hdd_root_partition}; # man newfs to explain everything;
• mount ${external_hdd_root_partition} to /mnt; cd /mnt
• dump -0f - / |restore -rf -
• modify /etc/fstab to set root dir to /dev/${external_hdd_root_partition}. Comment everything related to /dev/mmcsd* because you might need them later;
• modify /etc/fstab to activate the previous created swap partition;
• cp /etc/fstab /mnt/etc/fstab;
• umount /mnt
• reboot

---

So far at power-up the RPI3 should read ${YOUR_FAVOURITE_SDCARD}, load the required files from msdos partition, read the ${rootdir}/boot/loader.conf and load the freebsd12-current kernel.
Later it will read ${YOUR_FAVOURITE_SDCARD_ROOTDIR}/etc/fstab which instructs it to use /dev/${external_hdd_root_partition} as root directory and /dev/${external_hdd_swap_partition} as swap.

---

• ignore the message saying the swap is larger than recommended size in blocks;

At this point my RPI3 managed to boot the hybrid BSD11 world + bsd12 kernel without any cg->checksum failures.

I am currently compiling firefox from /usr/ports to stress the system.
cd /usr/ports/www/firefox && make config - I removed all references to video playing and sound, leaving just SNDIO as sound support and BUNDLED_CAIRO (first option activated by default).
I pkg-installed all required dependencies which exist in the repository.
/usr/ports/lang/rust is required and it does not come as a precompiled package. Stop the compiling process as soon as configuration files are generated. Use /usr/local/bin/mc to find all files containing "-march=native", remove all occurrences (F4-replace with blank space) because compiler does not support this flag on aarch64.
Firefox compiling fails at startup due to some core dump of python2.7 which I don't want to solve.
Same happens for /usr/ports/www/chromium.

Switching to /usr/ports/www/seamonkey

So far in the last 24 hours no more cylinder group (cg) CRC failures occurred.

Plans for the future:
if everything works fine, my plan is as following:
bsdinstall partedit to modify sdcard: delete root partition, add a smaller one and leave some space for some swap;
newfs -t -O2 -U -j -f 512 -b 4096 ${sdcard_root_partition}; (see man newfs)
mound ${sdcard_root_partition} to /mnt, cd /mnt && dump -0f - / |restore -rf -
umount /mnt
Create a backup image file of the sdcard - to avoid re-working all of these in case of any unforseen errors;
Continue to stress the RPI and the external HDD by compiling heavy stuff such as blender, audacity, gimp or OpenOffice;
Keep a FreeBSD-based RPI3 in working condition because it is much simple to make it serve you - Linux is a pain.

---

Please recreate the experiment in order to confirm.