Skip to content
This repository has been archived by the owner on Jun 4, 2024. It is now read-only.

support files

Jump to bottom
rgaudin edited this page May 6, 2021 · 14 revisions

pibox-installer as well as ansiblecube inside it depends on a number of external files which are downloaded and used during the installation process.

This document is a list of such files, their use and how to create/update them when that's available.

Bare copies

Following are files publicly available from their upstream source which we copied into our mirror to prevent network issues during builds (CI or image).

QEMU kernel (pibox)

Faster than the stock kernel on modern hosts.

Support up to 4 cores and 30GiB or RAM.

This vexpress boot can be compiled on linux using make-vexpress-boot python3 script.

requirements: zip git bc bison flex libssl-dev make libc6-dev libncurses5-dev crossbuild-essential-armhf

run: python3 make-vexpress-boot

Static aria2c (linux)

docker run -v $(pwd):/data -it ubuntu bash

apt-get update
apt-get install -y build-essential make libgnutls28-dev nettle-dev libgmp-dev libssh2-1-dev libc-ares-dev libxml2-dev zlib1g-dev libsqlite3-dev pkg-config curl
cd /data
tar xf aria2-1.35.0.tar.bz2
cd aria2-1.35.0
./configure --prefix /data/dist --with-ca-bundle='/etc/ssl/certs/ca-certificates.crt' ARIA2_STATIC=yes
make
make install

Static QEMU (pibox)

Used in .gitlab-ci.yml to skip the compilation step of QEMU.

Stock QEMU compiled statically for linux x86_64.

./configure --target-list=arm-softmmu \
    --disable-debug-info \
    --enable-system \
    --enable-tools \
    --disable-werror \
    --disable-linux-user \
    --static

Following dependencies needed (alpine) make samurai perl python3 gcc libc-dev pkgconf linux-headers glib-dev glib-static zlib-dev zlib-static apk add bash xz git patch pixman-dev pixman-static

macOS QEMU

Used in .travis.yml as part of macOS bundle. Travis would frequently fail to build due to timeouts in brew install or compilation of QEMU.

brew update
brew install qemu
git clone https://github.com/auriamg/macdylibbundler/
cd macdylibbundler/
git checkout 84440587e1b2e4e9e0727cbba7a455b96820e40a
make
cd ..
mkdir qemu
cd qemu
cp /usr/local/bin/qemu-img .
cp /usr/local/bin/qemu-system-arm .
chmod u+w qemu-img
chmod u+w qemu-system-arm
../macdylibbundler/dylibbundler -d . -p @executable_path/ -b -x ./qemu-system-arm  -x ./qemu-img
chmod u-w qemu-img
chmod u-w qemu-system-arm
otool -L *
cd ..
tar cf qemu-2.12.0_macOS.tar qemu/

Python 3.6 formulas for brew

On travis, Python needs to be installed via brew (for pygobject3). Brew does not provide a way to pick the python version and always uses the latest stable one (3.7.0 now).

Since pyinstaller is not yet 3.7 compatible, we have to use python 3.6.5 and, for that, change the brew formulas of python and some dependencies.

We used to do a git checkout into `` but now that travis updated its base macos image, this commit is beyond the limited history.

cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core
zip brew-py36.zip Formula/python.rb Formula/pygobject.rb Formula/gtk+.rb Formula/glib.rb Formula/adwaita-icon-theme.rb

GoRC

Stock copy of the binary (GoRC.exe) from http://www.godevtool.com/Gorc.zip

Used to compile resource file for Windows.

Custom ideascube build (ansiblecube)

A deb of ideascube with our patches:

  • move zim from --package-cache instead of copying.
  • --skip-sha256 option.
  • Debian stretch package (python3.5).

Use rgaudin/ideascube until those patches gets integrated upstream.

xz 5.2.4

Used only in linux's build .travis.yml: the build system is trusty (14.04) due to the py34 requirement but trust's LZMA library is too old. See #176.

Stock copy of lzmautils's distribution on sourceforge. SF raises frequent download issues from travis.

Exfat-fuse (ansiblecube)

Stock copy of debian bower package as stretch version is obsolete and fetching directly from archive sometimes fail.

Exfat Kernel modules (ansiblecube)

A pair of exfat-nofuse kernel modules for the raspberryPi.

  • Compiled against the same kernel tree as the raspbian kernel in use.
  • Archive contains module for the two rPi flavors: regular (Pi2) and -v7 (Pi3)

Creation steps

You can alternatively compile the modules on raspberry hardware running the same raspbian version you are targetting. You will need to do it twice thought (first on a Pi2 then on a P3) and copy the modules from there.

The following instructions are for a cross-compilation.

  • Get kernel commit from .info file of raspbian-liste curl http://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-04-19/2018-04-18-raspbian-stretch-lite.info

  • Download the raspberry version of the kernel for that commit

wget https://github.com/raspberrypi/linux/archive/f70eae405b5d75f7c41ea300b9f790656f99a203.zip
unzip f70eae405b5d75f7c41ea300b9f790656f99a203.zip
mv linux-f70eae405b5d75f7c41ea300b9f790656f99a203/ linux-4.14.34
cd linux-4.14.34
  • Download exfat-nofuse and place it in tree
wget https://github.com/dorimanx/exfat-nofuse/archive/master.zip
unzip master.zip
mv exfat-nofuse-master/ fs/exfat
  • Add exfat to fs/Makefile obj-$(CONFIG_EXFAT_FS) += exfat/
  • Add exfat to fs/Kconfig (in filesystems section) source "fs/exfat/Kconfig"

RaspberryPi 2

mkdir -p ../pi2
kernel="kernel"
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig

Edit .config

CONFIG_EXFAT_FS=m
CONFIG_EXFAT_DISCARD=y
CONFIG_EXFAT_DELAYED_SYNC=n
CONFIG_EXFAT_KERNEL_DEBUG=n
CONFIG_EXFAT_DEBUG_MSG=n
CONFIG_EXFAT_DEFAULT_CODEPAGE=437
CONFIG_EXFAT_DEFAULT_IOCHARSET="utf8"

Compile modules

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules

Install files

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=../pi2 modules_install

RaspberryPi 3

mkdir -p ../pi3
kernel="kernel7"
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig

Edit .config

CONFIG_EXFAT_FS=m
CONFIG_EXFAT_DISCARD=y
CONFIG_EXFAT_DELAYED_SYNC=n
CONFIG_EXFAT_KERNEL_DEBUG=n
CONFIG_EXFAT_DEBUG_MSG=n
CONFIG_EXFAT_DEFAULT_CODEPAGE=437
CONFIG_EXFAT_DEFAULT_IOCHARSET="utf8"

Compile modules

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules

Install files

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=../pi3 modules_install

Prepare archive

cp ../pi2/lib/modules/4.14.34/kernel/fs/exfat/exfat.ko exfat_4.14.34.ko
cp ../pi3/lib/modules/4.14.34-v7/kernel/fs/exfat/exfat.ko exfat_4.14.34-v7.ko
tar cvf exfat_4.14.34.tar exfat_4.14.34*.ko

Imdisk Installer

Stock copy of ImDisk 2.0.9 installer

Exfat-fuse

Static version of exfat-fuse binaries.

Compilation instructions

sudo apt install autoconf automake pkg-config libfuse-dev gcc make
wget https://github.com/relan/exfat/archive/master.zip
unzip master.zip
cd exfat-master
FUSE_LIBS="/usr/lib/x86_64-linux-gnu/libfuse.a /usr/lib/x86_64-linux-gnu/libulockmgr.a -ldl -pthread" ./configure --prefix=`pwd`/dist
make
make install
cd dist/sbin
tar cf exfat-fuse-1.2.8-linux_x86_64.tar mount.exfat-fuse mkexfatfs exfatlabel exfatfsck dumpexfat

Content-related (ansiblecube)

Most content downloads are available through our mirror.

Note: When creating tar archive, exclude symlinks (and hard links) as those are not supported on ExFAT.

find . -type l
find . -type l | while read f; do /bin/cp -rf --remove-destination $(readlink -f $(find . -wholename "${f}")) "${f}";done;

Ka-Lite (/other/kalite)

Language packs are copies of the stock KA-Lite version available here.

Videos tar files are archives of the extracted videos (folder) available as torrents.

Language packs and videos archives are tied to specific version of ka-lite.

Aflatoun (/other/aflatoun)

Aflatoun is only available from this mirror. While it is based on kalite, it includes numerous undocumented changes to the kalite code, templates and resources.

Wikifundi (/other/wikifundi)

Wikifundi is based on Kiwix-specific archives for both parsoid and the content. The content package contains the PHP code (mediawiki), configuration and the database.

fr.africapack.kiwix.org_2017-01.tar,bz2 is made from fr.africapack.kiwix.org_2017-01.tar.xz. It's a bzip2-compressed version which also renames the (14) files/folders which are incompatible with ExFAT.

tar xf fr.africapack.kiwix.org_2017-01.tar.xz
pax -M norm -rw -s '/[?<>\\:*|\"]/_/gp' fr.africapack.kiwix.org target-folder
cd target-folder
tar -cjf fr.africapack.kiwix.org_2017-01.tar.bz2 fr.africapack.kiwix.org

Installation instructions on github.

Clone this wiki locally