dist/tools: Add Doxygen as a Tool for CI

[crasbe]

Contribution description

This is an alternative approach for #21295. Instead of trying to handle Doxygen inside of the Docker container, we just add Doxygen as a package in RIOT. Therefore it can be managed inside the RIOT repository.

The changes to the CI script(s) are minimal, only the target has to be changed from make doc to make doc-ci and Doxygen does not have to be installed as a package anymore (but it could be so that all dependencies are there).

TODO: It would probably be a good idea to pass the DOXYGEN_MIN_VERSION from doc/doxygen/Makefile to dist/tools/doxygen/Makefile, so that we really truly only have one single spot where the Doxygen version is maintained. Done.

RFC: Would it be better to build Doxygen instead of downloading the binaries? IMO that would add a lot of unncessary compile time to the CI process.

Testing procedure

Execute make doc-ci and observe Doxygen being downloaded and the documentation being built.

Issues/PRs references

Alternative approach to fix #21106.

[riot-ci]

Murdock results

✔️ PASSED

ba58b18 fixup! fixup! dist/tools: add doxygen as a tool for CI

Success	Failures	Total	Runtime
10285	0	10287	14m:22s

Artifacts

• Documentation preview

[mcr]

So if I want/insist on using the local doxygen, then I should just export DOXYGEN=/my/path/to/doxygen, and I'm good?
This seems fine to me.

[crasbe]

So if I want/insist on using the local doxygen, then I should just export DOXYGEN=/my/path/to/doxygen, and I'm good? This seems fine to me.

The default is always the local doxygen, which usually points to /usr/bin/doxygen. Only for the doc-ci target the Doxygen dist package will be downloaded and used. But you can set the DOXYGEN variable and use another Doxygen binary as well for the normal doc, doc-man and doc-latex targets as well.

One downside of this PR is that it can only use released versions, not the latest master for example.

[Baertig]

I think this is a better way of doing it then #21295, because this makes it easier to replicate the CI behavior locally.

I just noticed one thing when i tried make-ci locally. There were a few errors like this:

/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atca_8h__dep__incl.dot:1: error: Problems running dot: exit code=127, command='dot', arguments='"/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atca_8h__dep__incl.dot" -Tsvg -o "/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atca_8h__dep__incl.svg" -Tcmapx -o "/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atca_8h__dep__incl.map"'
sh: 1: dot: not found
/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atmega256rfr2-xpro_2include_2board_8h__dep__incl.dot:1: error: Problems running dot: exit code=127, command='dot', arguments='"/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atmega256rfr2-xpro_2include_2board_8h__dep__incl.dot" -Tsvg -o "/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atmega256rfr2-xpro_2include_2board_8h__dep__incl.svg" -Tcmapx -o "/home/georg-baer-dumont/tu-dresden/riot/RIOT/doc/doxygen/html/atmega256rfr2-xpro_2include_2board_8h__dep__incl.map"'
sh: 1: dot: not found

I think it is, because i don't have graphviz installed locally.
Would it make sense to download that as well, to ensure that all tool versions used for building the docs are pinned?

[crasbe]

I think it is, because i don't have graphviz installed locally.
Would it make sense to download that as well, to ensure that all tool versions used for building the docs are pinned?

We could add a general check in the docs/doxygen/Makefile to see if graphviz is installed. But since graphviz is not that version dependant, I wouldn't really want to download it as well. It is much easier to have it installed by the Docker scripts in the CI. IMO it would even make sense to install Doxygen too and just don't use it to make sure all dependencies are there. (However I don't know if Doxygen has that many external dependencies?)

[crasbe]

I added a check to the doc/doxygen/Makefile because it's not only relevant for CI but for all users. Having an explicit (red) error message instead of the somewhat cryptic Doxygen error is better IMO. This is what the error looks like:

@Baertig it would be nice if you could test that on your system :)

[crasbe]

I don't really want to feature-creep myself (again), but I thought about another option:
Instead of adding another target, it would be possible to set DOXYGEN=ci (or similar, I thought about download or latest, but especially latest is a bit difficult to implement and would imply that other versions could be used, which I really don't want to implement) and use that as a trigger to fetch the Doxygen binary from GitHub.

Then the Version warning introduced in #21277 could be extended to hint the user to that possibility and it would be possible to use all targets (doc, doc-man and doc-latex) with the downloaded Doxygen version.
That contradicts the idea of it's only meant for CI, but maybe it would improve the quality of life a bit.

[OlegHahm]

Wouldn't that only work for x86 Linux machines or am I missing anything?

[crasbe]

Wouldn't that only work for x86 Linux machines or am I missing anything?

It would only work for x64 Linux machines, yes.

[crasbe]

Inspired by the idea to set a version in Doxygen and the question that @OlegHahm had, I added a rather extensive fixup that performs a check if the system is Linux x86_64 and if it is, it tries to download the binary package.
If that fails, it will fallback to building the source.

Furthermore, there is an option to use DOXYGEN_VERSION=latest, which will pull the current master. Perhaps it should be called master to avoid confusion with "latest release"? idk

Calling make doc-ci for the first time or after a make docclean. This will try to download the binary release version of Doxygen that is specified in DOXYGEN_MIN_VERSION:

cbuec@W11nMate:~/RIOTstuff/riot-doxygen/RIOT$ make doc-ci
"make" -BC doc/doxygen doc-ci
make[1]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
make[2]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[3]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[4]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
rm -rf /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen*
make[4]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
[INFO] Downloading Doxygen binary...
[INFO] Unpacking Doxygen...
make[3]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[2]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make -BC /home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen DOXYGEN=/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen doc
make[2]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
awk 'NR == 1 {print $0,"{#coc}"} NR > 1 {print $0}' ../../CODE_OF_CONDUCT.md > src/coc.md
( cat riot.doxyfile ; echo "GENERATE_HTML = yes" ) | /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen -
warning: Tag 'OUTPUT_TEXT_DIRECTION' at line 102 of file '-' has become obsolete.
         To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'HTML_TIMESTAMP' at line 1365 of file '-' has become obsolete.
         To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'FORMULA_TRANSPARENT' at line 1670 of file '-' has become obsolete.
...

Calling make doc-ci for the second time will avoid downloading Doxygen again if the versions match:

cbuec@W11nMate:~/RIOTstuff/riot-doxygen/RIOT$ make doc-ci
"make" -BC doc/doxygen doc-ci
make[1]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
make[2]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make -BC /home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen DOXYGEN=/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen doc
make[2]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
awk 'NR == 1 {print $0,"{#coc}"} NR > 1 {print $0}' ../../CODE_OF_CONDUCT.md > src/coc.md
( cat riot.doxyfile ; echo "GENERATE_HTML = yes" ) | /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen -
warning: Tag 'OUTPUT_TEXT_DIRECTION' at line 102 of file '-' has become obsolete.
         To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'HTML_TIMESTAMP' at line 1365 of file '-' has become obsolete.
...

Calling DOXYGEN_VERSION=1.13.1 make doc-ci will download Doxygen again because the versions mismatch:

cbuec@W11nMate:~/RIOTstuff/riot-doxygen/RIOT$ DOXYGEN_VERSION=1.13.1 make doc-ci
"make" -BC doc/doxygen doc-ci
make[1]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
make[2]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[3]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[4]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
rm -rf /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen*
make[4]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
[INFO] Downloading Doxygen binary...
[INFO] Unpacking Doxygen...
make[3]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[2]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make -BC /home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen DOXYGEN=/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen doc
make[2]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
awk 'NR == 1 {print $0,"{#coc}"} NR > 1 {print $0}' ../../CODE_OF_CONDUCT.md > src/coc.md
( cat riot.doxyfile ; echo "GENERATE_HTML = yes" ) | /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen -
warning: Tag 'OUTPUT_TEXT_DIRECTION' at line 102 of file '-' has become obsolete.
         To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
...

Calling DOXYGEN_VERSION=latest will always trigger a rebuild:

cbuec@W11nMate:~/RIOTstuff/riot-doxygen/RIOT$ DOXYGEN_VERSION=latest make doc-ci
"make" -BC doc/doxygen doc-ci
make[1]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
make[2]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[3]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
make[4]: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
rm -rf /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen*
make[4]: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen'
[INFO] Cloning and building Doxygen from source...
Cloning into '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen-latest'...
remote: Enumerating objects: 1352, done.
remote: Counting objects: 100% (1352/1352), done.
remote: Compressing objects: 100% (1064/1064), done.
remote: Total 1352 (delta 347), reused 698 (delta 206), pack-reused 0 (from 0)
Receiving objects: 100% (1352/1352), 7.23 MiB | 13.13 MiB/s, done.
Resolving deltas: 100% (347/347), done.
cd /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen/doxygen-latest && mkdir build && cd build && \
  cmake -G "Unix Makefiles" .. && make && cp bin/doxygen /home/cbuec/RIOTstuff/riot-doxygen/RIOT/dist/tools/doxygen
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
...

Building Doxygen inside of my WSL on a single thread takes FOR EVER and on multiple threads it takes so much RAM that the compiler crashes 😅
So it is definitively a good idea not to build Doxygen on every CI run...

[Baertig]

@Baertig it would be nice if you could test that on your system :)

works for me. i.e. the make script reports an error.
Thank you :)

[crasbe]

I added some documentation to the dist/tools/doxygen/Makefile to make it clearer how to use it and what it is meant for.
Also some minor improvements like not calling rm -rf in the ifneq-block in the beginning to delete the binary to trigger a rebuild but rather a REBUILD variable that contains the clean target which does the dirty work for me :)

This PKG is somewhat unusual in that it doesn't use the provided makefiles/pkg/pkg.mk, but I'm not sure if it would be possible to do what this does with the classic approach.

---

Long story short: this is ready for review and testing. You can follow the procedure I showed here: #21300 (comment)