Merge remote-tracking branch 'upstream/master' into relationalai-mast…

…er-20240320

.github/workflows/test.yml

@@ -4,11 +4,11 @@ on:
   push:
 jobs:
   tests:
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
       with:
         fetch-depth: 0
-    - uses: cachix/install-nix-action@v16
+    - uses: cachix/install-nix-action@v17
     #- run: nix flake check
     - run: nix-build -A checks.x86_64-linux.build -A checks.x86_64-linux.validate-openapi

.gitignore

@@ -38,6 +38,7 @@ t/jobs/declarative/project.json
 hydra-config.h
 hydra-config.h.in
 result
+result-*
 outputs
 config
 stamp-h1

Makefile.am

@@ -1,8 +1,12 @@
-SUBDIRS = src t doc
+SUBDIRS = src doc
+if CAN_DO_CHECK
+  SUBDIRS += t
+endif
+
 BOOTCLEAN_SUBDIRS = $(SUBDIRS)
 DIST_SUBDIRS      = $(SUBDIRS)
-EXTRA_DIST        = hydra-module.nix
+EXTRA_DIST        = nixos-modules/hydra.nix
 
-install-data-local: hydra-module.nix
+install-data-local: nixos-modules/hydra.nix
 	$(INSTALL) -d $(DESTDIR)$(datadir)/nix
-	$(INSTALL_DATA) hydra-module.nix $(DESTDIR)$(datadir)/nix/
+	$(INSTALL_DATA) nixos-modules/hydra.nix $(DESTDIR)$(datadir)/nix/hydra-module.nix

README.md

@@ -80,7 +80,7 @@ $ nix-build
 You can use the provided shell.nix to get a working development environment:
 ```
 $ nix-shell
-$ ./bootstrap
+$ autoreconfPhase
 $ configurePhase # NOTE: not ./configure
 $ make
 ```

configure.ac

@@ -10,8 +10,6 @@ AC_PROG_LN_S
 AC_PROG_LIBTOOL
 AC_PROG_CXX
 
-CXXFLAGS+=" -std=c++17"
-
 AC_PATH_PROG([XSLTPROC], [xsltproc])
 
 AC_ARG_WITH([docbook-xsl],
@@ -55,9 +53,6 @@ PKG_CHECK_MODULES([NIX], [nix-main nix-expr nix-store])
 testPath="$(dirname $(type -p expr))"
 AC_SUBST(testPath)
 
-jobsPath="$(realpath ./t/jobs)"
-AC_SUBST(jobsPath)
-
 CXXFLAGS+=" -include nix/config.h"
 
 AC_CONFIG_FILES([
@@ -73,11 +68,22 @@ AC_CONFIG_FILES([
   src/lib/Makefile
   src/root/Makefile
   src/script/Makefile
-  t/Makefile
-  t/jobs/config.nix
-  t/jobs/declarative/project.json
 ])
 
+# Tests might be filtered out
+AM_CONDITIONAL([CAN_DO_CHECK], [test -f "$srcdir/t/api-test.t"])
+AM_COND_IF(
+  [CAN_DO_CHECK],
+  [
+    jobsPath="$(realpath ./t/jobs)"
+    AC_SUBST(jobsPath)
+    AC_CONFIG_FILES([
+      t/Makefile
+      t/jobs/config.nix
+      t/jobs/declarative/project.json
+    ])
+  ])
+
 AC_CONFIG_COMMANDS([executable-scripts], [])
 
 AC_CONFIG_HEADER([hydra-config.h])

doc/architecture.md

@@ -0,0 +1,129 @@
+This is a rough overview from informal discussions and explanations of inner workings of Hydra.
+You can use it as a guide to navigate the codebase or ask questions.
+
+## Architecture
+
+### Components
+
+- Postgres database
+    - configuration
+    - build queue
+        - what is already built
+        - what is going to build
+- `hydra-server`
+    - Perl, Catalyst
+    - web frontend
+- `hydra-evaluator`
+    - Perl, C++
+    - fetches repositories
+    - evaluates job sets
+        - pointers to a repository
+    - adds builds to the queue
+- `hydra-queue-runner`
+    - C++
+    - monitors the queue
+    - executes build steps
+    - uploads build results
+        - copy to a Nix store
+- Nix store
+    - contains `.drv`s
+    - populated by `hydra-evaluator`
+    - read by `hydra-queue-runner`
+- destination Nix store
+    - can be a binary cache
+    - e.g. `[cache.nixos.org](http://cache.nixos.org)` or the same store again (for small Hydra instances)
+- plugin architecture
+    - extend evaluator for new kinds of repositories
+        - e.g. fetch from `git`
+
+### Database Schema
+
+[https://github.com/NixOS/hydra/blob/master/src/sql/hydra.sql](https://github.com/NixOS/hydra/blob/master/src/sql/hydra.sql)
+
+- `Jobsets`
+    - populated by calling Nix evaluator
+    - every Nix derivation in `release.nix` is a Job
+    - `flake`
+        - URL to flake, if job is from a flake
+        - single-point of configuration for flake builds
+        - flake itself contains pointers to dependencies
+        - for other builds we need more configuration data
+- `JobsetInputs`
+    - more configuration for a Job
+- `JobsetInputAlts`
+    - historical, where you could have more than one alternative for each input
+    - it would have done the cross product of all possibilities
+    - not used any more, as now every input is unique
+    - originally that was to have alternative values for the system parameter
+        - `x86-linux`, `x86_64-darwin`
+        - turned out not to be a good idea, as job set names did not uniquely identify output
+- `Builds`
+    - queue: scheduled and finished builds
+    - instance of a Job
+    - corresponds to a top-level derivation
+        - can have many dependencies that don’t have a corresponding build
+        - dependencies represented as `BuildSteps`
+    - a Job is all the builds with a particular name, e.g.
+        - `git.x86_64-linux` is a job
+        - there maybe be multiple builds for that job
+            - build ID: just an auto-increment number
+    - building one thing can actually cause many (hundreds of) derivations to be built
+    - for queued builds, the `drv` has to be present in the store
+        - otherwise build will fail, e.g. after garbage collection
+- `BuildSteps`
+    - corresponds to a derivation or substitution
+    - are reused through the Nix store
+    - may be duplicated for unique derivations due to how they relate to `Jobs`
+- `BuildStepOutputs`
+    - corresponds directly to derivation outputs
+        - `out`, `dev`, ...
+- `BuildProducts`
+    - not a Nix concept
+    - populated from a special file `$out/nix-support/hydra-build-producs`
+    - used to scrape parts of build results out to the web frontend
+        - e.g. manuals, ISO images, etc.
+- `BuildMetrics`
+    - scrapes data from magic location, similar to `BuildProducts` to show fancy graphs
+        - e.g. test coverage, build times, CPU utilization for build
+    - `$out/nix-support/hydra-metrics`
+- `BuildInputs`
+    - probably obsolute
+- `JobsetEvalMembers`
+    - joins evaluations with jobs
+    - huge table, 10k’s of entries for one `nixpkgs` evaluation
+    - can be imagined as a subset of the eval cache
+        - could in principle use the eval cache
+
+### `release.nix`
+
+- hydra-specific convention to describe the build
+- should evaluate to an attribute set that contains derivations
+- hydra considers every attribute in that set a job
+- every job needs a unique name
+    - if you want to build for multiple platforms, you need to reflect that in the name
+- hydra does a deep traversal of the attribute set
+    - just evaluating the names may take half an hour
+
+## FAQ
+
+Can we imagine Hydra to be a persistence layer for the build graph?
+
+- partially, it lacks a lot of information
+  - does not keep edges of the build graph
+
+How does Hydra relate to `nix build`?
+
+- reimplements the top level Nix build loop, scheduling, etc.
+- Hydra has to persist build results
+- Hydra has more sophisticated remote build execution and scheduling than Nix
+
+Is it conceptually possible to unify Hydra’s capabilities with regular Nix?
+
+- Nix does not have any scheduling, it just traverses the build graph
+- Hydra has scheduling in terms of job set priorities, tracks how much of a job set it has worked on
+    - makes sure jobs don’t starve each other
+- Nix cannot dynamically add build jobs at runtime
+    - [RFC 92](https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md) should enable that
+    - internally it is already possible, but there is no interface to do that
+- Hydra queue runner is a long running process
+    - Nix takes a static set of jobs, working it off at once

doc/manual/src/SUMMARY.md

@@ -7,6 +7,7 @@
 - [Hydra jobs](./jobs.md)
 - [Plugins](./plugins/README.md)
   - [Declarative Projects](./plugins/declarative-projects.md)
+  - [RunCommand](./plugins/RunCommand.md)
 - [Using the external API](api.md)
 - [Webhooks](webhooks.md)
 - [Monitoring Hydra](./monitoring/README.md)

doc/manual/src/configuration.md

@@ -74,6 +74,30 @@ following:
         }
     }
 
+Populating a Cache
+------------------
+
+A common use for Hydra is to pre-build and cache derivations which
+take a long time to build. While it is possible to direcly access the
+Hydra server's store over SSH, a more scalable option is to upload
+built derivations to a remote store like an [S3-compatible object
+store](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-help-stores.html#s3-binary-cache-store). Setting
+the `store_uri` parameter will cause Hydra to sign and upload
+derivations as they are built:
+
+```
+store_uri = s3://cache-bucket-name?compression=zstd&parallel-compression=true&write-nar-listing=1&ls-compression=br&log-compression=br&secret-key=/path/to/cache/private/key
+```
+
+This example uses [Zstandard](https://github.com/facebook/zstd)
+compression on derivations to reduce CPU usage on the server, but
+[Brotli](https://brotli.org/) compression for derivation listings and
+build logs because it has better browser support.
+
+See [`nix help
+stores`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-help-stores.html)
+for a description of the store URI format.
+
 Statsd Configuration
 --------------------
 
@@ -102,6 +126,26 @@ in the hydra configuration file, as below:
 </hydra_notify>
 ```
 
+hydra-queue-runner's Prometheus service
+---------------------------------------
+
+hydra-queue-runner supports running a Prometheus webserver for metrics. The
+exporter's address defaults to exposing on `127.0.0.1:9198`, but is also
+configurable through the hydra configuration file and a command line argument,
+as below. A port of `:0` will make the exposer choose a random, available port.
+
+```conf
+queue_runner_metrics_address = 127.0.0.1:9198
+# or
+queue_runner_metrics_address = [::]:9198
+```
+
+```shell
+$ hydra-queue-runner --prometheus-address 127.0.0.1:9198
+# or
+$ hydra-queue-runner --prometheus-address [::]:9198
+```
+
 Using LDAP as authentication backend (optional)
 -----------------------------------------------
 
@@ -111,8 +155,8 @@ use LDAP to manage roles and users.
 This is configured by defining the `<ldap>` block in the configuration file.
 In this block it's possible to configure the authentication plugin in the
 `<config>` block. All options are directly passed to `Catalyst::Authentication::Store::LDAP`.
-The documentation for the available settings can be found [here]
-(https://metacpan.org/pod/Catalyst::Authentication::Store::LDAP#CONFIGURATION-OPTIONS).
+The documentation for the available settings can be found
+[here](https://metacpan.org/pod/Catalyst::Authentication::Store::LDAP#CONFIGURATION-OPTIONS).
 
 Note that the bind password (if needed) should be supplied as an included file to
 prevent it from leaking to the Nix store.
@@ -159,13 +203,14 @@ Example configuration:
       <role_search_options>
         deref = always
       </role_search_options>
+    </store>
   </config>
   <role_mapping>
     # Make all users in the hydra_admin group Hydra admins
     hydra_admin = admin
     # Allow all users in the dev group to restart jobs and cancel builds
     dev = restart-jobs
-    dev = cancel-builds
+    dev = cancel-build
   </role_mapping>
 </ldap>
 ```

doc/manual/src/hacking.md

@@ -18,7 +18,7 @@ $ nix-shell
 To build Hydra, you should then do:
 
 ```console
-[nix-shell]$ ./bootstrap
+[nix-shell]$ autoreconfPhase
 [nix-shell]$ configurePhase
 [nix-shell]$ make
 ```
@@ -92,7 +92,7 @@ On NixOS:
 
 ```nix
 {
-  nix.trustedUsers = [ "YOURUSER" ];
+  nix.settings.trusted-users = [ "YOURUSER" ];
 }
 ```
 

doc/manual/src/plugins/README.md

@@ -172,17 +172,6 @@ Sets Gitlab CI status.
 
 - `gitlab_authorization.<projectId>`
 
-## HipChat notification
-
-Sends hipchat chat notifications when a build finish.
-
-### Configuration options
-
-- `hipchat.[].jobs`
-- `hipchat.[].builds`
-- `hipchat.[].token`
-- `hipchat.[].notify`
-
 ## InfluxDB notification
 
 Writes InfluxDB events when a builds finished.
@@ -192,10 +181,12 @@ Writes InfluxDB events when a builds finished.
 - `influxdb.url`
 - `influxdb.db`
 
-## Run command
+## RunCommand
 
 Runs a shell command when the build is finished.
 
+See [The RunCommand Plugin](./RunCommand.md) for more information.
+
 ### Configuration options:
 
 - `runcommand.[].job`