Create eval-jobset role and guard /api/push route

(cherry picked from commit f730433789)
api: Require POST for /api/push
2024-08-27 21:53:17 +02:00 · 2024-08-27 21:53:13 +02:00 · 2024-04-21 17:36:16 +02:00 · 2024-04-21 16:14:24 +02:00 · 2024-04-21 01:55:19 +02:00 · 2024-04-20 23:05:26 +02:00
74 changed files with 1087 additions and 867 deletions
--- a/README.md
+++ b/README.md
@@ -140,7 +140,7 @@ You can also interface with Hydra through a JSON API. The API is defined in [hyd
 ## Additional Resources

 - [Hydra User's Guide](https://nixos.org/hydra/manual/)
- [Hydra on the NixOS Wiki](https://wiki.nixos.org/wiki/Hydra)
+- [Hydra on the NixOS Wiki](https://nixos.wiki/wiki/Hydra)
 - [hydra-cli](https://github.com/nlewo/hydra-cli)
 - [Peter Simons - Hydra: Setting up your own build farm (NixOS)](https://www.youtube.com/watch?v=RXV0Y5Bn-QQ)

--- a/default.nix
+++ b/default.nix
@@ -1,6 +1,6 @@
 # The `default.nix` in flake-compat reads `flake.nix` and `flake.lock` from `src` and
 # returns an attribute set of the shape `{ defaultNix, shellNix }`

-(import (fetchTarball "https://github.com/edolstra/flake-compat/archive/master.tar.gz") {
+(import (fetchTarball https://github.com/edolstra/flake-compat/archive/master.tar.gz) {
  src = ./.;
 }).defaultNix
--- a/doc/manual/src/hacking.md
+++ b/doc/manual/src/hacking.md
@@ -15,18 +15,12 @@ and dependencies can be found:
 $ nix-shell
 ```

-of when flakes are enabled:
-
-```console
-$ nix develop
-```
-
 To build Hydra, you should then do:

 ```console
 [nix-shell]$ autoreconfPhase
 [nix-shell]$ configurePhase
-[nix-shell]$ make -j$(nproc)
+[nix-shell]$ make
 ```

 You start a local database, the webserver, and other components with
@@ -36,8 +30,6 @@ foreman:
 $ foreman start
 ```

-The Hydra interface will be available on port 63333, with an admin user named "alice" with password "foobar"
-
 You can run just the Hydra web server in your source tree as follows:

 ```console
--- a/doc/manual/src/plugins/README.md
+++ b/doc/manual/src/plugins/README.md
@@ -42,7 +42,7 @@ Sets CircleCI status.

 ## Compress build logs

-Compresses build logs after a build with bzip2 or zstd.
+Compresses build logs after a build with bzip2.

 ### Configuration options

@@ -50,14 +50,6 @@ Compresses build logs after a build with bzip2 or zstd.

 Enable log compression

- `compress_build_logs_compression`
-
-Which compression format to use. Valid values are bzip2 (default) and zstd.
-
- `compress_build_logs_silent`
-
-Whether to compress logs silently.
-
 ### Example

 ```xml
--- a/doc/manual/src/webhooks.md
+++ b/doc/manual/src/webhooks.md
@@ -1,12 +1,9 @@
 # Webhooks

-Hydra can be notified by github or gitea with webhooks to trigger a new evaluation when a
+Hydra can be notified by github's webhook to trigger a new evaluation when a
 jobset has a github repo in its input.
-
-## GitHub
-
-To set up a webhook for a GitHub repository go to `https://github.com/<yourhandle>/<yourrepo>/settings`
-and in the `Webhooks` tab click on `Add webhook`.
+To set up a github webhook go to `https://github.com/<yourhandle>/<yourrepo>/settings` and in the `Webhooks` tab
+click on `Add webhook`.

 - In `Payload URL` fill in `https://<your-hydra-domain>/api/push-github`.
 - In `Content type` switch to `application/json`.
@@ -14,14 +11,3 @@ and in the `Webhooks` tab click on `Add webhook`.
 - For `Which events would you like to trigger this webhook?` keep the default option for events on `Just the push event.`.

 Then add the hook with `Add webhook`.
-
-## Gitea
-
-To set up a webhook for a Gitea repository go to the settings of the repository in your Gitea instance
-and in the `Webhooks` tab click on `Add Webhook` and choose `Gitea` in the drop down.
-
- In `Target URL` fill in `https://<your-hydra-domain>/api/push-gitea`.
- Keep HTTP method `POST`, POST Content Type `application/json` and Trigger On `Push Events`.
- Change the branch filter to match the git branch hydra builds.
-
-Then add the hook with `Add webhook`.
--- a/flake.lock
+++ b/flake.lock
@@ -16,96 +16,74 @@
        "type": "github"
      }
    },
-    "flake-parts": {
-      "inputs": {
-        "nixpkgs-lib": [
-          "nix",
-          "nixpkgs"
-        ]
-      },
-      "locked": {
-        "lastModified": 1712014858,
-        "narHash": "sha256-sB4SWl2lX95bExY2gMFG5HIzvva5AVMJd4Igm+GpZNw=",
-        "owner": "hercules-ci",
-        "repo": "flake-parts",
-        "rev": "9126214d0a59633752a136528f5f3b9aa8565b7d",
-        "type": "github"
-      },
-      "original": {
-        "owner": "hercules-ci",
-        "repo": "flake-parts",
-        "type": "github"
-      }
-    },
-    "flake-utils": {
-      "locked": {
-        "lastModified": 1667395993,
-        "narHash": "sha256-nuEHfE/LcWyuSWnS8t12N1wc105Qtau+/OdUAjtQ0rA=",
-        "owner": "numtide",
-        "repo": "flake-utils",
-        "rev": "5aed5285a952e0b949eb3ba02c12fa4fcfef535f",
-        "type": "github"
-      },
-      "original": {
-        "owner": "numtide",
-        "repo": "flake-utils",
-        "type": "github"
-      }
-    },
-    "libgit2": {
+    "lowdown-src": {
      "flake": false,
      "locked": {
-        "lastModified": 1697646580,
-        "narHash": "sha256-oX4Z3S9WtJlwvj0uH9HlYcWv+x1hqp8mhXl7HsLu2f0=",
-        "owner": "libgit2",
-        "repo": "libgit2",
-        "rev": "45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5",
+        "lastModified": 1633514407,
+        "narHash": "sha256-Dw32tiMjdK9t3ETl5fzGrutQTzh2rufgZV4A/BbxuD4=",
+        "owner": "kristapsdz",
+        "repo": "lowdown",
+        "rev": "d2c2b44ff6c27b936ec27358a2653caaef8f73b8",
        "type": "github"
      },
      "original": {
-        "owner": "libgit2",
-        "repo": "libgit2",
+        "owner": "kristapsdz",
+        "repo": "lowdown",
        "type": "github"
      }
    },
    "nix": {
      "inputs": {
        "flake-compat": "flake-compat",
-        "flake-parts": "flake-parts",
-        "libgit2": "libgit2",
+        "lowdown-src": "lowdown-src",
        "nixpkgs": [
          "nixpkgs"
        ],
-        "nixpkgs-regression": "nixpkgs-regression",
-        "pre-commit-hooks": "pre-commit-hooks"
+        "nixpkgs-regression": "nixpkgs-regression"
      },
      "locked": {
-        "lastModified": 1727132408,
-        "narHash": "sha256-esYY9tT7Q79rG4+Xvf32L3AZvVGgdVeT1n0uK6Gd6ls=",
+        "lastModified": 1706208340,
+        "narHash": "sha256-wNyHUEIiKKVs6UXrUzhP7RSJQv0A8jckgcuylzftl8k=",
        "owner": "NixOS",
        "repo": "nix",
-        "rev": "5ffd239adc9b7fddca7a2a59a8b87da5af14ec4d",
+        "rev": "2c4bb93ba5a97e7078896ebc36385ce172960e4e",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
-        "ref": "2.23-maintenance",
+        "ref": "2.19-maintenance",
        "repo": "nix",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
-        "lastModified": 1712848736,
-        "narHash": "sha256-CzZwhqyLlebljv1zFS2KWVH/3byHND0LfaO1jKsGuVo=",
+        "lastModified": 1701615100,
+        "narHash": "sha256-7VI84NGBvlCTduw2aHLVB62NvCiZUlALLqBe5v684Aw=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "1d6a23f11e44d0fb64b3237569b87658a9eb5643",
+        "rev": "e9f06adb793d1cca5384907b3b8a4071d5d7cb19",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
-        "ref": "nixos-23.11-small",
+        "ref": "nixos-23.05",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-for-fileset": {
+      "locked": {
+        "lastModified": 1706098335,
+        "narHash": "sha256-r3dWjT8P9/Ah5m5ul4WqIWD8muj5F+/gbCdjiNVBKmU=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "a77ab169a83a4175169d78684ddd2e54486ac651",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixos-23.11",
        "repo": "nixpkgs",
        "type": "github"
      }
@@ -126,42 +104,11 @@
        "type": "github"
      }
    },
-    "pre-commit-hooks": {
-      "inputs": {
-        "flake-compat": [
-          "nix"
-        ],
-        "flake-utils": "flake-utils",
-        "gitignore": [
-          "nix"
-        ],
-        "nixpkgs": [
-          "nix",
-          "nixpkgs"
-        ],
-        "nixpkgs-stable": [
-          "nix",
-          "nixpkgs"
-        ]
-      },
-      "locked": {
-        "lastModified": 1712897695,
-        "narHash": "sha256-nMirxrGteNAl9sWiOhoN5tIHyjBbVi5e2tgZUgZlK3Y=",
-        "owner": "cachix",
-        "repo": "pre-commit-hooks.nix",
-        "rev": "40e6053ecb65fcbf12863338a6dcefb3f55f1bf8",
-        "type": "github"
-      },
-      "original": {
-        "owner": "cachix",
-        "repo": "pre-commit-hooks.nix",
-        "type": "github"
-      }
-    },
    "root": {
      "inputs": {
        "nix": "nix",
-        "nixpkgs": "nixpkgs"
+        "nixpkgs": "nixpkgs",
+        "nixpkgs-for-fileset": "nixpkgs-for-fileset"
      }
    }
  },
--- a/flake.nix
+++ b/flake.nix
@@ -1,11 +1,16 @@
 {
  description = "A Nix-based continuous build system";

-  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11-small";
-  inputs.nix.url = "github:NixOS/nix/2.23-maintenance";
+  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
+  inputs.nix.url = "github:NixOS/nix/2.19-maintenance";
  inputs.nix.inputs.nixpkgs.follows = "nixpkgs";

-  outputs = { self, nixpkgs, nix }:
+  # TODO get rid of this once https://github.com/NixOS/nix/pull/9546 is
+  # mered and we upgrade or Nix, so the main `nixpkgs` input is at least
+  # 23.11 and has `lib.fileset`.
+  inputs.nixpkgs-for-fileset.url = "github:NixOS/nixpkgs/nixos-23.11";
+
+  outputs = { self, nixpkgs, nix, nixpkgs-for-fileset }:
    let
      systems = [ "x86_64-linux" "aarch64-linux" ];
      forEachSystem = nixpkgs.lib.genAttrs systems;
@@ -17,13 +22,52 @@
        overlays = overlayList;
      });

+      # NixOS configuration used for VM tests.
+      hydraServer =
+        { config, pkgs, ... }:
+        {
+          imports = [ self.nixosModules.hydraTest ];
+
+          virtualisation.memorySize = 1024;
+          virtualisation.writableStore = true;
+
+          environment.systemPackages = [ pkgs.perlPackages.LWP pkgs.perlPackages.JSON ];
+
+          nix = {
+            # Without this nix tries to fetch packages from the default
+            # cache.nixos.org which is not reachable from this sandboxed NixOS test.
+            binaryCaches = [ ];
+          };
+        };
+
    in
    rec {

      # A Nixpkgs overlay that provides a 'hydra' package.
      overlays.default = final: prev: {
+
+        # Add LDAP dependencies that aren't currently found within nixpkgs.
+        perlPackages = prev.perlPackages // {
+
+          PrometheusTiny = final.perlPackages.buildPerlPackage {
+            pname = "Prometheus-Tiny";
+            version = "0.007";
+            src = final.fetchurl {
+              url = "mirror://cpan/authors/id/R/RO/ROBN/Prometheus-Tiny-0.007.tar.gz";
+              sha256 = "0ef8b226a2025cdde4df80129dd319aa29e884e653c17dc96f4823d985c028ec";
+            };
+            buildInputs = with final.perlPackages; [ HTTPMessage Plack TestException ];
+            meta = {
+              homepage = "https://github.com/robn/Prometheus-Tiny";
+              description = "A tiny Prometheus client";
+              license = with final.lib.licenses; [ artistic1 gpl1Plus ];
+            };
+          };
+
+        };
+
        hydra = final.callPackage ./package.nix {
-          inherit (nixpkgs.lib) fileset;
+          inherit (nixpkgs-for-fileset.lib) fileset;
          rawSrc = self;
        };
      };
@@ -49,9 +93,286 @@
              echo "doc manual $out/share/doc/hydra" >> $out/nix-support/hydra-build-products
            '');

-        tests = import ./nixos-tests.nix {
-          inherit forEachSystem nixpkgs pkgsBySystem nixosModules;
-        };
+        tests.install = forEachSystem (system:
+          with import (nixpkgs + "/nixos/lib/testing-python.nix") { inherit system; };
+          simpleTest {
+            name = "hydra-install";
+            nodes.machine = hydraServer;
+            testScript =
+              ''
+                machine.wait_for_job("hydra-init")
+                machine.wait_for_job("hydra-server")
+                machine.wait_for_job("hydra-evaluator")
+                machine.wait_for_job("hydra-queue-runner")
+                machine.wait_for_open_port(3000)
+                machine.succeed("curl --fail http://localhost:3000/")
+              '';
+          });
+
+        tests.notifications = forEachSystem (system:
+          let pkgs = pkgsBySystem.${system}; in
+          with import (nixpkgs + "/nixos/lib/testing-python.nix") { inherit system; };
+          simpleTest {
+            name = "hydra-notifications";
+            nodes.machine = { pkgs, ... }: {
+              imports = [ hydraServer ];
+              services.hydra-dev.extraConfig = ''
+                <influxdb>
+                  url = http://127.0.0.1:8086
+                  db = hydra
+                </influxdb>
+              '';
+              services.influxdb.enable = true;
+            };
+            testScript = ''
+              machine.wait_for_job("hydra-init")
+
+              # Create an admin account and some other state.
+              machine.succeed(
+                  """
+                      su - hydra -c "hydra-create-user root --email-address 'alice@example.org' --password foobar --role admin"
+                      mkdir /run/jobset
+                      chmod 755 /run/jobset
+                      cp ${./t/jobs/api-test.nix} /run/jobset/default.nix
+                      chmod 644 /run/jobset/default.nix
+                      chown -R hydra /run/jobset
+              """
+              )
+
+              # Wait until InfluxDB can receive web requests
+              machine.wait_for_job("influxdb")
+              machine.wait_for_open_port(8086)
+
+              # Create an InfluxDB database where hydra will write to
+              machine.succeed(
+                  "curl -XPOST 'http://127.0.0.1:8086/query' "
+                  + "--data-urlencode 'q=CREATE DATABASE hydra'"
+              )
+
+              # Wait until hydra-server can receive HTTP requests
+              machine.wait_for_job("hydra-server")
+              machine.wait_for_open_port(3000)
+
+              # Setup the project and jobset
+              machine.succeed(
+                  "su - hydra -c 'perl -I ${pkgs.hydra.perlDeps}/lib/perl5/site_perl ${./t/setup-notifications-jobset.pl}' >&2"
+              )
+
+              # Wait until hydra has build the job and
+              # the InfluxDBNotification plugin uploaded its notification to InfluxDB
+              machine.wait_until_succeeds(
+                  "curl -s -H 'Accept: application/csv' "
+                  + "-G 'http://127.0.0.1:8086/query?db=hydra' "
+                  + "--data-urlencode 'q=SELECT * FROM hydra_build_status' | grep success"
+              )
+            '';
+          });
+
+        tests.gitea = forEachSystem (system:
+          let pkgs = pkgsBySystem.${system}; in
+          with import (nixpkgs + "/nixos/lib/testing-python.nix") { inherit system; };
+          makeTest {
+            name = "hydra-gitea";
+            nodes.machine = { pkgs, ... }: {
+              imports = [ hydraServer ];
+              services.hydra-dev.extraConfig = ''
+                <gitea_authorization>
+                root=d7f16a3412e01a43a414535b16007c6931d3a9c7
+                </gitea_authorization>
+              '';
+              nix = {
+                distributedBuilds = true;
+                buildMachines = [{
+                  hostName = "localhost";
+                  systems = [ system ];
+                }];
+                binaryCaches = [ ];
+              };
+              services.gitea = {
+                enable = true;
+                database.type = "postgres";
+                disableRegistration = true;
+                httpPort = 3001;
+              };
+              services.openssh.enable = true;
+              environment.systemPackages = with pkgs; [ gitea git jq gawk ];
+              networking.firewall.allowedTCPPorts = [ 3000 ];
+            };
+            skipLint = true;
+            testScript =
+              let
+                scripts.mktoken = pkgs.writeText "token.sql" ''
+                  INSERT INTO access_token (id, uid, name, created_unix, updated_unix, token_hash, token_salt, token_last_eight) VALUES (1, 1, 'hydra', 1617107360, 1617107360, 'a930f319ca362d7b49a4040ac0af74521c3a3c3303a86f327b01994430672d33b6ec53e4ea774253208686c712495e12a486', 'XRjWE9YW0g', '31d3a9c7');
+                '';
+
+                scripts.git-setup = pkgs.writeShellScript "setup.sh" ''
+                  set -x
+                  mkdir -p /tmp/repo $HOME/.ssh
+                  cat ${snakeoilKeypair.privkey} > $HOME/.ssh/privk
+                  chmod 0400 $HOME/.ssh/privk
+                  git -C /tmp/repo init
+                  cp ${smallDrv} /tmp/repo/jobset.nix
+                  git -C /tmp/repo add .
+                  git config --global user.email test@localhost
+                  git config --global user.name test
+                  git -C /tmp/repo commit -m 'Initial import'
+                  git -C /tmp/repo remote add origin gitea@machine:root/repo
+                  GIT_SSH_COMMAND='ssh -i $HOME/.ssh/privk -o StrictHostKeyChecking=no' \
+                    git -C /tmp/repo push origin master
+                  git -C /tmp/repo log >&2
+                '';
+
+                scripts.hydra-setup = pkgs.writeShellScript "hydra.sh" ''
+                  set -x
+                  su -l hydra -c "hydra-create-user root --email-address \
+                    'alice@example.org' --password foobar --role admin"
+
+                  URL=http://localhost:3000
+                  USERNAME="root"
+                  PASSWORD="foobar"
+                  PROJECT_NAME="trivial"
+                  JOBSET_NAME="trivial"
+                  mycurl() {
+                    curl --referer $URL -H "Accept: application/json" \
+                      -H "Content-Type: application/json" $@
+                  }
+
+                  cat >data.json <<EOF
+                  { "username": "$USERNAME", "password": "$PASSWORD" }
+                  EOF
+                  mycurl -X POST -d '@data.json' $URL/login -c hydra-cookie.txt
+
+                  cat >data.json <<EOF
+                  {
+                    "displayname":"Trivial",
+                    "enabled":"1",
+                    "visible":"1"
+                  }
+                  EOF
+                  mycurl --silent -X PUT $URL/project/$PROJECT_NAME \
+                    -d @data.json -b hydra-cookie.txt
+
+                  cat >data.json <<EOF
+                  {
+                    "description": "Trivial",
+                    "checkinterval": "60",
+                    "enabled": "1",
+                    "visible": "1",
+                    "keepnr": "1",
+                    "enableemail": true,
+                    "emailoverride": "hydra@localhost",
+                    "type": 0,
+                    "nixexprinput": "git",
+                    "nixexprpath": "jobset.nix",
+                    "inputs": {
+                      "git": {"value": "http://localhost:3001/root/repo.git", "type": "git"},
+                      "gitea_repo_name": {"value": "repo", "type": "string"},
+                      "gitea_repo_owner": {"value": "root", "type": "string"},
+                      "gitea_status_repo": {"value": "git", "type": "string"},
+                      "gitea_http_url": {"value": "http://localhost:3001", "type": "string"}
+                    }
+                  }
+                  EOF
+
+                  mycurl --silent -X PUT $URL/jobset/$PROJECT_NAME/$JOBSET_NAME \
+                    -d @data.json -b hydra-cookie.txt
+                '';
+
+                api_token = "d7f16a3412e01a43a414535b16007c6931d3a9c7";
+
+                snakeoilKeypair = {
+                  privkey = pkgs.writeText "privkey.snakeoil" ''
+                    -----BEGIN EC PRIVATE KEY-----
+                    MHcCAQEEIHQf/khLvYrQ8IOika5yqtWvI0oquHlpRLTZiJy5dRJmoAoGCCqGSM49
+                    AwEHoUQDQgAEKF0DYGbBwbj06tA3fd/+yP44cvmwmHBWXZCKbS+RQlAKvLXMWkpN
+                    r1lwMyJZoSGgBHoUahoYjTh9/sJL7XLJtA==
+                    -----END EC PRIVATE KEY-----
+                  '';
+
+                  pubkey = pkgs.lib.concatStrings [
+                    "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHA"
+                    "yNTYAAABBBChdA2BmwcG49OrQN33f/sj+OHL5sJhwVl2Qim0vkUJQCry1zFpKTa"
+                    "9ZcDMiWaEhoAR6FGoaGI04ff7CS+1yybQ= sakeoil"
+                  ];
+                };
+
+                smallDrv = pkgs.writeText "jobset.nix" ''
+                  { trivial = builtins.derivation {
+                      name = "trivial";
+                      system = "${system}";
+                      builder = "/bin/sh";
+                      allowSubstitutes = false;
+                      preferLocalBuild = true;
+                      args = ["-c" "echo success > $out; exit 0"];
+                    };
+                   }
+                '';
+              in
+              ''
+                import json
+
+                machine.start()
+                machine.wait_for_unit("multi-user.target")
+                machine.wait_for_open_port(3000)
+                machine.wait_for_open_port(3001)
+
+                machine.succeed(
+                    "su -l gitea -c 'GITEA_WORK_DIR=/var/lib/gitea gitea admin user create "
+                    + "--username root --password root --email test@localhost'"
+                )
+                machine.succeed("su -l postgres -c 'psql gitea < ${scripts.mktoken}'")
+
+                machine.succeed(
+                    "curl --fail -X POST http://localhost:3001/api/v1/user/repos "
+                    + "-H 'Accept: application/json' -H 'Content-Type: application/json' "
+                    + f"-H 'Authorization: token ${api_token}'"
+                    + ' -d \'{"auto_init":false, "description":"string", "license":"mit", "name":"repo", "private":false}\'''
+                )
+
+                machine.succeed(
+                    "curl --fail -X POST http://localhost:3001/api/v1/user/keys "
+                    + "-H 'Accept: application/json' -H 'Content-Type: application/json' "
+                    + f"-H 'Authorization: token ${api_token}'"
+                    + ' -d \'{"key":"${snakeoilKeypair.pubkey}","read_only":true,"title":"SSH"}\'''
+                )
+
+                machine.succeed(
+                    "${scripts.git-setup}"
+                )
+
+                machine.succeed(
+                    "${scripts.hydra-setup}"
+                )
+
+                machine.wait_until_succeeds(
+                    'curl -Lf -s http://localhost:3000/build/1 -H "Accept: application/json" '
+                    + '|  jq .buildstatus | xargs test 0 -eq'
+                )
+
+                data = machine.succeed(
+                    'curl -Lf -s "http://localhost:3001/api/v1/repos/root/repo/statuses/$(cd /tmp/repo && git show | head -n1 | awk "{print \\$2}")" '
+                    + "-H 'Accept: application/json' -H 'Content-Type: application/json' "
+                    + f"-H 'Authorization: token ${api_token}'"
+                )
+
+                response = json.loads(data)
+
+                assert len(response) == 2, "Expected exactly two status updates for latest commit!"
+                assert response[0]['status'] == "success", "Expected latest status to be success!"
+                assert response[1]['status'] == "pending", "Expected first status to be pending!"
+
+                machine.shutdown()
+              '';
+          });
+
+        tests.validate-openapi = forEachSystem (system:
+          let pkgs = pkgsBySystem.${system}; in
+          pkgs.runCommand "validate-openapi"
+          { buildInputs = [ pkgs.openapi-generator-cli ]; }
+          ''
+            openapi-generator-cli validate -i ${./hydra-api.yaml}
+            touch $out
+          '');

        container = nixosConfigurations.container.config.system.build.toplevel;
      };
@@ -75,8 +396,6 @@
        system = "x86_64-linux";
        modules =
          [
-            self.nixosModules.hydra
-            self.nixosModules.overlayNixpkgsForThisHydra
            self.nixosModules.hydraTest
            self.nixosModules.hydraProxy
            {
--- a/nixos-modules/default.nix
+++ b/nixos-modules/default.nix
@@ -1,14 +1,14 @@
 { overlays }:

-{
-  hydra = import ./hydra.nix;
-
-  overlayNixpkgsForThisHydra = { pkgs, ... }: {
+rec {
+  hydra = {
+    imports = [ ./hydra.nix ];
    nixpkgs = { inherit overlays; };
-    services.hydra.package = pkgs.hydra;
  };

  hydraTest = { pkgs, ... }: {
+    imports = [ hydra ];
+
    services.hydra-dev.enable = true;
    services.hydra-dev.hydraURL = "http://hydra.example.org";
    services.hydra-dev.notificationSender = "admin@hydra.example.org";
@@ -16,7 +16,7 @@
    systemd.services.hydra-send-stats.enable = false;

    services.postgresql.enable = true;
-    services.postgresql.package = pkgs.postgresql_12;
+    services.postgresql.package = pkgs.postgresql_11;

    # The following is to work around the following error from hydra-server:
    #   [error] Caught exception in engine "Cannot determine local time zone"
--- a/nixos-modules/hydra.nix
+++ b/nixos-modules/hydra.nix
@@ -68,7 +68,7 @@ in

      package = mkOption {
        type = types.path;
-        default = pkgs.hydra_unstable;
+        default = pkgs.hydra;
        defaultText = literalExpression "pkgs.hydra";
        description = "The Hydra package.";
      };
@@ -233,7 +233,7 @@ in
      gc-keep-outputs = true;
      gc-keep-derivations = true;
    };
-
+    
    services.hydra-dev.extraConfig =
      ''
        using_frontend_proxy = 1
@@ -408,7 +408,6 @@ in
        requires = [ "hydra-init.service" ];
        after = [ "hydra-init.service" ];
        restartTriggers = [ hydraConf ];
-        path = [ pkgs.zstd ];
        environment = env // {
          PGPASSFILE = "${baseDir}/pgpass-queue-runner"; # grrr
          HYDRA_DBI = "${env.HYDRA_DBI};application_name=hydra-notify";
@@ -459,17 +458,10 @@ in
    # logs automatically after a step finishes, but this doesn't work
    # if the queue runner is stopped prematurely.
    systemd.services.hydra-compress-logs =
-      { path = [ pkgs.bzip2 pkgs.zstd ];
+      { path = [ pkgs.bzip2 ];
        script =
          ''
-            set -eou pipefail
-            compression=$(sed -nr 's/compress_build_logs_compression = ()/\1/p' ${baseDir}/hydra.conf)
-            if [[ $compression == "" ]]; then
-              compression="bzip2"
-            elif [[ $compression == zstd ]]; then
-              compression="zstd --rm"
-            fi
-            find ${baseDir}/build-logs -type f -name "*.drv" -mtime +3 -size +0c | xargs -r "$compression" --force --quiet
+            find ${baseDir}/build-logs -type f -name "*.drv" -mtime +3 -size +0c | xargs -r bzip2 -v -f
          '';
        startAt = "Sun 01:45";
      };
--- a/nixos-tests.nix
+++ b/nixos-tests.nix
@@ -1,309 +0,0 @@
-{ forEachSystem, nixpkgs, pkgsBySystem, nixosModules }:
-
-let
-  # NixOS configuration used for VM tests.
-  hydraServer =
-    { config, pkgs, ... }:
-    {
-      imports = [
-        nixosModules.hydra
-        nixosModules.overlayNixpkgsForThisHydra
-        nixosModules.hydraTest
-      ];
-
-      virtualisation.memorySize = 1024;
-      virtualisation.writableStore = true;
-
-      environment.systemPackages = [ pkgs.perlPackages.LWP pkgs.perlPackages.JSON ];
-
-      nix = {
-        # Without this nix tries to fetch packages from the default
-        # cache.nixos.org which is not reachable from this sandboxed NixOS test.
-        settings.substituters = [ ];
-      };
-    };
-
-in
-
-{
-
-  install = forEachSystem (system:
-    with import (nixpkgs + "/nixos/lib/testing-python.nix") { inherit system; };
-    simpleTest {
-      name = "hydra-install";
-      nodes.machine = hydraServer;
-      testScript =
-        ''
-          machine.wait_for_job("hydra-init")
-          machine.wait_for_job("hydra-server")
-          machine.wait_for_job("hydra-evaluator")
-          machine.wait_for_job("hydra-queue-runner")
-          machine.wait_for_open_port(3000)
-          machine.succeed("curl --fail http://localhost:3000/")
-        '';
-    });
-
-  notifications = forEachSystem (system:
-    let pkgs = pkgsBySystem.${system}; in
-    with import (nixpkgs + "/nixos/lib/testing-python.nix") { inherit system; };
-    simpleTest {
-      name = "hydra-notifications";
-      nodes.machine = { pkgs, ... }: {
-        imports = [ hydraServer ];
-        services.hydra-dev.extraConfig = ''
-          <influxdb>
-            url = http://127.0.0.1:8086
-            db = hydra
-          </influxdb>
-        '';
-        services.influxdb.enable = true;
-      };
-      testScript = ''
-        machine.wait_for_job("hydra-init")
-
-        # Create an admin account and some other state.
-        machine.succeed(
-            """
-                su - hydra -c "hydra-create-user root --email-address 'alice@example.org' --password foobar --role admin"
-                mkdir /run/jobset
-                chmod 755 /run/jobset
-                cp ${./t/jobs/api-test.nix} /run/jobset/default.nix
-                chmod 644 /run/jobset/default.nix
-                chown -R hydra /run/jobset
-        """
-        )
-
-        # Wait until InfluxDB can receive web requests
-        machine.wait_for_job("influxdb")
-        machine.wait_for_open_port(8086)
-
-        # Create an InfluxDB database where hydra will write to
-        machine.succeed(
-            "curl -XPOST 'http://127.0.0.1:8086/query' "
-            + "--data-urlencode 'q=CREATE DATABASE hydra'"
-        )
-
-        # Wait until hydra-server can receive HTTP requests
-        machine.wait_for_job("hydra-server")
-        machine.wait_for_open_port(3000)
-
-        # Setup the project and jobset
-        machine.succeed(
-            "su - hydra -c 'perl -I ${pkgs.hydra.perlDeps}/lib/perl5/site_perl ${./t/setup-notifications-jobset.pl}' >&2"
-        )
-
-        # Wait until hydra has build the job and
-        # the InfluxDBNotification plugin uploaded its notification to InfluxDB
-        machine.wait_until_succeeds(
-            "curl -s -H 'Accept: application/csv' "
-            + "-G 'http://127.0.0.1:8086/query?db=hydra' "
-            + "--data-urlencode 'q=SELECT * FROM hydra_build_status' | grep success"
-        )
-      '';
-    });
-
-  gitea = forEachSystem (system:
-    let pkgs = pkgsBySystem.${system}; in
-    with import (nixpkgs + "/nixos/lib/testing-python.nix") { inherit system; };
-    makeTest {
-      name = "hydra-gitea";
-      nodes.machine = { pkgs, ... }: {
-        imports = [ hydraServer ];
-        services.hydra-dev.extraConfig = ''
-          <gitea_authorization>
-          root=d7f16a3412e01a43a414535b16007c6931d3a9c7
-          </gitea_authorization>
-        '';
-        nixpkgs.config.permittedInsecurePackages = [ "gitea-1.19.4" ];
-        nix = {
-          settings.substituters = [ ];
-        };
-        services.gitea = {
-          enable = true;
-          database.type = "postgres";
-          settings = {
-            service.DISABLE_REGISTRATION = true;
-            server.HTTP_PORT = 3001;
-          };
-        };
-        services.openssh.enable = true;
-        environment.systemPackages = with pkgs; [ gitea git jq gawk ];
-        networking.firewall.allowedTCPPorts = [ 3000 ];
-      };
-      skipLint = true;
-      testScript =
-        let
-          scripts.mktoken = pkgs.writeText "token.sql" ''
-            INSERT INTO access_token (id, uid, name, created_unix, updated_unix, token_hash, token_salt, token_last_eight, scope) VALUES (1, 1, 'hydra', 1617107360, 1617107360, 'a930f319ca362d7b49a4040ac0af74521c3a3c3303a86f327b01994430672d33b6ec53e4ea774253208686c712495e12a486', 'XRjWE9YW0g', '31d3a9c7', 'all');
-          '';
-
-          scripts.git-setup = pkgs.writeShellScript "setup.sh" ''
-            set -x
-            mkdir -p /tmp/repo $HOME/.ssh
-            cat ${snakeoilKeypair.privkey} > $HOME/.ssh/privk
-            chmod 0400 $HOME/.ssh/privk
-            git -C /tmp/repo init
-            cp ${smallDrv} /tmp/repo/jobset.nix
-            git -C /tmp/repo add .
-            git config --global user.email test@localhost
-            git config --global user.name test
-            git -C /tmp/repo commit -m 'Initial import'
-            git -C /tmp/repo remote add origin gitea@machine:root/repo
-            GIT_SSH_COMMAND='ssh -i $HOME/.ssh/privk -o StrictHostKeyChecking=no' \
-              git -C /tmp/repo push origin master
-            git -C /tmp/repo log >&2
-          '';
-
-          scripts.hydra-setup = pkgs.writeShellScript "hydra.sh" ''
-            set -x
-            su -l hydra -c "hydra-create-user root --email-address \
-              'alice@example.org' --password foobar --role admin"
-
-            URL=http://localhost:3000
-            USERNAME="root"
-            PASSWORD="foobar"
-            PROJECT_NAME="trivial"
-            JOBSET_NAME="trivial"
-            mycurl() {
-              curl --referer $URL -H "Accept: application/json" \
-                -H "Content-Type: application/json" $@
-            }
-
-            cat >data.json <<EOF
-            { "username": "$USERNAME", "password": "$PASSWORD" }
-            EOF
-            mycurl -X POST -d '@data.json' $URL/login -c hydra-cookie.txt
-
-            cat >data.json <<EOF
-            {
-              "displayname":"Trivial",
-              "enabled":"1",
-              "visible":"1"
-            }
-            EOF
-            mycurl --silent -X PUT $URL/project/$PROJECT_NAME \
-              -d @data.json -b hydra-cookie.txt
-
-            cat >data.json <<EOF
-            {
-              "description": "Trivial",
-              "checkinterval": "60",
-              "enabled": "1",
-              "visible": "1",
-              "keepnr": "1",
-              "enableemail": true,
-              "emailoverride": "hydra@localhost",
-              "type": 0,
-              "nixexprinput": "git",
-              "nixexprpath": "jobset.nix",
-              "inputs": {
-                "git": {"value": "http://localhost:3001/root/repo.git", "type": "git"},
-                "gitea_repo_name": {"value": "repo", "type": "string"},
-                "gitea_repo_owner": {"value": "root", "type": "string"},
-                "gitea_status_repo": {"value": "git", "type": "string"},
-                "gitea_http_url": {"value": "http://localhost:3001", "type": "string"}
-              }
-            }
-            EOF
-
-            mycurl --silent -X PUT $URL/jobset/$PROJECT_NAME/$JOBSET_NAME \
-              -d @data.json -b hydra-cookie.txt
-          '';
-
-          api_token = "d7f16a3412e01a43a414535b16007c6931d3a9c7";
-
-          snakeoilKeypair = {
-            privkey = pkgs.writeText "privkey.snakeoil" ''
-              -----BEGIN EC PRIVATE KEY-----
-              MHcCAQEEIHQf/khLvYrQ8IOika5yqtWvI0oquHlpRLTZiJy5dRJmoAoGCCqGSM49
-              AwEHoUQDQgAEKF0DYGbBwbj06tA3fd/+yP44cvmwmHBWXZCKbS+RQlAKvLXMWkpN
-              r1lwMyJZoSGgBHoUahoYjTh9/sJL7XLJtA==
-              -----END EC PRIVATE KEY-----
-            '';
-
-            pubkey = pkgs.lib.concatStrings [
-              "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHA"
-              "yNTYAAABBBChdA2BmwcG49OrQN33f/sj+OHL5sJhwVl2Qim0vkUJQCry1zFpKTa"
-              "9ZcDMiWaEhoAR6FGoaGI04ff7CS+1yybQ= sakeoil"
-            ];
-          };
-
-          smallDrv = pkgs.writeText "jobset.nix" ''
-            { trivial = builtins.derivation {
-                name = "trivial";
-                system = "${system}";
-                builder = "/bin/sh";
-                allowSubstitutes = false;
-                preferLocalBuild = true;
-                args = ["-c" "echo success > $out; exit 0"];
-              };
-             }
-          '';
-        in
-        ''
-          import json
-
-          machine.start()
-          machine.wait_for_unit("multi-user.target")
-          machine.wait_for_open_port(3000)
-          machine.wait_for_open_port(3001)
-
-          machine.succeed(
-              "su -l gitea -c 'GITEA_WORK_DIR=/var/lib/gitea gitea admin user create "
-              + "--username root --password root --email test@localhost'"
-          )
-          machine.succeed("su -l postgres -c 'psql gitea < ${scripts.mktoken}'")
-
-          machine.succeed(
-              "curl --fail -X POST http://localhost:3001/api/v1/user/repos "
-              + "-H 'Accept: application/json' -H 'Content-Type: application/json' "
-              + f"-H 'Authorization: token ${api_token}'"
-              + ' -d \'{"auto_init":false, "description":"string", "license":"mit", "name":"repo", "private":false}\'''
-          )
-
-          machine.succeed(
-              "curl --fail -X POST http://localhost:3001/api/v1/user/keys "
-              + "-H 'Accept: application/json' -H 'Content-Type: application/json' "
-              + f"-H 'Authorization: token ${api_token}'"
-              + ' -d \'{"key":"${snakeoilKeypair.pubkey}","read_only":true,"title":"SSH"}\'''
-          )
-
-          machine.succeed(
-              "${scripts.git-setup}"
-          )
-
-          machine.succeed(
-              "${scripts.hydra-setup}"
-          )
-
-          machine.wait_until_succeeds(
-              'curl -Lf -s http://localhost:3000/build/1 -H "Accept: application/json" '
-              + '|  jq .buildstatus | xargs test 0 -eq'
-          )
-
-          data = machine.succeed(
-              'curl -Lf -s "http://localhost:3001/api/v1/repos/root/repo/statuses/$(cd /tmp/repo && git show | head -n1 | awk "{print \\$2}")" '
-              + "-H 'Accept: application/json' -H 'Content-Type: application/json' "
-              + f"-H 'Authorization: token ${api_token}'"
-          )
-
-          response = json.loads(data)
-
-          assert len(response) == 2, "Expected exactly three status updates for latest commit (queued, finished)!"
-          assert response[0]['status'] == "success", "Expected finished status to be success!"
-          assert response[1]['status'] == "pending", "Expected queued status to be pending!"
-
-          machine.shutdown()
-        '';
-    });
-
-  validate-openapi = forEachSystem (system:
-    let pkgs = pkgsBySystem.${system}; in
-    pkgs.runCommand "validate-openapi"
-    { buildInputs = [ pkgs.openapi-generator-cli ]; }
-    ''
-      openapi-generator-cli validate -i ${./hydra-api.yaml}
-      touch $out
-    '');
-
-}
--- a/package.nix
+++ b/package.nix
@@ -87,6 +87,7 @@ let
        DateTime
        DBDPg
        DBDSQLite
+        DBIxClassHelpers
        DigestSHA1
        EmailMIME
        EmailSender
--- a/src/hydra-eval-jobs/hydra-eval-jobs.cc
+++ b/src/hydra-eval-jobs/hydra-eval-jobs.cc
@@ -89,7 +89,7 @@ struct MyArgs : MixEvalArgs, MixCommonArgs, RootArgs

 static MyArgs myArgs;

-static std::string queryMetaStrings(EvalState & state, PackageInfo & drv, const std::string & name, const std::string & subAttribute)
+static std::string queryMetaStrings(EvalState & state, DrvInfo & drv, const std::string & name, const std::string & subAttribute)
 {
    Strings res;
    std::function<void(Value & v)> rec;
@@ -102,8 +102,8 @@ static std::string queryMetaStrings(EvalState & state, PackageInfo & drv, const
            for (unsigned int n = 0; n < v.listSize(); ++n)
                rec(*v.listElems()[n]);
        else if (v.type() == nAttrs) {
-            auto a = v.attrs()->find(state.symbols.create(subAttribute));
-            if (a != v.attrs()->end())
+            auto a = v.attrs->find(state.symbols.create(subAttribute));
+            if (a != v.attrs->end())
                res.push_back(std::string(state.forceString(*a->value, a->pos, "while evaluating meta attributes")));
        }
    };
@@ -138,12 +138,12 @@ static void worker(

        callFlake(state, lockedFlake, *vFlake);

-        auto vOutputs = vFlake->attrs()->get(state.symbols.create("outputs"))->value;
+        auto vOutputs = vFlake->attrs->get(state.symbols.create("outputs"))->value;
        state.forceValue(*vOutputs, noPos);

-        auto aHydraJobs = vOutputs->attrs()->get(state.symbols.create("hydraJobs"));
+        auto aHydraJobs = vOutputs->attrs->get(state.symbols.create("hydraJobs"));
        if (!aHydraJobs)
-            aHydraJobs = vOutputs->attrs()->get(state.symbols.create("checks"));
+            aHydraJobs = vOutputs->attrs->get(state.symbols.create("checks"));
        if (!aHydraJobs)
            throw Error("flake '%s' does not provide any Hydra jobs or checks", flakeRef);

@@ -181,11 +181,11 @@ static void worker(
                // CA derivations do not have static output paths, so we
                // have to defensively not query output paths in case we
                // encounter one.
-                PackageInfo::Outputs outputs = drv->queryOutputs(
+                DrvInfo::Outputs outputs = drv->queryOutputs(
                    !experimentalFeatureSettings.isEnabled(Xp::CaDerivations));

                if (drv->querySystem() == "unknown")
-                    state.error<EvalError>("derivation must have a 'system' attribute").debugThrow();
+                    throw EvalError("derivation must have a 'system' attribute");

                auto drvPath = state.store->printStorePath(drv->requireDrvPath());

@@ -204,11 +204,11 @@ static void worker(
                job["isChannel"] = drv->queryMetaBool("isHydraChannel", false);

                /* If this is an aggregate, then get its constituents. */
-                auto a = v->attrs()->get(state.symbols.create("_hydraAggregate"));
+                auto a = v->attrs->get(state.symbols.create("_hydraAggregate"));
                if (a && state.forceBool(*a->value, a->pos, "while evaluating the `_hydraAggregate` attribute")) {
-                    auto a = v->attrs()->get(state.symbols.create("constituents"));
+                    auto a = v->attrs->get(state.symbols.create("constituents"));
                    if (!a)
-                        state.error<EvalError>("derivation must have a ‘constituents’ attribute").debugThrow();
+                        throw EvalError("derivation must have a ‘constituents’ attribute");

                    NixStringContext context;
                    state.coerceToString(a->pos, *a->value, context, "while evaluating the `constituents` attribute", true, false);
@@ -260,7 +260,7 @@ static void worker(
            else if (v->type() == nAttrs) {
                auto attrs = nlohmann::json::array();
                StringSet ss;
-                for (auto & i : v->attrs()->lexicographicOrder(state.symbols)) {
+                for (auto & i : v->attrs->lexicographicOrder(state.symbols)) {
                    std::string name(state.symbols[i->name]);
                    if (name.find(' ') != std::string::npos) {
                        printError("skipping job with illegal name '%s'", name);
@@ -274,7 +274,7 @@ static void worker(
            else if (v->type() == nNull)
                ;

-            else state.error<TypeError>("attribute '%s' is %s, which is not supported", attrPath, showType(*v)).debugThrow();
+            else throw TypeError("attribute '%s' is %s, which is not supported", attrPath, showType(*v));

        } catch (EvalError & e) {
            auto msg = e.msg();
@@ -368,7 +368,7 @@ int main(int argc, char * * argv)
                            ]()
                            {
                                try {
-                                    EvalState state(myArgs.lookupPath, openStore());
+                                    EvalState state(myArgs.searchPath, openStore());
                                    Bindings & autoArgs = *myArgs.getAutoArgs(state);
                                    worker(state, autoArgs, *to, *from);
                                } catch (Error & e) {
--- a/src/hydra-evaluator/hydra-evaluator.cc
+++ b/src/hydra-evaluator/hydra-evaluator.cc
@@ -38,7 +38,7 @@ class JobsetId {
    friend bool operator!= (const JobsetId & lhs, const JobsetName & rhs);

    std::string display() const {
-        return boost::str(boost::format("%1%:%2% (jobset#%3%)") % project % jobset % id);
+        return str(format("%1%:%2% (jobset#%3%)") % project % jobset % id);
    }
 };
 bool operator==(const JobsetId & lhs, const JobsetId & rhs)
--- a/src/hydra-queue-runner/build-remote.cc
+++ b/src/hydra-queue-runner/build-remote.cc
@@ -8,7 +8,6 @@
 #include "build-result.hh"
 #include "path.hh"
 #include "serve-protocol.hh"
-#include "serve-protocol-impl.hh"
 #include "state.hh"
 #include "current-process.hh"
 #include "processes.hh"
@@ -21,6 +20,12 @@

 using namespace nix;

+
+static void append(Strings & dst, const Strings & src)
+{
+    dst.insert(dst.end(), src.begin(), src.end());
+}
+
 namespace nix::build_remote {

 static Strings extraStoreArgs(std::string & machine)
@@ -43,31 +48,63 @@ static Strings extraStoreArgs(std::string & machine)
    return result;
 }

-static std::unique_ptr<SSHMaster::Connection> openConnection(
-    ::Machine::ptr machine, SSHMaster & master)
+static void openConnection(::Machine::ptr machine, Path tmpDir, int stderrFD, SSHMaster::Connection & child)
 {
-    Strings command = {"nix-store", "--serve", "--write"};
+    std::string pgmName;
+    Pipe to, from;
+    to.create();
+    from.create();
+
+    Strings argv;
    if (machine->isLocalhost()) {
-        command.push_back("--builders");
-        command.push_back("");
+        pgmName = "nix-store";
+        argv = {"nix-store", "--builders", "", "--serve", "--write"};
    } else {
-        command.splice(command.end(), extraStoreArgs(machine->sshName));
+        pgmName = "ssh";
+        auto sshName = machine->sshName;
+        Strings extraArgs = extraStoreArgs(sshName);
+        argv = {"ssh", sshName};
+        if (machine->sshKey != "") append(argv, {"-i", machine->sshKey});
+        if (machine->sshPublicHostKey != "") {
+            Path fileName = tmpDir + "/host-key";
+            auto p = machine->sshName.find("@");
+            std::string host = p != std::string::npos ? std::string(machine->sshName, p + 1) : machine->sshName;
+            writeFile(fileName, host + " " + machine->sshPublicHostKey + "\n");
+            append(argv, {"-oUserKnownHostsFile=" + fileName});
+        }
+        append(argv,
+            { "-x", "-a", "-oBatchMode=yes", "-oConnectTimeout=60", "-oTCPKeepAlive=yes"
+            , "--", "nix-store", "--serve", "--write" });
+        append(argv, extraArgs);
    }

-    auto ret = master.startCommand(std::move(command), {
-        "-a", "-oBatchMode=yes", "-oConnectTimeout=60", "-oTCPKeepAlive=yes"
+    child.sshPid = startProcess([&]() {
+        restoreProcessContext();
+
+        if (dup2(to.readSide.get(), STDIN_FILENO) == -1)
+            throw SysError("cannot dup input pipe to stdin");
+
+        if (dup2(from.writeSide.get(), STDOUT_FILENO) == -1)
+            throw SysError("cannot dup output pipe to stdout");
+
+        if (dup2(stderrFD, STDERR_FILENO) == -1)
+            throw SysError("cannot dup stderr");
+
+        execvp(argv.front().c_str(), (char * *) stringsToCharPtrs(argv).data()); // FIXME: remove cast
+
+        throw SysError("cannot start %s", pgmName);
    });

+    to.readSide = -1;
+    from.writeSide = -1;
+
+    child.in = to.writeSide.release();
+    child.out = from.readSide.release();
+
    // XXX: determine the actual max value we can use from /proc.
-
-    // FIXME: Should this be upstreamed into `startCommand` in Nix?
-
    int pipesize = 1024 * 1024;
-
-    fcntl(ret->in.get(), F_SETPIPE_SZ, &pipesize);
-    fcntl(ret->out.get(), F_SETPIPE_SZ, &pipesize);
-
-    return ret;
+    fcntl(child.in.get(), F_SETPIPE_SZ, &pipesize);
+    fcntl(child.out.get(), F_SETPIPE_SZ, &pipesize);
 }


@@ -85,10 +122,13 @@ static void copyClosureTo(
       garbage-collect paths that are already there. Optionally, ask
       the remote host to substitute missing paths. */
    // FIXME: substitute output pollutes our build log
+    conn.to << ServeProto::Command::QueryValidPaths << 1 << useSubstitutes;
+    ServeProto::write(destStore, conn, closure);
+    conn.to.flush();
+
    /* Get back the set of paths that are already valid on the remote
       host. */
-    auto present = conn.queryValidPaths(
-        destStore, true, closure, useSubstitutes);
+    auto present = ServeProto::Serialise<StorePathSet>::read(destStore, conn);

    if (present.size() == closure.size()) return;

@@ -113,7 +153,7 @@ static void copyClosureTo(


 // FIXME: use Store::topoSortPaths().
-static StorePaths reverseTopoSortPaths(const std::map<StorePath, UnkeyedValidPathInfo> & paths)
+static StorePaths reverseTopoSortPaths(const std::map<StorePath, ValidPathInfo> & paths)
 {
    StorePaths sorted;
    StorePathSet visited;
@@ -154,6 +194,28 @@ static std::pair<Path, AutoCloseFD> openLogFile(const std::string & logDir, cons
    return {std::move(logFile), std::move(logFD)};
 }

+/**
+ * @param conn is not fully initialized; it is this functions job to set
+ * the `remoteVersion` field after the handshake is completed.
+ * Therefore, no `ServeProto::Serialize` functions can be used until
+ * that field is set.
+ */
+static void handshake(::Machine::Connection & conn, unsigned int repeats)
+{
+    conn.to << SERVE_MAGIC_1 << 0x206;
+    conn.to.flush();
+
+    unsigned int magic = readInt(conn.from);
+    if (magic != SERVE_MAGIC_2)
+        throw Error("protocol mismatch with ‘nix-store --serve’ on ‘%1%’", conn.machine->sshName);
+    conn.remoteVersion = readInt(conn.from);
+    // Now `conn` is initialized.
+    if (GET_PROTOCOL_MAJOR(conn.remoteVersion) != 0x200)
+        throw Error("unsupported ‘nix-store --serve’ protocol version on ‘%1%’", conn.machine->sshName);
+    if (GET_PROTOCOL_MINOR(conn.remoteVersion) < 3 && repeats > 0)
+        throw Error("machine ‘%1%’ does not support repeating a build; please upgrade it to Nix 1.12", conn.machine->sshName);
+}
+
 static BasicDerivation sendInputs(
    State & state,
    Step & step,
@@ -224,11 +286,21 @@ static BuildResult performBuild(
    Store & localStore,
    StorePath drvPath,
    const BasicDerivation & drv,
-    const ServeProto::BuildOptions & options,
+    const State::BuildOptions & options,
    counter & nrStepsBuilding
 )
 {
-    conn.putBuildDerivationRequest(localStore, drvPath, drv, options);
+    conn.to << ServeProto::Command::BuildDerivation << localStore.printStorePath(drvPath);
+    writeDerivation(conn.to, localStore, drv);
+    conn.to << options.maxSilentTime << options.buildTimeout;
+    if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 2)
+        conn.to << options.maxLogSize;
+    if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 3) {
+        conn.to
+            << options.repeats // == build-repeat
+            << options.enforceDeterminism;
+    }
+    conn.to.flush();

    BuildResult result;

@@ -278,7 +350,7 @@ static BuildResult performBuild(
    return result;
 }

-static std::map<StorePath, UnkeyedValidPathInfo> queryPathInfos(
+static std::map<StorePath, ValidPathInfo> queryPathInfos(
    ::Machine::Connection & conn,
    Store & localStore,
    StorePathSet & outputs,
@@ -287,18 +359,30 @@ static std::map<StorePath, UnkeyedValidPathInfo> queryPathInfos(
 {

    /* Get info about each output path. */
-    std::map<StorePath, UnkeyedValidPathInfo> infos;
+    std::map<StorePath, ValidPathInfo> infos;
    conn.to << ServeProto::Command::QueryPathInfos;
    ServeProto::write(localStore, conn, outputs);
    conn.to.flush();
    while (true) {
        auto storePathS = readString(conn.from);
        if (storePathS == "") break;
-
-        auto storePath = localStore.parseStorePath(storePathS);
-        auto info = ServeProto::Serialise<UnkeyedValidPathInfo>::read(localStore, conn);
+        auto deriver = readString(conn.from); // deriver
+        auto references = ServeProto::Serialise<StorePathSet>::read(localStore, conn);
+        readLongLong(conn.from); // download size
+        auto narSize = readLongLong(conn.from);
+        auto narHash = Hash::parseAny(readString(conn.from), htSHA256);
+        auto ca = ContentAddress::parseOpt(readString(conn.from));
+        readStrings<StringSet>(conn.from); // sigs
+        ValidPathInfo info(localStore.parseStorePath(storePathS), narHash);
+        assert(outputs.count(info.path));
+        info.references = references;
+        info.narSize = narSize;
        totalNarSize += info.narSize;
-        infos.insert_or_assign(std::move(storePath), std::move(info));
+        info.narHash = narHash;
+        info.ca = ca;
+        if (deriver != "")
+            info.deriver = localStore.parseStorePath(deriver);
+        infos.insert_or_assign(info.path, info);
    }

    return infos;
@@ -339,16 +423,14 @@ static void copyPathsFromRemote(
    NarMemberDatas & narMembers,
    Store & localStore,
    Store & destStore,
-    const std::map<StorePath, UnkeyedValidPathInfo> & infos
+    const std::map<StorePath, ValidPathInfo> & infos
 )
 {
      auto pathsSorted = reverseTopoSortPaths(infos);

      for (auto & path : pathsSorted) {
          auto & info = infos.find(path)->second;
-          copyPathFromRemote(
-              conn, narMembers, localStore, destStore,
-              ValidPathInfo { path, info });
+          copyPathFromRemote(conn, narMembers, localStore, destStore, info);
      }

 }
@@ -412,10 +494,21 @@ void RemoteResult::updateWithBuildResult(const nix::BuildResult & buildResult)

 }

+/* Utility guard object to auto-release a semaphore on destruction. */
+template <typename T>
+class SemaphoreReleaser {
+public:
+    SemaphoreReleaser(T* s) : sem(s) {}
+    ~SemaphoreReleaser() { sem->release(); }
+
+private:
+    T* sem;
+};

 void State::buildRemote(ref<Store> destStore,
+    MachineReservation::ptr & reservation,
    ::Machine::ptr machine, Step::ptr step,
-    const ServeProto::BuildOptions & buildOptions,
+    const BuildOptions & buildOptions,
    RemoteResult & result, std::shared_ptr<ActiveStep> activeStep,
    std::function<void(StepState)> updateStep,
    NarMemberDatas & narMembers)
@@ -426,26 +519,21 @@ void State::buildRemote(ref<Store> destStore,
    AutoDelete logFileDel(logFile, false);
    result.logFile = logFile;

+    nix::Path tmpDir = createTempDir();
+    AutoDelete tmpDirDel(tmpDir, true);
+
    try {

        updateStep(ssConnecting);

-        SSHMaster master {
-            machine->sshName,
-            machine->sshKey,
-            machine->sshPublicHostKey,
-            false, // no SSH master yet
-            false, // no compression yet
-            logFD.get(),
-        };
-
        // FIXME: rewrite to use Store.
-        auto child = build_remote::openConnection(machine, master);
+        SSHMaster::Connection child;
+        build_remote::openConnection(machine, tmpDir, logFD.get(), child);

        {
            auto activeStepState(activeStep->state_.lock());
            if (activeStepState->cancelled) throw Error("step cancelled");
-            activeStepState->pid = child->sshPid;
+            activeStepState->pid = child.sshPid;
        }

        Finally clearPid([&]() {
@@ -461,13 +549,9 @@ void State::buildRemote(ref<Store> destStore,
        });

        ::Machine::Connection conn {
-            {
-                .to = child->in.get(),
-                .from = child->out.get(),
-                /* Handshake. */
-                .remoteVersion = 0xdadbeef, // FIXME avoid dummy initialize
-            },
-            /*.machine =*/ machine,
+            .from = child.out.get(),
+            .to = child.in.get(),
+            .machine = machine,
        };

        Finally updateStats([&]() {
@@ -475,26 +559,14 @@ void State::buildRemote(ref<Store> destStore,
            bytesSent += conn.to.written;
        });

-        constexpr ServeProto::Version our_version = 0x206;
-
        try {
-            conn.remoteVersion = decltype(conn)::handshake(
-                conn.to,
-                conn.from,
-                our_version,
-                machine->sshName);
+          build_remote::handshake(conn, buildOptions.repeats);
        } catch (EndOfFile & e) {
-            child->sshPid.wait();
+            child.sshPid.wait();
            std::string s = chomp(readFile(result.logFile));
            throw Error("cannot connect to ‘%1%’: %2%", machine->sshName, s);
        }

-        // Do not attempt to speak a newer version of the protocol.
-        //
-        // Per https://github.com/NixOS/nix/issues/9584 should be handled as
-        // part of `handshake` in upstream nix.
-        conn.remoteVersion = std::min(conn.remoteVersion, our_version);
-
        {
            auto info(machine->state->connectInfo.lock());
            info->consecutiveFailures = 0;
@@ -551,6 +623,23 @@ void State::buildRemote(ref<Store> destStore,
            result.logFile = "";
        }

+        /* Throttle CPU-bound work. Opportunistically skip updating the current
+         * step, since this requires a DB roundtrip. */
+        if (!localWorkThrottler.try_acquire()) {
+            updateStep(ssWaitingForLocalSlot);
+            localWorkThrottler.acquire();
+        }
+        SemaphoreReleaser releaser(&localWorkThrottler);
+
+        /* Once we've started copying outputs, release the machine reservation
+         * so further builds can happen. We do not release the machine earlier
+         * to avoid situations where the queue runner is bottlenecked on
+         * copying outputs and we end up building too many things that we
+         * haven't been able to allow copy slots for. */
+        assert(reservation.unique());
+        reservation = 0;
+        wakeDispatcher();
+
        StorePathSet outputs;
        for (auto & [_, realisation] : buildResult.builtOutputs)
            outputs.insert(realisation.outPath);
@@ -597,8 +686,8 @@ void State::buildRemote(ref<Store> destStore,
        }

        /* Shut down the connection. */
-        child->in = -1;
-        child->sshPid.wait();
+        child.in = -1;
+        child.sshPid.wait();

    } catch (Error & e) {
        /* Disable this machine until a certain period of time has
--- a/src/hydra-queue-runner/builder.cc
+++ b/src/hydra-queue-runner/builder.cc
@@ -46,10 +46,12 @@ void State::builder(MachineReservation::ptr reservation)
        }
    }

-    /* Release the machine and wake up the dispatcher. */
-    assert(reservation.unique());
-    reservation = 0;
-    wakeDispatcher();
+    /* If the machine hasn't been released yet, release and wake up the dispatcher. */
+    if (reservation) {
+        assert(reservation.unique());
+        reservation = 0;
+        wakeDispatcher();
+    }

    /* If there was a temporary failure, retry the step after an
       exponentially increasing interval. */
@@ -72,11 +74,11 @@ void State::builder(MachineReservation::ptr reservation)


 State::StepResult State::doBuildStep(nix::ref<Store> destStore,
-    MachineReservation::ptr reservation,
+    MachineReservation::ptr & reservation,
    std::shared_ptr<ActiveStep> activeStep)
 {
-    auto & step(reservation->step);
-    auto & machine(reservation->machine);
+    auto step(reservation->step);
+    auto machine(reservation->machine);

    {
        auto step_(step->state.lock());
@@ -98,13 +100,10 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
       it). */
    BuildID buildId;
    std::optional<StorePath> buildDrvPath;
-    // Other fields set below
-    nix::ServeProto::BuildOptions buildOptions {
-        .maxLogSize = maxLogSize,
-        .nrRepeats = step->isDeterministic ? 1u : 0u,
-        .enforceDeterminism = step->isDeterministic,
-        .keepFailed = false,
-    };
+    BuildOptions buildOptions;
+    buildOptions.repeats = step->isDeterministic ? 1 : 0;
+    buildOptions.maxLogSize = maxLogSize;
+    buildOptions.enforceDeterminism = step->isDeterministic;

    auto conn(dbPool.get());

@@ -139,7 +138,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
            {
                auto i = jobsetRepeats.find(std::make_pair(build2->projectName, build2->jobsetName));
                if (i != jobsetRepeats.end())
-                    buildOptions.nrRepeats = std::max(buildOptions.nrRepeats, i->second);
+                    buildOptions.repeats = std::max(buildOptions.repeats, i->second);
            }
        }
        if (!build) build = *dependents.begin();
@@ -150,7 +149,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
        buildOptions.buildTimeout = build->buildTimeout;

        printInfo("performing step ‘%s’ %d times on ‘%s’ (needed by build %d and %d others)",
-            localStore->printStorePath(step->drvPath), buildOptions.nrRepeats + 1, machine->sshName, buildId, (dependents.size() - 1));
+            localStore->printStorePath(step->drvPath), buildOptions.repeats + 1, machine->sshName, buildId, (dependents.size() - 1));
    }

    if (!buildOneDone)
@@ -211,7 +210,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,

        try {
            /* FIXME: referring builds may have conflicting timeouts. */
-            buildRemote(destStore, machine, step, buildOptions, result, activeStep, updateStep, narMembers);
+            buildRemote(destStore, reservation, machine, step, buildOptions, result, activeStep, updateStep, narMembers);
        } catch (Error & e) {
            if (activeStep->state_.lock()->cancelled) {
                printInfo("marking step %d of build %d as cancelled", stepNr, buildId);
--- a/src/hydra-queue-runner/dispatcher.cc
+++ b/src/hydra-queue-runner/dispatcher.cc
@@ -39,13 +39,15 @@ void State::dispatcher()
            printMsg(lvlDebug, "dispatcher woken up");
            nrDispatcherWakeups++;

-            auto now1 = std::chrono::steady_clock::now();
+            auto t_before_work = std::chrono::steady_clock::now();

            auto sleepUntil = doDispatch();

-            auto now2 = std::chrono::steady_clock::now();
+            auto t_after_work = std::chrono::steady_clock::now();

-            dispatchTimeMs += std::chrono::duration_cast<std::chrono::milliseconds>(now2 - now1).count();
+            prom.dispatcher_time_spent_running.Increment(
+                std::chrono::duration_cast<std::chrono::microseconds>(t_after_work - t_before_work).count());
+            dispatchTimeMs += std::chrono::duration_cast<std::chrono::milliseconds>(t_after_work - t_before_work).count();

            /* Sleep until we're woken up (either because a runnable build
               is added, or because a build finishes). */
@@ -59,6 +61,10 @@ void State::dispatcher()
                *dispatcherWakeup_ = false;
            }

+            auto t_after_sleep = std::chrono::steady_clock::now();
+            prom.dispatcher_time_spent_waiting.Increment(
+                std::chrono::duration_cast<std::chrono::microseconds>(t_after_sleep - t_after_work).count());
+
        } catch (std::exception & e) {
            printError("dispatcher: %s", e.what());
            sleep(1);
@@ -127,6 +133,8 @@ system_time State::doDispatch()
           comparator is a partial ordering (see MachineInfo). */
        int highestGlobalPriority;
        int highestLocalPriority;
+        size_t numRequiredSystemFeatures;
+        size_t numRevDeps;
        BuildID lowestBuildID;

        StepInfo(Step::ptr step, Step::State & step_) : step(step)
@@ -135,6 +143,8 @@ system_time State::doDispatch()
                lowestShareUsed = std::min(lowestShareUsed, jobset->shareUsed());
            highestGlobalPriority = step_.highestGlobalPriority;
            highestLocalPriority = step_.highestLocalPriority;
+            numRequiredSystemFeatures = step->requiredSystemFeatures.size();
+            numRevDeps = step_.rdeps.size();
            lowestBuildID = step_.lowestBuildID;
        }
    };
@@ -187,6 +197,8 @@ system_time State::doDispatch()
                a.highestGlobalPriority != b.highestGlobalPriority ? a.highestGlobalPriority > b.highestGlobalPriority :
                a.lowestShareUsed != b.lowestShareUsed ? a.lowestShareUsed < b.lowestShareUsed :
                a.highestLocalPriority != b.highestLocalPriority ? a.highestLocalPriority > b.highestLocalPriority :
+                a.numRequiredSystemFeatures != b.numRequiredSystemFeatures ? a.numRequiredSystemFeatures > b.numRequiredSystemFeatures :
+                a.numRevDeps != b.numRevDeps ? a.numRevDeps > b.numRevDeps :
                a.lowestBuildID < b.lowestBuildID;
        });

@@ -231,11 +243,11 @@ system_time State::doDispatch()
        sort(machinesSorted.begin(), machinesSorted.end(),
            [](const MachineInfo & a, const MachineInfo & b) -> bool
            {
-                float ta = std::round(a.currentJobs / a.machine->speedFactor);
-                float tb = std::round(b.currentJobs / b.machine->speedFactor);
+                float ta = std::round(a.currentJobs / a.machine->speedFactorFloat);
+                float tb = std::round(b.currentJobs / b.machine->speedFactorFloat);
                return
                    ta != tb ? ta < tb :
-                    a.machine->speedFactor != b.machine->speedFactor ? a.machine->speedFactor > b.machine->speedFactor :
+                    a.machine->speedFactorFloat != b.machine->speedFactorFloat ? a.machine->speedFactorFloat > b.machine->speedFactorFloat :
                    a.currentJobs > b.currentJobs;
            });

--- a/src/hydra-queue-runner/hydra-queue-runner.cc
+++ b/src/hydra-queue-runner/hydra-queue-runner.cc
@@ -70,10 +70,31 @@ State::PromMetrics::PromMetrics()
            .Register(*registry)
            .Add({})
    )
-    , queue_max_id(
-        prometheus::BuildGauge()
-            .Name("hydraqueuerunner_queue_max_build_id_info")
-            .Help("Maximum build record ID in the queue")
+    , dispatcher_time_spent_running(
+        prometheus::BuildCounter()
+            .Name("hydraqueuerunner_dispatcher_time_spent_running")
+            .Help("Time (in micros) spent running the dispatcher")
+            .Register(*registry)
+            .Add({})
+    )
+    , dispatcher_time_spent_waiting(
+        prometheus::BuildCounter()
+            .Name("hydraqueuerunner_dispatcher_time_spent_waiting")
+            .Help("Time (in micros) spent waiting for the dispatcher to obtain work")
+            .Register(*registry)
+            .Add({})
+    )
+    , queue_monitor_time_spent_running(
+        prometheus::BuildCounter()
+            .Name("hydraqueuerunner_queue_monitor_time_spent_running")
+            .Help("Time (in micros) spent running the queue monitor")
+            .Register(*registry)
+            .Add({})
+    )
+    , queue_monitor_time_spent_waiting(
+        prometheus::BuildCounter()
+            .Name("hydraqueuerunner_queue_monitor_time_spent_waiting")
+            .Help("Time (in micros) spent waiting for the queue monitor to obtain work")
            .Register(*registry)
            .Add({})
    )
@@ -85,6 +106,7 @@ State::State(std::optional<std::string> metricsAddrOpt)
    : config(std::make_unique<HydraConfig>())
    , maxUnsupportedTime(config->getIntOption("max_unsupported_time", 0))
    , dbPool(config->getIntOption("max_db_connections", 128))
+    , localWorkThrottler(config->getIntOption("max_local_worker_threads", std::min(maxSupportedLocalWorkers, std::max(4u, std::thread::hardware_concurrency()) - 2)))
    , maxOutputSize(config->getIntOption("max_output_size", 2ULL << 30))
    , maxLogSize(config->getIntOption("max_log_size", 64ULL << 20))
    , uploadLogsToBinaryCache(config->getBoolOption("upload_logs_to_binary_cache", false))
@@ -155,16 +177,16 @@ void State::parseMachines(const std::string & contents)
        auto machine = std::make_shared<::Machine>(nix::Machine {
            // `storeUri`, not yet used
            "",
-            // `systemTypes`
-            tokenizeString<StringSet>(tokens[1], ","),
+            // `systemTypes`, not yet used
+            {},
            // `sshKey`
            tokens[2] == "-" ? "" : tokens[2],
            // `maxJobs`
            tokens[3] != ""
                ? string2Int<MaxJobs>(tokens[3]).value()
                : 1,
-            // `speedFactor`
-            std::stof(tokens[4].c_str()),
+            // `speedFactor`, not yet used
+            1,
            // `supportedFeatures`
            std::move(supportedFeatures),
            // `mandatoryFeatures`
@@ -176,6 +198,8 @@ void State::parseMachines(const std::string & contents)
        });

        machine->sshName = tokens[0];
+        machine->systemTypesSet = tokenizeString<StringSet>(tokens[1], ",");
+        machine->speedFactorFloat = atof(tokens[4].c_str());

        /* Re-use the State object of the previous machine with the
           same name. */
@@ -639,7 +663,7 @@ void State::dumpStatus(Connection & conn)

                json machine = {
                    {"enabled",  m->enabled},
-                    {"systemTypes", m->systemTypes},
+                    {"systemTypes", m->systemTypesSet},
                    {"supportedFeatures", m->supportedFeatures},
                    {"mandatoryFeatures", m->mandatoryFeatures},
                    {"nrStepsDone", s->nrStepsDone.load()},
--- a/src/hydra-queue-runner/nar-extractor.cc
+++ b/src/hydra-queue-runner/nar-extractor.cc
@@ -6,46 +6,7 @@

 using namespace nix;

-
-struct NarMemberConstructor : CreateRegularFileSink
-{
-    NarMemberData & curMember;
-
-    HashSink hashSink = HashSink { HashAlgorithm::SHA256 };
-
-    std::optional<uint64_t> expectedSize;
-
-    NarMemberConstructor(NarMemberData & curMember)
-        : curMember(curMember)
-    { }
-
-    void isExecutable() override
-    {
-    }
-
-    void preallocateContents(uint64_t size) override
-    {
-        expectedSize = size;
-    }
-
-    void operator () (std::string_view data) override
-    {
-        assert(expectedSize);
-        *curMember.fileSize += data.size();
-        hashSink(data);
-        if (curMember.contents) {
-            curMember.contents->append(data);
-        }
-        assert(curMember.fileSize <= expectedSize);
-        if (curMember.fileSize == expectedSize) {
-            auto [hash, len] = hashSink.finish();
-            assert(curMember.fileSize == len);
-            curMember.sha256 = hash;
-        }
-    }
-};
-
-struct Extractor : FileSystemObjectSink
+struct Extractor : ParseSink
 {
    std::unordered_set<Path> filesToKeep {
        "/nix-support/hydra-build-products",
@@ -54,6 +15,7 @@ struct Extractor : FileSystemObjectSink
    };

    NarMemberDatas & members;
+    NarMemberData * curMember = nullptr;
    Path prefix;

    Extractor(NarMemberDatas & members, const Path & prefix)
@@ -65,22 +27,53 @@ struct Extractor : FileSystemObjectSink
        members.insert_or_assign(prefix + path, NarMemberData { .type = SourceAccessor::Type::tDirectory });
    }

-    void createRegularFile(const Path & path, std::function<void(CreateRegularFileSink &)> func) override
+    void createRegularFile(const Path & path) override
    {
-        NarMemberConstructor nmc {
-            members.insert_or_assign(prefix + path, NarMemberData {
-                .type = SourceAccessor::Type::tRegular,
-                .fileSize = 0,
-                .contents = filesToKeep.count(path) ? std::optional("") : std::nullopt,
-            }).first->second,
-        };
-        func(nmc);
+        curMember = &members.insert_or_assign(prefix + path, NarMemberData {
+            .type = SourceAccessor::Type::tRegular,
+            .fileSize = 0,
+            .contents = filesToKeep.count(path) ? std::optional("") : std::nullopt,
+        }).first->second;
+    }
+
+    std::optional<uint64_t> expectedSize;
+    std::unique_ptr<HashSink> hashSink;
+
+    void preallocateContents(uint64_t size) override
+    {
+        expectedSize = size;
+        hashSink = std::make_unique<HashSink>(htSHA256);
+    }
+
+    void receiveContents(std::string_view data) override
+    {
+        assert(expectedSize);
+        assert(curMember);
+        assert(hashSink);
+        *curMember->fileSize += data.size();
+        (*hashSink)(data);
+        if (curMember->contents) {
+            curMember->contents->append(data);
+        }
+        assert(curMember->fileSize <= expectedSize);
+        if (curMember->fileSize == expectedSize) {
+            auto [hash, len] = hashSink->finish();
+            assert(curMember->fileSize == len);
+            curMember->sha256 = hash;
+            hashSink.reset();
+        }
    }

    void createSymlink(const Path & path, const std::string & target) override
    {
        members.insert_or_assign(prefix + path, NarMemberData { .type = SourceAccessor::Type::tSymlink });
    }
+
+    void isExecutable() override
+    { }
+
+    void closeRegularFile() override
+    { }
 };


--- a/src/hydra-queue-runner/queue-monitor.cc
+++ b/src/hydra-queue-runner/queue-monitor.cc
@@ -1,6 +1,7 @@
 #include "state.hh"
 #include "hydra-build-result.hh"
 #include "globals.hh"
+#include "thread-pool.hh"

 #include <cstring>

@@ -37,16 +38,21 @@ void State::queueMonitorLoop(Connection & conn)

    auto destStore = getDestStore();

-    unsigned int lastBuildId = 0;
-
    bool quit = false;
    while (!quit) {
+        auto t_before_work = std::chrono::steady_clock::now();
+
        localStore->clearPathInfoCache();

-        bool done = getQueuedBuilds(conn, destStore, lastBuildId);
+        bool done = getQueuedBuilds(conn, destStore);

        if (buildOne && buildOneDone) quit = true;

+        auto t_after_work = std::chrono::steady_clock::now();
+
+        prom.queue_monitor_time_spent_running.Increment(
+            std::chrono::duration_cast<std::chrono::microseconds>(t_after_work - t_before_work).count());
+
        /* Sleep until we get notification from the database about an
           event. */
        if (done && !quit) {
@@ -56,12 +62,10 @@ void State::queueMonitorLoop(Connection & conn)
            conn.get_notifs();

        if (auto lowestId = buildsAdded.get()) {
-            lastBuildId = std::min(lastBuildId, static_cast<unsigned>(std::stoul(*lowestId) - 1));
            printMsg(lvlTalkative, "got notification: new builds added to the queue");
        }
        if (buildsRestarted.get()) {
            printMsg(lvlTalkative, "got notification: builds restarted");
-            lastBuildId = 0; // check all builds
        }
        if (buildsCancelled.get() || buildsDeleted.get() || buildsBumped.get()) {
            printMsg(lvlTalkative, "got notification: builds cancelled or bumped");
@@ -71,6 +75,10 @@ void State::queueMonitorLoop(Connection & conn)
            printMsg(lvlTalkative, "got notification: jobset shares changed");
            processJobsetSharesChange(conn);
        }
+
+        auto t_after_sleep = std::chrono::steady_clock::now();
+        prom.queue_monitor_time_spent_waiting.Increment(
+            std::chrono::duration_cast<std::chrono::microseconds>(t_after_sleep - t_after_work).count());
    }

    exit(0);
@@ -84,20 +92,18 @@ struct PreviousFailure : public std::exception {


 bool State::getQueuedBuilds(Connection & conn,
-    ref<Store> destStore, unsigned int & lastBuildId)
+    ref<Store> destStore)
 {
    prom.queue_checks_started.Increment();

-    printInfo("checking the queue for builds > %d...", lastBuildId);
+    printInfo("checking the queue for builds...");

    /* Grab the queued builds from the database, but don't process
       them yet (since we don't want a long-running transaction). */
    std::vector<BuildID> newIDs;
-    std::map<BuildID, Build::ptr> newBuildsByID;
+    std::unordered_map<BuildID, Build::ptr> newBuildsByID;
    std::multimap<StorePath, BuildID> newBuildsByPath;

-    unsigned int newLastBuildId = lastBuildId;
-
    {
        pqxx::work txn(conn);

@@ -106,17 +112,12 @@ bool State::getQueuedBuilds(Connection & conn,
             "jobsets.name as jobset, job, drvPath, maxsilent, timeout, timestamp, "
             "globalPriority, priority from Builds "
             "inner join jobsets on builds.jobset_id = jobsets.id "
-             "where builds.id > $1 and finished = 0 order by globalPriority desc, builds.id",
-            lastBuildId);
+             "where finished = 0 order by globalPriority desc, random()");

        for (auto const & row : res) {
            auto builds_(builds.lock());
            BuildID id = row["id"].as<BuildID>();
            if (buildOne && id != buildOne) continue;
-            if (id > newLastBuildId) {
-                newLastBuildId = id;
-                prom.queue_max_id.Set(id);
-            }
            if (builds_->count(id)) continue;

            auto build = std::make_shared<Build>(
@@ -298,7 +299,7 @@ bool State::getQueuedBuilds(Connection & conn,
        try {
            createBuild(build);
        } catch (Error & e) {
-            e.addTrace({}, HintFmt("while loading build %d: ", build->id));
+            e.addTrace({}, hintfmt("while loading build %d: ", build->id));
            throw;
        }

@@ -318,15 +319,13 @@ bool State::getQueuedBuilds(Connection & conn,

        /* Stop after a certain time to allow priority bumps to be
           processed. */
-        if (std::chrono::system_clock::now() > start + std::chrono::seconds(600)) {
+        if (std::chrono::system_clock::now() > start + std::chrono::seconds(60)) {
            prom.queue_checks_early_exits.Increment();
            break;
        }
    }

    prom.queue_checks_finished.Increment();
-
-    lastBuildId = newBuildsByID.empty() ? newLastBuildId : newBuildsByID.begin()->first - 1;
    return newBuildsByID.empty();
 }

@@ -405,6 +404,34 @@ void State::processQueueChange(Connection & conn)
 }


+std::map<DrvOutput, std::optional<StorePath>> State::getMissingRemotePaths(
+    ref<Store> destStore,
+    const std::map<DrvOutput, std::optional<StorePath>> & paths)
+{
+    Sync<std::map<DrvOutput, std::optional<StorePath>>> missing_;
+    ThreadPool tp;
+
+    for (auto & [output, maybeOutputPath] : paths) {
+        if (!maybeOutputPath) {
+            auto missing(missing_.lock());
+            missing->insert({output, maybeOutputPath});
+        } else {
+            tp.enqueue([&] {
+                if (!destStore->isValidPath(*maybeOutputPath)) {
+                    auto missing(missing_.lock());
+                    missing->insert({output, maybeOutputPath});
+                }
+            });
+        }
+    }
+
+    tp.process();
+
+    auto missing(missing_.lock());
+    return *missing;
+}
+
+
 Step::ptr State::createStep(ref<Store> destStore,
    Connection & conn, Build::ptr build, const StorePath & drvPath,
    Build::ptr referringBuild, Step::ptr referringStep, std::set<StorePath> & finishedDrvs,
@@ -485,16 +512,15 @@ Step::ptr State::createStep(ref<Store> destStore,

    /* Are all outputs valid? */
    auto outputHashes = staticOutputHashes(*localStore, *(step->drv));
-    bool valid = true;
-    std::map<DrvOutput, std::optional<StorePath>> missing;
+    std::map<DrvOutput, std::optional<StorePath>> paths;
    for (auto & [outputName, maybeOutputPath] : destStore->queryPartialDerivationOutputMap(drvPath, &*localStore)) {
        auto outputHash = outputHashes.at(outputName);
-        if (maybeOutputPath && destStore->isValidPath(*maybeOutputPath))
-            continue;
-        valid = false;
-        missing.insert({{outputHash, outputName}, maybeOutputPath});
+        paths.insert({{outputHash, outputName}, maybeOutputPath});
    }

+    auto missing = getMissingRemotePaths(destStore, paths);
+    bool valid = missing.empty();
+
    /* Try to copy the missing paths from the local store or from
       substitutes. */
    if (!missing.empty()) {
@@ -700,7 +726,7 @@ BuildOutput State::getBuildOutputCached(Connection & conn, nix::ref<nix::Store>
                product.fileSize = row[2].as<off_t>();
            }
            if (!row[3].is_null())
-                product.sha256hash = Hash::parseAny(row[3].as<std::string>(), HashAlgorithm::SHA256);
+                product.sha256hash = Hash::parseAny(row[3].as<std::string>(), htSHA256);
            if (!row[4].is_null())
                product.path = row[4].as<std::string>();
            product.name = row[5].as<std::string>();
--- a/src/hydra-queue-runner/state.hh
+++ b/src/hydra-queue-runner/state.hh
@@ -7,6 +7,7 @@
 #include <memory>
 #include <queue>
 #include <regex>
+#include <semaphore>

 #include <prometheus/counter.h>
 #include <prometheus/gauge.h>
@@ -22,8 +23,6 @@
 #include "sync.hh"
 #include "nar-extractor.hh"
 #include "serve-protocol.hh"
-#include "serve-protocol-impl.hh"
-#include "serve-protocol-connection.hh"
 #include "machines.hh"


@@ -58,6 +57,7 @@ typedef enum {
    ssConnecting = 10,
    ssSendingInputs = 20,
    ssBuilding = 30,
+    ssWaitingForLocalSlot = 35,
    ssReceivingOutputs = 40,
    ssPostProcessing = 50,
 } StepState;
@@ -245,6 +245,14 @@ struct Machine : nix::Machine
       we are not yet used to, but once we are, we don't need this. */
    std::string sshName;

+    /* TODO Get rid once `nix::Machine::systemTypes` is a set not
+       vector. */
+    std::set<std::string> systemTypesSet;
+
+    /* TODO Get rid once `nix::Machine::systemTypes` is a `float` not
+       an `int`. */
+    float speedFactorFloat = 1.0;
+
    struct State {
        typedef std::shared_ptr<State> ptr;
        counter currentJobs{0};
@@ -271,7 +279,7 @@ struct Machine : nix::Machine
    {
        /* Check that this machine is of the type required by the
           step. */
-        if (!systemTypes.count(step->drv->platform == "builtin" ? nix::settings.thisSystem : step->drv->platform))
+        if (!systemTypesSet.count(step->drv->platform == "builtin" ? nix::settings.thisSystem : step->drv->platform))
            return false;

        /* Check that the step requires all mandatory features of this
@@ -301,9 +309,29 @@ struct Machine : nix::Machine
    }

    // A connection to a machine
-    struct Connection : nix::ServeProto::BasicClientConnection {
+    struct Connection {
+        nix::FdSource from;
+        nix::FdSink to;
+        nix::ServeProto::Version remoteVersion;
+
        // Backpointer to the machine
        ptr machine;
+
+        operator nix::ServeProto::ReadConn ()
+        {
+            return {
+                .from = from,
+                .version = remoteVersion,
+            };
+        }
+
+        operator nix::ServeProto::WriteConn ()
+        {
+            return {
+                .to = to,
+                .version = remoteVersion,
+            };
+        }
    };
 };

@@ -361,6 +389,10 @@ private:
    typedef std::map<std::string, Machine::ptr> Machines;
    nix::Sync<Machines> machines; // FIXME: use atomic_shared_ptr

+    /* Throttler for CPU-bound local work. */
+    static constexpr unsigned int maxSupportedLocalWorkers = 1024;
+    std::counting_semaphore<maxSupportedLocalWorkers> localWorkThrottler;
+
    /* Various stats. */
    time_t startedAt;
    counter nrBuildsRead{0};
@@ -438,7 +470,7 @@ private:

    /* How often the build steps of a jobset should be repeated in
       order to detect non-determinism. */
-    std::map<std::pair<std::string, std::string>, size_t> jobsetRepeats;
+    std::map<std::pair<std::string, std::string>, unsigned int> jobsetRepeats;

    bool uploadLogsToBinaryCache;

@@ -458,7 +490,12 @@ private:
        prometheus::Counter& queue_steps_created;
        prometheus::Counter& queue_checks_early_exits;
        prometheus::Counter& queue_checks_finished;
-        prometheus::Gauge& queue_max_id;
+
+        prometheus::Counter& dispatcher_time_spent_running;
+        prometheus::Counter& dispatcher_time_spent_waiting;
+
+        prometheus::Counter& queue_monitor_time_spent_running;
+        prometheus::Counter& queue_monitor_time_spent_waiting;

        PromMetrics();
    };
@@ -467,6 +504,12 @@ private:
 public:
    State(std::optional<std::string> metricsAddrOpt);

+    struct BuildOptions {
+        unsigned int maxSilentTime, buildTimeout, repeats;
+        size_t maxLogSize;
+        bool enforceDeterminism;
+    };
+
 private:

    nix::MaintainCount<counter> startDbUpdate();
@@ -502,8 +545,7 @@ private:
    void queueMonitorLoop(Connection & conn);

    /* Check the queue for new builds. */
-    bool getQueuedBuilds(Connection & conn,
-        nix::ref<nix::Store> destStore, unsigned int & lastBuildId);
+    bool getQueuedBuilds(Connection & conn, nix::ref<nix::Store> destStore);

    /* Handle cancellation, deletion and priority bumps. */
    void processQueueChange(Connection & conn);
@@ -511,6 +553,12 @@ private:
    BuildOutput getBuildOutputCached(Connection & conn, nix::ref<nix::Store> destStore,
        const nix::StorePath & drvPath);

+    /* Returns paths missing from the remote store. Paths are processed in
+     * parallel to work around the possible latency of remote stores. */
+    std::map<nix::DrvOutput, std::optional<nix::StorePath>> getMissingRemotePaths(
+        nix::ref<nix::Store> destStore,
+        const std::map<nix::DrvOutput, std::optional<nix::StorePath>> & paths);
+
    Step::ptr createStep(nix::ref<nix::Store> store,
        Connection & conn, Build::ptr build, const nix::StorePath & drvPath,
        Build::ptr referringBuild, Step::ptr referringStep, std::set<nix::StorePath> & finishedDrvs,
@@ -546,12 +594,13 @@ private:
       retried. */
    enum StepResult { sDone, sRetry, sMaybeCancelled };
    StepResult doBuildStep(nix::ref<nix::Store> destStore,
-        MachineReservation::ptr reservation,
+        MachineReservation::ptr & reservation,
        std::shared_ptr<ActiveStep> activeStep);

    void buildRemote(nix::ref<nix::Store> destStore,
+        MachineReservation::ptr & reservation,
        Machine::ptr machine, Step::ptr step,
-        const nix::ServeProto::BuildOptions & buildOptions,
+        const BuildOptions & buildOptions,
        RemoteResult & result, std::shared_ptr<ActiveStep> activeStep,
        std::function<void(StepState)> updateStep,
        NarMemberDatas & narMembers);
--- a/src/lib/Hydra/Base/Controller/NixChannel.pm
+++ b/src/lib/Hydra/Base/Controller/NixChannel.pm
@@ -4,6 +4,7 @@ use strict;
 use warnings;
 use base 'Hydra::Base::Controller::REST';
 use List::SomeUtils qw(any);
+use Nix::Store;
 use Hydra::Helper::Nix;
 use Hydra::Helper::CatalystUtils;

@@ -29,7 +30,7 @@ sub getChannelData {
        my $outputs = {};
        foreach my $output (@outputs) {
            my $outPath = $output->get_column("outpath");
-            next if $checkValidity && !$MACHINE_LOCAL_STORE->isValidPath($outPath);
+            next if $checkValidity && !isValidPath($outPath);
            $outputs->{$output->get_column("outname")} = $outPath;
            push @storePaths, $outPath;
            # Put the system type in the manifest (for top-level
--- a/src/lib/Hydra/Controller/API.pm
+++ b/src/lib/Hydra/Controller/API.pm
@@ -292,23 +292,6 @@ sub push_github : Chained('api') PathPart('push-github') Args(0) {
    $c->response->body("");
 }

-sub push_gitea : Chained('api') PathPart('push-gitea') Args(0) {
-    my ($self, $c) = @_;
-
-    $c->{stash}->{json}->{jobsetsTriggered} = [];
-
-    my $in = $c->request->{data};
-    my $url = $in->{repository}->{clone_url} or die;
-    $url =~ s/.git$//;
-    print STDERR "got push from Gitea repository $url\n";
-
-    triggerJobset($self, $c, $_, 0) foreach $c->model('DB::Jobsets')->search(
-        { 'project.enabled' => 1, 'me.enabled' => 1 },
-        { join => 'project'
-        , where => \ [ 'me.flake like ? or exists (select 1 from JobsetInputAlts where project = me.project and jobset = me.name and value like ?)', [ 'flake', "%$url%"], [ 'value', "%$url%" ] ]
-        });
-    $c->response->body("");
-}


 1;
--- a/src/lib/Hydra/Controller/Build.pm
+++ b/src/lib/Hydra/Controller/Build.pm
@@ -10,6 +10,8 @@ use File::Basename;
 use File::LibMagic;
 use File::stat;
 use Data::Dump qw(dump);
+use Nix::Store;
+use Nix::Config;
 use List::SomeUtils qw(all);
 use Encode;
 use JSON::PP;
@@ -81,9 +83,9 @@ sub build_GET {
    # false because `$_->path` will be empty
    $c->stash->{available} =
        $c->stash->{isLocalStore}
-        ? all { $_->path && $MACHINE_LOCAL_STORE->isValidPath($_->path) } $build->buildoutputs->all
+        ? all { $_->path && isValidPath($_->path) } $build->buildoutputs->all
        : 1;
-    $c->stash->{drvAvailable} = $MACHINE_LOCAL_STORE->isValidPath($build->drvpath);
+    $c->stash->{drvAvailable} = isValidPath $build->drvpath;

    if ($build->finished && $build->iscachedbuild) {
        my $path = ($build->buildoutputs)[0]->path or undef;
@@ -310,7 +312,7 @@ sub output : Chained('buildChain') PathPart Args(1) {
    error($c, "This build is not finished yet.") unless $build->finished;
    my $output = $build->buildoutputs->find({name => $outputName});
    notFound($c, "This build has no output named ‘$outputName’") unless defined $output;
-    gone($c, "Output is no longer available.") unless $MACHINE_LOCAL_STORE->isValidPath($output->path);
+    gone($c, "Output is no longer available.") unless isValidPath $output->path;

    $c->response->header('Content-Disposition', "attachment; filename=\"build-${\$build->id}-${\$outputName}.nar.bz2\"");
    $c->stash->{current_view} = 'NixNAR';
@@ -427,7 +429,7 @@ sub getDependencyGraph {
            };
        $$done{$path} = $node;
        my @refs;
-        foreach my $ref ($MACHINE_LOCAL_STORE->queryReferences($path)) {
+        foreach my $ref (queryReferences($path)) {
            next if $ref eq $path;
            next unless $runtime || $ref =~ /\.drv$/;
            getDependencyGraph($self, $c, $runtime, $done, $ref);
@@ -435,7 +437,7 @@ sub getDependencyGraph {
        }
        # Show in reverse topological order to flatten the graph.
        # Should probably do a proper BFS.
-        my @sorted = reverse $MACHINE_LOCAL_STORE->topoSortPaths(@refs);
+        my @sorted = reverse topoSortPaths(@refs);
        $node->{refs} = [map { $$done{$_} } @sorted];
    }

@@ -448,7 +450,7 @@ sub build_deps : Chained('buildChain') PathPart('build-deps') {
    my $build = $c->stash->{build};
    my $drvPath = $build->drvpath;

-    error($c, "Derivation no longer available.") unless $MACHINE_LOCAL_STORE->isValidPath($drvPath);
+    error($c, "Derivation no longer available.") unless isValidPath $drvPath;

    $c->stash->{buildTimeGraph} = getDependencyGraph($self, $c, 0, {}, $drvPath);

@@ -463,7 +465,7 @@ sub runtime_deps : Chained('buildChain') PathPart('runtime-deps') {

    requireLocalStore($c);

-    error($c, "Build outputs no longer available.") unless all { $MACHINE_LOCAL_STORE->isValidPath($_) } @outPaths;
+    error($c, "Build outputs no longer available.") unless all { isValidPath($_) } @outPaths;

    my $done = {};
    $c->stash->{runtimeGraph} = [ map { getDependencyGraph($self, $c, 1, $done, $_) } @outPaths ];
@@ -483,7 +485,7 @@ sub nix : Chained('buildChain') PathPart('nix') CaptureArgs(0) {
    if (isLocalStore) {
        foreach my $out ($build->buildoutputs) {
            notFound($c, "Path " . $out->path . " is no longer available.")
-                unless $MACHINE_LOCAL_STORE->isValidPath($out->path);
+                unless isValidPath($out->path);
        }
    }

--- a/src/lib/Hydra/Controller/Jobset.pm
+++ b/src/lib/Hydra/Controller/Jobset.pm
@@ -364,6 +364,21 @@ sub evals_GET {
    );
 }

+sub errors :Chained('jobsetChain') :PathPart('errors') :Args(0) :ActionClass('REST') { }
+
+sub errors_GET {
+    my ($self, $c) = @_;
+
+    $c->stash->{template} = 'eval-error.tt';
+
+    my $jobsetName = $c->stash->{params}->{name};
+    $c->stash->{jobset} = $c->stash->{project}->jobsets->find(
+        { name => $jobsetName },
+        { '+columns' => { 'errormsg' => 'errormsg' } }
+    );
+
+    $self->status_ok($c, entity => $c->stash->{jobset});
+}

 # Redirect to the latest finished evaluation of this jobset.
 sub latest_eval : Chained('jobsetChain') PathPart('latest-eval') {
--- a/src/lib/Hydra/Controller/JobsetEval.pm
+++ b/src/lib/Hydra/Controller/JobsetEval.pm
@@ -86,6 +86,17 @@ sub view_GET {
    );
 }

+sub errors :Chained('evalChain') :PathPart('errors') :Args(0) :ActionClass('REST') { }
+
+sub errors_GET {
+    my ($self, $c) = @_;
+
+    $c->stash->{template} = 'eval-error.tt';
+
+    $c->stash->{eval} = $c->model('DB::JobsetEvals')->find($c->stash->{eval}->id, { prefetch => 'evaluationerror' });
+
+    $self->status_ok($c, entity => $c->stash->{eval});
+}

 sub create_jobset : Chained('evalChain') PathPart('create-jobset') Args(0) {
    my ($self, $c) = @_;
--- a/src/lib/Hydra/Controller/Root.pm
+++ b/src/lib/Hydra/Controller/Root.pm
@@ -35,7 +35,6 @@ sub noLoginNeeded {

  return $whitelisted ||
         $c->request->path eq "api/push-github" ||
-         $c->request->path eq "api/push-gitea" ||
         $c->request->path eq "google-login" ||
         $c->request->path eq "github-redirect" ||
         $c->request->path eq "github-login" ||
@@ -81,7 +80,7 @@ sub begin :Private {
    $_->supportedInputTypes($c->stash->{inputTypes}) foreach @{$c->hydra_plugins};

    # XSRF protection: require POST requests to have the same origin.
-    if ($c->req->method eq "POST" && $c->req->path ne "api/push-github" && $c->req->path ne "api/push-gitea") {
+    if ($c->req->method eq "POST" && $c->req->path ne "api/push-github") {
        my $referer = $c->req->header('Referer');
        $referer //= $c->req->header('Origin');
        my $base = $c->req->base;
@@ -161,7 +160,7 @@ sub status_GET {
            { "buildsteps.busy" => { '!=', 0 } },
            { order_by => ["globalpriority DESC", "id"],
              join => "buildsteps",
-              columns => [@buildListColumns]
+              columns => [@buildListColumns, 'buildsteps.drvpath', 'buildsteps.type']
            })]
    );
 }
@@ -330,7 +329,7 @@ sub nar :Local :Args(1) {
    else {
        $path = $Nix::Config::storeDir . "/$path";

-        gone($c, "Path " . $path . " is no longer available.") unless $MACHINE_LOCAL_STORE->isValidPath($path);
+        gone($c, "Path " . $path . " is no longer available.") unless isValidPath($path);

        $c->stash->{current_view} = 'NixNAR';
        $c->stash->{storePath} = $path;
@@ -368,7 +367,7 @@ sub realisations :Path('realisations') :Args(StrMatch[REALISATIONS_REGEX]) {

    else {
        my ($rawDrvOutput) = $realisation =~ REALISATIONS_REGEX;
-        my $rawRealisation = $MACHINE_LOCAL_STORE->queryRawRealisation($rawDrvOutput);
+        my $rawRealisation = queryRawRealisation($rawDrvOutput);

        if (!$rawRealisation) {
            $c->response->status(404);
@@ -397,7 +396,7 @@ sub narinfo :Path :Args(StrMatch[NARINFO_REGEX]) {
        my ($hash) = $narinfo =~ NARINFO_REGEX;

        die("Hash length was not 32") if length($hash) != 32;
-        my $path = $MACHINE_LOCAL_STORE->queryPathFromHashPart($hash);
+        my $path = queryPathFromHashPart($hash);

        if (!$path) {
            $c->response->status(404);
--- a/src/lib/Hydra/Helper/BuildDiff.pm
+++ b/src/lib/Hydra/Helper/BuildDiff.pm
@@ -37,7 +37,16 @@ sub buildDiff {

    my $n = 0;
    foreach my $build (@{$builds}) {
-        my $aborted = $build->finished != 0 && ($build->buildstatus == 3 || $build->buildstatus == 4);
+        my $aborted = $build->finished != 0 && (
+            # aborted
+            $build->buildstatus == 3
+            # cancelled
+            || $build->buildstatus == 4
+            # timeout
+            || $build->buildstatus == 7
+            # log limit exceeded
+            || $build->buildstatus == 10
+        );
        my $d;
        my $found = 0;
        while ($n < scalar(@{$builds2})) {
@@ -79,4 +88,4 @@ sub buildDiff {
    return $ret;
 }

-1;
+1;
--- a/src/lib/Hydra/Helper/Nix.pm
+++ b/src/lib/Hydra/Helper/Nix.pm
@@ -40,11 +40,8 @@ our @EXPORT = qw(
    registerRoot
    restartBuilds
    run
-    $MACHINE_LOCAL_STORE
    );

-our $MACHINE_LOCAL_STORE = Nix::Store->new();
-

 sub getHydraHome {
    my $dir = $ENV{"HYDRA_HOME"} or die "The HYDRA_HOME directory does not exist!\n";
@@ -174,9 +171,6 @@ sub getDrvLogPath {
    for ($fn . $bucketed, $fn . $bucketed . ".bz2") {
        return $_ if -f $_;
    }
-    for ($fn . $bucketed, $fn . $bucketed . ".zst") {
-        return $_ if -f $_;
-    }
    return undef;
 }

@@ -193,10 +187,6 @@ sub findLog {

    return undef if scalar @outPaths == 0;

-    # Filter out any NULLs. Content-addressed derivations
-    # that haven't built yet or failed to build may have a NULL outPath.
-    @outPaths = grep {defined} @outPaths;
-
    my @steps = $c->model('DB::BuildSteps')->search(
        { path => { -in => [@outPaths] } },
        { select => ["drvpath"]
@@ -296,8 +286,7 @@ sub getEvals {

    my @evals = $evals_result_set->search(
        { hasnewbuilds => 1 },
-        { order_by => "$me.id DESC", rows => $rows, offset => $offset
-        , prefetch => { evaluationerror => [ ] } });
+        { order_by => "$me.id DESC", rows => $rows, offset => $offset });
    my @res = ();
    my $cache = {};

@@ -504,7 +493,7 @@ sub restartBuilds {
    $builds = $builds->search({ finished => 1 });

    foreach my $build ($builds->search({}, { columns => ["drvpath"] })) {
-        next if !$MACHINE_LOCAL_STORE->isValidPath($build->drvpath);
+        next if !isValidPath($build->drvpath);
        registerRoot $build->drvpath;
    }

--- a/src/lib/Hydra/Plugin/BazaarInput.pm
+++ b/src/lib/Hydra/Plugin/BazaarInput.pm
@@ -7,6 +7,7 @@ use Digest::SHA qw(sha256_hex);
 use File::Path;
 use Hydra::Helper::Exec;
 use Hydra::Helper::Nix;
+use Nix::Store;

 sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
@@ -37,9 +38,9 @@ sub fetchInput {
    (my $cachedInput) = $self->{db}->resultset('CachedBazaarInputs')->search(
        {uri => $uri, revision => $revision});

-    $MACHINE_LOCAL_STORE->addTempRoot($cachedInput->storepath) if defined $cachedInput;
+    addTempRoot($cachedInput->storepath) if defined $cachedInput;

-    if (defined $cachedInput && $MACHINE_LOCAL_STORE->isValidPath($cachedInput->storepath)) {
+    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
@@ -57,7 +58,7 @@ sub fetchInput {
        ($sha256, $storePath) = split ' ', $stdout;

        # FIXME: time window between nix-prefetch-bzr and addTempRoot.
-        $MACHINE_LOCAL_STORE->addTempRoot($storePath);
+        addTempRoot($storePath);

        $self->{db}->txn_do(sub {
            $self->{db}->resultset('CachedBazaarInputs')->create(
--- a/src/lib/Hydra/Plugin/CompressLog.pm
+++ b/src/lib/Hydra/Plugin/CompressLog.pm
@@ -9,24 +9,11 @@ use Hydra::Helper::CatalystUtils;
 sub stepFinished {
    my ($self, $step, $logPath) = @_;

-    my $doCompress = $self->{config}->{'compress_build_logs'} // '1';
-    my $silent = $self->{config}->{'compress_build_logs_silent'} // '0';
-    my $compression = $self->{config}->{'compress_build_logs_compression'} // 'bzip2';
+    my $doCompress = $self->{config}->{'compress_build_logs'} // "1";

-    if (not -e $logPath or $doCompress ne "1") {
-        return;
-    }
-
-    if ($silent ne '1') {
-        print STDERR "compressing '$logPath' with $compression...\n";
-    }
-
-    if ($compression eq 'bzip2') {
-        system('bzip2', '--force', $logPath);
-    } elsif ($compression eq 'zstd') {
-        system('zstd', '--rm', '--quiet', '-T0', $logPath);
-    } else {
-        print STDERR "unknown compression type '$compression'\n";
+    if ($doCompress eq "1" && -e $logPath) {
+        print STDERR "compressing ‘$logPath’...\n";
+        system("bzip2", "--force", $logPath);
    }
 }

--- a/src/lib/Hydra/Plugin/DarcsInput.pm
+++ b/src/lib/Hydra/Plugin/DarcsInput.pm
@@ -7,6 +7,7 @@ use Digest::SHA qw(sha256_hex);
 use File::Path;
 use Hydra::Helper::Exec;
 use Hydra::Helper::Nix;
+use Nix::Store;

 sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
@@ -57,7 +58,7 @@ sub fetchInput {
        {uri => $uri, revision => $revision},
        {rows => 1});

-    if (defined $cachedInput && $MACHINE_LOCAL_STORE->isValidPath($cachedInput->storepath)) {
+    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
        $revision = $cachedInput->revision;
@@ -74,8 +75,8 @@ sub fetchInput {
        die "darcs changes --count failed" if $? != 0;

        system "rm", "-rf", "$tmpDir/export/_darcs";
-        $storePath = $MACHINE_LOCAL_STORE->addToStore("$tmpDir/export", 1, "sha256");
-        $sha256 = $MACHINE_LOCAL_STORE->queryPathHash($storePath);
+        $storePath = addToStore("$tmpDir/export", 1, "sha256");
+        $sha256 = queryPathHash($storePath);
        $sha256 =~ s/sha256://;

        $self->{db}->txn_do(sub {
--- a/src/lib/Hydra/Plugin/GitInput.pm
+++ b/src/lib/Hydra/Plugin/GitInput.pm
@@ -186,9 +186,9 @@ sub fetchInput {
        {uri => $uri, branch => $branch, revision => $revision, isdeepclone => defined($deepClone) ? 1 : 0},
        {rows => 1});

-    $MACHINE_LOCAL_STORE->addTempRoot($cachedInput->storepath) if defined $cachedInput;
+    addTempRoot($cachedInput->storepath) if defined $cachedInput;

-    if (defined $cachedInput && $MACHINE_LOCAL_STORE->isValidPath($cachedInput->storepath)) {
+    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
        $revision = $cachedInput->revision;
@@ -217,7 +217,7 @@ sub fetchInput {
        ($sha256, $storePath) = split ' ', grab(cmd => ["nix-prefetch-git", $clonePath, $revision], chomp => 1);

        # FIXME: time window between nix-prefetch-git and addTempRoot.
-        $MACHINE_LOCAL_STORE->addTempRoot($storePath);
+        addTempRoot($storePath);

        $self->{db}->txn_do(sub {
            $self->{db}->resultset('CachedGitInputs')->update_or_create(
--- a/src/lib/Hydra/Plugin/GiteaStatus.pm
+++ b/src/lib/Hydra/Plugin/GiteaStatus.pm
@@ -88,6 +88,10 @@ sub buildQueued {
    common(@_, [], 0);
 }

+sub buildStarted {
+    common(@_, [], 1);
+}
+
 sub buildFinished {
    common(@_, 2);
 }
--- a/src/lib/Hydra/Plugin/MercurialInput.pm
+++ b/src/lib/Hydra/Plugin/MercurialInput.pm
@@ -7,6 +7,7 @@ use Digest::SHA qw(sha256_hex);
 use File::Path;
 use Hydra::Helper::Nix;
 use Hydra::Helper::Exec;
+use Nix::Store;
 use Fcntl qw(:flock);

 sub supportedInputTypes {
@@ -67,9 +68,9 @@ sub fetchInput {
    (my $cachedInput) = $self->{db}->resultset('CachedHgInputs')->search(
        {uri => $uri, branch => $branch, revision => $revision});

-    $MACHINE_LOCAL_STORE->addTempRoot($cachedInput->storepath) if defined $cachedInput;
+    addTempRoot($cachedInput->storepath) if defined $cachedInput;

-    if (defined $cachedInput && $MACHINE_LOCAL_STORE->isValidPath($cachedInput->storepath)) {
+    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
@@ -84,7 +85,7 @@ sub fetchInput {
        ($sha256, $storePath) = split ' ', $stdout;

        # FIXME: time window between nix-prefetch-hg and addTempRoot.
-        $MACHINE_LOCAL_STORE->addTempRoot($storePath);
+        addTempRoot($storePath);

        $self->{db}->txn_do(sub {
            $self->{db}->resultset('CachedHgInputs')->update_or_create(
--- a/src/lib/Hydra/Plugin/PathInput.pm
+++ b/src/lib/Hydra/Plugin/PathInput.pm
@@ -5,6 +5,7 @@ use warnings;
 use parent 'Hydra::Plugin';
 use POSIX qw(strftime);
 use Hydra::Helper::Nix;
+use Nix::Store;

 sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
@@ -29,7 +30,7 @@ sub fetchInput {
        {srcpath => $uri, lastseen => {">", $timestamp - $timeout}},
        {rows => 1, order_by => "lastseen DESC"});

-    if (defined $cachedInput && $MACHINE_LOCAL_STORE->isValidPath($cachedInput->storepath)) {
+    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
        $timestamp = $cachedInput->timestamp;
@@ -45,7 +46,7 @@ sub fetchInput {
        }
        chomp $storePath;

-        $sha256 = ($MACHINE_LOCAL_STORE->queryPathInfo($storePath, 0))[1] or die;
+        $sha256 = (queryPathInfo($storePath, 0))[1] or die;

        ($cachedInput) = $self->{db}->resultset('CachedPathInputs')->search(
            {srcpath => $uri, sha256hash => $sha256});
--- a/src/lib/Hydra/Plugin/S3Backup.pm
+++ b/src/lib/Hydra/Plugin/S3Backup.pm
@@ -92,7 +92,7 @@ sub buildFinished {
        my $hash = substr basename($path), 0, 32;
        my ($deriver, $narHash, $time, $narSize, $refs) = queryPathInfo($path, 0);
        my $system;
-        if (defined $deriver and $MACHINE_LOCAL_STORE->isValidPath($deriver)) {
+        if (defined $deriver and isValidPath($deriver)) {
            $system = derivationFromPath($deriver)->{platform};
        }
        foreach my $reference (@{$refs}) {
--- a/src/lib/Hydra/Plugin/SubversionInput.pm
+++ b/src/lib/Hydra/Plugin/SubversionInput.pm
@@ -7,6 +7,7 @@ use Digest::SHA qw(sha256_hex);
 use Hydra::Helper::Exec;
 use Hydra::Helper::Nix;
 use IPC::Run;
+use Nix::Store;

 sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
@@ -44,9 +45,9 @@ sub fetchInput {
    (my $cachedInput) = $self->{db}->resultset('CachedSubversionInputs')->search(
        {uri => $uri, revision => $revision});

-    $MACHINE_LOCAL_STORE->addTempRoot($cachedInput->storepath) if defined $cachedInput;
+    addTempRoot($cachedInput->storepath) if defined $cachedInput;

-    if (defined $cachedInput && $MACHINE_LOCAL_STORE->isValidPath($cachedInput->storepath)) {
+    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
@@ -61,16 +62,16 @@ sub fetchInput {
        die "error checking out Subversion repo at `$uri':\n$stderr" if $res;

        if ($type eq "svn-checkout") {
-            $storePath = $MACHINE_LOCAL_STORE->addToStore($wcPath, 1, "sha256");
+            $storePath = addToStore($wcPath, 1, "sha256");
        } else {
            # Hm, if the Nix Perl bindings supported filters in
            # addToStore(), then we wouldn't need to make a copy here.
            my $tmpDir = File::Temp->newdir("hydra-svn-export.XXXXXX", CLEANUP => 1, TMPDIR => 1) or die;
            (system "svn", "export", $wcPath, "$tmpDir/source", "--quiet") == 0 or die "svn export failed";
-            $storePath = $MACHINE_LOCAL_STORE->addToStore("$tmpDir/source", 1, "sha256");
+            $storePath = addToStore("$tmpDir/source", 1, "sha256");
        }

-        $sha256 = $MACHINE_LOCAL_STORE->queryPathHash($storePath); $sha256 =~ s/sha256://;
+        $sha256 = queryPathHash($storePath); $sha256 =~ s/sha256://;

        $self->{db}->txn_do(sub {
            $self->{db}->resultset('CachedSubversionInputs')->update_or_create(
--- a/src/lib/Hydra/Schema/Result/EvaluationErrors.pm
+++ b/src/lib/Hydra/Schema/Result/EvaluationErrors.pm
@@ -105,4 +105,6 @@ __PACKAGE__->add_column(
    "+id" => { retrieve_on_insert => 1 }
 );

+__PACKAGE__->mk_group_accessors('column' => 'has_error');
+
 1;
--- a/src/lib/Hydra/Schema/Result/Jobsets.pm
+++ b/src/lib/Hydra/Schema/Result/Jobsets.pm
@@ -386,6 +386,8 @@ __PACKAGE__->add_column(
    "+id" => { retrieve_on_insert => 1 }
 );

+__PACKAGE__->mk_group_accessors('column' => 'has_error');
+
 sub supportsDynamicRunCommand {
  my ($self) = @_;

--- a/src/lib/Hydra/Schema/ResultSet/EvaluationErrors.pm
+++ b/src/lib/Hydra/Schema/ResultSet/EvaluationErrors.pm
@@ -0,0 +1,30 @@
+package Hydra::Schema::ResultSet::EvaluationErrors;
+
+use strict;
+use utf8;
+use warnings;
+
+use parent 'DBIx::Class::ResultSet';
+
+use Storable qw(dclone);
+
+__PACKAGE__->load_components('Helper::ResultSet::RemoveColumns');
+
+# Exclude expensive error message values unless explicitly requested, and
+# replace them with a summary field describing their presence/absence.
+sub search_rs {
+  my ( $class, $query, $attrs ) = @_;
+
+  if ($attrs) {
+    $attrs = dclone($attrs);
+  }
+
+  unless (exists $attrs->{'select'} || exists $attrs->{'columns'}) {
+    $attrs->{'+columns'}->{'has_error'} = "errormsg != ''";
+  }
+  unless (exists $attrs->{'+columns'}->{'errormsg'}) {
+    push @{ $attrs->{'remove_columns'} }, 'errormsg';
+  }
+
+  return $class->next::method($query, $attrs);
+}
--- a/src/lib/Hydra/Schema/ResultSet/Jobsets.pm
+++ b/src/lib/Hydra/Schema/ResultSet/Jobsets.pm
@@ -0,0 +1,30 @@
+package Hydra::Schema::ResultSet::Jobsets;
+
+use strict;
+use utf8;
+use warnings;
+
+use parent 'DBIx::Class::ResultSet';
+
+use Storable qw(dclone);
+
+__PACKAGE__->load_components('Helper::ResultSet::RemoveColumns');
+
+# Exclude expensive error message values unless explicitly requested, and
+# replace them with a summary field describing their presence/absence.
+sub search_rs {
+  my ( $class, $query, $attrs ) = @_;
+
+  if ($attrs) {
+    $attrs = dclone($attrs);
+  }
+
+  unless (exists $attrs->{'select'} || exists $attrs->{'columns'}) {
+    $attrs->{'+columns'}->{'has_error'} = "errormsg != ''";
+  }
+  unless (exists $attrs->{'+columns'}->{'errormsg'}) {
+    push @{ $attrs->{'remove_columns'} }, 'errormsg';
+  }
+
+  return $class->next::method($query, $attrs);
+}
--- a/src/lib/Hydra/View/NARInfo.pm
+++ b/src/lib/Hydra/View/NARInfo.pm
@@ -8,7 +8,6 @@ use MIME::Base64;
 use Nix::Manifest;
 use Nix::Store;
 use Nix::Utils;
-use Hydra::Helper::Nix;
 use base qw/Catalyst::View/;

 sub process {
@@ -18,7 +17,7 @@ sub process {

    $c->response->content_type('text/x-nix-narinfo'); # !!! check MIME type

-    my ($deriver, $narHash, $time, $narSize, $refs) = $MACHINE_LOCAL_STORE->queryPathInfo($storePath, 1);
+    my ($deriver, $narHash, $time, $narSize, $refs) = queryPathInfo($storePath, 1);

    my $info;
    $info .= "StorePath: $storePath\n";
@@ -29,8 +28,8 @@ sub process {
    $info .= "References: " . join(" ", map { basename $_ } @{$refs}) . "\n";
    if (defined $deriver) {
        $info .= "Deriver: " . basename $deriver . "\n";
-        if ($MACHINE_LOCAL_STORE->isValidPath($deriver)) {
-            my $drv = $MACHINE_LOCAL_STORE->derivationFromPath($deriver);
+        if (isValidPath($deriver)) {
+            my $drv = derivationFromPath($deriver);
            $info .= "System: $drv->{platform}\n";
        }
    }
--- a/src/lib/Hydra/View/NixLog.pm
+++ b/src/lib/Hydra/View/NixLog.pm
@@ -16,10 +16,7 @@ sub process {

    my $tail = int($c->stash->{tail} // "0");

-    if ($logPath =~ /\.zst$/) {
-        my $doTail = $tail ? "| tail -n '$tail'" : "";
-        open($fh, "-|", "zstd -dc < '$logPath' $doTail") or die;
-    } elsif ($logPath =~ /\.bz2$/) {
+    if ($logPath =~ /\.bz2$/) {
        my $doTail = $tail ? "| tail -n '$tail'" : "";
        open($fh, "-|", "bzip2 -dc < '$logPath' $doTail") or die;
    } else {
--- a/src/root/build.tt
+++ b/src/root/build.tt
@@ -61,21 +61,7 @@ END;
            <td>[% IF step.busy != 0 || ((step.machine || step.starttime) && (step.status == 0 || step.status == 1 || step.status == 3 || step.status == 4 || step.status == 7)); INCLUDE renderMachineName machine=step.machine; ELSE; "<em>n/a</em>"; END %]</td>
            <td class="step-status">
              [% IF step.busy != 0 %]
-                [% IF step.busy == 1 %]
-                  <strong>Preparing</strong>
-                [% ELSIF step.busy == 10 %]
-                  <strong>Connecting</strong>
-                [% ELSIF step.busy == 20 %]
-                  <strong>Sending inputs</strong>
-                [% ELSIF step.busy == 30 %]
-                  <strong>Building</strong>
-                [% ELSIF step.busy == 40 %]
-                  <strong>Receiving outputs</strong>
-                [% ELSIF step.busy == 50 %]
-                  <strong>Post-processing</strong>
-                [% ELSE %]
-                  <strong>Unknown state</strong>
-                [% END %]
+                [% INCLUDE renderBusyStatus %]
              [% ELSIF step.status == 0 %]
                [% IF step.isnondeterministic %]
                  <span class="warn">Succeeded with non-determistic result</span>
--- a/src/root/common.tt
+++ b/src/root/common.tt
@@ -91,6 +91,17 @@ BLOCK renderDuration;
  duration % 60 %]s[%
 END;

+BLOCK renderDrvInfo;
+  drvname = step.drvpath
+    .substr(11) # strip `/nix/store/`
+    .split('-').slice(1).join("-") # strip hash part
+    .substr(0, -4); # strip `.drv`
+  IF drvname != releasename;
+    IF step.type == 0; action = "Build"; ELSE; action = "Substitution"; END;
+    IF drvname; %]<em> ([% action %] of [% drvname %])</em>[% END;
+  END;
+END;
+

 BLOCK renderBuildListHeader %]
  <table class="table table-striped table-condensed clickable-rows">
@@ -131,7 +142,12 @@ BLOCK renderBuildListBody;
      [% END %]
      <td><a class="row-link" href="[% link %]">[% build.id %]</a></td>
      [% IF !hideJobName %]
-        <td><a href="[%link%]">[% IF !hideJobsetName %][%build.jobset.get_column("project")%]:[%build.jobset.get_column("name")%]:[% END %][%build.get_column("job")%]</td>
+        <td>
+          <a href="[%link%]">[% IF !hideJobsetName %][%build.jobset.get_column("project")%]:[%build.jobset.get_column("name")%]:[% END %][%build.get_column("job")%]</a>
+          [% IF showStepName %]
+            [% INCLUDE renderDrvInfo step=build.buildsteps releasename=build.nixname %]
+          [% END %]
+        </td>
      [% END %]
      <td class="nowrap">[% t = showSchedulingInfo ? build.timestamp : build.stoptime; IF t; INCLUDE renderRelativeDate timestamp=(showSchedulingInfo ? build.timestamp : build.stoptime); ELSE; "-"; END %]</td>
      <td>[% !showSchedulingInfo and build.get_column('releasename') ? build.get_column('releasename') : build.nixname %]</td>
@@ -245,6 +261,27 @@ BLOCK renderBuildStatusIcon;
 END;


+BLOCK renderBusyStatus;
+  IF step.busy == 1 %]
+    <strong>Preparing</strong>
+  [% ELSIF step.busy == 10 %]
+    <strong>Connecting</strong>
+  [% ELSIF step.busy == 20 %]
+    <strong>Sending inputs</strong>
+  [% ELSIF step.busy == 30 %]
+    <strong>Building</strong>
+  [% ELSIF step.busy == 35 %]
+    <strong>Waiting to receive outputs</strong>
+  [% ELSIF step.busy == 40 %]
+    <strong>Receiving outputs</strong>
+  [% ELSIF step.busy == 50 %]
+    <strong>Post-processing</strong>
+  [% ELSE %]
+    <strong>Unknown state</strong>
+  [% END;
+END;
+
+
 BLOCK renderStatus;
  IF build.finished;
    buildstatus = build.buildstatus;
@@ -374,7 +411,7 @@ BLOCK renderInputDiff; %]
              [% ELSIF bi1.uri == bi2.uri && bi1.revision != bi2.revision %]
                [% IF bi1.type == "git" %]
                  <tr><td>
-                    <b>[% bi1.name %]</b></td><td><tt>[% INCLUDE renderDiffUri contents=(bi1.revision.substr(0, 12) _ ' to ' _ bi2.revision.substr(0, 12)) %]</tt>
+                    <b>[% bi1.name %]</b></td><td><tt>[% INCLUDE renderDiffUri contents=(bi1.revision.substr(0, 8) _ ' to ' _ bi2.revision.substr(0, 8)) %]</tt>
                  </td></tr>
                [% ELSE %]
                  <tr><td>
@@ -476,7 +513,7 @@ BLOCK renderEvals %]
            ELSE %]
             -
            [% END %]
-            [% IF eval.evaluationerror.errormsg %]
+            [% IF eval.evaluationerror.has_error %]
              <span class="badge badge-warning">Eval Errors</span>
            [% END %]
          </td>
@@ -602,7 +639,7 @@ BLOCK renderJobsetOverview %]
        <td>[% HTML.escape(j.description) %]</td>
        <td>[% IF j.lastcheckedtime;
                 INCLUDE renderDateTime timestamp = j.lastcheckedtime;
-                 IF j.errormsg || j.fetcherrormsg; %]&nbsp;<span class = 'badge badge-warning'>Error</span>[% END;
+                 IF j.has_error || j.fetcherrormsg; %]&nbsp;<span class = 'badge badge-warning'>Error</span>[% END;
                 ELSE; "-";
               END %]</td>
        [% IF j.get_column('nrtotal') > 0 %]
--- a/src/root/eval-error.tt
+++ b/src/root/eval-error.tt
@@ -0,0 +1,26 @@
+[% PROCESS common.tt %]
+<!DOCTYPE html>
+
+<html lang="en">
+
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <meta http-equiv="X-UA-Compatible" content="IE=Edge" />
+    [% INCLUDE style.tt %]
+  </head>
+
+  <body>
+
+    <div class="tab-content tab-pane">
+        <div id="tabs-errors" class="">
+          [% IF jobset %]
+              <p>Errors occurred at [% INCLUDE renderDateTime timestamp=(jobset.errortime || jobset.lastcheckedtime) %].</p>
+              <div class="card bg-light"><div class="card-body"><pre>[% HTML.escape(jobset.fetcherrormsg || jobset.errormsg) %]</pre></div></div>
+          [% ELSIF eval %]
+              <p>Errors occurred at [% INCLUDE renderDateTime timestamp=(eval.evaluationerror.errortime || eval.timestamp) %].</p>
+              <div class="card bg-light"><div class="card-body"><pre>[% HTML.escape(eval.evaluationerror.errormsg) %]</pre></div></div>
+          [% END %]
+        </div>
+    </div>
+  </body>
+</html>
--- a/src/root/jobset-eval.tt
+++ b/src/root/jobset-eval.tt
@@ -65,7 +65,7 @@ c.uri_for(c.controller('JobsetEval').action_for('view'),
  [% END %]

  [% IF aborted.size > 0 %]
-    <li class="nav-item"><a class="nav-link" href="#tabs-aborted" data-toggle="tab"><span class="text-warning">Aborted Jobs ([% aborted.size %])</span></a></li>
+    <li class="nav-item"><a class="nav-link" href="#tabs-aborted" data-toggle="tab"><span class="text-warning">Aborted / Timed out Jobs ([% aborted.size %])</span></a></li>
  [% END %]
  [% IF nowFail.size > 0 %]
    <li class="nav-item"><a class="nav-link" href="#tabs-now-fail" data-toggle="tab"><span class="text-warning">Newly Failing Jobs ([% nowFail.size %])</span></a></li>
@@ -90,7 +90,7 @@ c.uri_for(c.controller('JobsetEval').action_for('view'),
  [% END %]
  <li class="nav-item"><a class="nav-link" href="#tabs-inputs" data-toggle="tab">Inputs</a></li>

-  [% IF eval.evaluationerror.errormsg %]
+  [% IF eval.evaluationerror.has_error %]
    <li class="nav-item"><a class="nav-link" href="#tabs-errors" data-toggle="tab"><span class="text-warning">Evaluation Errors</span></a></li>
  [% END %]
 </ul>
@@ -108,13 +108,6 @@ c.uri_for(c.controller('JobsetEval').action_for('view'),

 <div class="tab-content">

-  [% IF eval.evaluationerror.errormsg %]
-    <div id="tabs-errors" class="tab-pane">
-      <p>Errors occurred at [% INCLUDE renderDateTime timestamp=(eval.evaluationerror.errortime || eval.timestamp) %].</p>
-      <div class="card bg-light"><div class="card-body"><pre>[% HTML.escape(eval.evaluationerror.errormsg) %]</pre></div></div>
-    </div>
-  [% END %]
-
  <div id="tabs-aborted" class="tab-pane">
    [% INCLUDE renderSome builds=aborted tabname="#tabs-aborted" %]
  </div>
@@ -172,10 +165,9 @@ c.uri_for(c.controller('JobsetEval').action_for('view'),
    [% END %]
  </div>

-  [% IF eval.evaluationerror.errormsg %]
+  [% IF eval.evaluationerror.has_error %]
    <div id="tabs-errors" class="tab-pane">
-      <p>Errors occurred at [% INCLUDE renderDateTime timestamp=(eval.evaluationerror.errortime || eval.timestamp) %].</p>
-      <div class="card bg-light"><div class="card-body"><pre>[% HTML.escape(eval.evaluationerror.errormsg) %]</pre></div></div>
+      <iframe src="[% c.uri_for(c.controller('JobsetEval').action_for('errors'), [eval.id], params) %]" loading="lazy" frameBorder="0" width="100%"></iframe>
    </div>
  [% END %]
 </div>
--- a/src/root/jobset.tt
+++ b/src/root/jobset.tt
@@ -61,7 +61,7 @@
  [% END %]

  <li class="nav-item"><a class="nav-link active" href="#tabs-evaluations" data-toggle="tab">Evaluations</a></li>
-  [% IF jobset.errormsg || jobset.fetcherrormsg %]
+  [% IF jobset.has_error || jobset.fetcherrormsg %]
    <li class="nav-item"><a class="nav-link" href="#tabs-errors" data-toggle="tab"><span class="text-warning">Evaluation Errors</span></a></li>
  [% END %]
  <li class="nav-item"><a class="nav-link" href="#tabs-jobs" data-toggle="tab">Jobs</a></li>
@@ -79,7 +79,7 @@
        <th>Last checked:</th>
        <td>
          [% IF jobset.lastcheckedtime %]
-            [% INCLUDE renderDateTime timestamp = jobset.lastcheckedtime %], [% IF jobset.errormsg || jobset.fetcherrormsg %]<em class="text-warning">with errors!</em>[% ELSE %]<em>no errors</em>[% END %]
+            [% INCLUDE renderDateTime timestamp = jobset.lastcheckedtime %], [% IF jobset.has_error || jobset.fetcherrormsg %]<em class="text-warning">with errors!</em>[% ELSE %]<em>no errors</em>[% END %]
          [% ELSE %]
            <em>never</em>
          [% END %]
@@ -117,10 +117,9 @@

  </div>

-  [% IF jobset.errormsg || jobset.fetcherrormsg %]
+  [% IF jobset.has_error || jobset.fetcherrormsg %]
    <div id="tabs-errors" class="tab-pane">
-      <p>Errors occurred at [% INCLUDE renderDateTime timestamp=(jobset.errortime || jobset.lastcheckedtime) %].</p>
-      <div class="card bg-light"><div class="card-body"><pre>[% HTML.escape(jobset.fetcherrormsg || jobset.errormsg) %]</pre></div></div>
+      <iframe src="[% c.uri_for('/jobset' project.name jobset.name "errors") %]" loading="lazy" frameBorder="0" width="100%"></iframe>
    </div>
  [% END %]

--- a/src/root/layout.tt
+++ b/src/root/layout.tt
@@ -10,31 +10,7 @@

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=Edge" />
-
-    <script type="text/javascript" src="[% c.uri_for("/static/js/jquery/jquery-3.4.1.min.js") %]"></script>
-    <script type="text/javascript" src="[% c.uri_for("/static/js/jquery/jquery-ui-1.10.4.min.js") %]"></script>
-    <script type="text/javascript" src="[% c.uri_for("/static/js/moment/moment-2.24.0.min.js") %]"></script>
-
-    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-
-    <link href="[% c.uri_for("/static/fontawesome/css/all.css") %]" rel="stylesheet" />
-    <script type="text/javascript" src="[% c.uri_for("/static/js/popper.min.js") %]"></script>
-    <script type="text/javascript" src="[% c.uri_for("/static/bootstrap/js/bootstrap.min.js") %]"></script>
-    <link href="[% c.uri_for("/static/bootstrap/css/bootstrap.min.css") %]" rel="stylesheet" />
-
-    <!-- hydra.css may need to be moved to before boostrap to make the @media rule work. -->
-    <link rel="stylesheet" href="[% c.uri_for("/static/css/hydra.css") %]" type="text/css" />
-    <link rel="stylesheet" href="[% c.uri_for("/static/css/rotated-th.css") %]" type="text/css" />
-
-    <style>
-      .popover { max-width: 40%; }
-    </style>
-
-    <script type="text/javascript" src="[% c.uri_for("/static/js/bootbox.min.js") %]"></script>
-
-    <link rel="stylesheet" href="[% c.uri_for("/static/css/tree.css") %]" type="text/css" />
-
-    <script type="text/javascript" src="[% c.uri_for("/static/js/common.js") %]"></script>
+    [% INCLUDE style.tt %]

    [% IF c.config.enable_google_login %]
      <meta name="google-signin-client_id" content="[% c.config.google_client_id %]">
--- a/src/root/machine-status.tt
+++ b/src/root/machine-status.tt
@@ -6,10 +6,10 @@
  <thead>
    <tr>
      <th>Job</th>
-      <th>System</th>
      <th>Build</th>
      <th>Step</th>
      <th>What</th>
+      <th>Status</th>
      <th>Since</th>
    </tr>
  </thead>
@@ -40,10 +40,10 @@
          [% idle = 0 %]
          <tr>
            <td><tt>[% INCLUDE renderFullJobName project=step.project jobset=step.jobset job=step.job %]</tt></td>
-            <td><tt>[% step.system %]</tt></td>
            <td><a href="[% c.uri_for('/build' step.build) %]">[% step.build %]</a></td>
            <td>[% IF step.busy >= 30 %]<a class="row-link" href="[% c.uri_for('/build' step.build 'nixlog' step.stepnr 'tail') %]">[% step.stepnr %]</a>[% ELSE; step.stepnr; END %]</td>
            <td><tt>[% step.drvpath.match('-(.*)').0 %]</tt></td>
+            <td>[% INCLUDE renderBusyStatus %]</td>
            <td style="width: 10em">[% INCLUDE renderDuration duration = curTime - step.starttime %] </td>
          </tr>
        [% END %]
--- a/src/root/static/js/common.js
+++ b/src/root/static/js/common.js
@@ -129,6 +129,12 @@ $(document).ready(function() {
            el.addClass("is-local");
        }
    });
+
+    [...document.getElementsByTagName("iframe")].forEach((element) => {
+        element.contentWindow.addEventListener("DOMContentLoaded", (_) => {
+            element.style.height = element.contentWindow.document.body.scrollHeight + 'px';
+        })
+    })
 });

 var tabsLoaded = {};
--- a/src/root/status.tt
+++ b/src/root/status.tt
@@ -7,7 +7,7 @@

 [% ELSE %]

-  [% INCLUDE renderBuildList builds=resource showSchedulingInfo=1 hideResultInfo=1 busy=1 %]
+  [% INCLUDE renderBuildList builds=resource showSchedulingInfo=1 hideResultInfo=1 busy=1 showStepName=1 %]

 [% END %]

--- a/src/root/style.tt
+++ b/src/root/style.tt
@@ -0,0 +1,24 @@
+<script type="text/javascript" src="[% c.uri_for("/static/js/jquery/jquery-3.4.1.min.js") %]"></script>
+<script type="text/javascript" src="[% c.uri_for("/static/js/jquery/jquery-ui-1.10.4.min.js") %]"></script>
+<script type="text/javascript" src="[% c.uri_for("/static/js/moment/moment-2.24.0.min.js") %]"></script>
+
+<meta name="viewport" content="width=device-width, initial-scale=1.0" />
+
+<link href="[% c.uri_for("/static/fontawesome/css/all.css") %]" rel="stylesheet" />
+<script type="text/javascript" src="[% c.uri_for("/static/js/popper.min.js") %]"></script>
+<script type="text/javascript" src="[% c.uri_for("/static/bootstrap/js/bootstrap.min.js") %]"></script>
+<link href="[% c.uri_for("/static/bootstrap/css/bootstrap.min.css") %]" rel="stylesheet" />
+
+<!-- hydra.css may need to be moved to before boostrap to make the @media rule work. -->
+<link rel="stylesheet" href="[% c.uri_for("/static/css/hydra.css") %]" type="text/css" />
+<link rel="stylesheet" href="[% c.uri_for("/static/css/rotated-th.css") %]" type="text/css" />
+
+<style>
+  .popover { max-width: 40%; }
+</style>
+
+<script type="text/javascript" src="[% c.uri_for("/static/js/bootbox.min.js") %]"></script>
+
+<link rel="stylesheet" href="[% c.uri_for("/static/css/tree.css") %]" type="text/css" />
+
+<script type="text/javascript" src="[% c.uri_for("/static/js/common.js") %]"></script>
--- a/src/script/hydra-eval-jobset
+++ b/src/script/hydra-eval-jobset
@@ -85,14 +85,14 @@ sub attrsToSQL {
 # Fetch a store path from 'eval_substituter' if not already present.
 sub getPath {
    my ($path) = @_;
-    return 1 if $MACHINE_LOCAL_STORE->isValidPath($path);
+    return 1 if isValidPath($path);

    my $substituter = $config->{eval_substituter};

    system("nix", "--experimental-features", "nix-command", "copy", "--from", $substituter, "--", $path)
        if defined $substituter;

-    return $MACHINE_LOCAL_STORE->isValidPath($path);
+    return isValidPath($path);
 }


@@ -143,7 +143,7 @@ sub fetchInputBuild {
        , version => $version
        , outputName => $mainOutput->name
        };
-    if ($MACHINE_LOCAL_STORE->isValidPath($prevBuild->drvpath)) {
+    if (isValidPath($prevBuild->drvpath)) {
        $result->{drvPath} = $prevBuild->drvpath;
    }

@@ -233,7 +233,7 @@ sub fetchInputEval {
        my $out = $build->buildoutputs->find({ name => "out" });
        next unless defined $out;
        # FIXME: Should we fail if the path is not valid?
-        next unless $MACHINE_LOCAL_STORE->isValidPath($out->path);
+        next unless isValidPath($out->path);
        $jobs->{$build->get_column('job')} = $out->path;
    }

--- a/src/script/hydra-update-gc-roots
+++ b/src/script/hydra-update-gc-roots
@@ -5,6 +5,7 @@ use warnings;
 use File::Path;
 use File::stat;
 use File::Basename;
+use Nix::Store;
 use Hydra::Config;
 use Hydra::Schema;
 use Hydra::Helper::Nix;
@@ -46,7 +47,7 @@ sub keepBuild {
        $build->finished && ($build->buildstatus == 0 || $build->buildstatus == 6))
    {
        foreach my $path (split / /, $build->get_column('outpaths')) {
-            if ($MACHINE_LOCAL_STORE->isValidPath($path)) {
+            if (isValidPath($path)) {
                addRoot $path;
            } else {
                print STDERR "    warning: output ", $path, " has disappeared\n" if $build->finished;
@@ -54,7 +55,7 @@ sub keepBuild {
        }
    }
    if (!$build->finished || ($keepFailedDrvs && $build->buildstatus != 0)) {
-        if ($MACHINE_LOCAL_STORE->isValidPath($build->drvpath)) {
+        if (isValidPath($build->drvpath)) {
            addRoot $build->drvpath;
        } else {
            print STDERR "    warning: derivation ", $build->drvpath, " has disappeared\n";
--- a/t/Hydra/Controller/Job/builds.t
+++ b/t/Hydra/Controller/Job/builds.t
@@ -54,14 +54,13 @@ subtest "/job/PROJECT/JOBSET/JOB/shield" => sub {

 subtest "/job/PROJECT/JOBSET/JOB/prometheus" => sub {
    my $response = request(GET '/job/' . $project->name . '/' . $jobset->name . '/' . $build->job . '/prometheus');
-
    ok($response->is_success, "The page showing the job's prometheus data returns 200.");
-
    my $metrics = $response->content;
-    like($metrics, qr/hydra_job_failed\{.*\} 0/);
-    like($metrics, qr/hydra_job_completion_time\{.*\} [\d]+/);
-    like($metrics, qr/hydra_build_closure_size\{.*\} 96/);
-    like($metrics, qr/hydra_build_output_size\{.*\} 96/);
+
+    ok($metrics =~ m/hydra_job_failed\{.*\} 0/);
+    ok($metrics =~ m/hydra_job_completion_time\{.*\} [\d]+/);
+    ok($metrics =~ m/hydra_build_closure_size\{.*\} 96/);
+    ok($metrics =~ m/hydra_build_output_size\{.*\} 96/);
 };

 done_testing;
--- a/t/Hydra/Controller/Jobset/evals.t
+++ b/t/Hydra/Controller/Jobset/evals.t
@@ -32,4 +32,9 @@ subtest "/jobset/PROJECT/JOBSET/evals" => sub {
    ok($jobsetevals->is_success, "The page showing the jobset evals returns 200.");
 };

+subtest "/jobset/PROJECT/JOBSET/errors" => sub {
+    my $jobsetevals = request(GET '/jobset/' . $project->name . '/' . $jobset->name . '/errors');
+    ok($jobsetevals->is_success, "The page showing the jobset eval errors returns 200.");
+};
+
 done_testing;
--- a/t/Hydra/Controller/Jobset/http.t
+++ b/t/Hydra/Controller/Jobset/http.t
@@ -186,7 +186,7 @@ subtest 'Update jobset "job" to have an invalid input type' => sub {
      })
  );
  ok(!$jobsetupdate->is_success);
-  like($jobsetupdate->content, qr/Invalid input type.*valid types:/);
+  ok($jobsetupdate->content =~ m/Invalid input type.*valid types:/);
 };


--- a/t/Hydra/Controller/JobsetEval/fetch.t
+++ b/t/Hydra/Controller/JobsetEval/fetch.t
@@ -35,6 +35,10 @@ subtest "Fetching the eval's overview" => sub {
    is($fetch->code, 200, "channel page is 200");
 };

+subtest "Fetching the eval's overview" => sub {
+    my $fetch = request(GET '/eval/' . $eval->id, '/errors');
+    is($fetch->code, 200, "errors page is 200");
+};


 done_testing;
--- a/t/Hydra/Controller/User/dashboard.t
+++ b/t/Hydra/Controller/User/dashboard.t
@@ -24,7 +24,7 @@ my $cookie = $login->header("set-cookie");
 my $my_jobs = request(GET '/dashboard/alice/my-jobs-tab', Accept => 'application/json', Cookie => $cookie);
 ok($my_jobs->is_success);
 my $content = $my_jobs->content();
-like($content, qr/empty_dir/);
+ok($content =~ /empty_dir/);
 ok(!($content =~ /fails/));
 ok(!($content =~ /succeed_with_failed/));
 done_testing;
--- a/t/api-test.t
+++ b/t/api-test.t
@@ -115,7 +115,7 @@ subtest "evaluation" => sub {
        my $build = decode_json(request_json({ uri => "/build/" . $evals->[0]->{builds}->[0] })->content());
        is($build->{job}, "job", "The build's job name is job");
        is($build->{finished}, 0, "The build isn't finished yet");
-        like($build->{buildoutputs}->{out}->{path}, qr/\/nix\/store\/[a-zA-Z0-9]{32}-job$/, "The build's outpath is in the Nix store and named 'job'");
+        ok($build->{buildoutputs}->{out}->{path} =~ /\/nix\/store\/[a-zA-Z0-9]{32}-job$/, "The build's outpath is in the Nix store and named 'job'");

        subtest "search" => sub {
            my $search_project = decode_json(request_json({ uri => "/search/?query=sample" })->content());
--- a/t/content-addressed/basic.t
+++ b/t/content-addressed/basic.t
@@ -27,13 +27,13 @@ my $project = $db->resultset('Projects')->create({name => "tests", displayname =
 my $jobset = createBaseJobset("content-addressed", "content-addressed.nix", $ctx{jobsdir});

 ok(evalSucceeds($jobset), "Evaluating jobs/content-addressed.nix should exit with return code 0");
-is(nrQueuedBuildsForJobset($jobset), 6, "Evaluating jobs/content-addressed.nix should result in 6 builds");
+is(nrQueuedBuildsForJobset($jobset), 5, "Evaluating jobs/content-addressed.nix should result in 4 builds");

 for my $build (queuedBuildsForJobset($jobset)) {
    ok(runBuild($build), "Build '".$build->job."' from jobs/content-addressed.nix should exit with code 0");
    my $newbuild = $db->resultset('Builds')->find($build->id);
    is($newbuild->finished, 1, "Build '".$build->job."' from jobs/content-addressed.nix should be finished.");
-    my $expected = $build->job eq "fails" ? 1 : $build->job =~ /with_failed/ ? 6 : $build->job =~ /FailingCA/ ? 2 : 0;
+    my $expected = $build->job eq "fails" ? 1 : $build->job =~ /with_failed/ ? 6 : 0;
    is($newbuild->buildstatus, $expected, "Build '".$build->job."' from jobs/content-addressed.nix should have buildstatus $expected.");

    my $response = request("/build/".$build->id);
@@ -55,8 +55,6 @@ for my $build (queuedBuildsForJobset($jobset)) {

 }

-# XXX: deststoredir is undefined: Use of uninitialized value $ctx{"deststoredir"} in concatenation (.) or string at t/content-addressed/basic.t line 58.
-# XXX: This test seems to not do what it seems to be doing. See documentation: https://metacpan.org/pod/Test2::V0#isnt($got,-$do_not_want,-$name)
 isnt(<$ctx{deststoredir}/realisations/*>, "", "The destination store should have the realisations of the built derivations registered");

 done_testing;
--- a/t/evaluator/evaluate-constituents-broken.t
+++ b/t/evaluator/evaluate-constituents-broken.t
@@ -22,7 +22,7 @@ like(
    "The stderr record includes a relevant error message"
 );

-$jobset->discard_changes;  # refresh from DB
+$jobset->discard_changes({ '+columns' => {'errormsg' => 'errormsg'} });  # refresh from DB
 like(
    $jobset->errormsg,
    qr/aggregate job ‘mixed_aggregate’ failed with the error: constituentA: does not exist/,
--- a/t/evaluator/evaluate-oom-job.t
+++ b/t/evaluator/evaluate-oom-job.t
@@ -31,10 +31,6 @@ if ($sd_res != 0) {
  skip_all("`systemd-run` returned non-zero when executing `true` (expected 0)");
 }

-# XXX(Mindavi): We should think about how to fix this.
-#               Note that it was always skipped on ofborg/h.n.o (nixos hydra) since systemd-run is not present in the ambient environment there.
-skip_all("Always fails, an error about 'oom' being a string is logged and the process never OOMs. Needs a way to use more memory.");
-
 my $ctx = test_context();

 # Contain the memory usage to 25 MegaBytes using `systemd-run`
--- a/t/jobs/content-addressed.nix
+++ b/t/jobs/content-addressed.nix
@@ -25,13 +25,6 @@ rec {
      FOO = empty_dir;
    };

-  caDependingOnFailingCA =
-    cfg.mkContentAddressedDerivation {
-      name = "ca-depending-on-failing-ca";
-      builder = ./dir-with-file-builder.sh;
-      FOO = fails;
-    };
-
  nonCaDependingOnCA =
    cfg.mkDerivation {
      name = "non-ca-depending-on-ca";
--- a/t/lib/CliRunners.pm
+++ b/t/lib/CliRunners.pm
@@ -14,7 +14,7 @@ our @EXPORT = qw(
 sub evalSucceeds {
    my ($jobset) = @_;
    my ($res, $stdout, $stderr) = captureStdoutStderr(60, ("hydra-eval-jobset", $jobset->project->name, $jobset->name));
-    $jobset->discard_changes;  # refresh from DB
+    $jobset->discard_changes({ '+columns' => {'errormsg' => 'errormsg'} });  # refresh from DB
    if ($res) {
        chomp $stdout; chomp $stderr;
        utf8::decode($stdout) or die "Invalid unicode in stdout.";
@@ -29,7 +29,7 @@ sub evalSucceeds {
 sub evalFails {
    my ($jobset) = @_;
    my ($res, $stdout, $stderr) = captureStdoutStderr(60, ("hydra-eval-jobset", $jobset->project->name, $jobset->name));
-    $jobset->discard_changes;  # refresh from DB
+    $jobset->discard_changes({ '+columns' => {'errormsg' => 'errormsg'} });  # refresh from DB
    if (!$res) {
        chomp $stdout; chomp $stderr;
        utf8::decode($stdout) or die "Invalid unicode in stdout.";
--- a/t/lib/HydraTestContext.pm
+++ b/t/lib/HydraTestContext.pm
@@ -92,7 +92,7 @@ sub new {
        $opts{'before_init'}->($self);
    }

-    expectOkay(30, ("hydra-init"));
+    expectOkay(5, ("hydra-init"));

    return $self;
 }
--- a/t/lib/LDAPContext.pm
+++ b/t/lib/LDAPContext.pm
@@ -70,7 +70,7 @@ sub add_user {
    my $email = $opts{'email'} // "$name\@example";
    my $password = $opts{'password'} // rand_chars();

-    my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("slappasswd", "-s", $password));
+    my ($res, $stdout, $stderr) = captureStdoutStderr(1, ("slappasswd", "-s", $password));
    if ($res) {
        die "Failed to execute slappasswd ($res): $stderr, $stdout";
    }
@@ -178,7 +178,7 @@ sub start {
 sub validateConfig {
    my ($self) = @_;

-    expectOkay(5, ("slaptest", "-u", "-F", $self->{"_slapd_dir"}));
+    expectOkay(1, ("slaptest", "-u", "-F", $self->{"_slapd_dir"}));
 }

 sub _spawn {
@@ -218,7 +218,7 @@ sub load_ldif {

    my $path = "${\$self->{'_tmpdir'}}/load.ldif";
    write_file($path, $content);
-    expectOkay(5, ("slapadd", "-F", $self->{"_slapd_dir"}, "-b", $suffix, "-l", $path));
+    expectOkay(1, ("slapadd", "-F", $self->{"_slapd_dir"}, "-b", $suffix, "-l", $path));
    $self->validateConfig();
 }

--- a/t/queue-runner/build-locally-with-substitutable-path.t
+++ b/t/queue-runner/build-locally-with-substitutable-path.t
@@ -39,7 +39,7 @@ subtest "Building, caching, and then garbage collecting the underlying job" => s

    ok(unlink(Hydra::Helper::Nix::gcRootFor($path)), "Unlinking the GC root for underlying Dependency succeeds");

-    (my $ret, my $stdout, my $stderr) = captureStdoutStderr(15, "nix-store", "--delete", $path);
+    (my $ret, my $stdout, my $stderr) = captureStdoutStderr(5, "nix-store", "--delete", $path);
    is($ret, 0, "Deleting the underlying dependency should succeed");
 };

--- a/t/queue-runner/direct-indirect-constituents.t
+++ b/t/queue-runner/direct-indirect-constituents.t
@@ -13,7 +13,7 @@ my $constituentBuildA = $builds->{"constituentA"};
 my $constituentBuildB = $builds->{"constituentB"};

 my $eval = $constituentBuildA->jobsetevals->first();
-is($eval->evaluationerror->errormsg, "");
+is($eval->evaluationerror->has_error, 0);

 subtest "Verifying the direct aggregate" => sub {
    my $aggBuild = $builds->{"direct_aggregate"};
--- a/t/s3-backup-test.pl
+++ b/t/s3-backup-test.pl
@@ -3,6 +3,7 @@ use warnings;
 use File::Basename;
 use Hydra::Model::DB;
 use Hydra::Helper::Nix;
+use Nix::Store;
 use Cwd;

 my $db = Hydra::Model::DB->new;
--- a/t/scripts/hydra-create-user.t
+++ b/t/scripts/hydra-create-user.t
@@ -9,7 +9,7 @@ my $db = $ctx->db();

 subtest "Handling password and password hash creation" => sub {
    subtest "Creating a user with a plain text password (insecure) stores the password securely" => sub {
-        my ($res, $stdout, $stderr) = captureStdoutStderr(15, ("hydra-create-user", "plain-text-user", "--password", "foobar"));
+        my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("hydra-create-user", "plain-text-user", "--password", "foobar"));
        is($res, 0, "hydra-create-user should exit zero");
        like($stderr, qr/Submitting plaintext passwords as arguments is deprecated and will be removed/, "Submitting a plain text password is deprecated.");

@@ -23,7 +23,7 @@ subtest "Handling password and password hash creation" => sub {
    };

    subtest "Creating a user with a sha1 password (still insecure) stores the password as a hashed sha1" => sub {
-        my ($res, $stdout, $stderr) = captureStdoutStderr(15, ("hydra-create-user", "old-password-hash-user", "--password-hash", "8843d7f92416211de9ebb963ff4ce28125932878"));
+        my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("hydra-create-user", "old-password-hash-user", "--password-hash", "8843d7f92416211de9ebb963ff4ce28125932878"));
        is($res, 0, "hydra-create-user should exit zero");

        my $user = $db->resultset('Users')->find({ username => "old-password-hash-user" });
@@ -36,7 +36,7 @@ subtest "Handling password and password hash creation" => sub {
    };

    subtest "Creating a user with an argon2 password stores the password as given" => sub {
-        my ($res, $stdout, $stderr) = captureStdoutStderr(15, ("hydra-create-user", "argon2-hash-user", "--password-hash", '$argon2id$v=19$m=262144,t=3,p=1$tMnV5paYjmIrUIb6hylaNA$M8/e0i3NGrjhOliVLa5LqQ'));
+        my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("hydra-create-user", "argon2-hash-user", "--password-hash", '$argon2id$v=19$m=262144,t=3,p=1$tMnV5paYjmIrUIb6hylaNA$M8/e0i3NGrjhOliVLa5LqQ'));
        is($res, 0, "hydra-create-user should exit zero");

        my $user = $db->resultset('Users')->find({ username => "argon2-hash-user" });
@@ -50,7 +50,7 @@ subtest "Handling password and password hash creation" => sub {

    subtest "Creating a user by prompting for the password" => sub {
        subtest "with the same password twice" => sub {
-            my ($res, $stdout, $stderr) = captureStdoutStderrWithStdin(15, ["hydra-create-user", "prompted-pass-user", "--password-prompt"], "my-password\nmy-password\n");
+            my ($res, $stdout, $stderr) = captureStdoutStderrWithStdin(5, ["hydra-create-user", "prompted-pass-user", "--password-prompt"], "my-password\nmy-password\n");
            is($res, 0, "hydra-create-user should exit zero");

            my $user = $db->resultset('Users')->find({ username => "prompted-pass-user" });
@@ -62,7 +62,7 @@ subtest "Handling password and password hash creation" => sub {
        };

        subtest "With mismatched password confirmation" => sub {
-            my ($res, $stdout, $stderr) = captureStdoutStderrWithStdin(15, ["hydra-create-user", "prompted-pass-user", "--password-prompt"], "my-password\nnot-my-password\n");
+            my ($res, $stdout, $stderr) = captureStdoutStderrWithStdin(5, ["hydra-create-user", "prompted-pass-user", "--password-prompt"], "my-password\nnot-my-password\n");
            isnt($res, 0, "hydra-create-user should exit non-zero");
        };
    };
@@ -76,7 +76,7 @@ subtest "Handling password and password hash creation" => sub {
        );

        for my $case (@cases) {
-            my ($res, $stdout, $stderr) = captureStdoutStderr(15, (
+            my ($res, $stdout, $stderr) = captureStdoutStderr(5, (
                "hydra-create-user", "bogus-password-options", @{$case}));
            like($stderr, qr/please specify only one of --password-prompt or --password-hash/, "We get an error about specifying the password");
            isnt($res, 0, "hydra-create-user should exit non-zero with conflicting " . join(" ", @{$case}));
@@ -84,7 +84,7 @@ subtest "Handling password and password hash creation" => sub {
    };

    subtest "A password is not required for creating a Google-based account" => sub {
-        my ($res, $stdout, $stderr) = captureStdoutStderr(15, (
+        my ($res, $stdout, $stderr) = captureStdoutStderr(5, (
            "hydra-create-user", "google-account", "--type", "google"));
        is($res, 0, "hydra-create-user should exit zero");
    };
--- a/t/scripts/hydra-init.t
+++ b/t/scripts/hydra-init.t
@@ -28,7 +28,7 @@ subtest "hydra-init upgrades user's password hashes from sha1 to sha1 inside Arg
    $janet->setPassword("foobar");

    is($alice->password, "8843d7f92416211de9ebb963ff4ce28125932878", "Alices's sha1 is stored in the database");
-    my ($res, $stdout, $stderr) = captureStdoutStderr(30, ("hydra-init"));
+    my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("hydra-init"));
    if ($res != 0) {
        is($stdout, "");
        is($stderr, "");
@@ -55,7 +55,7 @@ subtest "hydra-init upgrades user's password hashes from sha1 to sha1 inside Arg
    };

    subtest "Running hydra-init don't break Alice or Janet's passwords" => sub {
-        my ($res, $stdout, $stderr) = captureStdoutStderr(30, ("hydra-init"));
+        my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("hydra-init"));
        is($res, 0, "hydra-init should exit zero");

        my $updatedAlice = $db->resultset('Users')->find({ username => "alice" });
--- a/t/test.pl
+++ b/t/test.pl
@@ -21,7 +21,7 @@ if (defined($ENV{"NIX_BUILD_CORES"})
    print STDERR "test.pl: Defaulting \$YATH_JOB_COUNT to \$NIX_BUILD_CORES (${\$ENV{'NIX_BUILD_CORES'}})\n";
 }

-system($^X, find_yath(), '-D', 'test', '--qvf', '--event-timeout', 240, '--default-search' => './', @ARGV);
+system($^X, find_yath(), '-D', 'test', '--default-search' => './', @ARGV);
 my $exit = $?;

 # This makes sure it works with prove.
Author	SHA1	Message	Date
Martin Weinelt	bc6f08d233	Create eval-jobset role and guard /api/push route (cherry picked from commit `f730433789`)	2024-08-27 21:53:17 +02:00
Janne Heß	88d9898588	api: Require POST for /api/push (cherry picked from commit `916531dc9c`)	2024-08-27 21:53:13 +02:00
Pierre Bourdon	b8d03adaf4	queue runner: attempt at slightly smarter scheduling criteria Instead of just going for "whatever is the oldest build we know of", use the following first: - Is the step more constrained? If so, schedule it first to avoid filling up "more desirable" build slots with less constrained builds. - Does the step have more dependents? If so, schedule it first to try and maximize open parallelism and breadth of scheduling options.	2024-04-21 17:36:16 +02:00
Pierre Bourdon	ee1a7a7813	web: serveFile: also serve a CSP putting served HTML in its own origin	2024-04-21 16:14:24 +02:00
Pierre Bourdon	5c3e508e55	queue-runner: release machine reservation while copying outputs This allows for better builder usage when the queue runner is busy. To avoid running into uncontrollable imbalances between builder/queue runner, we only release the machine reservation after the local throttler has found a slot to start copying the outputs for that build.	2024-04-21 01:55:19 +02:00
Pierre Bourdon	026e3a3103	queue-runner: switch to pseudorandom ordering of builds processing We don't rely on sequential / monotonic build IDs processing anymore, so randomizing actually has the advantage of mixing builds for different systems together, to avoid only one chunk of builds for a single system getting processed while builders for other systems are starved.	2024-04-20 23:05:26 +02:00
Pierre Bourdon	6606a7f86e	queue runner: introduce some parallelism for remote paths lookup Each output for a given step being ingested is looked up in parallel, which should basically multiply the speed of builds ingestion by the average number of outputs per derivation.	2024-04-20 22:28:18 +02:00
Pierre Bourdon	f31b95d371	queue-runner: reduce the time between queue monitor restarts This will induce more DB queries (though these are fairly cheap), but at the benefit of processing bumps within 1m instead of within 10m.	2024-04-20 16:58:10 +02:00
Pierre Bourdon	54f8daf6b1	queue-runner: remove id > X from new builds query Running the query with/without it shows that it makes no difference to postgres, since there's an index on finished=0 already. This allows a few simplifications, but also paves the way towards running multiple parallel monitor threads in the future.	2024-04-20 16:53:52 +02:00
Pierre Bourdon	cc6bafe538	queue-runner: add prom metrics to allow detecting internal bottlenecks By looking at the ratio of running vs. waiting for the dispatcher and the queue monitor, we should get better visibility into what hydra is currently bottlenecked on. There are other side effects we can try to measure to get to the same result, but having a simple way doesn't cost us much.	2024-04-20 16:48:03 +02:00
Pierre Bourdon	6189ba9c5e	web: replace 'errormsg' with 'errormsg IS NULL' in most cases This is implement in an extremely hacky way due to poor DBIx feature support. Ideally, what we'd need is a way to tell DBIx to ignore the errormsg column unless explicitly requested, and to automatically add a computed 'errormsg IS NULL' column in others. Since it does not support that, this commit instead hacks some support via method overrides while taking care to not break anything obvious.	2024-04-12 20:14:09 +02:00
Pierre Bourdon	258e9314a9	web: include current step status on /machines	2024-04-11 17:15:58 +02:00
Pierre Bourdon	a51bd392a2	queue-runner: limit parallelism of CPU intensive operations My current theory is that running more parallel xz than available CPU cores is reducing our overall throughput by requiring more scheduling overhead and more cache thrashing.	2024-04-11 16:43:01 +02:00
Maximilian Bosch	a596d6c3c1	Only show stepname if it doesn't equal the name of the drv When building e.g. nixpkgs, the "Running builds" view will mostly look like this hello.x86_64-linux (Build of hello-X.Y) exa.x86_64-linux (Build of exa-X.Y) ... This doesn't provide any useful information. Showing the step name only makes sense if it's not a child of the job's derivation. With this patch, that information will only be shown if the drv name (i.e. w/o `/nix/store/` prefix, .drv ext & hash) is not equal to the drv name of the job itself (build.nixname).	2024-03-18 18:46:01 +01:00
Maximilian Bosch	415f9f2daa	Running builds view: show build step names When using Hydra to build machine configurations, you'll often see "nixosConfigurations.foo" five times, i.e. for each build step being run. This isn't very helpful I think because in such a case, a single build step can also be compiling the Linux kernel. This change also fetches the `drvpath` and `type` from the `buildsteps` relation. We're already joining it, so this doesn't make much difference (confirmed via query logging that this doesn't cause extra SQL queries). Unfortunately build steps don't have a human readable name, so I'm deriving it from the drvpath by stripping away the hash (assuming that it'll never contain a `-` and that `/nix/store/` is used as prefix). I decided against using the Nix bindings for that to avoid too much overhead due to store operations for each build step.	2024-03-18 18:46:01 +01:00
Maximilian Bosch	9b465e7a67	Make "timed out" and "log limit exceeded" builds aborted In `73694087a0` I gave builds that failed because of a timeout or exceeded log limit a stop sign and I stand by that reasoning: with that it's possible to distinguish between actual build failures and rather transient things such as timeouts. Back then I considered it a feature that these are shown in a different tab, but I don't think that's a good idea anymore. When using a jobset to e.g. track the regressions from a mass rebuild (like a compiler or gcc update), "Newly failed builds" should exclusively display regressions (and flaky builds of course, not much I can do about that). Also, when a bunch of builds fail in such a jobset because of e.g. a broken connection to a builder that results in a timeout, I want to be able to restart them all w/o rebuilding actual regressions. To make it clear that we not only have "Aborted" builds in the tab, I renamed the label to "Aborted / Timed out".	2024-03-16 22:10:40 +01:00
Maximilian Bosch	9b62c52e5c	hydra-queue-runner: drop broken connections from pool Closes #1336 When restarting postgresql, the connections are still reused in `hydra-queue-runner` causing errors like this main thread: Lost connection to the database server. queue monitor: Lost connection to the database server. and no more builds being processed. `hydra-evaluator` doesn't have that issue since it crashes right away. We could let it retry indefinitely as well (see below), but I don't want to change too much. If the DB is still unreachable 10s later, the process will stop with a non-zero exit code because of a missing DB connection. This however isn't such a big deal because it will be immediately restarted afterwards. With the current configuration, Hydra will never give up, but restart (and retry) infinitely. To me that seems reasonable, i.e. to retry DB connections on a long-running process. If this doesn't work out, the monitoring should fire anyways because the queue fills up, but I'm open to discuss that. Please note that this isn't reproducible with the DB and the queue runner on the same machine when using `services.hydra-dev`, because of the `Requires=` dependency `hydra-queue-runner.service` -> `hydra-init.service` -> `postgresql.service` that causes the queue runner to be restarted on `systemctl restart postgresql`. Internally, Hydra uses Nix's pool data structure: it basically has N slots (here DB connections) and whenever a new one is requested, an idle slot is provided or a new one is created (when N slots are active, it'll be waited until one slot is free). The issue in the code here is however that whenever an error is encountered, the slot is released, however the same broken connection will be reused the next time. By using `Pool::Handle::markBad`, Nix will drop a broken slot. This is now being done when `pqxx::broken_connection` was caught.	2024-03-16 22:10:40 +01:00
Maximilian Bosch	ef6be80f54	Use `submit` event in login form It's a pet peeve from me when logging into my personal Hydra that I always have to press the button rather than hitting Return after entering my password. Reason for that is that the form doesn't have a "submit" button, so far it was always listened to the "click" event. Submit does that and you can hit Return alternatively.	2024-03-16 22:10:40 +01:00
K900	969eb3eeac	urlencode drv names when fetching logs Otherwise names with special characters like + break things.	2024-03-16 22:10:40 +01:00
Pierre Bourdon	18466e8326	queue-runner: try larger pipe buffer sizes	2024-03-16 22:10:40 +01:00
ajs124	6ed21490ee	lazy-load evaluation errors Closes #1362	2024-03-16 22:10:40 +01:00