Merge pull request #1460 from NixOS/nix-2.27

Nix 2.27
Fix build (due to C++ API changes)
2025-04-07 11:37:43 -04:00 · 2025-04-07 11:12:12 -04:00 · 2025-04-07 11:02:52 -04:00 · 2025-03-29 15:37:06 +01:00 · 2025-03-29 14:31:18 +00:00 · 2025-03-29 14:31:18 +00:00
29 changed files with 523 additions and 314 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,7 @@
 /src/sql/hydra-postgresql.sql
 /src/sql/hydra-sqlite.sql
 /src/sql/tmp.sqlite
+.hydra-data
 result
 result-*
 outputs
--- a/README.md
+++ b/README.md
@@ -72,17 +72,16 @@ Make sure **State** at the top of the page is set to "_Enabled_" and click on "_
 You can build Hydra via `nix-build` using the provided [default.nix](./default.nix):

 ```
-$ nix-build
+$ nix build
 ```

 ### Development Environment

 You can use the provided shell.nix to get a working development environment:
 ```
-$ nix-shell
-$ autoreconfPhase
-$ configurePhase # NOTE: not ./configure
-$ make
+$ nix develop
+$ mesonConfigurePhase
+$ ninja
 ```

 ### Executing Hydra During Development
@@ -91,9 +90,9 @@ When working on new features or bug fixes you need to be able to run Hydra from
 can be done using [foreman](https://github.com/ddollar/foreman):

 ```
-$ nix-shell
+$ nix develop
 $ # hack hack
-$ make
+$ ninja -C build
 $ foreman start
 ```

@@ -115,22 +114,24 @@ Start by following the steps in [Development Environment](#development-environme
 Then, you can run the tests and the perlcritic linter together with:

 ```console
-$ nix-shell
-$ make check
+$ nix develop
+$ ninja -C build test
 ```

 You can run a single test with:

 ```
-$ nix-shell
-$ yath test ./t/foo/bar.t
+$ nix develop
+$ cd build
+$ meson test --test-args=../t/Hydra/Event.t testsuite
 ```

 And you can run just perlcritic with:

 ```
-$ nix-shell
-$ make perlcritic
+$ nix develop
+$ cd build
+$ meson test perlcritic
 ```

 ### JSON API
--- a/doc/manual/src/hacking.md
+++ b/doc/manual/src/hacking.md
@@ -11,12 +11,6 @@ $ cd hydra
 To enter a shell in which all environment variables (such as `PERL5LIB`)
 and dependencies can be found:

-```console
-$ nix-shell
-```
-
-of when flakes are enabled:
-
 ```console
 $ nix develop
 ```
@@ -24,15 +18,15 @@ $ nix develop
 To build Hydra, you should then do:

 ```console
-[nix-shell]$ autoreconfPhase
-[nix-shell]$ configurePhase
-[nix-shell]$ make -j$(nproc)
+$ mesonConfigurePhase
+$ ninja
 ```

 You start a local database, the webserver, and other components with
 foreman:

 ```console
+$ ninja -C build
 $ foreman start
 ```

@@ -47,18 +41,11 @@ $ ./src/script/hydra-server
 You can run Hydra's test suite with the following:

 ```console
-[nix-shell]$ make check
-[nix-shell]$ # to run as many tests as you have cores:
-[nix-shell]$ make check YATH_JOB_COUNT=$NIX_BUILD_CORES
-[nix-shell]$ # or run yath directly:
-[nix-shell]$ yath test
-[nix-shell]$ # to run as many tests as you have cores:
-[nix-shell]$ yath test -j $NIX_BUILD_CORES
+$ meson test
+# to run as many tests as you have cores:
+$ YATH_JOB_COUNT=$NIX_BUILD_CORES meson test
 ```

-When using `yath` instead of `make check`, ensure you have run `make`
-in the root of the repository at least once.
-
 **Warning**: Currently, the tests can fail
 if run with high parallelism [due to an issue in
 `Test::PostgreSQL`](https://github.com/TJC/Test-postgresql/issues/40)
@@ -75,7 +62,7 @@ will reload the page every time you save.
 To build Hydra and its dependencies:

 ```console
-$ nix-build release.nix -A build.x86_64-linux
+$ nix build .#packages.x86_64-linux.default
 ```

 ## Development Tasks
--- a/flake.lock
+++ b/flake.lock
@@ -1,51 +1,10 @@
 {
  "nodes": {
-    "flake-parts": {
-      "inputs": {
-        "nixpkgs-lib": [
-          "nix-eval-jobs",
-          "nixpkgs"
-        ]
-      },
-      "locked": {
-        "lastModified": 1722555600,
-        "narHash": "sha256-XOQkdLafnb/p9ij77byFQjDf5m5QYl9b2REiVClC+x4=",
-        "owner": "hercules-ci",
-        "repo": "flake-parts",
-        "rev": "8471fe90ad337a8074e957b69ca4d0089218391d",
-        "type": "github"
-      },
-      "original": {
-        "owner": "hercules-ci",
-        "repo": "flake-parts",
-        "type": "github"
-      }
-    },
-    "libgit2": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1715853528,
-        "narHash": "sha256-J2rCxTecyLbbDdsyBWn9w7r3pbKRMkI9E7RvRgAqBdY=",
-        "owner": "libgit2",
-        "repo": "libgit2",
-        "rev": "36f7e21ad757a3dacc58cf7944329da6bc1d6e96",
-        "type": "github"
-      },
-      "original": {
-        "owner": "libgit2",
-        "ref": "v1.8.1",
-        "repo": "libgit2",
-        "type": "github"
-      }
-    },
    "nix": {
      "inputs": {
        "flake-compat": [],
        "flake-parts": [],
        "git-hooks-nix": [],
-        "libgit2": [
-          "libgit2"
-        ],
        "nixpkgs": [
          "nixpkgs"
        ],
@@ -53,88 +12,59 @@
        "nixpkgs-regression": []
      },
      "locked": {
-        "lastModified": 1726787955,
-        "narHash": "sha256-XFznzb8L4SdUm9u+w3DPpMWJhffuv+/6+aiVl00slns=",
+        "lastModified": 1744029599,
+        "narHash": "sha256-u6RhBWQ1XohTZ4Ub5ml1PTcaxQgtqFNng6Sohy1rojw=",
        "owner": "NixOS",
        "repo": "nix",
-        "rev": "a7fdef6858dd45b9d7bda7c92324c63faee7f509",
+        "rev": "d0f98c76f962147610489e84c10033ca92e9c532",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
-        "ref": "2.24-maintenance",
+        "ref": "2.27-maintenance",
        "repo": "nix",
        "type": "github"
      }
    },
    "nix-eval-jobs": {
-      "inputs": {
-        "flake-parts": "flake-parts",
-        "nix-github-actions": [],
-        "nixpkgs": [
-          "nixpkgs"
-        ],
-        "treefmt-nix": "treefmt-nix"
-      },
+      "flake": false,
      "locked": {
-        "lastModified": 1733814344,
-        "narHash": "sha256-3wwtKpS5tUBdjaGeSia7CotonbiRB6K5Kp0dsUt3nzU=",
+        "lastModified": 1744009874,
+        "narHash": "sha256-PypQspB7h7EENe4RQQUQj2Ay8J1+O49AKNO9JbAU4Ek=",
        "owner": "nix-community",
        "repo": "nix-eval-jobs",
-        "rev": "889ea1406736b53cf165b6c28398aae3969418d1",
+        "rev": "62f9c9e8d00d2ff6ab27a6197ab459a8e0808e59",
        "type": "github"
      },
      "original": {
        "owner": "nix-community",
-        "ref": "release-2.24",
+        "ref": "v2.27.0",
        "repo": "nix-eval-jobs",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
-        "lastModified": 1726688310,
-        "narHash": "sha256-Xc9lEtentPCEtxc/F1e6jIZsd4MPDYv4Kugl9WtXlz0=",
+        "lastModified": 1739461644,
+        "narHash": "sha256-1o1qR0KYozYGRrnqytSpAhVBYLNBHX+Lv6I39zGRzKM=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "dbebdd67a6006bb145d98c8debf9140ac7e651d0",
+        "rev": "97a719c9f0a07923c957cf51b20b329f9fb9d43f",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
-        "ref": "nixos-24.05-small",
+        "ref": "nixos-24.11-small",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
-        "libgit2": "libgit2",
        "nix": "nix",
        "nix-eval-jobs": "nix-eval-jobs",
        "nixpkgs": "nixpkgs"
      }
-    },
-    "treefmt-nix": {
-      "inputs": {
-        "nixpkgs": [
-          "nix-eval-jobs",
-          "nixpkgs"
-        ]
-      },
-      "locked": {
-        "lastModified": 1723303070,
-        "narHash": "sha256-krGNVA30yptyRonohQ+i9cnK+CfCpedg6z3qzqVJcTs=",
-        "owner": "numtide",
-        "repo": "treefmt-nix",
-        "rev": "14c092e0326de759e16b37535161b3cb9770cea3",
-        "type": "github"
-      },
-      "original": {
-        "owner": "numtide",
-        "repo": "treefmt-nix",
-        "type": "github"
-      }
    }
  },
  "root": "root",
--- a/flake.nix
+++ b/flake.nix
@@ -1,25 +1,25 @@
 {
  description = "A Nix-based continuous build system";

-  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05-small";
+  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11-small";

-  inputs.libgit2 = { url = "github:libgit2/libgit2/v1.8.1"; flake = false; };
-  inputs.nix.url = "github:NixOS/nix/2.24-maintenance";
-  inputs.nix.inputs.nixpkgs.follows = "nixpkgs";
-  inputs.nix.inputs.libgit2.follows = "libgit2";
+  inputs.nix = {
+    url = "github:NixOS/nix/2.27-maintenance";
+    inputs.nixpkgs.follows = "nixpkgs";

-  inputs.nix-eval-jobs.url = "github:nix-community/nix-eval-jobs/release-2.24";
-  inputs.nix-eval-jobs.inputs.nixpkgs.follows = "nixpkgs";
+    # hide nix dev tooling from our lock file
+    inputs.flake-parts.follows = "";
+    inputs.git-hooks-nix.follows = "";
+    inputs.nixpkgs-regression.follows = "";
+    inputs.nixpkgs-23-11.follows = "";
+    inputs.flake-compat.follows = "";
+  };

-  # hide nix dev tooling from our lock file
-  inputs.nix.inputs.flake-parts.follows = "";
-  inputs.nix.inputs.git-hooks-nix.follows = "";
-  inputs.nix.inputs.nixpkgs-regression.follows = "";
-  inputs.nix.inputs.nixpkgs-23-11.follows = "";
-  inputs.nix.inputs.flake-compat.follows = "";
-
-  # hide nix-eval-jobs dev tooling from our lock file
-  inputs.nix-eval-jobs.inputs.nix-github-actions.follows = "";
+  inputs.nix-eval-jobs = {
+    url = "github:nix-community/nix-eval-jobs/v2.27.0";
+    # We want to control the deps precisely
+    flake = false;
+  };

  outputs = { self, nixpkgs, nix, nix-eval-jobs, ... }:
    let
@@ -30,9 +30,9 @@

      # A Nixpkgs overlay that provides a 'hydra' package.
      overlays.default = final: prev: {
+        nix-eval-jobs = final.callPackage nix-eval-jobs {};
        hydra = final.callPackage ./package.nix {
          inherit (nixpkgs.lib) fileset;
-          nix-eval-jobs = nix-eval-jobs.packages.${final.system}.default;
          rawSrc = self;
          nix-perl-bindings = final.nixComponents.nix-perl-bindings;
        };
@@ -74,11 +74,19 @@
      });

      packages = forEachSystem (system: {
+        nix-eval-jobs = nixpkgs.legacyPackages.${system}.callPackage nix-eval-jobs {
+          nix = nix.packages.${system}.nix;
+        };
        hydra = nixpkgs.legacyPackages.${system}.callPackage ./package.nix {
          inherit (nixpkgs.lib) fileset;
-          nix-eval-jobs = nix-eval-jobs.packages.${system}.default;
+          inherit (self.packages.${system}) nix-eval-jobs;
          rawSrc = self;
-          nix = nix.packages.${system}.nix;
+          inherit (nix.packages.${system})
+            nix-util
+            nix-store
+            nix-main
+            nix-cli
+            ;
          nix-perl-bindings = nix.hydraJobs.perlBindings.${system};
        };
        default = self.packages.${system}.hydra;
--- a/meson.build
+++ b/meson.build
@@ -8,22 +8,22 @@ project('hydra', 'cpp',
  ],
 )

+nix_util_dep = dependency('nix-util', required: true)
 nix_store_dep = dependency('nix-store', required: true)
 nix_main_dep = dependency('nix-main', required: true)
-nix_expr_dep = dependency('nix-expr', required: true)
-nix_flake_dep = dependency('nix-flake', required: true)
-nix_cmd_dep = dependency('nix-cmd', required: true)

 # Nix need extra flags not provided in its pkg-config files.
 nix_dep = declare_dependency(
  dependencies: [
+    nix_util_dep,
    nix_store_dep,
    nix_main_dep,
-    nix_expr_dep,
-    nix_flake_dep,
-    nix_cmd_dep,
  ],
-  compile_args: ['-include', 'nix/config.h'],
+  compile_args: [
+    '-include', 'nix/config-util.hh',
+    '-include', 'nix/config-store.hh',
+    '-include', 'nix/config-main.hh',
+  ],
 )

 pqxx_dep = dependency('libpqxx', required: true)
--- a/nixos-modules/default.nix
+++ b/nixos-modules/default.nix
@@ -15,7 +15,6 @@
    systemd.services.hydra-send-stats.enable = false;

    services.postgresql.enable = true;
-    services.postgresql.package = pkgs.postgresql_12;

    # The following is to work around the following error from hydra-server:
    #   [error] Caught exception in engine "Cannot determine local time zone"
--- a/nixos-modules/hydra.nix
+++ b/nixos-modules/hydra.nix
@@ -468,7 +468,7 @@ in
            elif [[ $compression == zstd ]]; then
              compression="zstd --rm"
            fi
-            find ${baseDir}/build-logs -type f -name "*.drv" -mtime +3 -size +0c | xargs -r "$compression" --force --quiet
+            find ${baseDir}/build-logs -ignore_readdir_race -type f -name "*.drv" -mtime +3 -size +0c | xargs -r "$compression" --force --quiet
          '';
        startAt = "Sun 01:45";
      };
--- a/package.nix
+++ b/package.nix
@@ -8,7 +8,10 @@

 , perlPackages

-, nix
+, nix-util
+, nix-store
+, nix-main
+, nix-cli
 , nix-perl-bindings
 , git

@@ -162,7 +165,7 @@ stdenv.mkDerivation (finalAttrs: {
    nukeReferences
    pkg-config
    mdbook
-    nix
+    nix-cli
    perlDeps
    perl
    unzip
@@ -172,7 +175,9 @@ stdenv.mkDerivation (finalAttrs: {
    libpqxx
    openssl
    libxslt
-    nix
+    nix-util
+    nix-store
+    nix-main
    perlDeps
    perl
    boost
@@ -199,13 +204,14 @@ stdenv.mkDerivation (finalAttrs: {
    glibcLocales
    libressl.nc
    python3
+    nix-cli
  ];

  hydraPath = lib.makeBinPath (
    [
      subversion
      openssh
-      nix
+      nix-cli
      coreutils
      findutils
      pixz
@@ -235,7 +241,7 @@ stdenv.mkDerivation (finalAttrs: {
  shellHook = ''
    pushd $(git rev-parse --show-toplevel) >/dev/null

-    PATH=$(pwd)/src/hydra-evaluator:$(pwd)/src/script:$(pwd)/src/hydra-queue-runner:$PATH
+    PATH=$(pwd)/build/src/hydra-evaluator:$(pwd)/build/src/script:$(pwd)/build/src/hydra-queue-runner:$PATH
    PERL5LIB=$(pwd)/src/lib:$PERL5LIB
    export HYDRA_HOME="$(pwd)/src/"
    mkdir -p .hydra-data
@@ -266,7 +272,7 @@ stdenv.mkDerivation (finalAttrs: {
            --prefix PATH ':' $out/bin:$hydraPath \
            --set HYDRA_RELEASE ${version} \
            --set HYDRA_HOME $out/libexec/hydra \
-            --set NIX_RELEASE ${nix.name or "unknown"} \
+            --set NIX_RELEASE ${nix-cli.name or "unknown"} \
            --set NIX_EVAL_JOBS_RELEASE ${nix-eval-jobs.name or "unknown"}
    done
  '';
@@ -274,5 +280,5 @@ stdenv.mkDerivation (finalAttrs: {
  dontStrip = true;

  meta.description = "Build of Hydra on ${stdenv.system}";
-  passthru = { inherit perlDeps nix; };
+  passthru = { inherit perlDeps; };
 })
--- a/src/hydra-queue-runner/build-remote.cc
+++ b/src/hydra-queue-runner/build-remote.cc
@@ -7,6 +7,7 @@

 #include "build-result.hh"
 #include "path.hh"
+#include "legacy-ssh-store.hh"
 #include "serve-protocol.hh"
 #include "serve-protocol-impl.hh"
 #include "state.hh"
@@ -21,28 +22,23 @@

 using namespace nix;

-namespace nix::build_remote {
-
-static Strings extraStoreArgs(std::string & machine)
+bool ::Machine::isLocalhost() const
 {
-    Strings result;
-    try {
-        auto parsed = parseURL(machine);
-        if (parsed.scheme != "ssh") {
-            throw SysError("Currently, only (legacy-)ssh stores are supported!");
-        }
-        machine = parsed.authority.value_or("");
-        auto remoteStore = parsed.query.find("remote-store");
-        if (remoteStore != parsed.query.end()) {
-            result = {"--store", shellEscape(remoteStore->second)};
-        }
-    } catch (BadURL &) {
-        // We just try to continue with `machine->sshName` here for backwards compat.
-    }
-
-    return result;
+    return storeUri.params.empty() && std::visit(overloaded {
+        [](const StoreReference::Auto &) {
+            return true;
+        },
+        [](const StoreReference::Specified & s) {
+            return
+               (s.scheme == "local" || s.scheme == "unix") ||
+               ((s.scheme == "ssh" || s.scheme == "ssh-ng") &&
+                s.authority == "localhost");
+        },
+    }, storeUri.variant);
 }

+namespace nix::build_remote {
+
 static std::unique_ptr<SSHMaster::Connection> openConnection(
    ::Machine::ptr machine, SSHMaster & master)
 {
@@ -51,7 +47,11 @@ static std::unique_ptr<SSHMaster::Connection> openConnection(
        command.push_back("--builders");
        command.push_back("");
    } else {
-        command.splice(command.end(), extraStoreArgs(machine->sshName));
+        auto remoteStore = machine->storeUri.params.find("remote-store");
+        if (remoteStore != machine->storeUri.params.end()) {
+            command.push_back("--store");
+            command.push_back(shellEscape(remoteStore->second));
+        }
    }

    auto ret = master.startCommand(std::move(command), {
@@ -198,7 +198,7 @@ static BasicDerivation sendInputs(
        MaintainCount<counter> mc2(nrStepsCopyingTo);

        printMsg(lvlDebug, "sending closure of ‘%s’ to ‘%s’",
-            localStore.printStorePath(step.drvPath), conn.machine->sshName);
+            localStore.printStorePath(step.drvPath), conn.machine->storeUri.render());

        auto now1 = std::chrono::steady_clock::now();

@@ -278,32 +278,6 @@ static BuildResult performBuild(
    return result;
 }

-static std::map<StorePath, UnkeyedValidPathInfo> queryPathInfos(
-    ::Machine::Connection & conn,
-    Store & localStore,
-    StorePathSet & outputs,
-    size_t & totalNarSize
-)
-{
-
-    /* Get info about each output path. */
-    std::map<StorePath, UnkeyedValidPathInfo> infos;
-    conn.to << ServeProto::Command::QueryPathInfos;
-    ServeProto::write(localStore, conn, outputs);
-    conn.to.flush();
-    while (true) {
-        auto storePathS = readString(conn.from);
-        if (storePathS == "") break;
-
-        auto storePath = localStore.parseStorePath(storePathS);
-        auto info = ServeProto::Serialise<UnkeyedValidPathInfo>::read(localStore, conn);
-        totalNarSize += info.narSize;
-        infos.insert_or_assign(std::move(storePath), std::move(info));
-    }
-
-    return infos;
-}
-
 static void copyPathFromRemote(
    ::Machine::Connection & conn,
    NarMemberDatas & narMembers,
@@ -430,15 +404,23 @@ void State::buildRemote(ref<Store> destStore,

        updateStep(ssConnecting);

-        SSHMaster master {
-            machine->sshName,
-            machine->sshKey,
-            machine->sshPublicHostKey,
-            false, // no SSH master yet
-            false, // no compression yet
-            logFD.get(),
+        auto storeRef = machine->completeStoreReference();
+
+        auto * pSpecified = std::get_if<StoreReference::Specified>(&storeRef.variant);
+        if (!pSpecified || pSpecified->scheme != "ssh") {
+            throw Error("Currently, only (legacy-)ssh stores are supported!");
+        }
+
+        LegacySSHStoreConfig storeConfig {
+            pSpecified->scheme,
+            pSpecified->authority,
+            storeRef.params
        };

+        auto master = storeConfig.createSSHMaster(
+            false, // no SSH master yet
+            logFD.get());
+
        // FIXME: rewrite to use Store.
        auto child = build_remote::openConnection(machine, master);

@@ -482,19 +464,13 @@ void State::buildRemote(ref<Store> destStore,
                conn.to,
                conn.from,
                our_version,
-                machine->sshName);
+                machine->storeUri.render());
        } catch (EndOfFile & e) {
            child->sshPid.wait();
            std::string s = chomp(readFile(result.logFile));
-            throw Error("cannot connect to ‘%1%’: %2%", machine->sshName, s);
+            throw Error("cannot connect to ‘%1%’: %2%", machine->storeUri.render(), s);
        }

-        // Do not attempt to speak a newer version of the protocol.
-        //
-        // Per https://github.com/NixOS/nix/issues/9584 should be handled as
-        // part of `handshake` in upstream nix.
-        conn.remoteVersion = std::min(conn.remoteVersion, our_version);
-
        {
            auto info(machine->state->connectInfo.lock());
            info->consecutiveFailures = 0;
@@ -523,7 +499,7 @@ void State::buildRemote(ref<Store> destStore,
        /* Do the build. */
        printMsg(lvlDebug, "building ‘%s’ on ‘%s’",
            localStore->printStorePath(step->drvPath),
-            machine->sshName);
+            machine->storeUri.render());

        updateStep(ssBuilding);

@@ -546,7 +522,7 @@ void State::buildRemote(ref<Store> destStore,
           get a build log. */
        if (result.isCached) {
            printMsg(lvlInfo, "outputs of ‘%s’ substituted or already valid on ‘%s’",
-                localStore->printStorePath(step->drvPath), machine->sshName);
+                localStore->printStorePath(step->drvPath), machine->storeUri.render());
            unlink(result.logFile.c_str());
            result.logFile = "";
        }
@@ -563,8 +539,10 @@ void State::buildRemote(ref<Store> destStore,

            auto now1 = std::chrono::steady_clock::now();

+            auto infos = conn.queryPathInfos(*localStore, outputs);
+
            size_t totalNarSize = 0;
-            auto infos = build_remote::queryPathInfos(conn, *localStore, outputs, totalNarSize);
+            for (auto & [_, info] : infos) totalNarSize += info.narSize;

            if (totalNarSize > maxOutputSize) {
                result.stepStatus = bsNarSizeLimitExceeded;
@@ -573,7 +551,7 @@ void State::buildRemote(ref<Store> destStore,

            /* Copy each path. */
            printMsg(lvlDebug, "copying outputs of ‘%s’ from ‘%s’ (%d bytes)",
-                localStore->printStorePath(step->drvPath), machine->sshName, totalNarSize);
+                localStore->printStorePath(step->drvPath), machine->storeUri.render(), totalNarSize);

            build_remote::copyPathsFromRemote(conn, narMembers, *localStore, *destStore, infos);
            auto now2 = std::chrono::steady_clock::now();
@@ -612,7 +590,7 @@ void State::buildRemote(ref<Store> destStore,
            info->consecutiveFailures = std::min(info->consecutiveFailures + 1, (unsigned int) 4);
            info->lastFailure = now;
            int delta = retryInterval * std::pow(retryBackoff, info->consecutiveFailures - 1) + (rand() % 30);
-            printMsg(lvlInfo, "will disable machine ‘%1%’ for %2%s", machine->sshName, delta);
+            printMsg(lvlInfo, "will disable machine ‘%1%’ for %2%s", machine->storeUri.render(), delta);
            info->disabledUntil = now + std::chrono::seconds(delta);
        }
        throw;
--- a/src/hydra-queue-runner/builder.cc
+++ b/src/hydra-queue-runner/builder.cc
@@ -41,7 +41,7 @@ void State::builder(MachineReservation::ptr reservation)
        } catch (std::exception & e) {
            printMsg(lvlError, "uncaught exception building ‘%s’ on ‘%s’: %s",
                localStore->printStorePath(reservation->step->drvPath),
-                reservation->machine->sshName,
+                reservation->machine->storeUri.render(),
                e.what());
        }
    }
@@ -150,7 +150,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
        buildOptions.buildTimeout = build->buildTimeout;

        printInfo("performing step ‘%s’ %d times on ‘%s’ (needed by build %d and %d others)",
-            localStore->printStorePath(step->drvPath), buildOptions.nrRepeats + 1, machine->sshName, buildId, (dependents.size() - 1));
+            localStore->printStorePath(step->drvPath), buildOptions.nrRepeats + 1, machine->storeUri.render(), buildId, (dependents.size() - 1));
    }

    if (!buildOneDone)
@@ -178,7 +178,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
                    unlink(result.logFile.c_str());
                }
            } catch (...) {
-                ignoreException();
+                ignoreExceptionInDestructor();
            }
        }
    });
@@ -196,7 +196,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
        {
            auto mc = startDbUpdate();
            pqxx::work txn(*conn);
-            stepNr = createBuildStep(txn, result.startTime, buildId, step, machine->sshName, bsBusy);
+            stepNr = createBuildStep(txn, result.startTime, buildId, step, machine->storeUri.render(), bsBusy);
            txn.commit();
        }

@@ -253,7 +253,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
    /* Finish the step in the database. */
    if (stepNr) {
        pqxx::work txn(*conn);
-        finishBuildStep(txn, result, buildId, stepNr, machine->sshName);
+        finishBuildStep(txn, result, buildId, stepNr, machine->storeUri.render());
        txn.commit();
    }

@@ -261,7 +261,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore,
       issue). Retry a number of times. */
    if (result.canRetry) {
        printMsg(lvlError, "possibly transient failure building ‘%s’ on ‘%s’: %s",
-            localStore->printStorePath(step->drvPath), machine->sshName, result.errorMsg);
+            localStore->printStorePath(step->drvPath), machine->storeUri.render(), result.errorMsg);
        assert(stepNr);
        bool retry;
        {
@@ -452,7 +452,7 @@ void State::failStep(
                    build->finishedInDB)
                    continue;
                createBuildStep(txn,
-                    0, build->id, step, machine ? machine->sshName : "",
+                    0, build->id, step, machine ? machine->storeUri.render() : "",
                    result.stepStatus, result.errorMsg, buildId == build->id ? 0 : buildId);
            }

--- a/src/hydra-queue-runner/dispatcher.cc
+++ b/src/hydra-queue-runner/dispatcher.cc
@@ -256,7 +256,7 @@ system_time State::doDispatch()
                /* Can this machine do this step? */
                if (!mi.machine->supportsStep(step)) {
                    debug("machine '%s' does not support step '%s' (system type '%s')",
-                        mi.machine->sshName, localStore->printStorePath(step->drvPath), step->drv->platform);
+                        mi.machine->storeUri.render(), localStore->printStorePath(step->drvPath), step->drv->platform);
                    continue;
                }

--- a/src/hydra-queue-runner/hydra-queue-runner.cc
+++ b/src/hydra-queue-runner/hydra-queue-runner.cc
@@ -135,65 +135,26 @@ void State::parseMachines(const std::string & contents)
        oldMachines = *machines_;
    }

-    for (auto line : tokenizeString<Strings>(contents, "\n")) {
-        line = trim(std::string(line, 0, line.find('#')));
-        auto tokens = tokenizeString<std::vector<std::string>>(line);
-        if (tokens.size() < 3) continue;
-        tokens.resize(8);
-
-        if (tokens[5] == "-") tokens[5] = "";
-        auto supportedFeatures = tokenizeString<StringSet>(tokens[5], ",");
-
-        if (tokens[6] == "-") tokens[6] = "";
-        auto mandatoryFeatures = tokenizeString<StringSet>(tokens[6], ",");
-
-        for (auto & f : mandatoryFeatures)
-            supportedFeatures.insert(f);
-
-        using MaxJobs = std::remove_const<decltype(nix::Machine::maxJobs)>::type;
-
-        auto machine = std::make_shared<::Machine>(nix::Machine {
-            // `storeUri`, not yet used
-            "",
-            // `systemTypes`
-            tokenizeString<StringSet>(tokens[1], ","),
-            // `sshKey`
-            tokens[2] == "-" ? "" : tokens[2],
-            // `maxJobs`
-            tokens[3] != ""
-                ? string2Int<MaxJobs>(tokens[3]).value()
-                : 1,
-            // `speedFactor`
-            std::stof(tokens[4].c_str()),
-            // `supportedFeatures`
-            std::move(supportedFeatures),
-            // `mandatoryFeatures`
-            std::move(mandatoryFeatures),
-            // `sshPublicHostKey`
-            tokens[7] != "" && tokens[7] != "-"
-                ? tokens[7]
-                : "",
-        });
-
-        machine->sshName = tokens[0];
+    for (auto && machine_ : nix::Machine::parseConfig({}, contents)) {
+        auto machine = std::make_shared<::Machine>(std::move(machine_));

        /* Re-use the State object of the previous machine with the
           same name. */
-        auto i = oldMachines.find(machine->sshName);
+        auto i = oldMachines.find(machine->storeUri.variant);
        if (i == oldMachines.end())
-            printMsg(lvlChatty, "adding new machine ‘%1%’", machine->sshName);
+            printMsg(lvlChatty, "adding new machine ‘%1%’", machine->storeUri.render());
        else
-            printMsg(lvlChatty, "updating machine ‘%1%’", machine->sshName);
+            printMsg(lvlChatty, "updating machine ‘%1%’", machine->storeUri.render());
        machine->state = i == oldMachines.end()
            ? std::make_shared<::Machine::State>()
            : i->second->state;
-        newMachines[machine->sshName] = machine;
+        newMachines[machine->storeUri.variant] = machine;
    }

    for (auto & m : oldMachines)
        if (newMachines.find(m.first) == newMachines.end()) {
            if (m.second->enabled)
-                printInfo("removing machine ‘%1%’", m.first);
+                printInfo("removing machine ‘%1%’", m.second->storeUri.render());
            /* Add a disabled ::Machine object to make sure stats are
               maintained. */
            auto machine = std::make_shared<::Machine>(*(m.second));
@@ -657,7 +618,7 @@ void State::dumpStatus(Connection & conn)
                    machine["avgStepTime"] = (float) s->totalStepTime / s->nrStepsDone;
                    machine["avgStepBuildTime"] = (float) s->totalStepBuildTime / s->nrStepsDone;
                }
-                statusJson["machines"][m->sshName] = machine;
+                statusJson["machines"][m->storeUri.render()] = machine;
            }
        }

--- a/src/hydra-queue-runner/queue-monitor.cc
+++ b/src/hydra-queue-runner/queue-monitor.cc
@@ -1,6 +1,7 @@
 #include "state.hh"
 #include "hydra-build-result.hh"
 #include "globals.hh"
+#include "parsed-derivations.hh"

 #include <cstring>

@@ -463,14 +464,17 @@ Step::ptr State::createStep(ref<Store> destStore,
       it's not runnable yet, and other threads won't make it
       runnable while step->created == false. */
    step->drv = std::make_unique<Derivation>(localStore->readDerivation(drvPath));
-    step->parsedDrv = std::make_unique<ParsedDerivation>(drvPath, *step->drv);
+    {
+        auto parsedDrv = ParsedDerivation{drvPath, *step->drv};
+        step->drvOptions = std::make_unique<DerivationOptions>(DerivationOptions::fromParsedDerivation(parsedDrv));
+    }

-    step->preferLocalBuild = step->parsedDrv->willBuildLocally(*localStore);
+    step->preferLocalBuild = step->drvOptions->willBuildLocally(*localStore, *step->drv);
    step->isDeterministic = getOr(step->drv->env, "isDetermistic", "0") == "1";

    step->systemType = step->drv->platform;
    {
-        StringSet features = step->requiredSystemFeatures = step->parsedDrv->getRequiredSystemFeatures();
+        StringSet features = step->requiredSystemFeatures = step->drvOptions->getRequiredSystemFeatures(*step->drv);
        if (step->preferLocalBuild)
            features.insert("local");
        if (!features.empty()) {
--- a/src/hydra-queue-runner/state.hh
+++ b/src/hydra-queue-runner/state.hh
@@ -6,7 +6,6 @@
 #include <map>
 #include <memory>
 #include <queue>
-#include <regex>

 #include <prometheus/counter.h>
 #include <prometheus/gauge.h>
@@ -14,7 +13,8 @@

 #include "db.hh"

-#include "parsed-derivations.hh"
+#include "derivations.hh"
+#include "derivation-options.hh"
 #include "pathlocks.hh"
 #include "pool.hh"
 #include "build-result.hh"
@@ -168,7 +168,7 @@ struct Step

    nix::StorePath drvPath;
    std::unique_ptr<nix::Derivation> drv;
-    std::unique_ptr<nix::ParsedDerivation> parsedDrv;
+    std::unique_ptr<nix::DerivationOptions> drvOptions;
    std::set<std::string> requiredSystemFeatures;
    bool preferLocalBuild;
    bool isDeterministic;
@@ -241,10 +241,6 @@ struct Machine : nix::Machine
 {
    typedef std::shared_ptr<Machine> ptr;

-    /* TODO Get rid of: `nix::Machine::storeUri` is normalized in a way
-       we are not yet used to, but once we are, we don't need this. */
-    std::string sshName;
-
    struct State {
        typedef std::shared_ptr<State> ptr;
        counter currentJobs{0};
@@ -294,11 +290,7 @@ struct Machine : nix::Machine
        return true;
    }

-    bool isLocalhost()
-    {
-        std::regex r("^(ssh://|ssh-ng://)?localhost$");
-        return std::regex_search(sshName, r);
-    }
+    bool isLocalhost() const;

    // A connection to a machine
    struct Connection : nix::ServeProto::BasicClientConnection {
@@ -358,7 +350,7 @@ private:

    /* The build machines. */
    std::mutex machinesReadyLock;
-    typedef std::map<std::string, Machine::ptr> Machines;
+    typedef std::map<nix::StoreReference::Variant, Machine::ptr> Machines;
    nix::Sync<Machines> machines; // FIXME: use atomic_shared_ptr

    /* Various stats. */
--- a/src/lib/Hydra/View/TT.pm
+++ b/src/lib/Hydra/View/TT.pm
@@ -6,6 +6,7 @@ use base 'Catalyst::View::TT';
 use Template::Plugin::HTML;
 use Hydra::Helper::Nix;
 use Time::Seconds;
+use Digest::SHA qw(sha1_hex);

 __PACKAGE__->config(
    TEMPLATE_EXTENSION => '.tt',
@@ -25,8 +26,14 @@ __PACKAGE__->config(
    makeNameTextForJobset
    relativeDuration
    stripSSHUser
+    metricDivId
    /]);

+sub metricDivId {
+    my ($self, $c, $text) = @_;
+    return "metric-" . sha1_hex($text);
+}
+
 sub buildLogExists {
    my ($self, $c, $build) = @_;
    return 1 if defined $c->config->{log_prefix};
--- a/src/root/job-metrics-tab.tt
+++ b/src/root/job-metrics-tab.tt
@@ -18,8 +18,7 @@

    <h3>Metric: <a [% HTML.attributes(href => c.uri_for('/job' project.name jobset.name job 'metric' metric.name)) %]><tt>[%HTML.escape(metric.name)%]</tt></a></h3>

-    [% id = "metric-" _ metric.name;
-       id = id.replace('\.', '_');
+    [% id = metricDivId(metric.name);
       INCLUDE createChart dataUrl=c.uri_for('/job' project.name jobset.name job 'metric' metric.name); %]

  [% END %]
--- a/src/script/hydra-eval-jobset
+++ b/src/script/hydra-eval-jobset
@@ -773,6 +773,9 @@ sub checkJobsetWrapped {
    my $jobsetChanged = 0;
    my %buildMap;

+    my @jobs;
+    push @jobs, $_ while defined($_ = $jobsIter->());
+
    $db->txn_do(sub {
        my $prevEval = getPrevJobsetEval($db, $jobset, 1);

@@ -796,7 +799,7 @@ sub checkJobsetWrapped {

        my @jobsWithConstituents;

-        while (defined(my $job = $jobsIter->())) {
+        foreach my $job (@jobs) {
            if ($jobsetsJobset) {
                die "The .jobsets jobset must only have a single job named 'jobsets'"
                    unless $job->{attr} eq "jobsets";
--- a/t/evaluator/evaluate-constituents-broken.t
+++ b/t/evaluator/evaluate-constituents-broken.t
@@ -6,27 +6,55 @@ use Hydra::Helper::Exec;

 my $ctx = test_context();

-my $jobsetCtx = $ctx->makeJobset(
-    expression => 'constituents-broken.nix',
-);
-my $jobset = $jobsetCtx->{"jobset"};
+subtest "broken constituents expression" => sub {
+    my $jobsetCtx = $ctx->makeJobset(
+        expression => 'constituents-broken.nix',
+    );
+    my $jobset = $jobsetCtx->{"jobset"};

-my ($res, $stdout, $stderr) = captureStdoutStderr(60,
-    ("hydra-eval-jobset", $jobsetCtx->{"project"}->name, $jobset->name)
-);
-isnt($res, 0, "hydra-eval-jobset exits non-zero");
-ok(utf8::decode($stderr), "Stderr output is UTF8-clean");
-like(
-    $stderr,
-    qr/aggregate job ‘mixed_aggregate’ failed with the error: "constituentA": does not exist/,
-    "The stderr record includes a relevant error message"
-);
+    my ($res, $stdout, $stderr) = captureStdoutStderr(60,
+        ("hydra-eval-jobset", $jobsetCtx->{"project"}->name, $jobset->name)
+    );
+    isnt($res, 0, "hydra-eval-jobset exits non-zero");
+    ok(utf8::decode($stderr), "Stderr output is UTF8-clean");
+    like(
+        $stderr,
+        qr/aggregate job 'mixed_aggregate' references non-existent job 'constituentA'/,
+        "The stderr record includes a relevant error message"
+    );

-$jobset->discard_changes({ '+columns' => {'errormsg' => 'errormsg'} });  # refresh from DB
-like(
-    $jobset->errormsg,
-    qr/aggregate job ‘mixed_aggregate’ failed with the error: "constituentA": does not exist/,
-    "The jobset records a relevant error message"
-);
+    $jobset->discard_changes({ '+columns' => {'errormsg' => 'errormsg'} });  # refresh from DB
+    like(
+        $jobset->errormsg,
+        qr/aggregate job ‘mixed_aggregate’ failed with the error: constituentA: does not exist/,
+        "The jobset records a relevant error message"
+    );
+};
+
+subtest "no matches" => sub {
+    my $jobsetCtx = $ctx->makeJobset(
+        expression => 'constituents-no-matches.nix',
+    );
+    my $jobset = $jobsetCtx->{"jobset"};
+
+    my ($res, $stdout, $stderr) = captureStdoutStderr(60,
+        ("hydra-eval-jobset", $jobsetCtx->{"project"}->name, $jobset->name)
+    );
+    isnt($res, 0, "hydra-eval-jobset exits non-zero");
+    ok(utf8::decode($stderr), "Stderr output is UTF8-clean");
+    like(
+        $stderr,
+        qr/aggregate job 'non_match_aggregate' references constituent glob pattern 'tests\.\*' with no matches/,
+        "The stderr record includes a relevant error message"
+    );
+
+    $jobset->discard_changes({ '+columns' => {'errormsg' => 'errormsg'} });  # refresh from DB
+    like(
+        $jobset->errormsg,
+        qr/aggregate job ‘non_match_aggregate’ failed with the error: tests\.\*: constituent glob pattern had no matches/,
+        qr/in job ‘non_match_aggregate’:\ntests\.\*: constituent glob pattern had no matches/,
+        "The jobset records a relevant error message"
+    );
+};

 done_testing;
--- a/t/evaluator/evaluate-constituents-globbing.t
+++ b/t/evaluator/evaluate-constituents-globbing.t
@@ -0,0 +1,138 @@
+use strict;
+use warnings;
+use Setup;
+use Test2::V0;
+use Hydra::Helper::Exec;
+use Data::Dumper;
+
+my $ctx = test_context();
+
+subtest "general glob testing" => sub {
+    my $jobsetCtx = $ctx->makeJobset(
+        expression => 'constituents-glob.nix',
+    );
+    my $jobset = $jobsetCtx->{"jobset"};
+
+    my ($res, $stdout, $stderr) = captureStdoutStderr(60,
+        ("hydra-eval-jobset", $jobsetCtx->{"project"}->name, $jobset->name)
+    );
+    is($res, 0, "hydra-eval-jobset exits zero");
+
+    my $builds = {};
+    for my $build ($jobset->builds) {
+        $builds->{$build->job} = $build;
+    }
+
+    subtest "basic globbing works" => sub {
+        ok(defined $builds->{"ok_aggregate"}, "'ok_aggregate' is part of the jobset evaluation");
+        my @constituents = $builds->{"ok_aggregate"}->constituents->all;
+        is(2, scalar @constituents, "'ok_aggregate' has two constituents");
+
+        my @sortedConstituentNames = sort (map { $_->nixname } @constituents);
+
+        is($sortedConstituentNames[0], "empty-dir-A", "first constituent of 'ok_aggregate' is 'empty-dir-A'");
+        is($sortedConstituentNames[1], "empty-dir-B", "second constituent of 'ok_aggregate' is 'empty-dir-B'");
+    };
+
+    subtest "transitivity is OK" => sub {
+        ok(defined $builds->{"indirect_aggregate"}, "'indirect_aggregate' is part of the jobset evaluation");
+        my @constituents = $builds->{"indirect_aggregate"}->constituents->all;
+        is(1, scalar @constituents, "'indirect_aggregate' has one constituent");
+        is($constituents[0]->nixname, "direct_aggregate", "'indirect_aggregate' has 'direct_aggregate' as single constituent");
+    };
+};
+
+subtest "* selects all except current aggregate" => sub {
+    my $jobsetCtx = $ctx->makeJobset(
+        expression => 'constituents-glob-all.nix',
+    );
+    my $jobset = $jobsetCtx->{"jobset"};
+
+    my ($res, $stdout, $stderr) = captureStdoutStderr(60,
+        ("hydra-eval-jobset", $jobsetCtx->{"project"}->name, $jobset->name)
+    );
+
+    subtest "no eval errors" => sub {
+        ok(utf8::decode($stderr), "Stderr output is UTF8-clean");
+        ok(
+            $stderr !~ "aggregate job ‘ok_aggregate’ has a constituent .* that doesn't correspond to a Hydra build",
+            "Catchall wildcard must not select itself as constituent"
+        );
+
+        $jobset->discard_changes;  # refresh from DB
+        is(
+            $jobset->errormsg,
+            "",
+            "eval-errors non-empty"
+        );
+    };
+
+    my $builds = {};
+    for my $build ($jobset->builds) {
+        $builds->{$build->job} = $build;
+    }
+
+    subtest "two constituents" => sub {
+        ok(defined $builds->{"ok_aggregate"}, "'ok_aggregate' is part of the jobset evaluation");
+        my @constituents = $builds->{"ok_aggregate"}->constituents->all;
+        is(2, scalar @constituents, "'ok_aggregate' has two constituents");
+
+        my @sortedConstituentNames = sort (map { $_->nixname } @constituents);
+
+        is($sortedConstituentNames[0], "empty-dir-A", "first constituent of 'ok_aggregate' is 'empty-dir-A'");
+        is($sortedConstituentNames[1], "empty-dir-B", "second constituent of 'ok_aggregate' is 'empty-dir-B'");
+    };
+};
+
+subtest "trivial cycle check" => sub {
+    my $jobsetCtx = $ctx->makeJobset(
+        expression => 'constituents-cycle.nix',
+    );
+    my $jobset = $jobsetCtx->{"jobset"};
+
+    my ($res, $stdout, $stderr) = captureStdoutStderr(60,
+        ("hydra-eval-jobset", $jobsetCtx->{"project"}->name, $jobset->name)
+    );
+
+    ok(
+        $stderr =~ "Found dependency cycle between jobs 'indirect_aggregate' and 'ok_aggregate'",
+        "Dependency cycle error is on stderr"
+    );
+
+    ok(utf8::decode($stderr), "Stderr output is UTF8-clean");
+
+    $jobset->discard_changes;  # refresh from DB
+    like(
+        $jobset->errormsg,
+        qr/Dependency cycle: indirect_aggregate <-> ok_aggregate/,
+        "eval-errors non-empty"
+    );
+
+    is(0, $jobset->builds->count, "No builds should be scheduled");
+};
+
+subtest "cycle check with globbing" => sub {
+    my $jobsetCtx = $ctx->makeJobset(
+        expression => 'constituents-cycle-glob.nix',
+    );
+    my $jobset = $jobsetCtx->{"jobset"};
+
+    my ($res, $stdout, $stderr) = captureStdoutStderr(60,
+        ("hydra-eval-jobset", $jobsetCtx->{"project"}->name, $jobset->name)
+    );
+
+    ok(utf8::decode($stderr), "Stderr output is UTF8-clean");
+
+    $jobset->discard_changes;  # refresh from DB
+    like(
+        $jobset->errormsg,
+        qr/aggregate job ‘indirect_aggregate’ failed with the error: Dependency cycle: indirect_aggregate <-> packages.constituentA/,
+        "packages.constituentA error missing"
+    );
+
+    # on this branch of Hydra, hydra-eval-jobset fails hard if an aggregate
+    # job is broken.
+    is(0, $jobset->builds->count, "Zero jobs are scheduled");
+};
+
+done_testing;
--- a/t/jobs/config.nix
+++ b/t/jobs/config.nix
@@ -0,0 +1,14 @@
+rec {
+  path = "/nix/store/l9mg93sgx50y88p5rr6x1vib6j1rjsds-coreutils-9.1/bin";
+
+  mkDerivation = args:
+    derivation ({
+      system = builtins.currentSystem;
+      PATH = path;
+    } // args);
+  mkContentAddressedDerivation = args: mkDerivation ({
+    __contentAddressed = true;
+    outputHashMode = "recursive";
+    outputHashAlgo = "sha256";
+  } // args);
+}
--- a/t/jobs/constituents-cycle-glob.nix
+++ b/t/jobs/constituents-cycle-glob.nix
@@ -0,0 +1,34 @@
+with import ./config.nix;
+{
+  packages.constituentA = mkDerivation {
+    name = "empty-dir-A";
+    builder = ./empty-dir-builder.sh;
+    _hydraAggregate = true;
+    _hydraGlobConstituents = true;
+    constituents = [ "*_aggregate" ];
+  };
+
+  packages.constituentB = mkDerivation {
+    name = "empty-dir-B";
+    builder = ./empty-dir-builder.sh;
+  };
+
+  ok_aggregate = mkDerivation {
+    name = "direct_aggregate";
+    _hydraAggregate = true;
+    _hydraGlobConstituents = true;
+    constituents = [
+      "packages.*"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+
+  indirect_aggregate = mkDerivation {
+    name = "indirect_aggregate";
+    _hydraAggregate = true;
+    constituents = [
+      "ok_aggregate"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+}
--- a/t/jobs/constituents-cycle.nix
+++ b/t/jobs/constituents-cycle.nix
@@ -0,0 +1,21 @@
+with import ./config.nix;
+{
+  ok_aggregate = mkDerivation {
+    name = "direct_aggregate";
+    _hydraAggregate = true;
+    _hydraGlobConstituents = true;
+    constituents = [
+      "indirect_aggregate"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+
+  indirect_aggregate = mkDerivation {
+    name = "indirect_aggregate";
+    _hydraAggregate = true;
+    constituents = [
+      "ok_aggregate"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+}
--- a/t/jobs/constituents-glob-all.nix
+++ b/t/jobs/constituents-glob-all.nix
@@ -0,0 +1,22 @@
+with import ./config.nix;
+{
+  packages.constituentA = mkDerivation {
+    name = "empty-dir-A";
+    builder = ./empty-dir-builder.sh;
+  };
+
+  packages.constituentB = mkDerivation {
+    name = "empty-dir-B";
+    builder = ./empty-dir-builder.sh;
+  };
+
+  ok_aggregate = mkDerivation {
+    name = "direct_aggregate";
+    _hydraAggregate = true;
+    _hydraGlobConstituents = true;
+    constituents = [
+      "*"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+}
--- a/t/jobs/constituents-glob.nix
+++ b/t/jobs/constituents-glob.nix
@@ -0,0 +1,31 @@
+with import ./config.nix;
+{
+  packages.constituentA = mkDerivation {
+    name = "empty-dir-A";
+    builder = ./empty-dir-builder.sh;
+  };
+
+  packages.constituentB = mkDerivation {
+    name = "empty-dir-B";
+    builder = ./empty-dir-builder.sh;
+  };
+
+  ok_aggregate = mkDerivation {
+    name = "direct_aggregate";
+    _hydraAggregate = true;
+    _hydraGlobConstituents = true;
+    constituents = [
+      "packages.*"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+
+  indirect_aggregate = mkDerivation {
+    name = "indirect_aggregate";
+    _hydraAggregate = true;
+    constituents = [
+      "ok_aggregate"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+}
--- a/t/jobs/constituents-no-matches.nix
+++ b/t/jobs/constituents-no-matches.nix
@@ -0,0 +1,20 @@
+with import ./config.nix;
+{
+  non_match_aggregate = mkDerivation {
+    name = "mixed_aggregate";
+    _hydraAggregate = true;
+    _hydraGlobConstituents = true;
+    constituents = [
+      "tests.*"
+    ];
+    builder = ./empty-dir-builder.sh;
+  };
+
+  # Without a second job no jobset is attempted to be created
+  # (the only job would be broken)
+  # and thus the constituent validation is never reached.
+  dummy = mkDerivation {
+    name = "dummy";
+    builder = ./empty-dir-builder.sh;
+  };
+}
--- a/t/jobs/declarative/project.json
+++ b/t/jobs/declarative/project.json
@@ -0,0 +1,24 @@
+{
+    "enabled": 1,
+    "hidden": false,
+    "description": "declarative-jobset-example",
+    "nixexprinput": "src",
+    "nixexprpath": "declarative/generator.nix",
+    "checkinterval": 300,
+    "schedulingshares": 100,
+    "enableemail": false,
+    "emailoverride": "",
+    "keepnr": 3,
+    "inputs": {
+        "src": {
+            "type": "path",
+            "value": "/home/ma27/Projects/hydra-cppnix/t/jobs",
+            "emailresponsible": false
+        },
+        "jobspath": {
+            "type": "string",
+            "value": "/home/ma27/Projects/hydra-cppnix/t/jobs",
+            "emailresponsible": false
+        }
+    }
+}
--- a/t/meson.build
+++ b/t/meson.build
@@ -27,6 +27,7 @@ testenv.prepend('PERL5LIB',
  separator: ':'
 )
 testenv.prepend('PATH',
+  fs.parent(find_program('nix').full_path()),
  fs.parent(hydra_evaluator.full_path()),
  fs.parent(hydra_queue_runner.full_path()),
  meson.project_source_root() / 'src/script',
--- a/t/queue-runner/constituents.t
+++ b/t/queue-runner/constituents.t
@@ -22,11 +22,11 @@ is(nrQueuedBuildsForJobset($jobset), 0, "Evaluating jobs/broken-constituent.nix

 like(
    $jobset->errormsg,
-    qr/^"does-not-exist": does not exist$/m,
+    qr/^does-not-exist: does not exist$/m,
    "Evaluating jobs/broken-constituent.nix should log an error for does-not-exist");
 like(
    $jobset->errormsg,
-    qr/^"does-not-evaluate": "error: assertion 'false' failed/m,
+    qr/^does-not-evaluate: error: assertion 'false' failed/m,
    "Evaluating jobs/broken-constituent.nix should log an error for does-not-evaluate");

 done_testing;
Author	SHA1	Message	Date
John Ericson	257b211832	Merge pull request #1460 from NixOS/nix-2.27 Nix 2.27	2025-04-07 11:37:43 -04:00
John Ericson	1cb1e139c4	Fix build (due to C++ API changes)	2025-04-07 11:12:12 -04:00
John Ericson	6b97e3ab7b	flake.lock: Update to nix and nix-eval-jobs 2.27 Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/e310c19a1aeb1ce1ed4d41d5ab2d02db596e0918?narHash=sha256-q/RgA4bB7zWai4oPySq9mch7qH14IEeom2P64SXdqHs%3D' (2025-02-18) → 'github:NixOS/nix/d0f98c76f962147610489e84c10033ca92e9c532?narHash=sha256-u6RhBWQ1XohTZ4Ub5ml1PTcaxQgtqFNng6Sohy1rojw%3D' (2025-04-07) • Updated input 'nix-eval-jobs': 'github:nix-community/nix-eval-jobs/f7418fc1fa45b96d37baa95ff3c016dd5be3876b?narHash=sha256-Lo4KFBNcY8tmBuCmEr2XV0IUZtxXHmbXPNLkov/QSU0%3D' (2025-03-26) → 'github:nix-community/nix-eval-jobs/62f9c9e8d00d2ff6ab27a6197ab459a8e0808e59?narHash=sha256-PypQspB7h7EENe4RQQUQj2Ay8J1%2BO49AKNO9JbAU4Ek%3D' (2025-04-07)	2025-04-07 11:02:52 -04:00
Jörg Thalheim	cad08f87d2	Merge pull request #1458 from NixOS/meson docs: fix contribution guide for new meson-based build	2025-03-29 15:37:06 +01:00
Jörg Thalheim	3fef32b364	gitignore hydra-data as created by foreman	2025-03-29 14:31:18 +00:00
Jörg Thalheim	ae18a7b3ae	fix development workflow after switching to meson-based build	2025-03-29 14:31:18 +00:00
Jörg Thalheim	b657bcdfb7	Merge pull request #1457 from dermetfan/fix-1429 hydra-eval-jobset: do not wait on n-e-j inside transaction	2025-03-29 11:36:13 +01:00
Jörg Thalheim	3b4c4972c2	Merge pull request #1449 from knedlsepp/fix-metrics-rendering-with-special-characters Fix rendering of metrics with special characters	2025-03-29 08:52:07 +01:00
John Ericson	b2fe3f5218	Merge pull request #1455 from qowoz/226-constituent nix-eval-jobs + constituent globs	2025-03-27 23:18:39 -04:00
Maximilian Bosch	9911f0107f	Reimplement (named) constituent jobs (+globbing) based on nix-eval-jobs Depends on https://github.com/nix-community/nix-eval-jobs/pull/349 & #1421. Almost equivalent to #1425, but with a small change: when having e.g. an aggregate job with a glob that matches nothing, the jobset evaluation is failed now. This was the intended behavior before (hydra-eval-jobset fails hard if an aggregate is broken), the code-path was never reached however since the aggregate was never marked as broken in this case before.	2025-03-28 11:12:54 +10:00
zowoq	feebb61897	flake.lock: Update Flake lock file updates: • Updated input 'nix-eval-jobs': 'github:nix-community/nix-eval-jobs/4b392b284877d203ae262e16af269f702df036bc?narHash=sha256-3wIReAqdTALv39gkWXLMZQvHyBOc3yPkWT2ZsItxedY%3D' (2025-02-14) → 'github:nix-community/nix-eval-jobs/f7418fc1fa45b96d37baa95ff3c016dd5be3876b?narHash=sha256-Lo4KFBNcY8tmBuCmEr2XV0IUZtxXHmbXPNLkov/QSU0%3D' (2025-03-26)	2025-03-28 11:12:54 +10:00
zowoq	4bcbed2f1b	hydraTest: remove outdated postgresql version error: postgresql_12 has been removed since it reached its EOL upstream	2025-03-28 11:12:48 +10:00
Robin Stumm	987dad3371	hydra-eval-jobset: do not wait on n-e-j inside transaction fixes #1429	2025-03-26 20:23:26 +01:00
John Ericson	d2db3c7446	Merge pull request #1450 from NixOS/hydra-compress-race Fix race condition in hydra-compress-logs	2025-03-16 14:37:39 -04:00
John Ericson	97dcdae068	Merge pull request #1451 from NixOS/revert-to-fix-hangs Revert "Use `LegacySSHStore`"	2025-03-03 10:18:28 -05:00
John Ericson	9a5bd39d4c	Revert "Use `LegacySSHStore`" There were some hangs caused by this. Need to fix them, ideally reproducing the issue in a test, before trying this again. This reverts commit `4a4a0f901c`.	2025-03-03 10:12:38 -05:00
Martin Weinelt	f1deb22c02	Fix race condition in hydra-compress-logs	2025-03-02 03:08:26 +01:00
Josef Kemetmüller	d22d030503	Fix rendering of metrics with special characters My main motivation here is to get metrics with brackets to work in order to support "pytest" test names: - test_foo.py::test_bar[1] - test_foo.py::test_bar[2] I couldn't find an "HTML escape"-style function that would generate valid html `id` attribute names from random strings, so I went with a hash digest instead.	2025-02-27 09:25:42 +01:00
John Ericson	18c0d76210	Merge pull request #1444 from NixOS/use-legacy-ssh-store Use `LegacySSHStore`	2025-02-18 14:37:17 -05:00
John Ericson	4a4a0f901c	Use `LegacySSHStore` In https://github.com/NixOS/nix/pull/10748 it is extended with everything we need.	2025-02-18 14:07:42 -05:00
John Ericson	881462bb4e	Merge pull request #1447 from NixOS/newer-2.6 Bump to newer 2.26.* Nix version	2025-02-18 13:00:40 -05:00
John Ericson	af72b694d8	Bump to newer 2.26.* Nix version Needed one more thing before trying out using `LegacySSHStore` directly. Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/674a87462cb93f605d4fbeef607d3453e7e5a7d8?narHash=sha256-TBoHqnIdVWhsBcL05vO2B1hSl9m//5Mz2NU%2BPMk3h3Y%3D' (2025-02-16) → 'github:NixOS/nix/e310c19a1aeb1ce1ed4d41d5ab2d02db596e0918?narHash=sha256-q/RgA4bB7zWai4oPySq9mch7qH14IEeom2P64SXdqHs%3D' (2025-02-18)	2025-02-18 12:43:31 -05:00
John Ericson	c92342d12f	Merge pull request #1446 from NixOS/newer-2.6 Bump to newer 2.26.* Nix version	2025-02-16 19:10:10 -05:00
John Ericson	df07670a21	Bump to newer 2.26.* Nix version Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/970942f45836172fda410a638853382952189eb9?narHash=sha256-jGFuyYKJjJZsBRoi7ZcaVKt1OYxusz/ld1HA7VD2w/0%3D' (2025-02-12) → 'github:NixOS/nix/674a87462cb93f605d4fbeef607d3453e7e5a7d8?narHash=sha256-TBoHqnIdVWhsBcL05vO2B1hSl9m//5Mz2NU%2BPMk3h3Y%3D' (2025-02-16)	2025-02-16 18:44:32 -05:00
John Ericson	51944a5fa5	Merge pull request #1443 from NixOS/nix-2.26 Nix 2.26	2025-02-13 22:13:32 -05:00
John Ericson	341b2f1309	Update build system to depend on Nix 2.26	2025-02-13 21:54:35 -05:00
John Ericson	4dc0f11379	Update flake.nix for Nix 2.26 Flake lock file updates: • Removed input 'libgit2' • Updated input 'nix': 'github:NixOS/nix/d652513e4519ed4eb48c92f8670e5a71c7793fc3?narHash=sha256-mIpJgIwPS4o4xYhN1B%2B/fHESEXoxpu6nVoZTzZ0MfTg%3D' (2025-02-12) → 'github:NixOS/nix/970942f45836172fda410a638853382952189eb9?narHash=sha256-jGFuyYKJjJZsBRoi7ZcaVKt1OYxusz/ld1HA7VD2w/0%3D' (2025-02-12) • Removed input 'nix/libgit2' • Updated input 'nix-eval-jobs': 'github:nix-community/nix-eval-jobs/6d4fd5a93d7bc953ffa4dcd6d53ad7056a71eff7?narHash=sha256-1dZLPw%2BnlFQzzswfyTxW%2B8VF1AJ4ZvoYvLTjlHiz1SA%3D' (2025-02-13) → 'github:nix-community/nix-eval-jobs/4b392b284877d203ae262e16af269f702df036bc?narHash=sha256-3wIReAqdTALv39gkWXLMZQvHyBOc3yPkWT2ZsItxedY%3D' (2025-02-14) • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/dbebdd67a6006bb145d98c8debf9140ac7e651d0?narHash=sha256-Xc9lEtentPCEtxc/F1e6jIZsd4MPDYv4Kugl9WtXlz0%3D' (2024-09-18) → 'github:NixOS/nixpkgs/97a719c9f0a07923c957cf51b20b329f9fb9d43f?narHash=sha256-1o1qR0KYozYGRrnqytSpAhVBYLNBHX%2BLv6I39zGRzKM%3D' (2025-02-13)	2025-02-13 21:54:31 -05:00
John Ericson	ea09952b7e	Merge pull request #1442 from NixOS/clean-up-flake-lockfile Clean up flake lockfile stuff	2025-02-13 20:52:40 -05:00
John Ericson	81d21979ef	Clean up flake lockfile stuff The `flake = false;` for `nix-eval-jobs` didn't fully take before. Flake lock file updates: • Removed input 'nix-eval-jobs/flake-parts' • Removed input 'nix-eval-jobs/flake-parts/nixpkgs-lib' • Removed input 'nix-eval-jobs/nix-github-actions' • Removed input 'nix-eval-jobs/nixpkgs' • Removed input 'nix-eval-jobs/treefmt-nix' • Removed input 'nix-eval-jobs/treefmt-nix/nixpkgs'	2025-02-13 20:23:08 -05:00
John Ericson	0ed9a82912	Merge pull request #1441 from NixOS/nix-2.25 Nix 2.25	2025-02-13 19:53:07 -05:00
John Ericson	80241fc8be	Make code change necessary for building with Nix 2.25	2025-02-13 19:10:09 -05:00
John Ericson	4347833f45	Rework to synchronize deps Bypass `nix-eval-job`'s flake, and just call-package it, for fine-grained control.	2025-02-13 19:03:37 -05:00
John Ericson	8835cbd10f	flake.lock: Update Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/a7fdef6858dd45b9d7bda7c92324c63faee7f509?narHash=sha256-XFznzb8L4SdUm9u%2Bw3DPpMWJhffuv%2B/6%2BaiVl00slns%3D' (2024-09-19) → 'github:NixOS/nix/d652513e4519ed4eb48c92f8670e5a71c7793fc3?narHash=sha256-mIpJgIwPS4o4xYhN1B%2B/fHESEXoxpu6nVoZTzZ0MfTg%3D' (2025-02-12) • Updated input 'nix-eval-jobs': 'github:nix-community/nix-eval-jobs/889ea1406736b53cf165b6c28398aae3969418d1?narHash=sha256-3wwtKpS5tUBdjaGeSia7CotonbiRB6K5Kp0dsUt3nzU%3D' (2024-12-10) → 'github:nix-community/nix-eval-jobs/6d4fd5a93d7bc953ffa4dcd6d53ad7056a71eff7?narHash=sha256-1dZLPw%2BnlFQzzswfyTxW%2B8VF1AJ4ZvoYvLTjlHiz1SA%3D' (2025-02-13)	2025-02-13 18:46:36 -05:00
John Ericson	9ad8ac586c	Merge pull request #1440 from NixOS/legacy-ssh-expose-ssh-master Use new `CommonSSHStoreConfig::createSSHMaster`	2025-02-13 18:30:41 -05:00
John Ericson	9a6928d93b	Use new `CommonSSHStoreConfig::createSSHMaster` This avoids some duplicated code, leveraging the same `StoreReference` type that also undergirds the machine file dedup we just did prior. By using `LegacySSHStoreConfig`, we're also taking a baby step towards using the store interface rather than messing around with the protocol internals.	2025-02-13 18:13:38 -05:00
John Ericson	810781a802	Merge pull request #1439 from NixOS/nix-next Dedup machine file parsing, and other improvements	2025-02-13 18:10:08 -05:00
John Ericson	af9b0663f2	Merge branch 'master' into nix-next	2025-02-13 17:54:15 -05:00
John Ericson	2feddd8511	flake.lock: Update Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/2c42e7b8d9ea32e59c01334852599b548b214d31' (2024-05-23) → 'github:NixOS/nix/ef5c846e257e1e284ad47ed6be4308d190fe6531' (2024-05-29)	2024-05-29 17:05:41 -04:00
John Ericson	cd925e876f	Merge branch 'master' into nix-next	2024-05-29 17:05:04 -04:00
John Ericson	91bb72e323	Merge pull request #1386 from NixOS/machine-dedup Dedup with nix: use `nix::Machine::parseConfig`	2024-05-23 11:21:41 -04:00
John Ericson	09a1e64ed2	Dedup with nix: use `nix::Machine::parseConfig` Companion to https://github.com/NixOS/nix/pull/10763	2024-05-23 09:59:46 -04:00
John Ericson	bede2a141a	flake.lock: Update Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/5845fd59c34198ad52a7f7bcb6d3ea7176ca437b' (2024-05-22) → 'github:NixOS/nix/2c42e7b8d9ea32e59c01334852599b548b214d31' (2024-05-23)	2024-05-23 09:59:32 -04:00
John Ericson	b75bf5c882	Merge pull request #1385 from NixOS/machine-dedup Utilize `nix::Machine` more fully	2024-05-23 00:00:58 -04:00
John Ericson	d55bea2a1e	Utilize `nix::Machine` more fully With https://github.com/NixOS/nix/pull/9839, the `storeUri` field is much better structured, so we can use it while still opening the SSH connection ourselves.	2024-05-22 22:02:46 -04:00
John Ericson	346badc66f	flake.lock: Update Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/a57abbd143f8ed44e823c3244e93507f64020878' (2024-05-20) → 'github:NixOS/nix/5845fd59c34198ad52a7f7bcb6d3ea7176ca437b' (2024-05-22)	2024-05-22 22:00:38 -04:00
John Ericson	a940450875	Merge branch 'master' into nix-next	2024-05-22 22:00:25 -04:00
John Ericson	af120e7195	Merge pull request #1384 from NixOS/more-serve-proto-factor-out Dedup more protocol code	2024-05-20 21:49:06 -04:00
John Ericson	71c4e2dc5b	Dedup more protocol code Use https://github.com/NixOS/nix/pull/10749	2024-05-20 18:19:59 -04:00
John Ericson	e4552ddf91	flake.lock: Update Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/beb3c2bc7ab781c1b8907b647c6e72b72fa9f56b' (2024-05-17) → 'github:NixOS/nix/a57abbd143f8ed44e823c3244e93507f64020878' (2024-05-20)	2024-05-20 18:11:37 -04:00
John Ericson	e4f2c84f8d	flake.lock: Update Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/0930058189f350a3729cd5aef2ffc8dae2ad436e' (2024-05-08) → 'github:NixOS/nix/beb3c2bc7ab781c1b8907b647c6e72b72fa9f56b' (2024-05-17)	2024-05-17 20:02:54 -04:00
John Ericson	e10fc2bd13	Merge branch 'master' into nix-next	2024-05-17 19:59:41 -04:00
John Ericson	5e910fa2ce	flake.lock: Update Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/00ca2b05b8fbbef09be5d1e4820857605d4c31b6' (2024-05-03) → 'github:NixOS/nix/0930058189f350a3729cd5aef2ffc8dae2ad436e' (2024-05-08)	2024-05-08 11:25:14 -04:00
John Ericson	4b767aa9a2	Merge branch 'master' into nix-next	2024-05-08 11:25:04 -04:00
John Ericson	2926aa1d64	Merge branch 'factor-out-tests' into nix-next	2024-05-03 12:44:51 -04:00
John Ericson	555ea44a7a	Merge branch 'master' into nix-next	2024-05-03 12:35:06 -04:00
John Ericson	410077a26e	Merge branch 'nix-2.22' into nix-next	2024-05-03 10:49:28 -04:00
John Ericson	39a4e4791e	Switch (back) to Nix master Re-creating `nix-next` after using it in #1375. Flake lock file updates: • Updated input 'nix': 'github:NixOS/nix/60824fa97c588a0faf68ea61260a47e388b0a4e5' (2024-04-11) → 'github:NixOS/nix/aa438b8fbaebbbdb922655127053c4e8ea3e55bb' (2024-04-12)	2024-04-12 17:30:57 -04:00