wip

[overwatch] grafana provision datasource and dashboards
[metrics] enable on all microvms
2025-01-21 22:17:16 +01:00 · 2025-01-20 22:31:36 +01:00 · 2025-01-20 21:32:07 +01:00 · 2025-01-20 21:32:07 +01:00 · 2025-01-20 21:32:07 +01:00 · 2025-01-20 21:32:07 +01:00
28 changed files with 26262 additions and 149 deletions
--- a/doc/src/Index.md
+++ b/doc/src/Index.md
@@ -1 +1,82 @@
-# Index
+# malobeo infrastructure
+
+this repository nxios configurations of the digital malobeo infrastructure. it should be used to setup, test, build and deploy different hosts in a reproducible manner.
+
+the file structure is based on this [blog post](https://samleathers.com/posts/2022-02-03-my-new-network-and-deploy-rs.html)
+
+### deploying configuration
+#### local deployment
+``` shell
+nixos-rebuild switch --use-remote-sudo
+```
+
+#### remote deployment
+you need the hostname and ip address of the host:
+``` shell
+ nixos-rebuild switch --flake .#<hostname> --target-host root@<ip_address> --build-host localhost
+```
+
+in this case 'localhost' is used as buildhost which can be usefull if the target host is low systemresources
+
+
+## development
+
+### requirements
+we use flake based configurations for our hosts. if you want to build configurations on you own machine you have to enable flakes first by adding the following to your *configuration.nix* or *nix.conf*
+``` nix
+nix.extraOptions = ''
+  experimental-features = nix-command flakes
+'';
+```
+
+More information about flakes can be found [here](https://nixos.wiki/wiki/Flakes)
+
+### dev shell
+a development shell with the correct environment can be created by running ```nix develop ```
+
+If you're using direnv you can add flake support by following those steps: [link](https://nixos.wiki/wiki/Flakes#Direnv_integration)
+
+### build a configuration
+
+to build a configuration run the following command (replace ```<hostname>``` with the actual hostname):
+
+``` shell
+nix build .#nixosConfigurations.<hostname>.config.system.build.toplevel
+```
+
+### building raspberry image
+
+for the raspberry it is possible to build the whole configuration as an sd-card image which then can be flashed directly. more information about building arm on nixos can be found [here](https://nixos.wiki/wiki/NixOS_on_ARM).
+
+to be able to build the image you need to enable qemu emulation on the machine you are building with. therefore it is necessary to add the following to your configuration.nix:
+
+``` nix
+boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
+```
+
+then you can build the image with:
+
+``` shell
+nix build .#nixosConfigurations.rpi1_base_image.config.system.build.sdImage
+```
+
+### run a configuration as vm
+
+to run a vm we have to build it first using the following command (replace ```<hostname>``` with the actual hostname):
+
+``` shell
+nix build .#nixosConfigurations.<hostname>.config.system.build.vm
+```
+
+afterwards run the following command to start the vm:
+
+``` shell
+./result/bin/run-<hostname>-vm
+```
+
+### documentation
+
+for documentation we currently just use README.md files.
+
+the devshell provides the python package ['grip'](https://github.com/joeyespo/grip) which can be used to preview different README.md files in the browser.
+the usage is simple, just run ```grip``` in the same folder as the README.md you wanna preview. then open your browser at ```http://localhost:6419 ```.
--- a/doc/src/anleitung/microvm.md
+++ b/doc/src/anleitung/microvm.md
@@ -12,13 +12,31 @@ Use durruti as orientation:
 "10.0.0.5" is the IP assigned to its tap interface.  

 ### Testing MicroVMs locally
-MicroVMs can be built and run easily on your local host, but they are not persistent!
-For durruti for example this is done by:
+MicroVMs can be built and run easily on your localhost for development.  
+We provide the script ```run-vm``` to handle stuff like development (dummy) secrets, sharing directories, ect. easily.
+Usage examples:
 ``` bash
-nix run .\#durruti-vm
+# run without args to get available options and usage info
+run-vm
+
+# run nextcloud locally with dummy secrets
+run-vm nextcloud --dummy-secrets
+
+# share a local folder as /var/lib dir so that nextcloud application data stays persistent between boots
+mkdir /tmp/nextcloud
+run-vm nextcloud --dummy-secrets --varlib /tmp/nextcloud
+
+# enable networking to provide connectivity between multiple vms
+# for that the malobeo hostBridge must be enabled on your host
+# this example deploys persistent grafana on overwatch and fetches metrics from infradocs
+mkdir overwatch
+run-vm overwatch --networking --varlib /tmp/overwatch
+run-vm infradocs --networking
 ```

-### Testing persistent microvms
+
+
+### Fully deploy microvms on local host
 In order to test persistent microvms locally we need to create them using the ```microvm``` command.  
 This is necessary to be able to mount persistent /etc and /var volumes on those hosts.  
 Do the following:
--- a/flake.lock
+++ b/flake.lock
@@ -7,11 +7,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1730135292,
-        "narHash": "sha256-CI27qHAbc3/tIe8sb37kiHNaeCqGxNimckCMj0lW5kg=",
+        "lastModified": 1736864502,
+        "narHash": "sha256-ItkIZyebGvNH2dK9jVGzJHGPtb6BSWLN8Gmef16NeY0=",
        "owner": "nix-community",
        "repo": "disko",
-        "rev": "ab58501b2341bc5e0fc88f2f5983a679b075ddf5",
+        "rev": "0141aabed359f063de7413f80d906e1d98c0c123",
        "type": "github"
      },
      "original": {
@@ -67,11 +67,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1733951536,
-        "narHash": "sha256-Zb5ZCa7Xj+0gy5XVXINTSr71fCfAv+IKtmIXNrykT54=",
+        "lastModified": 1736373539,
+        "narHash": "sha256-dinzAqCjenWDxuy+MqUQq0I4zUSfaCvN9rzuCmgMZJY=",
        "owner": "nix-community",
        "repo": "home-manager",
-        "rev": "1318c3f3b068cdcea922fa7c1a0a1f0c96c22f5f",
+        "rev": "bd65bc3cde04c16755955630b344bc9e35272c56",
        "type": "github"
      },
      "original": {
@@ -109,11 +109,11 @@
        "spectrum": "spectrum"
      },
      "locked": {
-        "lastModified": 1734041466,
-        "narHash": "sha256-51bhaMe8BZuNAStUHvo07nDO72wmw8PAqkSYH4U31Yo=",
+        "lastModified": 1736905611,
+        "narHash": "sha256-eW6SfZRaOnOybBzhvEzu3iRL8IhwE0ETxUpnkErlqkE=",
        "owner": "astro",
        "repo": "microvm.nix",
-        "rev": "3910e65c3d92c82ea41ab295c66df4c0b4f9e7b3",
+        "rev": "a18d7ba1bb7fd4841191044ca7a7f895ef2adf3b",
        "type": "github"
      },
      "original": {
@@ -124,11 +124,11 @@
    },
    "nixlib": {
      "locked": {
-        "lastModified": 1733620091,
-        "narHash": "sha256-5WoMeCkaXqTZwwCNLRzyLxEJn8ISwjx4cNqLgqKwg9s=",
+        "lastModified": 1736643958,
+        "narHash": "sha256-tmpqTSWVRJVhpvfSN9KXBvKEXplrwKnSZNAoNPf/S/s=",
        "owner": "nix-community",
        "repo": "nixpkgs.lib",
-        "rev": "f4dc9a6c02e5e14d91d158522f69f6ab4194eb5b",
+        "rev": "1418bc28a52126761c02dd3d89b2d8ca0f521181",
        "type": "github"
      },
      "original": {
@@ -145,11 +145,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1733965598,
-        "narHash": "sha256-0tlZU8xfQGPcBOdXZee7P3vJLyPjTrXw7WbIgXD34gM=",
+        "lastModified": 1737057290,
+        "narHash": "sha256-3Pe0yKlCc7EOeq1X/aJVDH0CtNL+tIBm49vpepwL1MQ=",
        "owner": "nix-community",
        "repo": "nixos-generators",
-        "rev": "d162ffdf0a30f3d19e67df5091d6744ab8b9229f",
+        "rev": "d002ce9b6e7eb467cd1c6bb9aef9c35d191b5453",
        "type": "github"
      },
      "original": {
@@ -160,11 +160,11 @@
    },
    "nixos-hardware": {
      "locked": {
-        "lastModified": 1733861262,
-        "narHash": "sha256-+jjPup/ByS0LEVIrBbt7FnGugJgLeG9oc+ivFASYn2U=",
+        "lastModified": 1736978406,
+        "narHash": "sha256-oMr3PVIQ8XPDI8/x6BHxsWEPBRU98Pam6KGVwUh8MPk=",
        "owner": "NixOS",
        "repo": "nixos-hardware",
-        "rev": "cf737e2eba82b603f54f71b10cb8fd09d22ce3f5",
+        "rev": "b678606690027913f3434dea3864e712b862dde5",
        "type": "github"
      },
      "original": {
@@ -192,11 +192,11 @@
    },
    "nixpkgs-unstable": {
      "locked": {
-        "lastModified": 1733759999,
-        "narHash": "sha256-463SNPWmz46iLzJKRzO3Q2b0Aurff3U1n0nYItxq7jU=",
+        "lastModified": 1737062831,
+        "narHash": "sha256-Tbk1MZbtV2s5aG+iM99U8FqwxU/YNArMcWAv6clcsBc=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "a73246e2eef4c6ed172979932bc80e1404ba2d56",
+        "rev": "5df43628fdf08d642be8ba5b3625a6c70731c19c",
        "type": "github"
      },
      "original": {
@@ -208,11 +208,11 @@
    },
    "nixpkgs_2": {
      "locked": {
-        "lastModified": 1733808091,
-        "narHash": "sha256-KWwINTQelKOoQgrXftxoqxmKFZb9pLVfnRvK270nkVk=",
+        "lastModified": 1736916166,
+        "narHash": "sha256-puPDoVKxkuNmYIGMpMQiK8bEjaACcCksolsG36gdaNQ=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "a0f3e10d94359665dba45b71b4227b0aeb851f8e",
+        "rev": "e24b4c09e963677b1beea49d411cd315a024ad3a",
        "type": "github"
      },
      "original": {
@@ -245,11 +245,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1733965552,
-        "narHash": "sha256-GZ4YtqkfyTjJFVCub5yAFWsHknG1nS/zfk7MuHht4Fs=",
+        "lastModified": 1737107480,
+        "narHash": "sha256-GXUE9+FgxoZU8v0p6ilBJ8NH7k8nKmZjp/7dmMrCv3o=",
        "owner": "Mic92",
        "repo": "sops-nix",
-        "rev": "2d73fc6ac4eba4b9a83d3cb8275096fbb7ab4004",
+        "rev": "4c4fb93f18b9072c6fa1986221f9a3d7bf1fe4b6",
        "type": "github"
      },
      "original": {
--- a/machines/.sops.yaml
+++ b/machines/.sops.yaml
@@ -11,7 +11,18 @@ keys:
  - &machine_durruti age1xu6kxpf8p0r8d6sgyl0m20p5hmw35nserl7rejuzm66eql0ur4mq03u0vp
  - &machine_vpn age1v6uxwej4nlrpfanr9js7x6059mtvyg4fw50pzt0a2kt3ahk7edlslafeuh
  - &machine_fanny age14dpm6vaycd6u34dkndcktpamqgdyj4aqccjnl5533dsza05hxuds0tjfnf
+  #this dummy key is used for testing.
+  - &machine_dummy age18jn5mrfs4gqrnv0e2sxsgh3kq4sgxx39hwr8z7mz9kt7wlgaasjqlr88ng
 creation_rules:
+  #provide fake secrets in a dummy.yaml file for each host
+  - path_regex: '.*dummy\.yaml$'
+    key_groups:
+    - pgp:
+      - *admin_kalipso
+      - *admin_kalipso_dsktp
+      age:
+      - *machine_dummy
+      - *admin_atlan
  - path_regex: moderatio/secrets/secrets.yaml$
    key_groups:
    - pgp:
@@ -73,3 +84,12 @@ creation_rules:
      - *admin_kalipso_dsktp
      age:
      - *admin_atlan
+
+  - path_regex: nextcloud/secrets.yaml$
+    key_groups:
+    - pgp:
+      - *admin_kalipso
+      - *admin_kalipso_dsktp
+      - *machine_durruti
+      age:
+      - *admin_atlan
--- a/machines/configuration.nix
+++ b/machines/configuration.nix
@@ -43,6 +43,7 @@ let
  defaultModules = baseModules;

  makeMicroVM = hostName: ipv4Addr: macAddr: modules: [
+    self.nixosModules.malobeo.metrics
    {
      microvm = {
        hypervisor = "cloud-hypervisor";
@@ -79,6 +80,13 @@ let
          }
        ];
      };
+
+      malobeo.metrics = {
+        enable = true;
+        enablePromtail = true;
+        logNginx = false;
+        lokiHost = "10.0.0.14";
+      };
      
      systemd.network.enable = true;
      
@@ -170,6 +178,24 @@ in
    ];
  };

+  nextcloud = nixosSystem {
+    system = "x86_64-linux";
+    specialArgs.inputs = inputs;
+    specialArgs.self = self;
+    modules = makeMicroVM "nextcloud" "10.0.0.13" "D0:E5:CA:F0:D7:E9" [
+      ./nextcloud/configuration.nix
+    ];
+  };
+
+  overwatch = nixosSystem {
+    system = "x86_64-linux";
+    specialArgs.inputs = inputs;
+    specialArgs.self = self;
+    modules = makeMicroVM "overwatch" "10.0.0.14" "D0:E5:CA:F0:D7:E0" [
+      ./overwatch/configuration.nix
+    ];
+  };
+
  testvm = nixosSystem {
    system = "x86_64-linux";
    specialArgs.inputs = inputs;
--- a/machines/durruti/configuration.nix
+++ b/machines/durruti/configuration.nix
@@ -3,12 +3,9 @@
 with lib;

 {
-  sops.defaultSopsFile = ./secrets.yaml;
-
  networking = {
    hostName = mkDefault "durruti";
    useDHCP = false;
-    nameservers = [ "1.1.1.1" ];
  };

  networking.firewall.allowedTCPPorts = [ 8080 ];
@@ -20,7 +17,6 @@ with lib;
  ];

  imports = [
-    inputs.ep3-bs.nixosModules.ep3-bs
    inputs.tasklist.nixosModules.malobeo-tasklist

    ./documentation.nix
--- a/machines/durruti/secrets.yaml
+++ b/machines/durruti/secrets.yaml
@@ -1,70 +0,0 @@
-hello: ENC[AES256_GCM,data:MKKsvoFlHX6h4qazxcjl/RE1ZsK64G926k4hgFW3AkoJgXO1QXmTaRG7ZBgS8A==,iv:hoFbcNRkge24xJfLZJH651jB4NnXCjYAdTrirkans+4=,tag:68AyEHamlGxdmSJGkTGbsA==,type:str]
-ep3bsDb: ENC[AES256_GCM,data:Z4ZYRaV/eCkaW5Ma+88hbl1o8qsI7PANrIHXoLdIOqIGFLPt7dw=,iv:BCVM+PeGm2NRcvBBy0kId1iVOD/uoiVKKBDA03p0QFM=,tag:CMypO3RLOhvHdVG5YvWewg==,type:str]
-ep3bsMail: ENC[AES256_GCM,data:rZhRb/+gs0Lm8Gdi2P2FMe15A344b88TRg==,iv:hEIG2CBcMslg3hmH3ST3bu6tmes01jncQ3V7h5KcuhA=,tag:XAHdMAlVZNyMdp4TznWDQQ==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age1xu6kxpf8p0r8d6sgyl0m20p5hmw35nserl7rejuzm66eql0ur4mq03u0vp
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEay9wZGM5elhUd2RqVFdJ
-            aHFhRVNiY0lzZEZzSkVvcVlMT1FmMXN4YzNrCkE3SnprNUJ6Ty9hUGZhbzNEVit4
-            THpoUnMyNmQ2Q3Z0SlR6cDFzeE9BaDAKLS0tIHFpbFJadTdtb2s2T2hmMWFBTlBV
-            azZzNXBTRVFoUGtJaGpPdzlDNVpYcjAKd/9v8gn3jbMEK+UPipI8cIufCoWwWfS/
-            kI9zLws/jtjhRZLNHJaXWz7CjAEwKA+6NOQA3pwZaeS1QKwSmeRdZA==
-            -----END AGE ENCRYPTED FILE-----
-        - recipient: age1ljpdczmg5ctqyeezn739hv589fwhssjjnuqf7276fqun6kc62v3qmhkd0c
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxc1o2eTlFc1l4YjVOUHdM
-            S1F2RG9PQWwyd2VYSmJmVzE4cWNSSEt5WUJZCjlwaWNJWFNHNnZkUVBwdVJUbVNi
-            WjdYZ2dENVIydWw4WHJmckF0ZjRLWXMKLS0tIDRsNXNSRnZkVzFkSHpDSWgrSEhv
-            bjBqRlYzcGIvNzhLbjdUbmFhMkU2RXMKsgkwNqQeP40boqriANQg13YKKwMz9iTZ
-            Vw1wYVeQmo4En7c4yAztqBriVoTNsbWkkvGw0P4z37B+6ll8kdEMSQ==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2024-06-26T10:07:26Z"
-    mac: ENC[AES256_GCM,data:TfN80Hffm+Lf/5Cz7T37bBxMgJCAnk2aBxxW1/lr89N2p3cckcSOGAKoLWNIsdOkqOjAs4kft0nQ+xyfdLehG1WPo6OlOwZhJexfUUcS7GJ0QGNEVntkehQiHGw9TIv08/WHRbjnKTOGHLn1vuJAIJmSyff0hncGR7nxcwghZUU=,iv:TfidjsiqDx4SCbtb6ksNYOSz/EwzwnYieeWOaBrvA7Y=,tag:e8Vaycv9bxrVBn2QjRyfSw==,type:str]
-    pgp:
-        - created_at: "2024-12-19T15:09:01Z"
-          enc: |-
-            -----BEGIN PGP MESSAGE-----
-
-            hQGMA5HdvEwzh/H7AQv+K+G7MhXO0RlQENydEstPcMV5vAgkzL06kiN3wXpeOPmj
-            2gwdNcbOLtcXV8a4mH6xGZPkKOV8xjkybp7Myicll6YDs+4Uw3qRTUmCyZ0BC2Wc
-            WDrTMz/lCx1gZGVa99KgHaLmALhZbEO/R08qW52Xkwmcvg1GdM22RtB12L+c8JPB
-            +RR/pLR4UCTfN21uS2CJ33bJnAayfi+s/maGYsElZkH/zoPtDBxF/ntk7g/xeN13
-            Jymg1Ofmjm8JT0FPe8RE7Er/qXlxsG46GVj964chCtljz3NgL76tgC207E8CLUJq
-            rVqGKU0PO6h924uNmVON+JI1CeyCsjejsFOGaS8kOEAwEgCoeICqiqkTbtUCU21K
-            4C7J3mFwhAL+F2IueOY8NZxEV4tMJoY6JZ8c8wtM4Gl6JePlkFRX8LhuO/Bw2VJ9
-            cuGlkIIg3pA94U6Hql7LwLZbIkquI7SWGx7IHOhk/4qtCUlEn4t40JdN4PbA0bz2
-            Cde3+6zFOkX0m1BXkj4f0lgBIOfcPsXmY8ho4isVd9+v7arbE2WSZ6IBG75cx0a1
-            4LYx3QWTLlujiDIc5arhBgpB2ceO8lFTARnoLLqG6y1T+w6UNoVHQZ4n987SpWkk
-            EKQxUDnO8Nvb
-            =1PHB
-            -----END PGP MESSAGE-----
-          fp: c4639370c41133a738f643a591ddbc4c3387f1fb
-        - created_at: "2024-12-19T15:09:01Z"
-          enc: |-
-            -----BEGIN PGP MESSAGE-----
-
-            hQIMA98TrrsQEbXUARAAhfUKm9iR11pU0U44IDfwa7NRRurim8GOPX4FWwJJORNL
-            q85xGM0jA/k8JRsOdsjfHb4/khHtG8cl+t09nEBxTeeb7mKdiOXfsxrvHEf6qeUw
-            F/DQGoaxk+ISXW4iMcV0CPYciLb7kSHCqVFovmmTGlI9fMXryKl3UpP/nzzz9Zk2
-            5cXLmbQqeQVsp17Dw5x7rglkTlx8+W7Z1tDHlHrycxzh6LYpJ7QX54EHM8JgMjw/
-            WREO0qnJMt6C0qp8e3KWhYhMHIidM3WexJR9ixBICxevy0QwvNult0ryOZMc+nTY
-            48sXxCTebnLspiFBS5OsagGxNgwMixydfKv0ci8E7FyB84jwq7XriiQRzYfzU/6L
-            wEPapKrXno0F7wyiiesl/HKdLkOujFIhAl7P1ZNHQLcDuzDCqSo2xd7dbUsbPLcR
-            BUNcfc0VK3TEJks1lXkO5C1PeYEy+NgsJnEQ2lrnAbmKDxpH6qOA2KSGh12uZnHp
-            7kk/hRclVnygkcQc6j71eOyprQms2VjU6fVy2dED+ucjvogrceWWSUkuP6GQEqZV
-            bPhLxpMMw6cIWcTLZIEqLRQv9EqibIFEohkUh9A2TL7XxPb6MEhsRXKTsmMqzdiH
-            /xUwxH3w0w8CrEheVvxGxQi7B4XWC9jHGN+KvJGisrLeGpl/wJ8NKcqOSasB4fLS
-            WAHQxsAnNtNj5rV/BQJHr8lvX+ebJkMpCEBmIdQUeX4WVegr3HkDF34EWoqVfzV2
-            T0ZUaCXNI+tdmvJji9MPd1ZFrTgF5XuFjQxMP1uPI6gannH9InvBXvY=
-            =5AlZ
-            -----END PGP MESSAGE-----
-          fp: aef8d6c7e4761fc297cda833df13aebb1011b5d4
-    unencrypted_suffix: _unencrypted
-    version: 3.8.1
--- a/machines/fanny/configuration.nix
+++ b/machines/fanny/configuration.nix
@@ -53,7 +53,7 @@ in
  };

  services.malobeo.microvm.enableHostBridge = true;
-  services.malobeo.microvm.deployHosts = [ "infradocs" ];
+  services.malobeo.microvm.deployHosts = [ "infradocs" "nextcloud" "overwatch" ];

  networking = {
    firewall = {
@@ -70,6 +70,14 @@ in
        '';
      };
    };
+
+    virtualHosts."cloud.malobeo.org" = {
+      locations."/" = {
+        proxyPass = "http://10.0.0.13";
+        extraConfig = ''
+        '';
+      };
+    };
  };

  services.tor = {
--- a/machines/fanny/dummy.yaml
+++ b/machines/fanny/dummy.yaml
@@ -0,0 +1,68 @@
+wg_private: ENC[AES256_GCM,data:YEmIfgtyHE9msYijva0Ye2w7shVmYBPZ3mcKRF7Cy20xa6yHEUQ0kC2OWnM=,iv:ouK6fHcrxrEtsmiPmtCz9Ca8Ec1algOifrgZSBNHi74=,tag:524e/SQt++hwVyeWruCsLg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age18jn5mrfs4gqrnv0e2sxsgh3kq4sgxx39hwr8z7mz9kt7wlgaasjqlr88ng
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIVnB0dDdQT0tNSUJDSlhx
+            QVFoVTZlb01MbVBwM2V2MGdGZFJTWm1FTW5nCkN5V0Y5MEp4K2FiU0xNVlRQM2xN
+            SFJEWFFwTGhQWWwzNjlFN3NiakNBMnMKLS0tIE9MRHdnVHVYTG5rR1lGazdlK0Nv
+            cmZiN0R5OW9vaitZb0JIa2srdmNMRjAKYlL4e8hfB0YuVNLM65yyvvCKl6EAF6E5
+            YkAidAO5MY/wo1SDFQMeDub0Uso1QuNexYUZt7kzotvuPOzgywUORA==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1ljpdczmg5ctqyeezn739hv589fwhssjjnuqf7276fqun6kc62v3qmhkd0c
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRZ25EdmdWVjAwWGhiVDRa
+            cU9saUxnSXVDN0NodzI3aXMrTDZRc1FOUUJZCmh6V3lhS1FER2lyMzk5eU1XbXVh
+            b3JFQ05GdEZTNVFTdFJjN3dTN2xBaXMKLS0tIG15YlVvVHZ5c2pYVmZCaktwRXFx
+            NjJ5cFdTVS9NZmVWMjcrcHo2WDZEZDgKiDwkuUn90cDmidwYGZBb5qp+4R1HafV0
+            vMQfjT9GrwB5K/O1GumOmvbzLNhvO2vRZJhfVHzyHLzQK64abQgF5Q==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2025-01-19T22:46:09Z"
+    mac: ENC[AES256_GCM,data:eU3SIqAGrgbO2tz4zH1tgYcif7oe5j+/wmdYl2xXXI+D6IhiKrTJGvzE3rd3ElEpb+Bg0UQId952U2Ut0yPTfxGLtdlbJA66CmhLAksByoJ8lOXUcp/qDyA4yMRSuwYG2v7uF2crvue9fyRfZ7hl7abE/Q7Z2UjOKqhSZC5cO3U=,iv:NmCVvtBWZRzhpr5nMLy+98VuQZWoUms7xFSxq8PMvBA=,tag:UWjA7oqoNWh4wb0myNg7FA==,type:str]
+    pgp:
+        - created_at: "2025-01-19T22:45:26Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQGMA5HdvEwzh/H7AQv/fLoi/LsDTN6tDdw0rWQg1iG/6oFFxcYO44XU2vCd13Ql
+            okvR1ZtvXtnM/FwTboK2KjxahkuAbapXXEvfWJ2W+d2L38aYxCPe/ryQhjrUP/jJ
+            4IjFSa5R6oWca9i5Apue5In71ACzGCF/v2oyNAF4fSDX7Q+YKOMiwaDatfOAKp5v
+            JlkcfIq6WBR+gKZsTCfLunURNJoKu7jz35OUJDmzyZl9u/xV0ENveQxaKa3hj87s
+            hb7RGXqph6WWhigy+rtTqYQNjycRDHuspb2GgGE5N7OYteZo/XxA3tDz2EWVCYx+
+            g8aEEvxHK3qEIcWbgmbKXNPNSH/CG1XQFaUhdSnkg5lMJsBuYToeNsPTS0o866k1
+            wmoiI3nT2KtnV2SeR3UUMNDqSDl5unLgBCrbDi0m3Sqt9ubjCfuOYN2T19WvAMZx
+            CwB33mVAevPy4Qs5IjPad1WtiaUFulkfJFd1iCM9dhA0RDxbIJRaVGCjqnaE4lW4
+            yP25uKoEUSitgr5nLk000lgBQBkE3obMFZ+DPoNaqupremevGJ71LyjJhXwgzQvk
+            7pwQXBZWybBFFcH7wDurSIJMqE1KP24Krshm7aR9yVNd3mEz7v2T5pbUywTP8H1I
+            TkFpiaZ7OG8G
+            =VXoo
+            -----END PGP MESSAGE-----
+          fp: c4639370c41133a738f643a591ddbc4c3387f1fb
+        - created_at: "2025-01-19T22:45:26Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQIMA98TrrsQEbXUARAAmhiwA8S+wZ0wiyIJwmRyLMWj3Xm95dTnEoeZmJ2I3O4g
+            0szCKLdW8eUWrjZML09ByPYXQINkuyUR+g72+/ALEr9F587GxWDdMwcwLSlIYlX0
+            3GwnJ7ACv/uTZjK24AXno3TkffPQy+rRQwXkpmUz7CMCeH/WRmVtf1LFUuxgbcrj
+            Kmx9x52dn+ae5JOeMkEu4t8lAtI1pv1JRPnm6RIqK2N7VBRGjiD9SiyJiwLqV2GN
+            7N+vepFhbBKPzt+CFpnPWnFePb+TtQmAdJVULedlFPLcJGsPMloEXSuunK2eKveB
+            Vj1NO80i8PEVup02IlEabp+H7eYV8wZOviAJ7HGVhpw6kxD1tqO98KeSFfhuqbul
+            ijaeF2COgf9lioR6Y8T+RhTqeEZK85U/OGXgiM7MdTdYQV9BrY5nR5XSYIrK6zl9
+            TlS24DdM/Sd2939o+wdtgpm0FNQjW3WwA3n2QE/rqjQ6z2pyCTH16yRalAgHKNk/
+            B3uDGxIO5ua6xZwPzFrOB7uKggB8W/lx1eyAT53Lv7MTRp9PW6mm+NoVkNIzmCYa
+            5G2Y/bluKRt39O6UuSVrN8YLcyYCC+xYUfQf4Lr6/CwZ/XbgMTYm29+IgkOkgoS0
+            UxPcmXUgxi98lu5IhdIwWTNtaWEvT9adwmd3bxebWgDmUvK5QxAc7BYUnGIe+C7S
+            WAHA1m5OEQrNFLKGTSha/K20cDAoV2f4IAykRRWD3zieBAP3rzsIv78mgrMBIWP6
+            z1L41UXlBToKfcw8TI9XKIlYId/asI7mR+bqT3oLSdni8qr32VpRjZ0=
+            =MPBp
+            -----END PGP MESSAGE-----
+          fp: aef8d6c7e4761fc297cda833df13aebb1011b5d4
+    unencrypted_suffix: _unencrypted
+    version: 3.9.2
--- a/machines/infradocs/configuration.nix
+++ b/machines/infradocs/configuration.nix
@@ -6,7 +6,6 @@ with lib;
  networking = {
    hostName = mkDefault "infradocs";
    useDHCP = false;
-    nameservers = [ "1.1.1.1" ];
  };

  imports = [
@@ -15,6 +14,12 @@ with lib;
    ../modules/sshd.nix
  ];

+  networking.firewall.allowedTCPPorts = [ 9002 ];
+
+  malobeo.metrics.logNginx = lib.mkForce true;
+
+  users.users.promtail.extraGroups = [ "nginx" "systemd-journal" ];
+
  system.stateVersion = "22.11"; # Did you read the comment?
 }

--- a/machines/lucia/dummy.yaml
+++ b/machines/lucia/dummy.yaml
@@ -0,0 +1,70 @@
+hello: ENC[AES256_GCM,data:ehp7eckur8THsbnSUcFYobA2SVDORUpqBcPTWC6/EvunlZbihaJoDoSfSh4Itg==,iv:nEHRg9TfYVdmJgrBs62Tek/3JhwFz8BMKHph4ThUqA8=,tag:1h2DSiOk4khxhRc7YX9ljg==,type:str]
+njala_api_key: ENC[AES256_GCM,data:vGH79aN2m1rZ0278ydoCQ0U5393HL0AZlajTVWcRbD+/V7QREN7ROW2LrdVK95I0cxobmJQ=,iv:vMpFTwWkC0R1/J9fZaks7c0G1Vj64/ryRkN5EgpWCdU=,tag:g2MJADBrJYTbmj2bhUQ8UA==,type:str]
+wireguard_private: ENC[AES256_GCM,data:T4c0qdFZdrwRU9i+nzAdg4ePEVXyeG4e/zNyn8G9Kd//Fwu1woNhQiyDuAo=,iv:VGPCSeU+RqjUdUlLA+RaCXQZK6AMdE4BwOdxM3whwaM=,tag:pXOwj3zxuFRpv2TInjISuw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age18jn5mrfs4gqrnv0e2sxsgh3kq4sgxx39hwr8z7mz9kt7wlgaasjqlr88ng
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArU0lpUjBadHc2c1lPMUtv
+            dmtqRTd0OEd6TnJrQjJaaFFaMjQ0MWlONEJZClBrdVNMb2xhK2RXRzlmN2dmTzZk
+            SStWSzVGbWdqNEFpMnc3RFdpYWNEcTQKLS0tIDY0SlBvcmJ5RjFKTHQyN3lpSEZ2
+            Z3hTOHN2VWVPMENVS1YzR0Z6Y0MxZmMKf0K43yWL7DE15wqEWb6Z0xsQ3nb1Ybyi
+            0gKxb3hTeoWJnJug3hWyeAJvAJ4pzaA5v8PonnSIJK4UxBUnr+5nGw==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1ljpdczmg5ctqyeezn739hv589fwhssjjnuqf7276fqun6kc62v3qmhkd0c
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIa0pnOGFlYU5OenJvWGhl
+            TjFhU0NyWmprRDY3ZWIrRHdmSVBMS3pER3dZCmVQdlVYQ0pFRTBwZXQ1Z1V5ODVK
+            dkREdEpsYk1MMm5kZU1hUEJYRWZDVjAKLS0tIHdsUjJTaURjaGErclJadTF0clhh
+            aStSbDZ6NWtFZ2NrNHY1a29DTmo4bGcKfZZjFA2j5RgMf0crK8TV67iVizzmXvBR
+            6tePJuCePnNDOoZ7WV5YThxYOPSTI1QvfEvcC1qo7l3Kca9jdkkfbQ==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2025-01-19T22:46:57Z"
+    mac: ENC[AES256_GCM,data:GIRj11bDZi38RobJvGoOf5geN42gaGk3294EvB21M/Y+lAsDOUUUbU1fQbBPRUsYvA/lyuHMQWRORTdy0LdjN9ejzwcuev8+j4i6A1zwPSmjIL2+Jp2pBqQj0F6th27hECJlh0wK3vU/aNcccRJP9kEgRME+7FS5uYw9r+ZPJWk=,iv:CUgdVr1b3O4niYTSFokA9uWR3ceiU+6qo+3N+K1BZ3Q=,tag:AERU8MZWHqVsZ+zbT27WIg==,type:str]
+    pgp:
+        - created_at: "2025-01-19T22:46:34Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQGMA5HdvEwzh/H7AQwAqm3KeXNDFMwVoz9qIIqtNHR+KIcdvdRTlG0GA5BEBiHp
+            hSGkyK/Ni3GGX0WYU5Cf85Id/eEoxqDqDd9kX6MSZK3LBnH+hFy4JiSPENVpmwfQ
+            eONaMwdfwI+/5ZfquVj/AApTXZ8ENhdBzLTuAfIa2hGPDwwkajzkVIg18TZOvKG6
+            f2wiEpnSOVHKnPcGhI1dGxN9TqqN74IUoPhThRzQ79l9RcTEZVClos9IPOhaPfk0
+            TvcBQez3G5Mn8W+s7kg4rl7g6XRZqjcaOuNwopB0x+Dx+alZExsTR6A+1Gf+29a+
+            ELwQYg2mkKUi48vrcKXxz+OMhW7V7wuXQIjLP4jULc7LI3ShRP8z4QKy7asOVRBh
+            XPjxRJ2RcTtEHYUoW3guzoZGSiPW/ex2fcupwaSRDG2GQ/ImiB5dmXSaG2va1MbX
+            z93Ej7Gy2IURglWCK29v4mJiqtDzq8/GNztT7zezHxyUAjjuJ59qXzFF/MQPibxY
+            p3dTfkNNrTqrCQnrFa8k0lgBtKmvz/HO3eCguXVMNgOt+BmfZbJqq+AD8HKYNxaM
+            W0A+9WRmYXcOEUqfaoX9IE9LhgqDd/xgpW4CujUZrXRYgf4IokSm/MFAD/ZnoxKm
+            n2My3wf5PCNd
+            =oUOe
+            -----END PGP MESSAGE-----
+          fp: c4639370c41133a738f643a591ddbc4c3387f1fb
+        - created_at: "2025-01-19T22:46:34Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQIMA98TrrsQEbXUAQ/9HD1muIaf8us6XE5hGfrXuy5axFeJNBtc8igo/OP3jCam
+            M2pjwDNIX3EjsgrK2WYo43Xt7aHW5bqnP+d1faLXJ3I0cMia3XxSLmaKswQRrBSi
+            P00ew43kBSx0Smwixf5zCSCzBpWrtOXI8monO8xYLtRnSpfKBf/4kc2gQiuAxByd
+            fxdE/x1et0XXiK038KgHMnYdOIvMTGcpymoSDHk0bDw+ruBqG93cmzOkT4Oc5CsF
+            oeFN83ku1cGFQr58hUhJ1q5eUTK/lDEVYElGJ2n0pGxThYyrUz3SIzZu3Jbxgs/Q
+            2Xok4KsNPEbY2VKz/d3nrwbg2S3VF/CHl9sKxoFK8g3WcEE+HO7BFqvz901H+aUJ
+            5mN7stKSs5pViDHusv3Kv2+eT2fPJ4lPU2IEvXkCt3jjB/G0UDz+t6Qn1Wr8PPcO
+            8u+QafpILgYTK7QOF88GYstq8iWOWHlN9VKFYfqHMGWkrMtnGRWCbXsNmg7lKh6D
+            MtdvROsESVDKtydZwBpoQ4ILLROhkoL+eOMzFOgc/i4PWFlva2RBuRnZQNlieq/R
+            9aYpGsZbD+YYGjQKhlwwakpWK5XOoqqSh6Fv6Qzonu2Y++Zf9c9zpe+LINlUhxEY
+            AA1YnxbqvVJCqoBuq5avAd0fivhFDes0OmR7jLZzluotgwUePwBOVGo5l2ow/03S
+            WAGO2443OWOfbmTi/mq1C/8It1WAkC70XSQbff9pHLd4pRA6XgjMXbJ8+5X0FXRM
+            H5nWOAaO4Dp6SnccT3Q9zytkm+lXdJL6Rou3PkmP4JBvU535bNv5a3k=
+            =732R
+            -----END PGP MESSAGE-----
+          fp: aef8d6c7e4761fc297cda833df13aebb1011b5d4
+    unencrypted_suffix: _unencrypted
+    version: 3.9.2
--- a/machines/modules/malobeo/metrics.nix
+++ b/machines/modules/malobeo/metrics.nix
@@ -0,0 +1,56 @@
+{ config, lib, pkgs, ... }:
+let 
+  cfg = config.malobeo.metrics;
+in
+{
+  options.malobeo.metrics = {
+    enable = lib.mkOption {
+      type = lib.types.bool;
+      default = false;
+      description = "Enable sharing metrics";
+    };
+    enablePromtail = lib.mkOption {
+      type = lib.types.bool;
+      default = true;
+      description = "Enable sharing logs";
+    };
+    logNginx = lib.mkOption {
+      type = lib.types.bool;
+      default = false;
+      description = "Share nginx logs";
+    };
+    lokiHost = lib.mkOption {
+      type = lib.types.str;
+      default = "10.0.0.14";
+      description = "Address of loki host";
+    };
+  };
+  
+  config = lib.mkIf (cfg.enable) {
+
+    networking.firewall.allowedTCPPorts = [ 9002 ];
+
+    services.prometheus = {
+      exporters = {
+        node = {
+          enable = true;
+          enabledCollectors = [ "systemd" "processes" ];
+          port = 9002;
+        };
+      };
+    };
+
+    services.promtail = {
+      enable = cfg.enablePromtail;
+      configFile = import ./promtail_config.nix { 
+        lokiAddress = cfg.lokiHost;
+        logNginx = cfg.logNginx;
+        config = config;
+        pkgs = pkgs;
+      };
+    };
+
+    users.users.promtail.extraGroups = [ "systemd-journal" ] ++ (lib.optionals cfg.logNginx [ "nginx" ]) ;
+
+  };
+}
--- a/machines/modules/malobeo/microvm_host.nix
+++ b/machines/modules/malobeo/microvm_host.nix
@@ -86,6 +86,12 @@ in
    in
    builtins.listToAttrs (map mapperFunc cfg.deployHosts);

+    systemd.tmpfiles.rules = builtins.concatLists (map (name: [
+      "d /var/lib/microvms/${name}/var 0755 root root - -"
+      "d /var/lib/microvms/${name}/etc 0755 root root - -"
+      "d /${name} 0755 root root - -"
+    ]) cfg.deployHosts);
+
    systemd.services = builtins.foldl' (services: name: services // {
      "microvm-update@${name}" = {
        description = "Update MicroVMs automatically";
--- a/machines/modules/malobeo/promtail_config.nix
+++ b/machines/modules/malobeo/promtail_config.nix
@@ -0,0 +1,49 @@
+{ logNginx, lokiAddress, config, pkgs, ... }:
+
+let
+  basecfg = ''
+    server:
+      http_listen_port: 9080
+      grpc_listen_port: 0
+    
+    positions:
+      filename: /tmp/positions.yaml
+    
+    clients:
+      - url: http://${lokiAddress}:3100/loki/api/v1/push
+  '';
+
+  withNginx = ''
+    scrape_configs:
+      - job_name: journal
+        journal:
+          max_age: 12h
+          labels:
+            job: systemd-journal
+            host: ${config.networking.hostName}
+        relabel_configs:
+          - source_labels: ["__journal__systemd_unit"]
+            target_label: "unit"
+      - job_name: nginx
+        static_configs:
+        - targets:
+            - localhost
+          labels:
+            job: nginx
+            __path__: /var/log/nginx/*log
+  '';
+
+  withoutNginx = ''
+    scrape_configs:
+      - job_name: journal
+        journal:
+          max_age: 12h
+          labels:
+            job: systemd-journal
+            host: ${config.networking.hostName}
+        relabel_configs:
+          - source_labels: ["__journal__systemd_unit"]
+            target_label: "unit"
+  '';
+in
+pkgs.writeText "promtailcfg.yaml" (if logNginx then ''${basecfg}${withNginx}'' else ''${basecfg}${withoutNginx}'')
--- a/machines/nextcloud/configuration.nix
+++ b/machines/nextcloud/configuration.nix
@@ -0,0 +1,57 @@
+{ config, lib, pkgs,  ... }:
+
+with lib;
+
+{
+  sops.defaultSopsFile = ./secrets.yaml;
+  sops.secrets = {
+    nextcloudAdminPass = {
+      owner = "nextcloud";
+      group = "nextcloud";
+    };
+  };
+
+  networking = {
+    hostName = mkDefault "nextcloud";
+    useDHCP = false;
+  };
+
+  imports = [
+    ../modules/malobeo_user.nix
+    ../modules/sshd.nix
+    ../modules/minimal_tools.nix
+    ../modules/autoupdate.nix
+  ];
+
+  services.nextcloud = {
+    enable = true;
+    package = pkgs.nextcloud30;
+    hostName = "cloud.malobeo.org";
+    config.adminpassFile = config.sops.secrets.nextcloudAdminPass.path;
+    #https = true; #disable for testing
+    database.createLocally = true;
+    config.dbtype = "pgsql";
+    configureRedis = true;
+    caching = {
+      redis = true;
+      apcu = true;
+    };
+    extraAppsEnable = true;
+    extraApps = {
+      inherit (config.services.nextcloud.package.packages.apps) contacts calendar;
+      collectives = pkgs.fetchNextcloudApp {
+        sha256 = "sha256-cj/8FhzxOACJaUEu0eG9r7iAQmnOG62yFHeyUICalFY=";
+        url = "https://github.com/nextcloud/collectives/releases/download/v2.15.2/collectives-2.15.2.tar.gz";
+        license = "agpl3Plus";
+      };
+    };
+    settings = {
+      trusted_domains = ["10.0.0.13"];
+    };
+  };
+
+  networking.firewall.allowedTCPPorts = [ 80 443 ];
+
+  system.stateVersion = "22.11"; # Did you read the comment?
+}
+
--- a/machines/nextcloud/dummy.yaml
+++ b/machines/nextcloud/dummy.yaml
@@ -0,0 +1,68 @@
+nextcloudAdminPass: ENC[AES256_GCM,data:4GvCg7g=,iv:3m2Vh86WzrVR7BG0xlNwRE9ebIGLWbVdcxoYC9x7dXo=,tag:t2bWTVlw9rHSVnkXW8ZTFQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age18jn5mrfs4gqrnv0e2sxsgh3kq4sgxx39hwr8z7mz9kt7wlgaasjqlr88ng
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvbTYrcUV3Wk0xSDM1Mm0w
+            TkoxZHBFUXFBSC80YzkwV3paWGpRaFY2WndZClh1c0xmNWpWMjFXOS9OYU9OU2Mx
+            c3NEREczaDkvNC90eERwb0RKUlNZemsKLS0tIEp1VWZISXZoWFNuRC9mVE1JUmc3
+            bUNFd2dyRGludFQ3MzdiRzFTcXUwWlkKFGd8Uvfu2W1LejgQFpF162JnVmfPxAuX
+            IQ3oopYXUBM3QqCXGLTY3DBffD4WZ4AXyGLsfUtwn3kcvjQ85ewidw==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1ljpdczmg5ctqyeezn739hv589fwhssjjnuqf7276fqun6kc62v3qmhkd0c
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBraEtDblNxMGY0NlhzcDdM
+            RFo2VGI1UFE4eVdZdDZ5ZTNKRUFCRWFHOWdBCkRBaGk2WmYxK2ovbHQrSGl3akVp
+            TUhxck83Q1NVQy9VU0lXOEVraGtOZ1UKLS0tIDYxS0hHSW1nZW9hOTFJNCtheU1x
+            ZXk2b1RVd1FoYk4xTGxKQ1cxZmVJalkKkC5XckyrgwfqaeVq+OjNCzAtKKiCf7Q9
+            sC9ZMlPoOAm8xpLEpWgNooOBa04YsDEe9XgN8S0HrVxt/NHlnS5+ow==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2025-01-19T22:23:37Z"
+    mac: ENC[AES256_GCM,data:ZVMA4qgliSASQ0LtuedU4pybVwJA0x4vdSlOspsTF22s9DjRbG2tA7PpxTqDBGliBqS4w5J6Rqp3OSF7zddZ23GOz72sOZv0WY5YGeYxIltT7RWSMRkhkwXoM8Pf3BOYCZ4Gy8zaMVnbwbhHZ9LZI6wulh19SDKBV965moUW+Z0=,iv:tmz8C1kGUZq8gfzTHoaU/8RfrT5ohLqA11H42l7TEv0=,tag:E3AV6t2bbKASeVI2G3kNYA==,type:str]
+    pgp:
+        - created_at: "2025-01-19T22:23:26Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQGMA5HdvEwzh/H7AQwAsbbUEzSg7iETqT40w/+qjJ+iuHUeeZ+NFMPVVedbLuZX
+            /gPPSJ3JSx3wxJke+QPX/OUiRxXwGBZ5yg7CDoc+RrrdaavazuOUGISL38KPQiin
+            TmUq4nY6nWjMqHPSaQ+NiGxRpIErCQCUx6YEdi4YgHhf9J4KEg0f7ueR5wWlfzx4
+            c7pXkEubFpPy4v45mh8sYXOQTa2yt2keprJ6iqmfL/HKnCJf1TBFxJAvwnPuKDfe
+            npvE4Pk5hcoKjFQNsqpnz7Etc8XfAYXSCXSLKfSaZNSeXQRhE1lHLH9OPnC46li7
+            Dw6SPW0/4MTJrg08JvrNeogQ1QohLT4mkfAnASVUUeGFaF7UhztVoodqMsG8kuhD
+            LtmYv625h3j/QqshpFPGe8nJG1DziE4YHngxvgToIC+6+Q2x47ciulpimlpt6b6w
+            s+mGEVlJLSoJMd9jwE53PMXteBYKWgSEL7osj1V5KIJDFSgwfjYEngCkuyxjf8e3
+            uYxb1WEadSyLhW8xnoyE0lYB6WI8m6j2Ls9TFG7kKtQqrnVSZd2nsKk/5wPVIAXO
+            nrMtb4Q88PLM1eusiLgCs592owKHoWVYqBRpi79EBlnGZwmTfDQzTsQlL7aQCbIo
+            kBI1swNH3g==
+            =K/IK
+            -----END PGP MESSAGE-----
+          fp: c4639370c41133a738f643a591ddbc4c3387f1fb
+        - created_at: "2025-01-19T22:23:26Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQIMA98TrrsQEbXUAQ/7BGOQn8xVX9IOFHnUXKdiw/+fcXHu0L7CLuhcEfpZRJNI
+            vIbztLTH5gT59SkS07cIEcYWqERO+gjK6pZLCxmYO/c15YDWCv1PtS4YHZAET+ow
+            YHzBTZDq2k+pV/cTbUtJgGYOxuqkqUtflyGZSQM+NjSXu4In83u+nwkEaKSxcaWt
+            q9rrYVkBUiika2FYtyehoCoDuyJzNmfGhZ9CPJaUzrSzuRLQZy6Hdg8TDPTzGqi4
+            awo53eOdsEpNu1AQ5rnJ19RFLU63IQlwEnApnXZZ+AX8J2//KgLOcAqjo5skJuN1
+            EJBh4DjfmOdCCN3uRccNiDn9YYrPP5Jl3jfnkMUQ2gVEE/c5ib84hvjc+y9Eh1jJ
+            s2JqhfX+ccV5kuva1q1CuTUCHQVEKahJAvghl77JQjtUY6Lf+5QLeKbchUnwxaIN
+            v6MwZmdCwBJOS40SnA+Ft2g9psho4MIPXtu4DR4t5VWvhrmw2cl7TCffQslWETxE
+            267PHQegcO40skEOYwkLSv7PWycL3YUg/EfPuElAobqlM45UQA/lcC5seV0WvEPB
+            Lj+Pk8stY3LZ5wpblmmA72PyJhTq7ghsPvxlyRByHIS8v94gu5Zasbi63WGmxgPA
+            mt/M/HIQEf2XOOKuJnGJ3yDwLTAHAgFa0jlIgvt4gkuhm5ibcHLmedBh4ecRNfDS
+            VgFA4UwqUKN1XYTWOvT1vlxE9UE+dgVO0QNMX8yw1ivmL2eyPUk8HtIpKbuDLFIC
+            iKwHMtUIW/q7UPmfKLRbDumimG/290GXxN24ouiN8AR0dCXK9oor
+            =EsRn
+            -----END PGP MESSAGE-----
+          fp: aef8d6c7e4761fc297cda833df13aebb1011b5d4
+    unencrypted_suffix: _unencrypted
+    version: 3.9.2
--- a/machines/nextcloud/secrets.yaml
+++ b/machines/nextcloud/secrets.yaml
@@ -0,0 +1,79 @@
+nextcloudAdminPass: ENC[AES256_GCM,data:es9hhtCcqBqPbV2L,iv:Kyq5kqao0uaMPs0GeRkJT9OWYSZfImBXngg51k0uQ0M=,tag:zN/u90/j4rmdo0HtY+cF9w==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age1ljpdczmg5ctqyeezn739hv589fwhssjjnuqf7276fqun6kc62v3qmhkd0c
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmVGxsNmZ3Z0RIYmMyL0Mr
+            UUpaMEZLTCtQaGFrL1YwOVBicEtNRTVaVGhRCmhDSUgxYXpRcldaMngvOWJDdnNo
+            b2ZFbUdmcE9EV2E3SkMvZ1RpKzZmeU0KLS0tIE5hNmVFTXpBZFZ3bHYwQlJQaUtw
+            UFJmTVFaOTJXN09QLzY4emh5Z3hqRjAKXk1PSwR2x0H2cMN06fyigiusz8v2IRIg
+            S4ZTq/JX39U4QQHgWA1dFPfC636LNBo+QKdl/2mjwnXW7duqDJ+5kA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2024-11-26T20:00:50Z"
+    mac: ENC[AES256_GCM,data:qoY9SfpoU+8HfvD5v/1S6BOkbnZUmHIbtwr0tTSuPETjnFNgr1VVw9mnRatJKPYYFb9/rMZQWIqTY+iUIEkcTVyVXhd6ki5CHW+uxCeBIyMzq33rtEa/btkEUoii4iPieamBCIY21W0znE+edxfR04yRJtLxMICEbuW4Hjf6bwk=,iv:nG42fRgjpuIjPMYnn/6egEdzYolcUBsspaZ8zMv4888=,tag:C6apGoAvVLsWdLWSCwrx6w==,type:str]
+    pgp:
+        - created_at: "2024-11-26T19:59:36Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQGMA5HdvEwzh/H7AQv+KkX46UGQzLvhrk/VUCnnMdLEcNbYfk4h+sZJzs1riOGA
+            LAKYNeaeN6iLLeZX+T2/s5OT4WkIEKGg8/gziurdx01BR70M96Faubp6EVtdK44+
+            6F5BLLrDhlEKDNOx48qPwJdFjbYW4wZLWmv5nzwPmmRCKO7MoI9UHKq69msCor1i
+            ralbjlVHyKSuRfvflKAlxFoEqeB6H+ryc54g3stk1j2eFiMNuF/oKDJZT+XI5LHZ
+            Ai80DAWoUBYgpP4aWiNC075GPutdPlZ3mrGf5+7QnNm7GmNUdJN5VAWmI2NUGr1J
+            BLopnPFo4juWNsZkLMj2aAuKvGTkhz2PuFKfLj6Erpu82RAjadpFWx239n+i4Ryq
+            wSquYshpuiecLEejntTBKLEacwp+aPx8IHKnOOKBTdJj+YYaISiznQAlkF7WS+lg
+            MTZR85BvCxiPogujL7uhYSx1wM5FVkuAIPf1JOJCRvQt30eRRrR0VMrmqQ1Kl5OT
+            VMzZRIGIoC5vrKGeIIjJ0lgBWQ3bYFh/LGrwKetku6TRAH29mp/XwQqBC97RsUYb
+            EOxft5sUWaYrXK+z2yzCxOQBWKJISPgcyhdoKfYGnRkHXHi2Uay84oQP4co72eVF
+            cAhEJOxMw36e
+            =bSaN
+            -----END PGP MESSAGE-----
+          fp: c4639370c41133a738f643a591ddbc4c3387f1fb
+        - created_at: "2024-11-26T19:59:36Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQIMA98TrrsQEbXUARAAmG88ZDt43zj6dCJkYYVj7MGIhIviJzilTvX4+EfNobtA
+            tll60GYfRotKnwbuqzSVaaIcV+6cDQ5I1hG5WNFJSXm7DpJ0W1Ir1x2hpxektXFa
+            fQ+9HiCOfEqUu5PEynCAD1jN6CQLdl87hLQx9TqbZnHuUYPSH1o9Y6kbA/Vp3bpy
+            evJc8qa66WHYH1kjdEw+qneD5HzZQOLOtXZ7xkxjGbyMcYex9JfyGHohO5dpLg6B
+            3XrLlIWWERVz04MlnzlaMKfzhoMCU9ByqJSQ3VBm9kblQqu54fOZD2sN8j9ACEfL
+            YNC7Jm2rasVSqv09G1kso9/VNDw3kNCLvjnpE5rJRP7Ckfj+4FxQN/zPVUwQ1e1k
+            upoQ8MHyf1bJr8vspm/prm9zp+PRRTUwY1Yyts/ffj+CF5ec9M3jr/RSeEAdswsL
+            6dLKBL1LuLAjKXOuVnQ7E6gN940Y994sDFkbqEmzzCUHGcfxSF3IDn/qpkQlqerU
+            B/D43Yef+rtsUDyTA5RUpxKleGORcS4sV0BhQrNXeFclaMTyMr+AbOei4Y77qlD1
+            x/fHB3IT4Intvp9k4m6jJ86RtLpVhEoA4cHEdCCiXHzUpA6aVtNHVAOqT/aBykrf
+            uSm1wu/nl6yKbIwTJueli1OfQYKEYcUdjOrEOwXb+UDQKSohWZrMg0sj7/S6Pl/S
+            WAG1BZ20HXD2ZrVqESV87Pl04nKMqswrio+BINfAT9X3ya7L3DF69MR18bDt+ZIB
+            0F3+9WUREGI5in4S3hXNxrgfLNFl1YLklfWLYcx0HXJN3z6F2eJOUvM=
+            =aT3U
+            -----END PGP MESSAGE-----
+          fp: aef8d6c7e4761fc297cda833df13aebb1011b5d4
+        - created_at: "2024-11-26T19:59:36Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQIMA1kR3vWkIYVnARAAqFyuCtvu6AidWg/9+btEcjWv0sBZaIRpYfX3p2QECwCu
+            UYAtssvSHgHdBEQzU27MA/5CGmEreB3NhWrjGquv88RojLO1JuhNHGYPZKeIcBKr
+            I2oS79RKuYs+d2Qu0KUYDaVoY9M5YJsfkju2FOXqMNYlbqX+lDuWnisigj3n2N4e
+            OEBnVIpfPBQE6c1Z9DaQJE7MyBbKfg5YeWjlwwh+fCf1dV/nGp+QdD88F3dzWMoK
+            xNGt69TwZ8JUVmElAIJqLJTpyDI5xHQUw2A6ddPSTk/u363eHhOnZZUNAAm3FdO5
+            0x+4QhcBaH59S8WDZhw4MVmZN7v4+3l3mf7Rx/TXSz4oJg+U7RMgvc291/gowNVm
+            /cVhBlMYz4Ogx/OYR/t+nzq35r+eBungTB+dRXw7qTTfkCtNgp34JMCkGAq5WWnY
+            57H2HtssGiMF0qN4SfWxw7317oUmqHI2XvG0yWt42G++jNgIGbDOtuc/7wATEbhK
+            SBX2aLqDIB1OUwLHQeawyKkB0qGmRSVPkPg8JLwRp43ICETH1WPkY5m/a2slVlDj
+            qgdw00clTI5Fgu/5G5QBD4Ds9f9ZwjrMD4v+NYfGxa0ajisXl1X6CL1+YvQ6Uicf
+            QmIRJYxyVd0VoXScZnsk0T/XTKjJB/fRLRalA2PmlZ1v+gisCUz2dhM+OHtSjGTS
+            WAG5znRbP8UMVt02O0PgbzHYtIUAtQLCuBnzfEKJn721rqCXf7DXU3jrR73Ys6ce
+            VJzkVBMnBszF71GN56t0PaUYIDOnaGvgjMtHHtOCLQHSK7asnm/Bc+E=
+            =Znii
+            -----END PGP MESSAGE-----
+          fp: 4095412245b6efc14cf92ca25911def5a4218567
+    unencrypted_suffix: _unencrypted
+    version: 3.8.1
--- a/machines/overwatch/configuration.nix
+++ b/machines/overwatch/configuration.nix
@@ -0,0 +1,115 @@
+{ config, lib, pkgs, inputs, ... }:
+
+with lib;
+
+{
+  networking = {
+    hostName = mkDefault "overwatch";
+    useDHCP = false;
+  };
+
+  imports = [
+    ../modules/malobeo_user.nix
+    ../modules/sshd.nix
+  ];
+
+  networking.firewall.allowedTCPPorts = [ 80 9080 9001 3100 ];
+
+  services.grafana = {
+    enable = true;
+    domain = "grafana.malobeo.org";
+    port = 2342;
+    addr = "127.0.0.1";
+
+    provision.datasources.settings = {
+      apiVersion = 1;
+
+      datasources = [
+        {
+          name = "loki";
+          type = "loki";
+          access = "proxy";
+          uid = "eeakiack8nqwwc";
+          url = "http://localhost:3100";
+          editable = false;
+        }
+        {
+          name = "prometheus";
+          type = "prometheus";
+          access = "proxy";
+          uid = "feakib1gq7ugwc";
+          url = "http://localhost:9001";
+          editable = false;
+
+        }
+      ];
+    };
+
+    provision.dashboards.settings = {
+      apiVersion = 1;
+      providers = [{
+        name = "default";
+        options.path = ./dashboards;
+      }];
+    };
+  };
+
+  services.nginx = {
+    enable = true;
+    virtualHosts.${config.services.grafana.domain} = {
+      locations."/" = {
+          proxyPass = "http://127.0.0.1:${toString config.services.grafana.port}";
+          proxyWebsockets = true;
+
+      extraConfig = ''
+        proxy_set_header Host $host;
+      '';
+    };
+      };
+  };
+
+  services.prometheus = {
+    enable = true;
+    port = 9001;
+
+    scrapeConfigs = [
+      {
+        job_name = "overwatch";
+        static_configs = [{
+          targets = [ "127.0.0.1:9002" ];
+        }];
+      }
+      {
+        job_name = "durruti";
+        static_configs = [{
+          targets = [ "10.0.0.5:9002" ];
+        }];
+      }
+      {
+        job_name = "infradocs";
+        static_configs = [{
+          targets = [ "10.0.0.11:9002" ];
+        }];
+      }
+      {
+        job_name = "nextcloud";
+        static_configs = [{
+          targets = [ "10.0.0.13:9002" ];
+        }];
+      }
+      # add vpn - check how to reach it first. most probably 10.100.0.1
+    ];
+  };
+
+  services.loki = {
+    enable = true;
+    configFile = ./loki.yaml;
+  };
+
+  users.users.promtail.extraGroups = [ "nginx" "systemd-journal" ];
+
+
+
+  system.stateVersion = "22.11"; # Did you read the comment?
+}
+
--- a/machines/overwatch/dashboards/main.json
+++ b/machines/overwatch/dashboards/main.json
--- a/machines/overwatch/dashboards/node_full.json
+++ b/machines/overwatch/dashboards/node_full.json
--- a/machines/overwatch/loki.yaml
+++ b/machines/overwatch/loki.yaml
@@ -0,0 +1,60 @@
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+  grpc_listen_port: 9096
+  log_level: debug
+  grpc_server_max_concurrent_streams: 1000
+
+common:
+  instance_addr: 127.0.0.1
+  path_prefix: /tmp/loki
+  storage:
+    filesystem:
+      chunks_directory: /tmp/loki/chunks
+      rules_directory: /tmp/loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+query_range:
+  results_cache:
+    cache:
+      embedded_cache:
+        enabled: true
+        max_size_mb: 100
+
+schema_config:
+  configs:
+    - from: 2020-10-24
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+pattern_ingester:
+  enabled: true
+  metric_aggregation:
+    loki_address: localhost:3100
+
+ruler:
+  alertmanager_url: http://localhost:9093
+
+frontend:
+  encoding: protobuf
+
+# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
+# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
+#
+# Statistics help us better understand how Loki is used, and they show us performance
+# levels for most users. This helps us prioritize features and documentation.
+# For more information on what's sent, look at
+# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
+# Refer to the buildReport method to see what goes into a report.
+#
+# If you would like to disable reporting, uncomment the following lines:
+analytics:
+  reporting_enabled: false
--- a/machines/overwatch/promtail.yaml
+++ b/machines/overwatch/promtail.yaml
@@ -0,0 +1,29 @@
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://10.0.0.13:3100/loki/api/v1/push
+
+  
+scrape_configs:
+  - job_name: journal
+    journal:
+      max_age: 12h
+      labels:
+        job: systemd-journal
+        host: overwatch
+    relabel_configs:
+      - source_labels: ["__journal__systemd_unit"]
+        target_label: "unit"
+  - job_name: nginx
+    static_configs:
+    - targets:
+        - localhost
+      labels:
+        job: nginx
+        __path__: /var/log/nginx/*log
+
--- a/machines/secrets/devkey_ed25519
+++ b/machines/secrets/devkey_ed25519
@@ -0,0 +1,7 @@
+-----BEGIN OPENSSH PRIVATE KEY-----
+b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
+QyNTUxOQAAACBpnsgZCWwvETJKoZf7QFKCLaUnj8hioci+SewK8cUUwgAAAJgdrbX3Ha21
+9wAAAAtzc2gtZWQyNTUxOQAAACBpnsgZCWwvETJKoZf7QFKCLaUnj8hioci+SewK8cUUwg
+AAAECaQfylNoG/uN8fozvq3loBLWQ3gIKPOGnZpwyHUlAMO2meyBkJbC8RMkqhl/tAUoIt
+pSePyGKhyL5J7ArxxRTCAAAADmthbGlwc29AY2VsaW5lAQIDBAUGBw==
+-----END OPENSSH PRIVATE KEY-----
--- a/machines/secrets/devkey_ed25519.pub
+++ b/machines/secrets/devkey_ed25519.pub
@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGmeyBkJbC8RMkqhl/tAUoItpSePyGKhyL5J7ArxxRTC kalipso@celine
--- a/machines/uptimekuma/configuration.nix
+++ b/machines/uptimekuma/configuration.nix
@@ -6,7 +6,6 @@ with lib;
  networking = {
    hostName = mkDefault "uptimekuma";
    useDHCP = false;
-    nameservers = [ "1.1.1.1" ];
  };

  imports = [
--- a/machines/vpn/configuration.nix
+++ b/machines/vpn/configuration.nix
@@ -17,6 +17,7 @@ with lib;
  };

  imports = [
+    inputs.self.nixosModules.malobeo.vpn
    ../modules/malobeo_user.nix
    ../modules/sshd.nix
    ../modules/minimal_tools.nix
--- a/machines/vpn/dummy.yaml
+++ b/machines/vpn/dummy.yaml
@@ -0,0 +1,68 @@
+wg_private: ENC[AES256_GCM,data:s+dZfKCfrdZnFKhmCl7u1LRzR5dMflJumh1uVQ5Dktb5teohxDo0zlOR7KE=,iv:N9WSEzGonWNkqix8yaImhvrxpcAEJraWEcTrXORASow=,tag:pKgOmtKJ933FEKZVDHCWWQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age18jn5mrfs4gqrnv0e2sxsgh3kq4sgxx39hwr8z7mz9kt7wlgaasjqlr88ng
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSMFk2bzE3OG9VR0VqOTIz
+            UEQySS9SUnRmMDFqVTg1dks3WTZvbE13VGxVCitHVE1SVlBlYkZwejNlWWNMTVhF
+            M2EzSFRmS3lFd1VPMHRpMjhtMVgyVDQKLS0tIGJObk1kcWlaeUhveHdrY1BEQkh4
+            WTJua1FvNFFtMDFGWE9ZaW9wWFoxcncKlYHjkzlUj+rBPmXK/jj9XCUoGrQ4vBXH
+            ZTItzrbCI30juPjy6dJ0ffZF2ILvJLUdwurz4lZFybNuUjhE2sAY+A==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1ljpdczmg5ctqyeezn739hv589fwhssjjnuqf7276fqun6kc62v3qmhkd0c
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5S1dhZUVpT3NMaGR4eEhV
+            dUxvUGVvMUtPbWhEQnpJd3Y1YTBYbm1QMTBVCmpQbkhvM3VWV2MvcmY2RVhVOWdy
+            MVZxK201bmcwVHlwUlFnb0p5eGFNNGsKLS0tIDlrc1ErS0NiRUJ0UFZnNHNNSk9m
+            U2xLQVhoS2NxNUVvcGZBYW9VVkZNOUEKeCpijhxpkAxCB9/iIQmek03mj7b14sqs
+            CuGKgoeq7C6eG1PK3I8MzGplQMyCpEFQ+33KMj0vGwktpv/eVzC8/w==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2025-01-19T22:47:26Z"
+    mac: ENC[AES256_GCM,data:DiriXLPnm+08q1Jp1YxjEdsJzFiewQxgu1JDdevo9aGdkq92Xu8cnSxLzWUkh8bEDx4uhjOXvZd3PSU9rWiTh899U3Ou99NiSOgR1+wr5ouR20viCZqIe86YqoZlLJnYs2dlZDhL+ggwFqYJ5wfWbq7OauIVEEdnM/57RyNI2qM=,iv:lwOJi4pVGGHn7+CGq7jAHorOTFtl7ONzzV35ec1uEsg=,tag:DhjloWlsGqM579NafaERIw==,type:str]
+    pgp:
+        - created_at: "2025-01-19T21:35:34Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQGMA5HdvEwzh/H7AQwAoO2gRS+Vwu+5KZ4V2ReXPcjFcEUb6aoYe8PNUhjBu0yE
+            hs4lhWGy53BdupQ+cyxk0U+L6UhOjTfIAYE9emqTDlF5OImN/379j9NfiPe/K0/8
+            ylSOuzNgVmTpsXlHGaXE2wk0ADp1P9mZUwbJ5vHCtm+ZFe5HCuTrB61drIU0fEYw
+            NVCaARK/IRn5eAlPCjPuW1mhKP+3HNGMQszqCRKMU5kLZPzjqsHmEITSFJ5bVtAu
+            fLRtF8SyJpHgvyw1AH1IX6I+/lrDRQro0oD/0LcC1Nay9n86WIWhA/VbotyFCkI2
+            gZtV0IQq05mxO11DycgxlDLQk6nqiqDDjWv/8kj23HnQ3BAO6SXLKhHWq9rH8EOX
+            wkee7RHc09GWNcGL93YMkjIHWJyitphpU/NtTmEpTptzry5vPfittPaZ+zU+MF8G
+            REyft2X9Aj7UWcL1w3kbX9BDWuxImcirWWCShHakSrzAlpuIoXVQA6MCl6/Jr2Ve
+            lLx3lDX+BiSpU01zY32q0lgBOGcSEcRVXTiYO00EoJbIiEMwqm1aAXQzaCvhCCpP
+            54NsNzZ4eoGL0DmioKwzLbv1CpIJs3w0k6StfOtTCPKdlL99k24Z8GyJ44UnkhCe
+            V32jGc/yCong
+            =OZSD
+            -----END PGP MESSAGE-----
+          fp: c4639370c41133a738f643a591ddbc4c3387f1fb
+        - created_at: "2025-01-19T21:35:34Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hQIMA98TrrsQEbXUARAAhnd9Z1gekp4XWw+gcWK5j9mXWP7XS/FGNXmmCUBKec1j
+            zXzMJjG1YZyCYmqj3XGFMFwg2Ex6pBPoOTzOL2VOGd9mZvHjh0MGtuUopg1GprE7
+            NoyrYlV2UikyBSzVlsvykyNCYWfEt0uDotnGIK0NXYzfWfqgw+ImAH/PvNRY4nIB
+            wxI/Ze100ITAN7Dop9d6MFUZbrYKZTMsO8w5Z7TWHRPzFWH//XZjY7UpxvNVP1oJ
+            RXqqo2I97P0c6H7s17+xw6ZjyE0Qoin1gq4XSMHc4l8o+3D7fWecoTLxcjma5gvY
+            SMVCYeSrI2kc8DJ2RVeXdDlEP7SS3bwPNaE4Tklxv1rE+CUuYQ1X6dkPVKnKLfRS
+            Lwy614LDAarZmvXc3jPgFkpG+grE80PAStzOze0eWyZA/oCAI3/CS+yaeBAI4viz
+            UEkNmCMTZu1eXyIurC/suTOdq4nehGlD/2F8EKU+Y/6f6J2wHUJdvLjNxOuAOHd0
+            lGSu61gt/b2PFy/aHqFgQaZCPUMJ8UfK8JQ66zOIUW3HzsXOsvVqo/8DMQRIw7/z
+            3tZ43LPjmIeCRFwPfbIbeThoZmq1SPejkzadxDEwD3U0YAiblBJ2E+AyEDiKqP/N
+            D+NRN5Ta0ySAmGOyYgDES8QDDBQGA4cSZak7pMidCrSbAagNyYNg3Qrowc0aG93S
+            WAFD/Q4EtOdM6kveLdbDkPX/bAiCFwhzSCtDVkLAPxfrkw/a+az6emPWmImML5FT
+            to3vvXendrd9+u6uSNK5acuwzW2cW8GM4gLC90+p/kRFJukiGJbl400=
+            =6Gw4
+            -----END PGP MESSAGE-----
+          fp: aef8d6c7e4761fc297cda833df13aebb1011b5d4
+    unencrypted_suffix: _unencrypted
+    version: 3.9.2
--- a/outputs.nix
+++ b/outputs.nix
@@ -12,8 +12,241 @@
 let filter_system = name: if name == utils.lib.system.i686-linux then false else true;
 in (utils.lib.eachSystem (builtins.filter filter_system utils.lib.defaultSystems) ( system:
  let
+    baseModules = [
+      # make flake inputs accessiable in NixOS
+      { _module.args.inputs = inputs; }
+      {
+        imports = [
+          ({ pkgs, ... }: {
+            nix = {
+              extraOptions = ''
+                experimental-features = nix-command flakes
+              '';
+
+              settings = {
+                substituters = [
+                  "https://cache.dynamicdiscord.de"
+                  "https://cache.nixos.org/"
+                ];
+                trusted-public-keys = [
+                  "cache.dynamicdiscord.de:DKueZicqi2NhJJXz9MYgUbiyobMs10fTyHCgAUibRP4="
+                ];
+                trusted-users = [ "root" "@wheel" ];
+              };
+            };
+          })
+          sops-nix.nixosModules.sops
+          #microvm.nixosModules.microvm
+        ];
+      }
+    ];
+    defaultModules = baseModules;
+
+  makeMicroVM = hostName: ipv4Addr: macAddr: modules: [
+    self.nixosModules.malobeo.metrics
+    {
+      microvm = {
+        hypervisor = "cloud-hypervisor";
+        mem = 2560;
+        shares = [
+          {
+            source = "/nix/store";
+            mountPoint = "/nix/.ro-store";
+            tag = "store";
+            proto = "virtiofs";
+            socket = "store.socket";
+          }
+          {
+            source = "/var/lib/microvms/${hostName}/etc";
+            mountPoint = "/etc";
+            tag = "etc";
+            proto = "virtiofs";
+            socket = "etc.socket";
+          }
+          {
+            source = "/var/lib/microvms/${hostName}/var";
+            mountPoint = "/var";
+            tag = "var";
+            proto = "virtiofs";
+            socket = "var.socket";
+          }
+        ];
+
+        interfaces = [
+          {
+            type = "tap";
+            id = "vm-${hostName}";
+            mac = "${macAddr}";
+          }
+        ];
+      };
+
+      malobeo.metrics = {
+        enable = true;
+        enablePromtail = true;
+        logNginx = false;
+        lokiHost = "10.0.0.14";
+      };
+      
+      systemd.network.enable = true;
+      
+      systemd.network.networks."20-lan" = {
+        matchConfig.Type = "ether";
+        networkConfig = {
+          Address = [ "${ipv4Addr}/24" ];
+          Gateway = "10.0.0.1";
+          DNS = ["1.1.1.1"];
+          DHCP = "no";
+        };
+      };
+    }
+  ] ++ defaultModules ++ modules;
+
+
    pkgs-unstable = nixpkgs-unstable.legacyPackages."${system}";
    pkgs = nixpkgs.legacyPackages."${system}";
+
+    vmMicroVMOverwrites = hostname: options: {
+      microvm = {
+        mem = pkgs.lib.mkForce 4096;
+        hypervisor = pkgs.lib.mkForce "qemu";
+        socket = pkgs.lib.mkForce null;
+
+
+        #needed for hosts that deploy imperative microvms (for example fanny)
+        writableStoreOverlay = pkgs.lib.mkIf options.writableStore "/nix/.rw-store";
+        volumes = pkgs.lib.mkIf options.writableStore [ {
+          image = "nix-store-overlay.img";
+          mountPoint = self.nixosConfigurations.${hostname}.config.microvm.writableStoreOverlay;
+          size = 2048;
+        } ];
+
+        shares = pkgs.lib.mkForce (pkgs.lib.optionals (!options.writableStore) [
+          {
+            tag = "ro-store";
+            source = "/nix/store";
+            mountPoint = "/nix/.ro-store";
+          }
+        ] ++ pkgs.lib.optionals (options.varPath != "") [
+          {
+            source = "${options.varPath}";
+            securityModel = "mapped";
+            mountPoint = "/var";
+            tag = "var";
+          }
+        ]);
+
+        interfaces = pkgs.lib.mkIf (!options.withNetworking) (pkgs.lib.mkForce [{
+          type = "user";
+          id = "eth0";
+          mac = "02:23:de:ad:be:ef";
+        }]);
+
+        #if networking is disabled forward port 80 to still have access to webservices
+        forwardPorts = pkgs.lib.mkIf (!options.withNetworking) (pkgs.lib.mkForce [
+          { from = "host"; host.port = 8080; guest.port = 80; }
+        ]);
+
+      };
+
+      fileSystems = {
+        "/".fsType = pkgs.lib.mkForce "tmpfs";
+
+        # prometheus uses a memory mapped file which doesnt seem supported by 9p shares
+        # therefore we mount a tmpfs inside the datadir
+        "/var/lib/prometheus2/data" = pkgs.lib.mkIf (hostname == "overwatch" && options.varPath != "") (pkgs.lib.mkForce {
+          fsType = pkgs.lib.mkForce "tmpfs";
+        });
+      };
+
+      boot.isContainer = pkgs.lib.mkForce false;
+      services.timesyncd.enable = false;
+      users.users.root.password = "";
+      services.getty.helpLine = ''
+        Log in as "root" with an empty password.
+        Use "reboot" to shut qemu down.
+      '';
+    };
+
+    vmDiskoOverwrites = {
+      boot.initrd = {
+        secrets = pkgs.lib.mkForce {};
+        network.ssh.enable = pkgs.lib.mkForce false;
+      };
+
+      malobeo.disks.enable = pkgs.lib.mkForce false;
+      networking.hostId = "a3c3101f";
+    };
+
+    vmSopsOverwrites = host: {
+      sops.defaultSopsFile = pkgs.lib.mkForce ./machines/${host}/dummy.yaml;
+
+      environment.etc = {
+        devHostKey = {
+          source = ./machines/secrets/devkey_ed25519;
+          mode = "0600";
+        };
+      };
+
+      services.openssh.hostKeys = [{
+        path = "/etc/devHostKey";
+        type = "ed25519";
+      }];
+    };
+
+    buildVM = host: networking: sopsDummy: disableDisko: varPath: writableStore: (self.nixosConfigurations.${host}.extendModules {
+        modules = [
+          (vmMicroVMOverwrites host { 
+            withNetworking = networking; 
+            varPath = "${varPath}"; 
+            writableStore = writableStore; })
+          (if sopsDummy then (vmSopsOverwrites host) else {})
+          (if disableDisko then vmDiskoOverwrites else {})
+        ] ++ pkgs.lib.optionals (! self.nixosConfigurations.${host}.config ? microvm) [
+          #microvm.nixosModules.microvm
+        ] ++ pkgs.lib.optionals (self.nixosConfigurations.${host}.config ? services.malobeo.microvm.deployHosts) [
+          #microvm.nixosModules.host
+          {
+          services.malobeo.microvm.deployHosts = pkgs.lib.mkForce [];
+          systemd.tmpfiles.rules = builtins.concatLists (map (name: [
+            "q /var/lib/microvms/${name}/var 0755 root root - -"
+            "q /var/lib/microvms/${name}/etc 0755 root root - -"
+            "q /var/${name}/wow/it/works 0755 root root - -"
+            "q /var/lib/${name} 0755 root root - -"
+            "d /${name} 0755 root root - -"
+          ]) self.nixosConfigurations.${host}.config.services.malobeo.microvm.deployHosts);
+
+
+          microvm.vms = 
+          let
+            # Map the values to each hostname to then generate an Attrset using listToAttrs
+            mapperFunc = name: { inherit name; value = {
+              #pkgs = import self.nixosConfigurations.${name}.config.nixpkgs;
+
+              #pkgs = (buildVM name networking sopsDummy false "" false).config.nixpkgs;
+              #config = (buildVM name networking sopsDummy false "" false);
+              #pkgs = pkgs;
+              #config = self.nixosConfigurations.${name};
+              specialArgs.inputs = inputs;
+              specialArgs.self = self;
+              config = {
+                imports = (makeMicroVM "${name}" "10.0.0.11" "D0:E5:CA:F0:D7:E7" [
+              
+                  #(vmMicroVMOverwrites name { 
+                  #  withNetworking = true; 
+                  #  varPath = ""; 
+                  #  writableStore = false; })
+                  (if sopsDummy then (vmSopsOverwrites name) else {})
+
+                  
+                ]);
+
+              };
+            }; };
+          in
+          builtins.listToAttrs (map mapperFunc self.nixosConfigurations.${host}.config.services.malobeo.microvm.deployHosts);
+        }];
+      });
  in
  {
    devShells.default =
@@ -38,13 +271,19 @@ in (utils.lib.eachSystem (builtins.filter filter_system utils.lib.defaultSystems
        pkgs.mdbook
        microvmpkg.microvm
      ];
+
      packages = builtins.map (pkgName: self.legacyPackages."${pkgs.system}".scripts.${pkgName}) installed;
      shellHook = ''echo "Available scripts: ${builtins.concatStringsSep " " installed}"'';
    };
+
    legacyPackages = {
      scripts.remote-install = pkgs.writeShellScriptBin "remote-install" (builtins.readFile ./scripts/remote-install-encrypt.sh);
      scripts.boot-unlock = pkgs.writeShellScriptBin "boot-unlock" (builtins.readFile ./scripts/unlock-boot.sh);
+      scripts.run-vm = self.packages.${system}.run-vm;
    };
+
+    vmBuilder = buildVM;
+
    packages = {
      docs = pkgs.stdenv.mkDerivation {
        name = "malobeo-docs";
@@ -62,57 +301,91 @@ in (utils.lib.eachSystem (builtins.filter filter_system utils.lib.defaultSystems
          cp -r ./book/* $dest
        '';
      };
-    } //

-    builtins.foldl'
-    (result: host:
-      let
-        inherit (self.nixosConfigurations.${host}) config;
-      in
-      result // {
-        # boot any machine in a microvm
-        "${host}-vm" = (self.nixosConfigurations.${host}.extendModules {
-          modules = [{
-            microvm = {
-              mem = pkgs.lib.mkForce 4096;
-              hypervisor = pkgs.lib.mkForce "qemu";
-              socket = pkgs.lib.mkForce null;
-              shares = pkgs.lib.mkForce [{
-                tag = "ro-store";
-                source = "/nix/store";
-                mountPoint = "/nix/.ro-store";
-              }];
-              interfaces = pkgs.lib.mkForce [{
-                type = "user";
-                id = "eth0";
-                mac = "02:23:de:ad:be:ef";
-              }];
-            };
-            boot.isContainer = pkgs.lib.mkForce false;
-            users.users.root.password = "";
-            fileSystems."/".fsType = pkgs.lib.mkForce "tmpfs";
-            services.getty.helpLine = ''
-              Log in as "root" with an empty password.
-              Use "reboot" to shut qemu down.
-            '';
-          }] ++ pkgs.lib.optionals (! config ? microvm) [
-            microvm.nixosModules.microvm
-          ];
-        }).config.microvm.declaredRunner;
-    })
-    { }
-    (builtins.attrNames self.nixosConfigurations);
+      run-vm = pkgs.writeShellScriptBin "run-vm" ''
+        usage() {
+            echo "Usage: run-vm <hostname> [--networking] [--dummy-secrets] [--no-disko]"
+            echo "ATTENTION: This script must be run from the flakes root directory"
+            echo "--networking setup interfaces. requires root and hostbridge enabled on the host"
+            echo "--dummy-secrets run vm with dummy sops secrets"
+            echo "--no-disko disable disko and initrd secrets. needed for real hosts like fanny"
+            echo "--writable-store enables writable store. necessary for host with nested imperative microvms like fanny"
+            echo "--var path to directory that should be shared as /var. may require root otherwise some systemd units fail within vm. if dir is empty vm will populate"
+            exit 1
+        }
+        
+        # check at least one arg was given
+        if [ "$#" -lt 1 ]; then
+            usage
+        fi
+        
+        HOSTNAME=$1
+        
+        # Optionale Argumente
+        NETWORK=false
+        DUMMY_SECRETS=false
+        NO_DISKO=false
+        RW_STORE=false
+        VAR_PATH=""
+        
+        # check argws
+        shift
+        while [[ "$#" -gt 0 ]]; do
+            case $1 in
+                --networking) NETWORK=true ;;
+                --dummy-secrets) DUMMY_SECRETS=true ;;
+                --no-disko) NO_DISKO=true ;;
+                --writable-store) RW_STORE=true ;;
+                --var) 
+                  if [[ -n "$2" && ! "$2" =~ ^- ]]; then
+                      VAR_PATH="$2"
+                      shift
+                  else
+                      echo "Error: --var requires a non-empty string argument."
+                      usage
+                  fi
+                  ;;
+                *) echo "Unknown argument: $1"; usage ;;
+            esac
+            shift
+        done
+        echo "starting host $HOSTNAME"
+        echo "enable networking: $NETWORK"
+        echo "deploy dummy secrets: $DUMMY_SECRETS"
+        echo "disable disko and initrd secrets: $NO_DISKO"
+        echo "use writable store: $RW_STORE"
+        if [ -n "$VAR_PATH" ]; then
+          echo "sharing var directory: $VAR_PATH"
+        fi
+        
+        ${pkgs.nix}/bin/nix run --show-trace --impure --expr "((builtins.getFlake \"$(pwd)\").vmBuilder.x86_64-linux \"$HOSTNAME\" $NETWORK $DUMMY_SECRETS $NO_DISKO \"$VAR_PATH\" $RW_STORE).config.microvm.declaredRunner"
+      '';
+    };

    apps = {
      docs = {
        type = "app";
        program = builtins.toString (pkgs.writeShellScript "docs" ''
+          ${pkgs.xdg-utils}/bin/xdg-open "${self.packages.${system}.docs}/share/doc/index.html"
+        '');
+      };
+
+
+      docsDev = {
+        type = "app";
+        program = builtins.toString (pkgs.writeShellScript "docs" ''
+          echo "needs to run from infrastuctre root folder"
          ${pkgs.mdbook}/bin/mdbook serve --open ./doc
        '');
      };
+
+      run-vm = {
+        type = "app";
+        program = "${self.packages.${system}.run-vm}/bin/run-vm";
+      };
    };

-  })) // rec {
+  })) // {
    nixosConfigurations = import ./machines/configuration.nix (inputs // {
      inherit inputs;
      self = self;
@@ -123,6 +396,7 @@ in (utils.lib.eachSystem (builtins.filter filter_system utils.lib.defaultSystems
      microvm.imports = [ ./machines/modules/malobeo/microvm_host.nix ];
      vpn.imports = [ ./machines/modules/malobeo/wireguard.nix ];
      initssh.imports = [ ./machines/modules/malobeo/initssh.nix ];
+      metrics.imports = [ ./machines/modules/malobeo/metrics.nix ];
      disko.imports = [ ./machines/modules/disko ];
    };
Author	SHA1	Message	Date
kalipso	af11810935	wip All checks were successful Check flake syntax / flake-check (push) Successful in 4m5s Details	2025-01-21 22:17:16 +01:00
kalipso	81f73a0704	[overwatch] grafana provision datasource and dashboards All checks were successful Check flake syntax / flake-check (push) Successful in 4m3s Details	2025-01-20 22:31:36 +01:00
kalipso	7730163b84	[metrics] enable on all microvms All checks were successful Check flake syntax / flake-check (push) Successful in 4m9s Details	2025-01-20 21:32:07 +01:00
kalipso	725efc3e0a	[metrics] init module	2025-01-20 21:32:07 +01:00
kalipso	724f14cfcd	[overwatch] backup dashboard	2025-01-20 21:32:07 +01:00
kalipso	cf3bfa1784	[infradocs] fix loki addr	2025-01-20 21:32:07 +01:00
kalipso	c71d7959f2	[run-vm] handle edgecase for prometheus mmaped file on 9p share	2025-01-20 21:31:43 +01:00
kalipso	155e78b519	[run-vm] use securityModel mapped to allow mounting /var 9p share	2025-01-20 21:31:43 +01:00
kalipso	c54f04cb62	[nix] change .#docs to just start browser and use .#docsDev for local development	2025-01-20 21:31:43 +01:00
kalipso	197a8427b7	[docs] add run-vm examples	2025-01-20 21:31:43 +01:00
kalipso	1877f6dc9c	[run-vm] allow sharing of /var/lib sharing /var somehow doesnt work. for example nginx fails because of lacking permissions to access /var/log/nginx. this also happens when run-vm is started as root. thats why only /var/lib is shared which still allows application persistency between tests	2025-01-20 21:31:43 +01:00
kalipso	c47e93170f	[run-vm] add flag to disable disko needed to run fanny as vm	2025-01-20 21:31:43 +01:00
kalipso	eea5d6924d	[nix] add run-vm script	2025-01-20 21:31:43 +01:00
kalipso	468106677c	[vpn] update dummy secrets	2025-01-20 21:31:43 +01:00
kalipso	df4d769f82	[lucia] init dummy secrets	2025-01-20 21:31:43 +01:00
kalipso	249eebf164	[fanny] init dummy secrets	2025-01-20 21:31:43 +01:00
kalipso	e3bcd250e0	[durruti] rm secrets.yaml - currently empty	2025-01-20 21:31:43 +01:00
kalipso	1b871f3860	[nextcloud] add dummy secrets	2025-01-20 21:31:43 +01:00
kalipso	3564436dfe	[sops] add a dummy key to allow secret usage within test vms	2025-01-20 21:31:43 +01:00
kalipso	4744324385	[sops] test sharing hostkey with vm	2025-01-20 21:31:43 +01:00
kalipso	88ef307a65	[fanny] proxypass cloud.malobeo.org	2025-01-20 21:31:43 +01:00
kalipso	dec968a4db	[fanny] deploy nextcloud	2025-01-20 21:31:43 +01:00
kalipso	283dc51d67	[nextcloud] rm obsolete nameserver	2025-01-20 21:31:43 +01:00
ahtlon	f4a6c40cd2	[nextcloud] nextcloud works now	2025-01-20 21:31:43 +01:00
ahtlon	23caa27d4e	[nextcloud] flake update because for some reason the sha changed	2025-01-20 21:31:43 +01:00
ahtlon	d6aee8657b	[nextcloud] enable postgress, redis, change domain	2025-01-20 21:31:43 +01:00
ahtlon	e7e05327e4	[nextcloud] rm discourse	2025-01-20 21:31:40 +01:00
kalipso	1fc3538e03	[nextcloud] fix hostname	2025-01-20 21:30:48 +01:00
ahtlon	1ebee6d886	add nextcloud collectives	2025-01-20 21:30:48 +01:00
ahtlon	94e439bf0b	login geht	2025-01-20 21:30:48 +01:00
ahtlon	e50f3349ba	nextcloud minimal	2025-01-20 21:30:48 +01:00
ahtlon	18b747a7df	sops....	2025-01-20 21:30:48 +01:00
ahtlon	ea6e019b64	Start over but right this time	2025-01-20 21:30:48 +01:00
ahtlon	8581f762a2	basic discourse example	2025-01-20 21:30:48 +01:00
ahtlon	b223f0cb0c	with nix flake check the hydraJobs output is evaluated in the same way as Hydra's hydra-eval-jobs	2025-01-20 21:30:48 +01:00
ahtlon	9ba607ce16	Only run on push	2025-01-20 21:30:48 +01:00
ahtlon	34c2661c53	[actions] Add flake check	2025-01-20 21:30:48 +01:00
ahtlon	211799b6b9	[modules] move microvm module import from makeMicroVM to baseModules	2025-01-20 21:30:48 +01:00
ahtlon	543c4ed49e	[scripts] move packages to legacyPackages	2025-01-20 21:30:48 +01:00
ahtlon	27085dd3e6	[scripts] check for flake.nix	2025-01-20 21:30:48 +01:00
ahtlon	8076956982	[packages] make scripts available in shell without nix run	2025-01-20 21:30:48 +01:00
kalipso	26829f9255	[infradocs] provide stats All checks were successful Evaluate Hydra Jobs / eval-hydra-jobs (push) Successful in 4m55s Details Evaluate Hydra Jobs / eval-hydra-jobs (pull_request) Successful in 5m57s Details	2025-01-18 20:01:27 +01:00
kalipso	0d93cad9d4	[overwatch] init All checks were successful Evaluate Hydra Jobs / eval-hydra-jobs (push) Successful in 4m19s Details	2025-01-18 11:40:27 +01:00
kalipso	02a57d98d0	[microvms] rm nameserver option	2025-01-18 11:39:36 +01:00
kalipso	4553c2c069	[nix] do not rm network interfaces of vms	2025-01-17 16:19:19 +01:00
kalipso	3f3dca3c7f	[docs] make readme the index still most of it is quite out of date...	2025-01-17 14:30:49 +01:00
				`@@ -0,0 +1 @@`
				`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGmeyBkJbC8RMkqhl/tAUoItpSePyGKhyL5J7ArxxRTC kalipso@celine`