malobeo/infrastructure
3
1
Fork 1
Code Issues 49 Pull Requests 5 Actions 1 Projects 3 Releases Wiki Activity

Hydra Migration #169

New Issue
Closed
opened 2026-08-15 11:50:23 +02:00 by kalipso · 16 comments
kalipso commented 2026-08-15 11:50:23 +02:00
Owner
Copy Link
Copy Source

malobeo job evaluation is getting to CPU intensive on the current host. It slows down other services which external users rely on. For this reason a migration to our own machine is needed.

I propose using one of the thinkcentres for this, which we got donated.

malobeo job evaluation is getting to CPU intensive on the current host. It slows down other services which external users rely on. For this reason a migration to our own machine is needed. I propose using one of the `thinkcentres` for this, which we got donated.
👍 1
kalipso added the MigrationLokal labels 2026-08-15 11:50:23 +02:00
kalipso changed title from Hydra Migratoin to Hydra Migration 2026-08-15 11:51:22 +02:00
kalipso commented 2026-08-28 15:50:54 +02:00
Author
Owner
Copy Link
Copy Source

I disabled hydra on the server. Till we have our own buildhost setup the hydra callback ci pipeline should be ignored on prs.

Starting the hydra-evaluator.service leads to consumption of roughly 26GB of RAM. The host itself only has 32GB in total.
I suspect the pull request evaluation to be the cause since this high amount of RAM usage is new. We should investigate if such high RAM usage is also happening on albert once its running.

I disabled hydra on the server. Till we have our own buildhost setup the hydra callback ci pipeline should be ignored on prs. Starting the `hydra-evaluator.service` leads to consumption of roughly 26GB of RAM. The host itself only has 32GB in total. I suspect the pull request evaluation to be the cause since this high amount of RAM usage is new. We should investigate if such high RAM usage is also happening on `albert` once its running.
ahtlon commented 2026-08-28 16:09:13 +02:00
Owner
Copy Link
Copy Source

It seems fine on albert for now image.png

It seems fine on albert for now <img width="553" alt="image.png" src="attachments/1ae0ebaa-8869-4ef3-8cc0-53492b68f502">
image.png
134 KiB
ahtlon commented 2026-08-28 16:09:49 +02:00
Owner
Copy Link
Copy Source

I'll probably go by malo tomorrow and plug it in so we can continue the switch

I'll probably go by malo tomorrow and plug it in so we can continue the switch
kalipso commented 2026-08-28 16:14:58 +02:00
Author
Owner
Copy Link
Copy Source

I now just disabled the malobeo2 jobset, which evaluated PRs, leading to a instant drop of RAM consumption.

So hydra is running again, just PR evaluation is disabled at the moment - means autoupdate continues to work as expected.

I now just disabled the `malobeo2` jobset, which evaluated PRs, leading to a instant drop of RAM consumption. So hydra is running again, just PR evaluation is disabled at the moment - means autoupdate continues to work as expected.
kalipso commented 2026-08-28 16:15:33 +02:00
Author
Owner
Copy Link
Copy Source

It seems fine on albert for now image.png

Did you setup the pr evaluation on this host already?

And if yes, how many PRs is it evaluating currently?

> It seems fine on albert for now <img width="553" alt="image.png" src="attachments/1ae0ebaa-8869-4ef3-8cc0-53492b68f502"> Did you setup the pr evaluation on this host already? And if yes, how many PRs is it evaluating currently?
ahtlon commented 2026-08-28 16:24:31 +02:00
Owner
Copy Link
Copy Source

Yes, image.png
Currently not evaling anything but you can see the peak was 13 G which is a lot but I also changed it to 6 parallels
Wait, isn't albert in the vpn? then we can already switch over by just using that ip

Yes, ![image.png](/attachments/9d9951b3-0350-46a4-b932-9ad8f60f1a95) Currently not evaling anything but you can see the peak was 13 G which is a lot but I also changed it to 6 parallels Wait, isn't albert in the vpn? then we can already switch over by just using that ip
image.png
119 KiB
ahtlon commented 2026-08-28 16:30:43 +02:00
Owner
Copy Link
Copy Source

Also, comparing to the cloud server, evals seem to be like 10x faster image.png
image.png

Also, comparing to the cloud server, evals seem to be like 10x faster <img width="361" alt="image.png" src="attachments/48efa9d5-0df6-48e9-8766-ce23176b5c5a"> <img width="427" alt="image.png" src="attachments/dedc6610-bf07-4c10-934e-353c3839768e">
image.png
16 KiB
image.png
16 KiB
kalipso commented 2026-08-28 16:44:52 +02:00
Author
Owner
Copy Link
Copy Source

Wait, isn't albert in the vpn? then we can already switch over by just using that ip

#170 needs to be merged before vpn will pickup albert.

If you want to start using albert through vpn proxy forward on server (host_config.nix) and vpn would need to be setup and dns configured to the servers public ip.

But this would need to be changed again as soon as albert is setup within malobeo. Therefore i suggest to wait with it till we have it running locally. But if you still want to go ahead with it thats fine for me.

> Wait, isn't albert in the vpn? then we can already switch over by just using that ip https://git.dynamicdiscord.de/malobeo/infrastructure/pulls/170 needs to be merged before vpn will pickup albert. If you want to start using albert through vpn proxy forward on server (host_config.nix) and vpn would need to be setup and dns configured to the servers public ip. But this would need to be changed again as soon as albert is setup within malobeo. Therefore i suggest to wait with it till we have it running locally. But if you still want to go ahead with it thats fine for me.
ahtlon commented 2026-08-28 17:10:15 +02:00
Owner
Copy Link
Copy Source

meh not worth it for the day

meh not worth it for the day
ahtlon commented 2026-08-29 22:03:27 +02:00
Owner
Copy Link
Copy Source

#170 is merged but fanny will not auto update image.png

#170 is merged but fanny will not auto update <img width="1020" alt="image.png" src="attachments/aedafcdc-d403-47f4-acb8-fea78f533bb8">
image.png
160 KiB
ahtlon commented 2026-08-30 00:36:23 +02:00
Owner
Copy Link
Copy Source

Uh apparently it worked at some point, theres a new problem now image.png which is probably caused by albert not being reachable for some reason...

Uh apparently it worked at some point, theres a new problem now <img width="1029" alt="image.png" src="attachments/8d7fe8fc-67cd-4691-8878-62acf88cb7cc"> which is probably caused by albert not being reachable for some reason...
image.png
115 KiB
kalipso commented 2026-08-31 15:20:00 +02:00
Author
Owner
Copy Link
Copy Source

The caches pubkey was never added to the trusted hosts:

infrastructure/machines/modules/host_builder.nix Line 33 in 60c12ce7af
trusted-public-keys = [

This should have been done before the migration. Once this is updated fanny/albert need to be deployed manually and afterwards the autoupdate hopefully starts working.

The caches pubkey was never added to the trusted hosts: https://git.dynamicdiscord.de/malobeo/infrastructure/src/commit/60c12ce7afa8f8e5f7b78a74b5eb035732e8eb36/machines/modules/host_builder.nix#L33 This should have been done before the migration. Once this is updated fanny/albert need to be deployed manually and afterwards the autoupdate hopefully starts working.
kalipso commented 2026-08-31 15:27:14 +02:00
Author
Owner
Copy Link
Copy Source

Also you forgot to set proxy_set_header :D It needs to be set as it is done for all other nginx virtualHosts on fanny:

infrastructure/machines/fanny/configuration.nix Line 220 in 60c12ce7af
proxy_set_header Host $host;

The fact its missing makes external requests fail since they redirect to alberts local IP:
image.png

I also think the nginx directive on albert it self should not have been removed as it was done here:
103c7730be

It should stay, only forceSSL and enableACME should be disabled. This way albert webinterface can still be used without depending on a secondary host like fanny.

Also you forgot to set `proxy_set_header` :D It needs to be set as it is done for all other nginx virtualHosts on fanny: https://git.dynamicdiscord.de/malobeo/infrastructure/src/commit/60c12ce7afa8f8e5f7b78a74b5eb035732e8eb36/machines/fanny/configuration.nix#L220 The fact its missing makes external requests fail since they redirect to alberts local IP: ![image.png](/attachments/0d93f31f-a5e3-4d48-bdc3-ab8a3ac064fa) I also think the nginx directive on albert it self should not have been removed as it was done here: https://git.dynamicdiscord.de/malobeo/infrastructure/commit/103c7730beb85da278cf086284cd807f77c5bb76 It should stay, only forceSSL and enableACME should be disabled. This way albert webinterface can still be used without depending on a secondary host like fanny.
image.png
6.8 KiB
ahtlon commented 2026-08-31 15:37:14 +02:00
Owner
Copy Link
Copy Source

But the inbound port 80/443 forwards to fanny anyway so theres no way to reach that nginx directive on albert. Or do you mean setting up services.nginx.virtualHosts."hydra.hq.malobeo.org"?

But the inbound port 80/443 forwards to fanny anyway so theres no way to reach that nginx directive on albert. Or do you mean setting up `services.nginx.virtualHosts."hydra.hq.malobeo.org"`?
kalipso commented 2026-08-31 15:59:14 +02:00
Author
Owner
Copy Link
Copy Source

No i was talking about the hydra.malobeo.org - but it is not really important for now. When migrating to HAProxy this will be needed again, for now we can leave it as it is.

No i was talking about the `hydra.malobeo.org` - but it is not really important for now. When migrating to HAProxy this will be needed again, for now we can leave it as it is.
ahtlon commented 2026-08-31 21:28:07 +02:00
Owner
Copy Link
Copy Source

Done! yay!

Done! yay!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Bug/Problem
Lokal
Migration
Module/Disks
Module/Initssh
Request
Security
Service
Updates
bump
nextcloud
No labels Lokal Migration
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Bender (Bender Bending Rodríguez) ahtlon kalipso quinn
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: malobeo/infrastructure#169