[overwatch] add the blackbox exporter to monitor websites #177

Manually merged
ahtlon merged 3 commits from add_blackbox_metrics into master 2026-09-07 17:18:32 +02:00
Owner
No description provided.
ahtlon force-pushed add_blackbox_metrics from f0de7390e9 to ae1c69277a 2026-09-01 19:53:23 +02:00 Compare
ahtlon added 1 commit 2026-09-01 21:02:41 +02:00
[overwatch] add the blackbox exporter to monitor websites
Check flake syntax / flake-check (push) Successful in 12m9s
Hydra callback / on_pr (pull_request) Successful in 6m40s
Hydra callback / on_push (pull_request) Skipped
d50a9ddbbf
ahtlon force-pushed add_blackbox_metrics from ae1c69277a to d50a9ddbbf 2026-09-01 21:02:41 +02:00 Compare
ahtlon requested review from kalipso 2026-09-02 09:19:20 +02:00
Author
Owner

You said you didn't wanna run too many services on one microvm. Should I make a new one for this or do we count this as in scope?

You said you didn't wanna run too many services on one microvm. Should I make a new one for this or do we count this as in scope?
Author
Owner

Also, I tested this on local already, it works

Also, I tested this on local already, it works
ahtlon added 1 commit 2026-09-04 18:21:51 +02:00
[overwatch] enable and scrape metrics from hydra
Check flake syntax / flake-check (push) Successful in 10m15s
Hydra callback / on_pr (pull_request) Successful in 4m41s
Hydra callback / on_push (pull_request) Skipped
4c8253e8e3
ahtlon force-pushed add_blackbox_metrics from 116db8851b to 4c8253e8e3 2026-09-04 18:46:07 +02:00 Compare
Author
Owner

Also, do you wanna add any more sites to monitor before merging?

Also, do you wanna add any more sites to monitor before merging?
kalipso left a comment
Owner

Other than that looks goot to me

Other than that looks goot to me
@@ -112,0 +123,4 @@
static_configs = [{
targets = [
"https://malobeo.org"
"https://cloud.malobeo.org"
Owner
There are many domains missing here: - https://antamap.malobeo.org - https://docs.malobeo.org - https://events.malobeo.org - https://hydra.malobeo.org - https://keys.malobeo.org - https://tasklist.malobeo.org - https://zines.malobeo.org
ahtlon marked this conversation as resolved
Owner

You said you didn't wanna run too many services on one microvm. Should I make a new one for this or do we count this as in scope?

No new host needed

> You said you didn't wanna run too many services on one microvm. Should I make a new one for this or do we count this as in scope? No new host needed
Owner

In general i think the addition of domain monitoring here is very nice - but at the same time it is a problem that it is coupled to our local infra. The previously used status.malobeo.org ran on an external host with very high uptime and kind of guaranteed network connectivity. This way it was able to send notifications even if our local infra went down.

The solution for domain monitoring proposed in this MR only works as long as our local network is up, and fanny is actually running. It can only send notifications if some single service fails, but a real network/electricity outage will never trigger notifications since grafana.malobeo.org will be also affected by it.

I think its fine for now, but in the future we should separate our monitoring from the actual local infra to keep it independent. This is either linked to #117 or some self hosted solution in another location

In general i think the addition of domain monitoring here is very nice - but at the same time it is a problem that it is coupled to our local infra. The previously used `status.malobeo.org` ran on an external host with very high uptime and kind of guaranteed network connectivity. This way it was able to send notifications even if our local infra went down. The solution for domain monitoring proposed in this MR only works as long as our local network is up, and fanny is actually running. It can only send notifications if some single service fails, but a real network/electricity outage will never trigger notifications since `grafana.malobeo.org` will be also affected by it. I think its fine for now, but in the future we should separate our monitoring from the actual local infra to keep it independent. This is either linked to https://git.dynamicdiscord.de/malobeo/infrastructure/issues/117 or some self hosted solution in another location
ahtlon added 1 commit 2026-09-07 16:59:09 +02:00
[overwatch] blackbox: add more domains
Check flake syntax / flake-check (push) Successful in 12m17s
Hydra callback / on_pr (pull_request) Successful in 3m31s
Hydra callback / on_push (pull_request) Skipped
8ee0a99ce7
ahtlon manually merged commit a6de2c394d into master 2026-09-07 17:18:32 +02:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: malobeo/infrastructure#177