lab/homelab
1
0
Fork 0
Code Issues 33 Projects Releases Packages Wiki Activity Actions

Setup lightweight monitoring: Beszel + Uptime Kuma (replace Grafana/Prometheus) #11

New issue
Open
opened 2026-01-17 22:16:54 +00:00 by lab · 4 comments
lab commented 2026-01-17 22:16:54 +00:00
Owner
Copy link

Goal: lightweight monitoring replacing Grafana/Prometheus

Current kube-prometheus-stack (Prometheus + Grafana + node-exporter + Alertmanager) uses ~1.5GB+ across pods. Replace with a lightweight stack, but keep Grafana running until the new stack is proven working.

Target stack (approved 2026-08-01)

Component What it does Resource cost
Beszel hub Dashboard, host metric history, alerts ~25-50 MB
Beszel agent (DaemonSet, all nodes) Host metrics: CPU, mem, disk, network, temp, SMART ~6-23 MB/node
Uptime Kuma Synthetic HTTP checks + cert expiry ~150 MB

Alerts: Beszel threshold alerts (CPU / mem / disk / agent status) -> ntfy via shoutrrr (ntfy already running in cluster).

Scope / caveats (decided 2026-08-01)

  • No per-container stats on K3s nodes -- Beszel agent only reads docker.sock/podman.sock, nodes run containerd. Host-level metrics only. If container stats become must-have later: nerdctl docker.sock bridge, or wait for Beszel K8s metrics integration (henrygd/beszel#1586, in progress).
  • No log aggregation -- Loki dropped, few logs flowing (see #41).
  • Pi (infra-pi) is currently powered off on purpose (low cluster load). Beszel agent DaemonSet will pick it up when it comes back.
  • K8s-native metrics (pod health, PVC usage, Longhorn) are intentionally not covered -- rely on native k8s features + Beszel host alerts.

Steps

  1. Deploy Beszel hub via official Helm chart.
  2. Deploy Beszel agent as DaemonSet (all nodes) + binary agent on PVE hosts.
  3. Expose beszel.yukselcloud.com via Caddy (DDNS + route done).
  4. Configure alerts in hub -> ntfy (CPU, memory, disk > 80%, status down). Beszel already has default notifications; set the shoutrrr URL to the homelab-alerts topic.
  5. Deploy Uptime Kuma (Helm) -- see #41.
  6. Add Caddy routes: uptime-kuma behind Authelia (see #27).
  7. Add Beszel + Kuma links to Homer dashboard (homer/config.yaml).
  8. Run both stacks in parallel for 2-4 weeks, verify Beszel captures everything Grafana did (history, node down, disk full, cert expiry).
  9. Tear down kube-prometheus-stack (helm uninstall kube-prometheus-stack -n monitoring), remove node-exporter, release ~1.5GB. Close #28 as moot.

Decisions still open

  • Beszel hub auth: local account vs Authelia OIDC (hub supports OIDC)
  • Uptime Kuma status page public vs private

Progress

  • Steps 1-3 (cert fixed 2026-08-01: removed crowdsec bouncer from beszel caddy block which was breaking ACME http-01)
  • Step 4 in progress: ntfy token created, shoutrrr URL ntfy://:tk_...@ntfy.yukselcloud.com/homelab-alerts (empty username + token = basic auth, works)
# Goal: lightweight monitoring replacing Grafana/Prometheus Current kube-prometheus-stack (Prometheus + Grafana + node-exporter + Alertmanager) uses ~1.5GB+ across pods. Replace with a lightweight stack, but **keep Grafana running until the new stack is proven working**. ## Target stack (approved 2026-08-01) | Component | What it does | Resource cost | |-----------|-------------|---------------| | **Beszel hub** | Dashboard, host metric history, alerts | ~25-50 MB | | **Beszel agent** (DaemonSet, all nodes) | Host metrics: CPU, mem, disk, network, temp, SMART | ~6-23 MB/node | | **Uptime Kuma** | Synthetic HTTP checks + cert expiry | ~150 MB | Alerts: Beszel threshold alerts (CPU / mem / disk / agent status) -> **ntfy** via shoutrrr (ntfy already running in cluster). ## Scope / caveats (decided 2026-08-01) - **No per-container stats on K3s nodes** -- Beszel agent only reads `docker.sock`/`podman.sock`, nodes run containerd. Host-level metrics only. If container stats become must-have later: nerdctl docker.sock bridge, or wait for Beszel K8s metrics integration (henrygd/beszel#1586, in progress). - **No log aggregation** -- Loki dropped, few logs flowing (see #41). - **Pi (infra-pi) is currently powered off** on purpose (low cluster load). Beszel agent DaemonSet will pick it up when it comes back. - K8s-native metrics (pod health, PVC usage, Longhorn) are intentionally not covered -- rely on native k8s features + Beszel host alerts. ## Steps 1. Deploy Beszel hub via official Helm chart. 2. Deploy Beszel agent as DaemonSet (all nodes) + binary agent on PVE hosts. 3. Expose beszel.yukselcloud.com via Caddy (DDNS + route done). 4. Configure alerts in hub -> ntfy (CPU, memory, disk > 80%, status down). Beszel already has default notifications; set the shoutrrr URL to the homelab-alerts topic. 5. Deploy Uptime Kuma (Helm) -- see #41. 6. Add Caddy routes: uptime-kuma behind Authelia (see #27). 7. Add Beszel + Kuma links to Homer dashboard (`homer/config.yaml`). 8. **Run both stacks in parallel for 2-4 weeks**, verify Beszel captures everything Grafana did (history, node down, disk full, cert expiry). 9. Tear down kube-prometheus-stack (`helm uninstall kube-prometheus-stack -n monitoring`), remove node-exporter, release ~1.5GB. Close #28 as moot. ## Decisions still open - Beszel hub auth: local account vs Authelia OIDC (hub supports OIDC) - Uptime Kuma status page public vs private ## Progress - [x] Steps 1-3 (cert fixed 2026-08-01: removed crowdsec bouncer from beszel caddy block which was breaking ACME http-01) - [ ] Step 4 in progress: ntfy token created, shoutrrr URL `ntfy://:tk_...@ntfy.yukselcloud.com/homelab-alerts` (empty username + token = basic auth, works)
lab commented 2026-07-25 19:37:12 +00:00
Author
Owner
Copy link

Monitoring stack (Prometheus + Grafana + node-exporter + Alertmanager → ntfy) is deployed and working. Remaining work: add Prometheus alerting rules, consider Loki for log aggregation, create cluster health Grafana dashboard.

Monitoring stack (Prometheus + Grafana + node-exporter + Alertmanager → ntfy) is deployed and working. Remaining work: add Prometheus alerting rules, consider Loki for log aggregation, create cluster health Grafana dashboard.
lab changed title from Setup monitoring stack to Setup lightweight monitoring: Beszel + Uptime Kuma (replace Grafana/Prometheus) 2026-08-01 19:15:47 +00:00
lab commented 2026-08-01 19:23:13 +00:00
Author
Owner
Copy link

Progress: step 1 done (2026-08-01)

Beszel hub deployed. Raw manifests in k3s-ha-cluster/monitoring/beszel/ (deployment + service + pvc).

  • Namespace: monitoring, image henrygd/beszel:0.18.7
  • Longhorn PVC beszel-data (2Gi) — Bound
  • MetalLB IP: 192.168.0.225 (service beszel)
  • GitOps workflow: .forgejo/workflows/deploy-beszel.yml (trigger: k3s-ha-cluster/monitoring/beszel/**)
  • Verified: pod Ready, http://192.168.0.225:8090 returns the Beszel dashboard (HTTP 200)
  • APP_URL currently http://192.168.0.225:8090 — flip to https://beszel.yukselcloud.com when the Caddy route lands (step 6)

Note: used raw manifests instead of the official Helm chart — the chart PR (henrygd/beszel#1582) is unmerged/broken (references undefined values, no env/APP_URL support, appVersion 0.17.0). Raw manifests match repo convention (caddy, ddns).

Next: step 2 — Beszel agent DaemonSet.

## Progress: step 1 done (2026-08-01) **Beszel hub deployed.** Raw manifests in `k3s-ha-cluster/monitoring/beszel/` (deployment + service + pvc). - Namespace: `monitoring`, image `henrygd/beszel:0.18.7` - Longhorn PVC `beszel-data` (2Gi) — Bound - MetalLB IP: `192.168.0.225` (service `beszel`) - GitOps workflow: `.forgejo/workflows/deploy-beszel.yml` (trigger: `k3s-ha-cluster/monitoring/beszel/**`) - Verified: pod Ready, `http://192.168.0.225:8090` returns the Beszel dashboard (HTTP 200) - `APP_URL` currently `http://192.168.0.225:8090` — flip to `https://beszel.yukselcloud.com` when the Caddy route lands (step 6) > Note: used raw manifests instead of the official Helm chart — the chart PR (henrygd/beszel#1582) is unmerged/broken (references undefined values, no env/APP_URL support, appVersion 0.17.0). Raw manifests match repo convention (caddy, ddns). Next: step 2 — Beszel agent DaemonSet.
lab commented 2026-08-01 19:38:08 +00:00
Author
Owner
Copy link

Progress: steps 1-2 done (2026-08-01)

Beszel hub + agents deployed. All manifests in k3s-ha-cluster/monitoring/beszel/.

  • Hub (step 1): deploy + svc + pvc, MetalLB 192.168.0.225, image henrygd/beszel:0.18.7 — done, HTTP 200
  • Agents (step 2): DaemonSet with hostNetwork + tolerations, image henrygd/beszel-agent:0.18.7 — 4/4 nodes Running (172/182/171/181), WebSocket connected to hub. infra-pi Pending (node offline, picks up when back)
  • Secret: monitoring/beszel/secret.yaml is a placeholder (GitHub mirror is public). Real values applied manually:
    kubectl -n monitoring create secret generic beszel-agent --from-literal=key="<hub public key>" --from-literal=token="<permanent universal token>"
  • Universal token set to permanent in hub settings (Settings -> Tokens & Fingerprints)
  • GitOps workflow: .forgejo/workflows/deploy-beszel.yml — applies deployment/service/pvc/agent-daemonset, explicitly excludes secret.yaml so the placeholder never overwrites the real secret

Next: verify systems appear in the hub dashboard, configure ntfy alerts (step 4), deploy Uptime Kuma (#41).

## Progress: steps 1-2 done (2026-08-01) **Beszel hub + agents deployed.** All manifests in `k3s-ha-cluster/monitoring/beszel/`. - **Hub** (step 1): deploy + svc + pvc, MetalLB `192.168.0.225`, image `henrygd/beszel:0.18.7` — done, HTTP 200 - **Agents** (step 2): DaemonSet with hostNetwork + tolerations, image `henrygd/beszel-agent:0.18.7` — 4/4 nodes Running (`172/182/171/181`), WebSocket connected to hub. `infra-pi` Pending (node offline, picks up when back) - **Secret**: `monitoring/beszel/secret.yaml` is a **placeholder** (GitHub mirror is public). Real values applied manually: `kubectl -n monitoring create secret generic beszel-agent --from-literal=key="<hub public key>" --from-literal=token="<permanent universal token>"` - Universal token set to **permanent** in hub settings (Settings -> Tokens & Fingerprints) - GitOps workflow: `.forgejo/workflows/deploy-beszel.yml` — applies deployment/service/pvc/agent-daemonset, **explicitly excludes secret.yaml** so the placeholder never overwrites the real secret Next: verify systems appear in the hub dashboard, configure ntfy alerts (step 4), deploy Uptime Kuma (#41).
lab commented 2026-08-01 19:41:37 +00:00
Author
Owner
Copy link

Progress: PVE hosts monitored (2026-08-01)

Beszel binary agents installed on both bare-metal Proxmox hosts (not via cluster DaemonSet — PVE is a minimal Debian host, no Docker):

  • lab-pve1 (192.168.0.51) — systemd service beszel-agent, active, WebSocket connected
  • lab-pve2 (192.168.0.52) — same

Install method (runs the official install script as root):

curl -sL https://get.beszel.dev -o /tmp/install-agent.sh && chmod +x /tmp/install-agent.sh && /tmp/install-agent.sh -p 45876 -k "<hub public key>" -t "<permanent universal token>" -url "http://192.168.0.225:8090" --auto-update=false

Auto-update disabled to keep versions pinned (consistent with #30). Same permanent universal token as the K8s agents, so they auto-registered as new systems lab-pve1 / lab-pve2.

Value here: .52 hosts the K3s external MariaDB datastore + NFS (media), so disk/S.M.A.R.T./network monitoring of both hypervisors is useful. Note: host metrics only, no container stats (no Docker socket).

Verify: hub dashboard should now show 6 systems (4 K8s nodes + 2 PVE hosts).

## Progress: PVE hosts monitored (2026-08-01) **Beszel binary agents installed on both bare-metal Proxmox hosts** (not via cluster DaemonSet — PVE is a minimal Debian host, no Docker): - `lab-pve1` (192.168.0.51) — `systemd` service `beszel-agent`, active, WebSocket connected - `lab-pve2` (192.168.0.52) — same Install method (runs the official install script as root): ```bash curl -sL https://get.beszel.dev -o /tmp/install-agent.sh && chmod +x /tmp/install-agent.sh && /tmp/install-agent.sh -p 45876 -k "<hub public key>" -t "<permanent universal token>" -url "http://192.168.0.225:8090" --auto-update=false ``` Auto-update disabled to keep versions pinned (consistent with #30). Same permanent universal token as the K8s agents, so they auto-registered as new systems `lab-pve1` / `lab-pve2`. **Value here**: .52 hosts the K3s external MariaDB datastore + NFS (media), so disk/S.M.A.R.T./network monitoring of both hypervisors is useful. Note: host metrics only, no container stats (no Docker socket). Verify: hub dashboard should now show 6 systems (4 K8s nodes + 2 PVE hosts).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
feature/42-ddns-automation
feature/41-uptime-kuma
feature/11-grafana-cleanup
feature/11-drop-crowdsec-beszel
feature/11-beszel-notify-fix
feature/11-beszel-workflow-dispatch
feature/11-beszel-monitoring
feature/forgejo-action
feature/monitoring
feature/metube
feature/crowdsec
feature/mysql
feature/termix
feature/forgejo-runner
feature/authelia
feature/lldap
No results found.
Labels
Clear labels   
Monitoring

Monitoring stack for environment

  
automation

Automation/CI tasks

  
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
feature

New feature

  
forgejo

Forgejo setup/config

  
help wanted

Need some help

  
invalid

Something is wrong

  
k3s

K3s related

  
keycloak

Keycloak related

  
ldap

Light LDAP implementation for authentication

  
question

More information is needed

  
wontfix

This won't be fixed

No labels
Monitoring
automation
bug
duplicate
feature
forgejo
help wanted
invalid
k3s
keycloak
ldap
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lab/homelab#11
No description provided.