nixos, mostly. It has a very steep learning curve but if it breaks I’m just "whatever, rollback now, fix properly later".

I actually don’t remember off the top of my head, huh. The output chain is not the one I use often. I'd think nft add rule tracing filter output ip protocol icmp icmp type echo-reply meta nftrace set 1 would do it. Just make sure the priority is low enough again.

systemd will try to save any kernel panics in pstore via systemd-pstore, if it's enabled in your kernel. I'd check /var/lib/systemd/pstore and see if anything in there on the next boot.

Can you elaborate on ping "running"? Do you get actual icmp replies coming back? Because there's no code path I can imagine where a ping would cause hdd activity (on a normally running system).

If ssh times out (and you don’t do anything fancy with the firewall), then it's not sshd dead, it's sshd not being able to respond. Grab a tcpdump for dst port 22 from your local machine while RPi is stuck and see if you get any replies whatsoever or it's just retransmits going into the void.

My first rough suspicion would be ram abuse. Something eats up all ram and the system locks up and semi-dies. Pstore would have OOMs. You could run a local script for telemetry recording too to see if ram use spikes up before the system gets unresponsive.

It is accepted just fine then. You might want to check the reverse, start in top of the output chain and trace the echo reply.

nftables offers a very decent debugging interface. First, you add a rule to trace the packet (a new chain with high priority works best). Usually I’d suggest to add those rules by hand instead of relying on declarative configs:

nft add table ip ping_trace
nft 'add chain ip ping_trace prerouting { type filter hook prerouting priority -301; policy accept; }'

Then you add the actual tracing rule there with the shape to match the incoming traffic:

nft 'add rule ip ping_trace prerouting icmp type { echo-request, echo-reply } meta nftrace set 1'

now you can run nft monitor trace and see the decisions made for the matching packets. Remember to delete the ping_trace table afterwards to clean up.

First on how to get an ASN: you can buy it for reasonably cheap from a LIR. Some will even toss a free /48 with that. Happy to offer names in private so that there's no advertising. Expect a budget quoted above.

Once you have an ASN, you need to get an upstream - actually two as RIPE mandates at least two (otherwise why’d you need an ASN). Some LIRs would offer transit with ASN purchase. You can upstream via your ISP, if they allow you to (that's very rare). Another option is a tunnel (there are free and paid ones) or a VM somewhere (some cloud providers offer to set up bgp with VMs they host). Generally, free ones are enough for basic stuff. Not much bandwidth and oftentimes IPv6 only, but you don’t pay anything either. Besides, you can ask around in various network related chats. Practically, I can offer ip transit with some marginally low burstable bandwidth, and that’s pretty common. You can look/ask around https://discord.gg/ipv6 for example.

For getting ASN to your homelab you’re looking at a tunnel option, most probably. Great if you have static ipv4 - allows you to use more common tunnels, but is still doable with a floating IP (e.g. check bgptunnel).

BGP anycast person here. If you have any presence in the region RIPE operates in then the pricing is within the homelab reach. ASN and a block of /48 would be about 70 EUR/year.

Alternatively, something like route64 would happily tunnel you IPs they announce for about 2 EUR/month.

I’ve used VM in my homelab for a while - it's genuinely a smaller ram footprint, especially on the ingress path. You can even scale query separately if required.

I'd suggest VictoriaMetrics over pure Prometheus - better ram footprint is pretty useful nowadays.

You probably want to mention that some prominent cryptographers think this just astroturfing?

I actually had lots of fun writing a clojure compiler in rust that would target wasm. This allows to write it all in rust and then get it to self-host bit by bit.

Or you could, idk, have some metrics. That's a wild idea, of course. Who uses monitoring when you can just ask an LLM?