This is something that seems to happen randomly to my little RaspberryPi 4B homeserver, like once every few months maybe.

What happens is that out of nowhere (meaning I didn't make any change to config or installation) I cannot reach any services I host on it (i.e. Jellyfin, Anchor notes...). I cannot access it via SSH at all, connection keeps timing out.

Ping however gives a signal. I can see/hear its running and doing something, since I hear the external HDD working. It is running headless and I have no monitor and/or keyboard I could connect.

So far my only way out is to just switch its power off, which is obviously not good for the hard drive. It has happened now like half a dozen times over the past few years, so I am wondering what a better way is to handle this. This was on RPI OS and now its happening on DietPi, so seems not related to the OS version.

I tried sending a shutdown signal via SSH but that also just times out, so how do I get it to shutdown properly?

Secondly, what kind of logs could I look at or start collecting to make sense of this?

all 5 comments

sorted by: hot top controversial new old
[–] 1 point 9 hours ago

Had this problem when my storage filled up

  • source
  • [–] 1 point 14 hours ago

    systemd will try to save any kernel panics in pstore via systemd-pstore, if it's enabled in your kernel. I'd check /var/lib/systemd/pstore and see if anything in there on the next boot.

    Can you elaborate on ping "running"? Do you get actual icmp replies coming back? Because there's no code path I can imagine where a ping would cause hdd activity (on a normally running system).

    If ssh times out (and you don’t do anything fancy with the firewall), then it's not sshd dead, it's sshd not being able to respond. Grab a tcpdump for dst port 22 from your local machine while RPi is stuck and see if you get any replies whatsoever or it's just retransmits going into the void.

    My first rough suspicion would be ram abuse. Something eats up all ram and the system locks up and semi-dies. Pstore would have OOMs. You could run a local script for telemetry recording too to see if ram use spikes up before the system gets unresponsive.

  • source
  • [–] 1 point 22 hours ago* (last edited 21 hours ago) (2 children)

    It sounds to me like ssdh may have stopped working. That may explain why you can't ssh into your server but pings still respond. I have a Raspberry Pi4 and a Pi5 and have had similar issues in the past.

    I would probably approach this issue by writing a small script that checks every so often if the process sshd is still alive and if not restart sshd. Maybe SystemD can so something similar but I am not familiar with SystemD.

    Edit: A quick and simple script looks like this

    #!/bin/sh
    
    # Check if `sshd` process is running; If not running, `pgrep` returns
    # an exit status of '1' and restarts `sshd`
    pgrep 'sshd' > /dev/null || systemctl restart sshd
    

    Make the script executable with chmod +x /home/user_name/sshd_check

    Add the following line to /etc/crontabs/root to run the script every 15 minutes

     */15    *       *       *       *       /home/user_name/sshd-check
    

    I don't use SystemD but I am pretty sure systemctl restart sshd is correct, otherwise it can be changed to whatever your operating system uses to control services

  • source
  • hideshow 2 child comments
  • [–] [S] 1 point 16 hours ago (1 child)

    Thanks, I'll try this. What speaks against this is that all my services turn unreachable, and those do not rely on ssh access. But still worth a try.

  • source
  • parent
  • hideshow 1 child comment
  • Aah you did mention that, my eyes just decided to skip that when I read your post.

    This reminds me of another issue I ran into but I use Alpine Linux so I don't know if it's a distribution specific issue. I'll share the issue and workaround solution anyways as something to consider.

    Networking on Alpine Linux is controlled by a process called networking and for reasons I don't understand and can't see by any logs, it just stops working. I can't ssh or access the reverse proxy port. I don't remember if ping was working or not as it's been a while since I dealt with it now.

    My work around was to have a script on my server ping a known location and restart networking if it couldn't ping out. If a second ping after restarting the process failed, it would then restart the device. This script would run every 15 minutes.

    It's a bandage solution that doesn't solve the problem but it does keep my server running. However it seems like pings still work with your server so you might need to get creative in how you test your server's connectivity.

  • source
  • parent