One day, our internet just stopped working entirely. Our ISP was fine, router good, but no outside connection and I couldn’t access local devices. I have two Unraid servers. I took a quick glance at them, they were powered on, all looked ok. I decide to power cycle everything. Still no internet. Then I realize that I can still access Google from my phone when on WiFi.. hmm. That would mean… an issue with pi-hole!

I begin to try to diagnose from my phone with a hotspot to my computer. I’m getting warmer but some things just aren’t making sense. I go back and stare at the server running pi-hole. Then it fucking hits me.

I have the pretty odd and totally not ideal home network. I have plans to upgrade everything but currently I have two wifi networks tied to the same WAN. One network is run off of access points so I can get Ethernet to odd areas of the property. The Unraid server hosting pi-hole was connected to an AP and the Ethernet cable I used didn’t have a clip, so it just barely unseated itself from it, breaking the connection. The router+pi-hole config meant also breaking the internet if anything happened to pi-hole or that server. I swapped it for a better cable and it all instantly started working.

Spent that weekend sorting out the network so it doesn’t happen again. It was definitely a head-scratcher.

For a while, the windows machine I was using kept randomly becoming unresponsive (both via network, as well as the keyboard+mouse) and I just could not figure out why. Everytime it did, I'd have to force a shutdown by holding the power button until it shutoff, then press it again to start back up. Sometimes this happened while I wasn't home, so it had to just stay offline until I was.

I got sick of having to do this; so I replaced the power button with a transistor and a RPI. The PI would ping the server every 5min; if it failed to get a response 3 times in a row, it would trigger the transistor for 10sec, release for 3sec, then press and release again; forcing a poweroff then starting the machine again. It'd also write these events to a log file. I think it was around twice a week ish.

Never did figure out what the cause of the lockup was; but it stopped when I replaced windows with Debian.

Well at home i only have problems with spiders in my racks, not like the racks i have seen at work. Highlights are toiletpaper, chipmunks+nuts and worst in a school server rack a gun.

Cat vomit... my cat hit my raspberry pi.

Nice chonky heatsink, goddam

Not self hosted but at work: after an update to both our firewall cluster nodes they both acted wonky, restarted 5 min after booting, ssh connection wasn't working half the time, dropped packages... the problem persisted through multiple reboots, support had no idea.

the fix was to pull the plug entirely, wait 10 seconds then power them on again. on both nodes we had this behavior and both worked again after losing power for a few seconds.

seems that even for business appliances "have you tried to turn it off and on again" is a viable solution.

At my first-ever tech job back in 2000, I was given a little Celeron desktop computer to do my work: mostly writing some ColdFusion and cutting up images in Photoshop. For the most part, the machine worked fine, except whenever I scrolled down on a webpage in Netscape, the box's PC speaker would start screaming. I tried to ignore it, but I was in a small room with two other nerds and it was getting really annoying.

One day, my frustrated colleague decided it was time to investigate. As part of the process, he noticed that there was very little air coming out of the power supply fan port. "Must be a busted fan and that's causing an overheating alarm" he declared. We unplugged the box, popped it onto my desk, and opened it up. The internals were pristine. No dust, nothing. The computer was practically brand new after all.

We were just about to crack open the power supply (not recommended, but we were getting desperate) when my colleague noticed that something was wedged inside the fan... it was the biggest (dead, thankfully) cockroach I have ever seen, at least 10cm long. For context, this was in Vancouver, Canada. Those just don't exist there.

My colleague jammed his screwdriver in there to grind up the cockroach carcass, then plugged the box back in. After a shittone of dessicated cockroach guts spewed out the back, my little computer was operating normally.

As best we could guess, the monster crawled into the fan in Malaysia when as was assembled and somehow got pinned in there, died of starvation, and then stowed away to Canada.

Had a problem where my ports would suddenly close. At first I thought it was Telekom, then the modem, then the NAS.

After much investigation I discovered that restarting my router would solve it.

After some more weeks I discovered that the crawling bots were triggering some stateful tplink firewall that would close everything until rebooted.

So the router in the middle of the setup was the problem, not the edges.

You don't need a mouse for a server. Just SSH into it, duh.

I lost a raspberry pi for a couple months. Just couldn't find the darn thing. It was working fine serving away.

Turns out it was in the basement. I forgot I moved it to be closer to my tilt Hydrometer.

for me it was the fact that the desktop case i used for the server had the power button on top and it was big enough to be pushed in, when my cat walked over it...

Mine would be when I had a 48 bay disk array / JBOD fail on me... badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.

These were all running bcache on top of mdraid... One single mount.

Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.

I just HAD to solve this one though.

In come several spare chassis / mobo etc.... get a bunch of drives powered and on /dev/ , move to the next.

A couple spare gigabit switches...

several gigabit NICs...

two explicit paths for each machine...

a bit of iSCSI magic, and one machine now had the physical disks all exposed to it... mdadm --assemble blah blah, bit of UUID chaos...

It's surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.

Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of "I can lose this and don't care" vs "this box gets RAID10 and/or offsite backups nightly".

back up your critical stuff people!