How to remotely reboot a Linux host if SSH fails to connect? (lemmy.nz)

submitted 2 years ago* (last edited 2 years ago) by sylverstream@lemmy.nz to c/selfhosted@lemmy.world

49 comments fedilink hide all child comments

Edit2: Thanks all for your responses! I have checked the logs, https://lemmy.nz/comment/6192604, and based on that removed tracker-miner-fs as it's a search/index tool which I don't need. No idea why it took over all memory. I'll also get a WiFi Smartplug as a kill switch. Hopefully that solves it. Thanks again heaps!

I've got a HP ProDesk G3 which I'm using as home server, I've installed Ubuntu on it. Earlier this week the services I host on it stopped (Immich & Frigate). I tried to SSH, but it just hung after asking for a password. I could ping it, but it was just unresponsive.

I had to force reboot it manually. This is fine, but I'm not always at home.

The chip has Intel vPro as far as I know, which could be an option, but I have no idea how this works. The documentation on the Intel site seems focused on enterprises. I tried to connect with RealVNC which does not work, so I think I've got to install/configure something on the server first.

I also asked Bing Chat but it came up with non existing packages & commands. Welcome your thoughts!

/edit: I just found this, which seems to be exactly what I need: https://manpages.ubuntu.com/manpages/focal/en/man7/amt-howto.7.html

you are viewing a single comment's thread
view the rest of the comments

[-] sylverstream@lemmy.nz 5 points 2 years ago

Yes, thanks for that. Good point. I checked the logs, and minutes before it crashed I can see below in the logs. Seems like either a GPU error or out of memory error. No idea what tracker-miner-f is by the way. It also shows a massive list of processes with their memory usage.

This goes beyond my knowledge :(

Feb 21 17:27:49 hppd600-g3 kernel: i915 0000:00:02.0: [drm] GPU HANG: ecode 9:0:00000000
Feb 21 17:32:43 hppd600-g3 kernel: 1305621 total pagecache pages
Feb 21 17:32:43 hppd600-g3 kernel: 16258 pages in swap cache
Feb 21 17:32:43 hppd600-g3 kernel: Free swap  = 0kB
Feb 21 17:32:43 hppd600-g3 kernel: Total swap = 1000444kB
Feb 21 17:32:43 hppd600-g3 kernel: 2065206 pages RAM
Feb 21 17:32:43 hppd600-g3 kernel: 0 pages HighMem/MovableOnly
Feb 21 17:32:43 hppd600-g3 kernel: 64196 pages reserved
Feb 21 17:32:43 hppd600-g3 kernel: 0 pages hwpoisoned

Feb 21 17:32:43 hppd600-g3 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-113.slice/user@113.service/background.slice/tracker-miner-fs-3.service,task=t>
Feb 21 17:32:43 hppd600-g3 kernel: Out of memory: Killed process 833 (tracker-miner-f) total-vm:625676kB, anon-rss:3144kB, file-rss:4816kB, shmem-rss:4kB, UID:113 pgtables:280kB oom_score_adj:200
Feb 21 17:32:43 hppd600-g3 kernel: i915 0000:00:02.0: [drm] Resetting rcs0 for stopped heartbeat on rcs0

[-] ryannathans@aussie.zone 6 points 2 years ago

Tracker miner fs generates thumbnails for files iirc. There was a recent vulnerability where malicious files could crash it and execute code just by being on disk. Make sure you haven't been hit by malware

[-] sylverstream@lemmy.nz 4 points 2 years ago

I've uninstalled it, it's an index/search tool. Don't need it :D

[-] ryannathans@aussie.zone 3 points 2 years ago

Usually comes with your DE, sometimes removing it breaks your DE

[-] sylverstream@lemmy.nz 1 points 2 years ago

Yeah tracker miner sounds dodgy. I've only installed Immich & Frigate on the box, and no dodgy repositories. It's also auto updating. Will do research how to check for malware, thought that was a Windows only thing :D

[-] constantokra@lemmy.one 1 points 2 years ago

I've previously had a problem with my server becoming unresponsive when running immich. It's been a while, but I remember there being some kind of memory leak having to do with immich. It was in their GitHub issues and everything. On my system it would take about a day and a half and then ssh, along with everything else, would become unresponsive. Rebooting would fix it for a day and a half. I stopped running immich and it hasn't happened since. I suppose you could try using a cron job to restart immich periodically and see if that resolves your problem.

[-] sylverstream@lemmy.nz 2 points 2 years ago

That is good to know! Will keep an eye on memory usage of immich. I really like it, so I'm reluctant to let it go.

this post was submitted on 22 Feb 2024

64 points (95.7% liked)

Selfhosted

60789 readers

235 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world