Have you heard of webtrees? Self hosted, similar enough to Ancestry and you can upload relevant media to their entry. Not sure if it can handle a few hundred gigs but you could probably host the media on a separate program to reduce the DB strain.

I've used it when my wife was getting into genealogy. Still running in the background for the last few years with little to no maintenance.

https://tngsitebuilding.com/index.php

Edit: Ah it's not foss though.

What you're looking for is genealogy software. You've got me curious about this as well now, hadn't thought about it before. I'm seeing some recommendations online for Gramps Web, don't know if it's a perfect fit but might be a good starting point.

It sounds like you want a combination of things, which is why you are getting such varied responses. Keep in mind, you probably aren’t going to find an all in one solution, but with some docker images you could run a couple different solutions, one for each task. Immich for photos, and maybe another for the family tree.

I was in a similar boat and stumbled across “digital gardens”. I made my own digital garden, and I think if it as a life wiki. It has tech notes, images, recipes, etc. Obviously it’s not a drop in replacement for good photo hosting software or ancestry software tho.

HTH!

Wordpress?

Self host dokuwiki. Simple, easy, effective. I do.

Which ones have you looked at?

First hit on my search was https://www.bookstackapp.com/ which seems to be a nice FOSS wiki with MIT licensing.

Mediawiki?

Nextcloud offers practically everything you ask for besides the family tree and even that might be available as a plugin (or "App" as Nextcloud calls tem). Or if you're willing to split photos in a different app, Immich works great.