Hello Friends!
The silent admin has come out of hibernation to give you an update. As we've been moderating, approving, and all around enjoying the content reddthat & the lemmyverse provides I figured we might as well have an update.
Reddthat has been upgraded to the latest lemmy verison 0.19.18. 🥳
See the link in the post for the main updates, as well as the 0.19.17 and 0.19.16 versions. But honestly... there hasn't been many changes that are relatable for end users. The Lemmy devs have been pushing out only serious fixes. The only one that is even slightly relevant is 'Display UserBadges for Bot, Banned and Deleted users in all PersonListings'
It looks like this:
We've been kicking along but have seen a huge increase in our image/video storage! Which has unfortunately meant that I'm getting charged close to $44USD/m for our storage, and we are now further away from breaking even. So all the savings we manged to find with our server migration has been offset with our increase in storage usages. This happened a couple months ago and I don't have a pretty graph showing the increase, but happened in Feb based on the financials.
We regular serve about 1 to 2 TB per month from the server and have apparently "saved" 1TB over the past 30 days.
Something else that is interesting (to me at least), and might be part of the current state of AI nonsense is that even though we have always blocked 'old.reddthat.com' from the search engines it is still significantly queried more than other hostnames. Including one of our old frontends 'next' which has been removed for a "long time".
The DNS query map shows the closest datacentre for those queries. Interestingly there are some tiny tiny tiny datacenters there.
The total number of connections also is an indication of how popular the lemmyverse is. As this includes the server to server connections, not just our regular users it is a bit polluted but shows how many connections we need to respond to on a regular basis.
And if you've made it to the end, I want to make a special shoutout to everyone for being around, having fun, and being respectful to our fellow netizens. Unfortunately there have been a few bad actors so I would like to reiterate our No. 1 Rule. Remember the human! (no harassment, threats, etc.) (Yeah... While it's certainly not number 1 in the list, it should be!) On a whole Reddthat users are always special in my heart and we have been very "stable" in the number of users. For those of you who regularly, comment, vote, post, and use reddthat thankyou!
An an extra special shoutout to those who "put their money where their mouth is" and support Reddthat financially. I honestly wouldn't be doing this after all these years without your continued support.
Cheers,
Tiff
Note: On Liberapay, donations are paid in advance, but you are more than welcome to make it recurring monthly instead of paying yearly. Don't worry too much about the "fees". It's just the cost of doing business via the credit card duopoly.
💸 "Expenses":
I expect future costs to stay around the 150 mark. We were so close to being self sufficient but as always, if we get close to our goal of $60/w we'll finally be in a solid position to stay forever online.
⭐ Donation "Statistics":
🥅 Goal: 27.36 / 60.00
Want a month dedicated to you? -> https://liberapay.com/reddthat
PS: don't like fees? Use Crypto (Litecoin/Monero) for even better transaction fees than credit cards for your donation. (See the main sidebar for addresses). And validate them again on liberapay too if you want to ensure I get those dollary doos!
I've made my position pretty clear how this was the wrong move by our government...
Services that eSafety considers will be age-restricted social media platforms
Facebook
Instagram
Kick
Reddit
Snapchat
Threads
TikTok
Twitch
X (formerly Twitter)
YouTube
Services that eSafety considers will not be age-restricted social media platforms
Discord
GitHub
Google Classroom
LEGO Play
Messenger
Pinterest
Roblox
Steam and Steam Chat
WhatsApp
YouTube Kids
Looks easy : https://www.ifixit.com/Guide/Steam+Deck+SSD+Replacement/148989
Edit: Is it worth 30-60minutes of your time, the screwdrivers, maybe the spatchula, and reinstalling steamOS onto the drive?
This is sso support as the client. So you could use any backend that supports the oauth backend (I assume, didn't look at it yet).
So you could use a forgejo instance, immediately making your git hosting instance a social platform, if you wanted.
Or use something as self hostable like hydra.
Or you can use the social platforms that already exist such as Google or Microsoft. Allowing faster onboarding to joining the fediverse. While allowing the issues that come with user creation to be passed onto a bigger player who already does verification. All of these features are up for your instance to decide on.
The best part, if you don't agree with what your instance decides on, you can migrate to one that has a policy that coincides with your values.
Hope that gives you an idea behind why this feature is warranted.
We enabled the CloudFlare AI bots and Crawlers mode around 0:00 UTC (20/Sept).
This was because we had a huge number of AI scrapers that were attempting to scan the whole lemmyverse.
It successfully blocked them... While also blocking federation 😴
I've disabled the block. Within the next hour we should see federation traffic come through.
Sorry for the unfortunate delay in new posts!
Tiff
That's a big decision I won't make without community input as it would affect all of us.
If we purely treated it as just another instance with no history then I believe our stance on it would be to allow them, as we are an allow-first type of instance. While there are plenty of people we might not want to interact with, that doesn't mean we should immediately hit that defederate button.
When taking history into account it becomes a whole different story. One may lean towards just saying no without thought.
All of our content (Lemmy/Fediverse) is public by default (at the present time) searchable by anyone and even if I were to block all of the robots and crawlers it wouldn't stop anyone from crawling one of the many other sites where all of that content is shared.
A recent feature being worked on is the private/local only communities. If a new Lemmy instance was created and they only used their local only communities, would we enact the same open first policy when their communities are closed for us to use? Or would we still allow them because they can still interact, view comments, vote and generate content for our communities etc?
What if someone created instances purely for profit? They create an instance corner stone piece of the "market" and then run ads? Or made their instance a subscription only instance where you have to pay per month for access?
What if there are instances right now federating with us and will use the comments and posts you make to create a shit-posting-post or to enhance their classification AI? (Obviously I would be personally annoyed, but we can't stop them)
An analogy of what threads is would be to say threads is a local only fediverse instance like mastodon, with a block on replies. It restricts federation to their users in USA, Canada and Japan and Users cannot see when you comment/reply to their posts and will only see votes. They cannot see your posts either and only allow other fediverse users to follow threads users.
With all of that in mind if we were to continue with our open policy, you would be able to follow threads users and get information from them, but any comments would stay local to the instance that comments on the post (and wouldn't make it back to threads).
While writing up to this point I was going to stay impartial... But I think the lack of two way communication is what tips the scales towards our next instance block. It might be a worthwhile for keeping up-to-date with people who are on threads who don't understand what the fediverse is. But still enabled the feature because it gives their content a "wider reach" so to speak. But in the context of Reddthat and people expressing views and opinions, having one sided communication doesn't match with what we are trying to achieve here.
Tiff
Source(s): https://help.instagram.com/169559812696339/about-threads-and-the-fediverse/
PS: As we have started the discussion I'll leave what I've said for the next week to allow everyone to reply and see what the rest of the community thinks before acting/ blocking them.
Edit1:(30/Mar) PPS: we are currently not federated with them, as no one has bothered to initiate following a threads account
I managed to streamline the exports and syncs so we performed them concurrently. Allowing us to finish just under 40 minutes! Enjoy the new hardware!
So it begins: (Federation "Queue")
Successfully migrated from Postgres 15 to Postgres 16 without issues.
It's a sad day when something like this happens. Unfortunately with how the Lemmy's All works it's possible a huge amount of the initial downvotes are regular people not wanting to see the content, as downvotes are federated. This constituted as part of my original choices for disabling it when I started my instance. We had the gripes people are displaying here and it probably constituted to a lack in Reddthat's growth potential.
There needs to be work done not only for flairs, which I like the idea of, but for a curated All/Frontpage (per-instance). Too many times I see people unable to find communities or new content that piques their interest. Having to "wade through" All-New to find content might attribute to the current detriment as instead of a general niche they might want to enjoy they are bombarded with things they dislike.
Tough problem to solve in a federated space. Hell... can't even get every instance to update to 0.18.5 so federated moderation actions happen. If we can't all decide on a common Lemmy instance version, I doubt we can ask our users to be subjected to not using the tools at their disposal. (up/down/report).
Keep on Keeping on!
Tiff - A fellow admin.
Don't forget & in community names and sidebars.
Constantly getting trolled by &
We are now using v0.18.3!
There was extended downtime because docker wouldn't cooperate AT ALL.
The nginx proxy container would not resolve the DNS. So after rebuilding the containers twice and investigating the docker network settings, a "simple" reboot of the server fixed it!
upstream lemmy-ui & upstream lemmy. These are DNS entries which are cached for a period of time. So if a new container comes online it doesn't actually find the new containers because it cached all the IPs that lemmy-ui resolves too. (In this example it would have been only 1, and then we add more containers the proxy would never find them).
4.1 You can read more here: http://forum.nginx.org/read.php?2,215830,215832#msg-215832I get notified whenever reddthat goes down, most of the time it coincided with me banning users and removing content. So I didn't look into it much, but honestly the uptime isn't great. (Red is <95% uptime, which means we were down for 1 hour!).
Actually, it is terrible.
With the changes we've made i'll be monitoring it over the next 48 hours and confirm that we no longer have any real issues. Then i'll make a real announcement.
Thanks all for joining our little adventure!
Tiff
These were because of recent spam bots.
I made some changes today. We now have 4 containers for the UI (we only had 1 before) and 4 for the backend (we only had 2)
It seems that when you delete a user, and you tell lemmy to also remove the content (the spam) it tells the database to mark all of the content as deleted.
Kbin.social had about 30 users who posted 20/30 posts each which I told Lemmy to delete.
This only marks it as deleted for Reddthat users until the mods mark the post as deleted and it federates out.
The UPDATE in the database (marking the spam content as deleted) takes a while and the backend waits(?) for the database to finish.
Even though the backend has 20 different connections to the database it uses 1 connection for the UPDATE, and then waits/gets stuck.
This is what is causing the outages unfortunately and it's really pissing me off to be honest. I can't remove content / action reports without someone seeing an error.
I don't see any solutions on the 0.18.3 release notes that would solve this.
So to combat this a little I've increased our backend processes from 2 to 4 and our front-end from 1 to 4.
My idea is that if 1 of the backend processes gets "locked" up while performing tasks, the other 3 processes should take care of it.
This unfortunately is an assumption because if the "removal" performs an UPDATE on the database and the /other/ backend processes are aware of this and wait as well... This would count as "locking" up the database and it won't matter how many processes I scale out too, the applications will lockup and cause us downtime.
Note: we are kinda doing #3 point already it does a round-robbin (tries each sequentially). But from what I've seen in part of the logs it can't differentiate between one that is down and one that is up. (From the nginx documentation, that feature is a paid one)
Cheers, Tiff
Hey! Sorry for the joke, I didn't expect it to be seen by a real user!
As we are one of the very few instances that has a no email policy there is very few ways in which we can determine if a person signing up is a bot or a regular user.
Recently a very very specific person or group of people have been abusing Reddthat to create accounts, then ask interesting questions (let's just say that), and then proceed to delete their account (which deletes all of their posts and comments). This makes it impossible to figure out what they have done unless someone quotes the reply or reports it before they delete it.
I'm sorry you got caught up in the little bit of fun us admins have with writing little anecdotes or fun catch phases!
You are welcome to come say hi on Reddthat any time!