12
Something’s up with federation from multiple instances (lemmy.ml)
submitted 1 month ago by Trying2KnowMyself@lemmy.ml to c/meta@lemmy.ml
27 comments fedilink hide all child comments

I have posts and comments on Hexbear from 4 hrs ago that don’t show up

I have a comment on Lemmygrad from 4 hrs ago that doesn’t show up

A recent series of posts from ml to other instances have almost no interaction locally. When viewed remotely, they have higher vote counts (no more comments to compare).

I haven’t extensively looked at the new feed to see if posts are outright missing, but I see the same general pattern spot checking posts that do appear: vote totals much lower than elsewhere, missing comments

I don’t see issues reported on https://phiresky.github.io/lemmy-federation-state/site

top 27 comments
sorted by: hot top new old
[-] Trying2KnowMyself@lemmy.ml 4 points 1 month ago

@nutomic@lemmy.ml @dessalines@lemmy.ml tagging because something’s broke

[-] dessalines@lemmy.ml 1 points 1 month ago
[-] Trying2KnowMyself@lemmy.ml 3 points 1 month ago* (last edited 1 month ago)

It doesn’t show anything different when I reverse them, but the problem isn’t outgoing it’s incoming.

3 upvotes, zero comments: https://lemmy.ml/post/48485322?scrollToComments=true

18 upvotes, 16 comments: https://hexbear.net/post/8719960

Same post.

Or, to the extent it’s also ~~incoming~~outgoing, I haven’t tried to diagnose that.

[-] nutomic@lemmy.ml 2 points 1 month ago

Dont know whats going on with these comments, but federation status shows up to date in both directions:

[-] Muehe@lemmy.ml 3 points 1 month ago

OP already mentioned in the post body that the problem doesn't seem to show up on that site. Just compare the TopDay/Active sort of different instances, lemmy.ml is missing a whole lot of full posts by now, not just comments.

[-] nutomic@lemmy.ml 5 points 1 month ago

I found it now, posts are coming in normally again.

[-] Muehe@lemmy.ml 4 points 1 month ago* (last edited 1 month ago)

Nice. Will old comments/votes also start coming in? I can see posts popping up, but many votes/comments still seem to be missing.

Edit:

Example of things still missing:

https://lemmy.ml/post/48501463
https://lemmy.world/post/47922684

Edit2:

Oh yeah, the post seems to be gaining stuff as time goes on, just added another upvote and a comment thread from 22 hours ago.

[-] nutomic@lemmy.ml 3 points 1 month ago

Old posts and comments will come in when we receive votes or replies for them. But missing votes won't be federated.

[-] Muehe@lemmy.ml 1 points 1 month ago

Hmm, yeah, the post I linked above is still missing some 50 comments, and most of the post and comment upvotes. Kind of interesting though that it acquired 92/148 comments (at time of writing), before you pushed the fix yesterday it had 2 upvotes and 0 comments. Guess the upvotes are just what has been added since then.

Would be nice if there was a contingency option for server admins for situations like this, no? Something like "Resynch federation events from to ", maybe limitable to certain instances/communities/posts. Not sure if that's possible with how the federation works currently, but if it is it would be a good feature I guess.

[-] nutomic@lemmy.ml 2 points 1 month ago

Normally Lemmy returns an error if activities cant be handled (eg instance is down). Then the sending instance keeps retrying until it is successful. But in this case its a bit different, and the sending looked successful.

Anyway I made a change to the federation library to add Host header automatically if missing, then this problem wont happen anymore in the future.

[-] Muehe@lemmy.ml 2 points 1 month ago

But in this case its a bit different, and the sending looked successful.

Yeah I got that. But couldn't you (theoretically) add a "resend activities from " option to the API? So you can ask other servers for things you missed. I mean what happened here is one problem where that may be useful, but another would be something like having to restore from day old backups, being a new instance with no history, a network outage, or missing things for whatever other reason.

Like I said I don't know the data structures involved, but in my layperson head it sounds possible, although you may have to throttle it to prevent abuse I guess.

[-] Trying2KnowMyself@lemmy.ml 1 points 1 month ago

internet-delenda-est

Yep. I checked that site and it definitely doesn’t show any issues, clearly everything is working.

Plenty of other folks are chiming in with the same experiences. It’s very clearly not just Hexbear. Is every other instance broken? Why are they federating just fine with each other, then?

https://lemmy.world/post/47958962

https://lemmy.ml/post/48512217

https://discuss.online/post/40968502

https://lemmy.ml/post/48508670

https://aussie.zone/post/33509821

https://lemmy.ml/post/48507829

https://piefed.blahaj.zone/c/womensstuff/p/792979/thoughts-on-this

https://lemmy.ml/post/48509886

https://startrek.website/post/40652133

https://lemmy.ml/post/48504964

https://lemmy.dbzer0.com/post/70286030

https://lemmy.ml/post/48512741

https://lemmy.ca/post/66154822

https://lemmy.ml/post/48503190

And yeah, that doesn’t even get into the posts that haven’t federated at all. I only went through a few pages of /new and I’m on posts that are hours old.

[-] nutomic@lemmy.ml 7 points 1 month ago* (last edited 1 month ago)

Found the problem, I had commented out the Host header in nginx thinking it would not be necessary. Turns out I was wrong. Adding the header back in fixed the problem.

Big thanks for notifying about this!

[-] Trying2KnowMyself@lemmy.ml 2 points 1 month ago

Thanks for fixing it!

[-] nutomic@lemmy.ml 5 points 1 month ago

You're right, its also really noticable if you just compare the All feed, on lemmy.ml almost everything from local users. Most posts from other instances are missing. In the server logs are a lot of errors like this:

lemmy-1  |    0: lemmy_server::root_span_builder::HTTP request
lemmy-1  |            with http.method=POST http.scheme="http" http.host=0.0.0.0:10633 http.target=/inbox otel.kind="server" request_id=780b6573-2672-4361-87c2-6ece46e272ee
lemmy-1  |              at src/root_span_builder.rs:16
lemmy-1  | 2026-06-09T17:34:25.537123Z  WARN lemmy_server::root_span_builder: Unknown: Incoming activity has invalid signature

I thought at first that it would be related to the new 0.19.19 release, but downgrading to 0.19.18 doesnt fix the problem.

Issue link: https://github.com/LemmyNet/lemmy/issues/6581

[-] dessalines@lemmy.ml 2 points 1 month ago

That means that something is going on with hexbear's outgoing federation. You'll need to ask them if there's something going wrong.

[-] Trying2KnowMyself@lemmy.ml 2 points 1 month ago

internet-delenda-est

Plenty of other folks are chiming in with the same experiences. It’s very clearly not just Hexbear. Is every other instance broken? Why are they federating just fine with each other, then?

https://lemmy.world/post/47958962

https://lemmy.ml/post/48512217

https://discuss.online/post/40968502

https://lemmy.ml/post/48508670

https://aussie.zone/post/33509821

https://lemmy.ml/post/48507829

https://piefed.blahaj.zone/c/womensstuff/p/792979/thoughts-on-this

https://lemmy.ml/post/48509886

https://startrek.website/post/40652133

https://lemmy.ml/post/48504964

https://lemmy.dbzer0.com/post/70286030

https://lemmy.ml/post/48512741

https://lemmy.ca/post/66154822

https://lemmy.ml/post/48503190

And yeah, that doesn’t even get into the posts that haven’t federated at all. I only went through a few pages of /new and I’m on posts that are hours old.

[-] dessalines@lemmy.ml 1 points 1 month ago

@nutomic@lemmy.ml figured it out, should be good now.

[-] Muehe@lemmy.ml 2 points 1 month ago* (last edited 1 month ago)

Nah, something's wrong here on lemmy.ml. Compare these sites:

https://lemmy.world/?dataType=Post&listingType=All&sort=TopDay
https://lemmy.ml/?dataType=Post&listingType=All&sort=TopDay

On world it's FuckAI post with just north of 900 upvotes, on ml it's mostly local posts with the first one having 62 upvotes...

Same is true when comparing other instances, they all look similar to worlds TopDay sort, ml is the odd one out.

[-] eldavi@lemmy.ml 2 points 1 month ago

it's not just hexbear, traffic here is noticeably & significantly down. i use lemmy at the same times every weekday and have developed a sense of normal based on comment frequency and it's somewhere between 2x (for today) and 4x (for yesterday)

[-] Trying2KnowMyself@lemmy.ml 3 points 1 month ago

Even this post has a comment that didn’t federate from Hexbear, it’ll be 2 soon enough: https://hexbear.net/comment/7223117

[-] solrize@lemmy.ml 3 points 1 month ago

Yeah I noticed a big slowdown starting a few hours ago, but I didn't try to identify specifics. Over on .world there are lots more newer posts.

[-] Trying2KnowMyself@lemmy.ml 1 points 1 month ago* (last edited 1 month ago)

I assume world is more broadly federated, but could be wrong. There’s definitely something specifically up with ml though.

[-] solrize@lemmy.ml 3 points 1 month ago

I'm used to world and ml having about the same stuff, at least with regard to my own subscriptions. Ml actually has a few rederations like hexbear that are missing from world, and I read hexbear's covid group.

[-] chgxvjh@hexbear.net 0 points 1 month ago

Not sure if normal is the right term but that's sort of what I learned to expect from activitypub.

[-] Trying2KnowMyself@hexbear.net 0 points 1 month ago

Guess what? I’m breaking my general rule about using one account per thread again because your comment didn’t federate to ml! Anyways, yeah, to quote myself griping previously:

federation sucks and I hate it.

[-] chgxvjh@hexbear.net 1 points 1 month ago

On Lemmy I don't really care if the federation works but on Mastodon&co I live in constant fear that I'm replying with the same comment as a 100 other accounts (ok more like 10, the fediverse isn't that big) because the replies never sync correctly.

this post was submitted on 08 Jun 2026
12 points (100.0% liked)

lemmy.ml meta

1551 readers
1 users here now

Anything about the lemmy.ml instance and its moderation.

For discussion about the Lemmy software project, go to !lemmy@lemmy.ml.

founded 5 years ago
MODERATORS
nutomic@lemmy.ml