I don't think ActivityPub is hard to implement, actually. It's fairly specific JSON with a weird structure, but that structure isn't hard to implement.
It does come with an oversight, though; the protocol doesn't define how to ensure activities propagate to every server. That's why following a Lemmy community from Mastodon will spam your feed with boosts, and why Mastodon instances all have different amounts of comments and likes on them. Some other things are also underspecified (like how to propagate creates through boosts, are those two events of is boosting a create enough?). These can use some fleshing out.
One thing that's definitely underspecced is the client-server API. It's practically impossible to take the spec and write a client for a C2S server that's also built around the spec, because crucial things like "authentication" simply aren't specified. Very few servers bother to implement C2S and even fewer of those cover more than the bare basics.
At some point I tried setting up a sort-of-tumblr clone by combining Fedibox with a C2S client, but I found it impossible to set everything up properly, and no other backends seemed to provide more than a basic C2S implementation. There's potential there, but it's hard to do it without an opinionated take on the spec that I feel shouldn't be necessary.