In the development industry, the concept of an API (application programming interface) is to give developers a way to interface with the backend". When we're referring to an API, like the Reddit API, a majority of the time we're referring to a public API - which are typically versioned, documented, and have specific rules or conditions unique to them.
When a website or service doesn't have a public API, or that public API doesn't suit the needs or is otherwise not applicable, we will often turn to scraping. Scraping can be done a number of ways, and I won't go into those details. But the short of it is that we're usually mapping data into a usable format, and acquiring it by interfacing with the website as though we're a normal user.
One of the drawbacks of scraping is that normal changes, even small, can break the logic we have scraping that website. This is also the case with projects like NewPipe, Piped, and LibreTube.
LibreTube, NewPipe, and Piped all use NewPipe's Extractor underneath the hood. Invidious appears to have it's own logic. The "Piped API" uses NewPipe Extractor, which is a Java library, and essentially converts it into a web API that other projects that aren't Java can utilize.
Sharing that logic is actually beneficial, if YouTube changes something that breaks all the open source frontends, developers from all 3 projects (and as it is OSS, unaffiliated ones as well) can identify what change is breaking the extractor and fix it, pushing out an update fixing their frontend rapidly. I'm not sure how it works in practice as I don't follow the project, so I'm just assuming that Invidious gets fixed slightly slower than the other three.
But back to your original question, YouTube can't "turn rogue" in the same way Reddit did, in a sense YouTube is already rogue. The developers of third party apps on Reddit utilize a public API with permission, and follow a certain set of guidelines and other requirements. Developers making alternate frontends for YouTube do not utilize any APIs with permission, and just the act of using one of those frontends is likely a terms of service violation on YouTube (not that it matters).
The reason we don't see the open source YouTube frontends breaking all the time is likely due to effort. YouTube could completely change one of it's APIs it uses, but then they would have to update all of it's software that relies on it. Their website, the Android app, the iOS app, probably the YouTube music apps, and not to mention every smart TV app. And this change would likely be identified and fixed by the OSS developers within days, if not hours. So YouTube can't really actively combat it on the development end, and would need to take the legal route. Publishing code for scraping isn't illegal (but they could, and have, use intimidation), so YouTube's legal recourse would be to get stuff using the code shutdown. Right now the user base for the frontends is probably small enough that they're tolerating it.
TL;DR: YouTube is already rogue, scrapiing bypasses that. Changes to YouTube already break open source frontends but are resolved rapidly by developers.
As for answering the followup question: "can Reddit developers do the same thing". Yes. Will they? Maybe, maybe not. Reddit had a great API which encouraged third party growth, now that Reddit has made that API prohibitively expensive, it's not really an option. Developers inthe past could utilize that API and monetize their projects (so long as it fit within guidelines, if any). Turning to scraping / "unsupported" methods of interfacing with the website means they realistically can't monetize their work as doing so puts them in legal hot water. They also would have trouble publishing their app (if it's an app) on platforms like Play Store and the App Store. And face intimidation and legal threats.