The primary problem is using agile all the time instead of when it is actually intended to be used: short term work that needs to be done quickly by a small team that are all on the same page already.
It sucks for any large group that requires a lot of coordination. Some parts of it are still helpful as part of a blended process, like more collaboration with the customer and responding to change, but those can easily derail a project if not everyone is on the same page through scope creep or losing sight of long term goals.