Yes, yes it is. You don’t judge a system by some ideal that can’t be achieved. If it’s a system meant for humans you judge it based on what it does to said humans.
If agile makes managers more insufferable, then maybe it’s not a good tool for the problem at hand, working in companies with managers.