Yah I think we should, it was literally just put in place because some people realized that a genuinely popular president could make serious positive changes that undermined extant power structures.

If a totalitarian president were to take control of the government enough to continuously win elections without popular support, they could certainly ignore a term limit, so all it really does is prevent the public from re-electing presidents they legitimately like.