Not at all. The notion of a purely military fortress in the sense of being primarily for the permanent garrisoning of troops only really developed after the medieval period.

Castles were overwhelmingly and in many polities exclusively built for defensive purposes, with the ability to reside in them being only an extension of that insofar as you can't hold out against a siege if you can't live there for at least a few nights. It's why permission to build a castle in the first place is so often restricted by medieval law, requiring the direct assent of the monarch in most medieval polities.