Yeah, and I'm not here to say Windows Server is always the wrong choice. Windows Server is the champion of directory services, because Active Directory is king. Windows Server can also do neat things like run WSUS and stuff. It has its place. In larger deployments, it's likely typical to run a variety of servers in GUI-less mode, and then have one GUI-equipped install for management.

That being said, my experience is that almost all Windows Server deployments I've found in the field have had the GUI, and have therefore been rather sluggish. It's just part of how it works.