The biggest issue was really driver support. Vista involved a breaking change to the Windows driver model (which was technically a good thing). But, that meant that a lot of hardware would never function with Vista. In the short term, this meant that you basically had to upgrade hardware to upgrade your OS. And this often included peripherals like printers and scanners. Software compatibility was also not great between XP and Vista.

Beyond that, Vista was a memory hog. People had become used to XP running well on systems with less than 4GB of memory. It was 32-bit after all. Even with the PAE switch enabled, you didn't exactly get the full 4GB. With Vista, it should have never been put on a system with less than 8GB, and more was better. So, performance was often terrible.

There were also UI changes, and those are never popular at first. Though, that sentiment usually fades with time (except the Win 8 Start Screen, that UI is still terrible). I still remember the "crayola UI" jokes about XP. People used to Windows 2000 thought the XP interface was a bad joke. Now, they often look back fondly on it.

In the light of day, Vista wasn't really all that bad. But, it represented some major changes to the OS and had a lot of issues at launch. Microsoft could have handled it better by ensuring that driver support and low end system performance was there in day 1. But, they fumbled that and so Vista has forever had the stink of being a bad OS.

It was really slow compared to XP, and while it attempted to improve the security situation (notoriously bad on XP) that came with significantly reduced compatibility. The Windows Glass (if I recall its name correctly) feature that was on by default was horrendously slow.