Still needs to be read from disk/flash, for the kernel to know it exists. And more relevantly every module has a number of dependencies that will all contribute to size/boot times in their own way.

Maybe even milliseconds is giving it too much, but bloat is really important especially near the release of new hardware.

I imagine the last thing anyone was worried about in crunch time was if they enabled enough random hardware in their config file.