Those are the mesa drivers, not the "amd" drivers.
Those very same drivers work on Intel cards and pre-20 series nvidia cards. Mesa is not an AMD project or an Intel project either, that is an independent team.
Even then, those drivers are for allowing the GPU to display to a screen and interact with the system. They are pretty much the same idea as the Microsoft basic display adapter. You still need the xf86 drivers to display X, the opengl drivers for opengl, cuda for cuda, vulkan for vulkan, etc. Those are all separate components because they have libraries included with them.
If all of those extras were built into the kernel, the kernel would be like 2 gigabytes, not 150ish megabytes. It is literally enough to get you going with a getty and that's about it.