It's not mad science, it was a natural development. Today a GPU is a computationally powerful add-on that is installed into computers. Plugging a more powerful co-processor doesn't make too much sense though.

Frequencies back then were also low enough to tolerate more common shenanigans. Accessing on-board or off-board memory didn't really make a difference etc.