Yes, this is exactly correct. And it's not actually too slow - the specialized models can be run quite quickly, and there's various speedups like Groq.

The issue is just more cost of multiple passes, so companies are trying to have it be "all-in-one" even though cognitive science in humans isn't an all-in-one process either.

For example, AI alignment would be much better if it took inspiration from the prefrontal cortex inhibiting intrusive thoughts rather than trying to prevent the generation of the equivalent of intrusive thoughts in the first place.