you really should have words for at least 6 distinct colours, those being red, yellow, blue, green, orange and purple.
Even if those six were the actual primary and secondary colours, it wouldn't be necessary to have a primary word for each. You can refer them by hue, or by referring to some object.
Also, note that the "true" colour wheel (based on our light reception) is more like red, yellow, green, cyan, blue and magenta. (No orange; and purple is kind of far from magenta.) The wheel that you're implying by those six colours is mostly an artistic "distortion" of the 18th century. And yet it isn't cross-linguistically common to take cyan or magenta as "named" colours, they're often seen as hues of blue/green and red/rose/purple respectively.
(Some societies live just fine with three colours - "dark", "light", and "red".)