People usually don't care consider white to be a colour. You wouldn't call a white wall colourful, would you?

The absence of colour is not a colour.

That said, white people are generally actually various shades of pinky beige, black people are generally various shades of brown, but our language uses colour-based terminology to refer to ethnicity anyway.

Complaining that white is a colour, so whites should be included as coloured people, is kind of silly in that context.

Especially if you remember that the term "coloured people" originated as a way to stop people who had black and white heritage from getting any of the benefits of their whiteness.