Would I have to use flowers that are closer to cyan, magenta, and yellow so it is closer to the cmyk color model or does that not help?

The thing about that is the "k" in CMYK is black, because it's very hard to mix other colors to get anywhere close to black. I think you'd want to look for a pigment which is black and bleaches under UV, but that's going to be hard, since bleaching from black fully to white so other colors can be applied will be crazy hard. Might want to just compromise to brown being your "black point".

Do I need to do it in multiple layers?

Yes. Most of these pigments are only going to be sensitive to UV light, so each layer is going to be a monochrome representation of the total light exposed on that layer.

Also I was planning on trying to use the paper as a replacement for photo paper in pinhole photography with it instead of a mask.

That makes this almost impossible, frankly. The mask method means that you have direct sunlight hitting the paper at full intensity. For a pinhole camera, for one thing you're not pointing at the sun, so the light is maybe 5-10% as intense, but the pinhole aperture dictates the amount of light. Say the paper is A4 for math's sake, and so is 62,370 mm^2. A typical pinhole camera's pinhole is probably a bit less than 1mm^2, but let's say it's 1.

The brightness of light on the pinhole is the same as it would be on a piece of paper held in the same place, but then once inside the camera the light spreads out to cover the whole paper. That means that relative to the mask method and leaving it in the sun, the pinhole method exposes the anthrotype to 0.05×1/6,2370 = 0.0000008 the light of the mask method. This is all too approximate to calculate what the exposure time might be, but it's unreasonably long, a couple hundred years.

I think the most practical method would be to take a digital photo, filter the colors into a monochrome for each dye you want to use, print that on a transparency, and use that as a mask.