Color is not invented by the brain but is socially constructed. You cannot look inside someone's brain and find a blob of green, unless idk you let the brain mold for awhile. All you can do is ask the person to think of "green" and then correlate whatever their brain patterns are that respond to that request, but everyone's brain patterns are different so the only thing that ties them all together is that we've all agreed as a society to associate a certain property in reality with "green."

If you were an alien who had no concept of green and had abducted a single person, if that person is thinking of "green," you would have no way to know because you have no concept of "green," you would just see arbitrary patterns in their brain that to you would seem meaningless. Without the ability to reference that back to the social system, you cannot identify anything "green" going on in their brain, or for any colors at all, or, in fact, for any concepts in general.

This was the point of Wittgenstein's rule-following problem, that ultimately it is impossible to tie any symbol (such as "green") back to a concrete meaning without referencing a social system. If you were on a deserted island and forgot what "green" meant and started to use it differently, there would be no one to correct you, so that new usage might as well be what "green" meant.

If you try to not change your usage by building up a basket of green items to remind you of what "green" is, there is no basket you could possibly construct that would have no ambiguity. If you put a green apple and a green lettuce in there, and you forget what "green" is so you look at the basket for reference, you might think, for example, that "green" just refers to healthy vegetation. No matter how many items you add to the basket, there will always be some ambiguity, some possible definition that is compatible with all your examples yet not your original intention.

Without a social system to reference for meaning and to correct your mistakes, there is no way to be sure that today you are even using symbols the same way you used them yesterday. Indeed, there would be no reason for someone born and grew up in complete isolation to even develop any symbols at all, because they would just all be fuzzy and meaningless. They would still have a brain and intelligence and be able to interpret the world, but they would not divide it up into rigid categories like "green" or "red" or "dogs" or "cats." They would think in a way where everything kind of merges together, a mode of thought that is very alien to social creatures and so we cannot actually imagine what it is like.