Freedom of speech is an ideal before it is a law. Even if we lived in a utopian classless society without a government we could still have the concept that everyone deserves the right to say what they wish.
If it can exist without the government then logically it cannot strictly refer to the government.