it's not rude but you will be an outcast because of it

we're social animals, if you don't show yourself as part of the pack you'll be treated as an other - with suspicion and distance. it's much better to first show that you want to be part of the pack, and then establish yourself as someone who keeps to themselves. this way you'll get even more peace and quiet, because people you're on friendly terms are more likely to respect your personal space

this is the difference between "that lonely guy over there (he likes peace, but is nice)" and "that lonely guy over there (he creeps me out)"