Finding albums "in the wild" at local record stores is a real thrill, but sometimes you have to buy online. Look at Discogs.com.
It's the biggest marketplace and community for vinyl online. You can buy from little shops all over the world, and they usually have everything! You can also catalogue your purchases and have them all on your phone, so when you're at the shop, you don't have to guess as to whether you already have a certain album
One trap people fall into is buying vinyl that is overpriced for no good reason. Shop around a little. A rule I made for myself is "never pay over $25 per disc." If it's a two record set, I wouldn't pay more than $40-50 for it. Unless my want for it overshadowed my thriftiness.