Yes it can be done on a budget, although the more you get into it, the more alluring the expensive stuff is.

Even without a license, you can get started with some listening only stuff. For me, I picked up an rtl-sdr off Amazon for pretty cheap and started listening to NOAA weather satellites.

Before you start transmitting, you'll need to get your license. Hamstudy.org is a great place to start practicing for the exam.