Beware you don't really "compress" the videos like zip files but rather re-encode them, which is not lossless. Encoding is a very computation-heavy task that can only be brute forced with more raw hardware power.
You should make more research and experiments before doing this, no one can give you a "one size fit them all" solution when it comes to encoding.(i.e. you want to prefer speed or quality, there are tons of option) For starters if your GPU supports hardware decoder+encoder you can use it to speed up the encoding process, this is far faster than using only CPU to decode and encode.
Have a look at what HEVC is if you want to have good balance between size while preserving quality.
My NAS is almost filled with used parts, mostly lightly used. Only the power supply is brand new as the form factor is hard to find.(flex ATX, I bought an FSP FlexGURU)
IMO excluding HDD, most hardwares can be used without much consequence: CPU, RAM(I only trust original/major grade sticks), case and perhaps power supply(from reputable brand and known good model)