And it wouldn’t affect your ability to download torrents if you don’t have port forwarding on VPN, just your ability to upload

This isn't quite true. Two peers who both lack port forwarding will not be able to connect to each other at all. Once a connection is established between two peers, both uploading and downloading should work just fine.

A significant portion of swarms are users like that, who can initiate connections but cannot receive incoming connections. This is especially problematic with smaller torrents. If you're working with well-supported torrents with dedicated seeders and thousands of users, then it won't really matter. But if you need something with just a couple seeders, you might find yourself stuck with zero accessible peers.