You should be concerned, because there is always a risk, but there is also a risk when you are just using internet even without selfhosting. It is good to remember some rules:
- Keep apps up to date
- Open only that ports, that you need (80, 443, maybe 22 for ssh)
- If you open ssh port, use keys authentication and cut possibility logging with password
- Do not expose apps, that you are not need in this form, for example Yunohost allows to hide apps behind Yunohost SSO logging page
- Using cloudflare or/and Fail2Ban is cool
- Maybe I am a little bit paranoic, but I do not expose apps, that can be used anonymously by everyone, for example link shorters or pastebin alternatives.
It is not all of course.