Spear phishing disagrees with you.

If you're targeting a specific individual, cloning their SIM or performing another number hijack or even intercepting their SMS in flight, are all viable.

For broader, more general attacks SMS is usually enough to keep anyone out.