Grab a thin needle or piece of wire, thin enough to easily insert into the USB-C port, and scratch all of the dirt and lint out of it. Always point the needle towards the outer surface so you don't scratch the electrical contacts in the middle.

There is often a surprising amount of junk inside even if you can't see it from the outside, and that can greatly affect the connection quality.

My phone recently had a similar issue where it would only charge if the cable was inserted in a specific way, and any movement would cause it to stop charging. The cable also wasn't really held well even though it looked like it was fully inserted. I cleaned out the port even though I couldn't see anything inside, and managed to pull out a bit of dust anyway. And now my phone no longer has charging issues and holds on to the cable much better.

USB-C unfortunately just seems to have a design that makes it very easy for dust to get stuck in it, while also having a relatively low tolerance for foreign material buildup before the connection quality gets affected, making this a quite common issue.