I expect Feynman’s answer, if he had a whiteboard and unlimited time, would’ve been to dive into Maxwell’s equations.

With that in mind, his answer makes complete sense. Good luck explaining coupled PDEs to people who aren’t mathy in a few sentences without visual aid. The analogy to the gravitational force isn’t on point; there’s a lot more to be said about how magnets tie to into E&M more broadly, compared to gravity.

Though you’re absolutely right that once you get deep enough into any topic in physics that the answer to “why?” inevitably becomes “it just be like that”.