I compiled my own kernel for a few years when I first started using Linux in the late 90s. "So much faster by removing things I don't need," I claimed. I cringe at that now.
I've been using stock kernels for years now. Any speed improvements from a custom kernel are more than offset by the time it takes to configure and compile it. Making one mistake could wipe out years of theoretical speed improvements as you recover, reconfigure, and recompile. Plus, if you have a problem with a piece of hardware down the road, how do you know the issue isn't your custom kernel?
I'd say, do it once or twice for the experience, or if there's some specific reason you need it (bleeding edge hardware, for example).
Some years back, I realized I'd rather be using my computers instead of administering them.