People like to think in black and white, but you're definitely right. Having your SSH server on port 36271 will likely stop a ton of drive by attacks because they simply won't check it. Having it only listen on IP6 would stop almost all of them because you can't trawl the IP6 space efficiently. These are "obscurity", but they have real benefits. The idea that "obscurity" doesn't help is just a meme that people love to quote because it's a great single sentence with some nice rhyming "security by obscurity". I assume the reason it became a meme is because tons of products fully relied on obscurity; I still see it all the time. As you said, it's all layers.