I'm not saying they always drive slowly, but in my experience in Japan, they do in those small, one-lane residential streets where there's no room for a sidewalk. Is that not what you've experienced? I never lived in Japan long-term, but I've visited a bunch of times, and whenever I was in areas like that the drivers were cautious.