I think the reading ruler effect is a big part of the accessibility being discussed here. If you want to separate the issues out, consider if browsers didn't show selected text by inverting the background color, but by e.g. underlining the selected text - you're right that it's still important to be able to do that, but the reading ruler is also important.

The use of text selection as a nearly ubiquitous reading ruler is a neat cultural thing that shouldn't be broken lightly, but it's not necessarily the only way browsers and websites could implement it.