That sounds like it could work pretty well, you could even just add it on to other comment sort styles. You don't need to necessarily remove downvotes if you really want them in specific instances.