The simple fact is that it is often easier to hoodwink a large group than a smaller, more invested group. The voters as a collective are that large group. In addition, with the spoiler effect, it is hard for voters to express their true opinion. Even when an alternative to first-past-the-post voting is used, that effect is still there, but reduced. When parties are elected instead of representatives, the problem is that parties, like representatives, can lie and not do what they promised. In addition, political parties and elected representatives are inherently partisan.
With voting, many remain rationally ignorant and do not vote. After all, a single voter rarely is the deciding factor; the voters know this, so many do not bother with voting.
While direct democracy appears like a viable alternative, it is not since it is difficult to get an issue on the ballot, and the voters cannot decide all the minutiae as they lack the bandwidth.
With sortition there is no active effort on the part of the electorate, unlike voting. The electorate can remain rationally ignorant and put forth no effort. For those that are selected through sortition, they will have a real incentive to be knowledgeable and care.
Imagine that you are selected at random to represent a district in a legislature and it comes with a high salary. No campaigning pressure, no donors are needed, and no pressure from an electorate. You could focus solely on what policies are the most beneficial to your constituents while not worrying about your popularity with anyone.
It has been said that those who seek power should be the last ones to hold it, and sortition acknowledges this reality.
Theories are great, but there needs to be a spark. The fundamental problem is not whether or not sortition works or how it would work, but actually getting sortition adopted. The catch-22 is that the same broken system that needs to be fixed is the same system that would need to vote themselves out with sortition. Since no place has sortition, people do not know what they are missing out on.
The solution: use direct democracy where it can be done. For instance, Nebraska has a unicameral state legislature, while every other state is bicameral. It was only possible to do so, as in Nebraska, voters can amend the state's constitution without the legislature's consent. There is no structural reason why the same could not turn one chamber of a state legislature into a sortition-based one.
With even one chamber of one state legislature changed, that could be the catalyst to bring sortition into the mainstream.
My flavor of sortition is delegated sortition. In delegated sortition, you would have approximately equally populous districts. A number of eligible voters would be selected at random for that district. They form that district's sortition board. With a majority vote, the sortition board selects that district's representative for that term. The sortition board then dissolves. This happens across all the districts. A representative can be reappointed or changed out in the next term.
While there will be some flukes here and there, overall it would allow the representatives to be much more aligned with their constituency. This is as close as you can get to every voter directly deciding on everything.
I understand your concern over many people lacking a voice under such a system. However, selecting people at random necessarily leads to average people deciding directly. You likely will never be selected, but someone like you will invariably get on a board.
The real problem is how hard it is for an average person to run for office. Do you have extra money laying around for campaign funds? Most people cannot self fund, thus they need donors. Holding office is for poor and middle class folks as well. Many people could never run as they cannot afford to miss work. That to me is undemocratic. This bypasses all of that.
I concur; referendums are a good tool, and some items should come with a required vote.