I am not even sure they are necessary to have in the hands of a singular person. I think a citizens review board could come in a lot of different forms of varying merits and still be more fair then having it in the hands of one partisan elected person.

If I were designing it I would make it double as almost like a jobs program. You create these boards out of random citizens and make it a full time 40hr/46 week job. They have to research each case, argue the merits, and then vote all on public record. The duration would maybe 3 to 4 years for each cohort, with the first year being education, training and analyzing and debating the decisions of one of the prior cohorts. The other 2-3 years would be their effective working years. I would also have a recruitment office, so after the cohorts terms are done they could transfer into parallel professions if they wish.

I would make it so it has to be representative of the general population too, meaning not just race, sex and age, but former incarceration status has to be taken into account. Make it a lottery system where every name is by default in the system, if they don't want the job (which will be most people) you move on to the next person until you fill your cohort.