Maybe it doesn't automatically expunge the record, but a pardon does fully restore one's civil rights. If they were pardoned then they received "…complete forgiveness of a crime and…" had their "…full rights of citizenship" restored.(Source.)

A pardon is an expression of the President’s forgiveness and ordinarily is granted in recognition of the applicant’s acceptance of responsibility for the crime and established good conduct for a significant period of time after conviction or completion of sentence. It does not signify innocence. It does, however, remove civil disabilities – e.g., restrictions on the right to vote, hold state or local office, or sit on a jury – imposed because of the conviction for which pardon is sought, and should lessen the stigma arising from the conviction. It may also be helpful in obtaining licenses, bonding, or employment. (Source.)

In other words, these pardon refusers are self-righteous twerps. Maybe they don't deserve whatever rights they've given up on that basis alone.