i give about equal odds that either there's some evolutionary downside to it (requires more energy, or makes you more vulnerable to brain injury somehow), or it's just completely irrelevant to our survival and so it's neither selected for nor against.

It would be very funny and miserable if the only thing holding humanity back from perfect memory is evolutionary indifference.