While expecting and worrying about facing ableism from the system and society at large is a legitimate and valid concern, I think that in this case, if by this point you've not only been unaware of any learning difficulty, but also not treated for one, it's not something that will show up in your records or be significant enough to make any difference. I think if you just leave it as what it is - a single remark made years ago that has had no impact on your life until you found out about it recently, it's likely to never have any further impact on you. Though keep in mind that your support needs might change with time, and if it does become relevant, getting the help you need might overweigh the potential complications having any formal diagnosis on your record might bring.

I agree with your edit, that the perception of our own, others, and others' of our so called intelligence can have real life impacts, but that doesn't make it right that that's the case, and we don't have to play along within an artificial framework that is harmful to us if we don't want to (I appreciate and mentioned before that in medical settings we might not have a choice, but I'm talking about how we think of ourselves, others, and the world around us in a more general sense). The more of us decline to participate or give it any weight, the less power the system has.

Bottom line - if you feel like you have an issue that you want to investigate further, or are struggling with certain things, go ahead and seek support. If you feel like you were doing perfectly fine before you found out about this note in a file, put it to the back of your mind and keep on living as you were, the likelihood of anyone but you ever finding out about it, unless you bring it up (and even then, unless you end up perusing diagnosis and support) is tiny.

E: I hope this isn't coming across as dismissive of your concerns, I can understand where you're coming from, I used to have similar concerns, but what helped me as I've grown older is the understanding that I don't have to conform to a society that already generally rejects me whether I play by its rules or not, and so I started prioritising my own needs over its expectations.