they don’t know as much as the primary school did

Or maybe they knew more? Because they were seeing you there as is, and they didn't see these issues, nor did you, nor do they seem to interfere with your daily life, so perhaps the teachers in your primary school were just ableist assholes who didn't care enough to learn about you beyond a possible difficulty you were having (or one they were having), which is perfectly normal for any kid that age, but especially ND kids, before they categorised you as not worth their time (I'm sorry they called you that).

In the grand scheme of things, no one ever in your life, unless you show them the document or this post, is likely to ever find out about that note from primary school. Even if there is a copy kept outside of the school, they wouldn't keep stuff like that forever, and there's definitely no "main file of your life" where this will be.

Maybe it's worth seeing this as the traumatic past experience that it was, process that, learn from it, and accept yourself a little more. Those teachers had their own issues, or were just shitty people, definitely not worthy educators whose opinions you should take to heart. I know it hurts and has a lasting impact because I had teachers like that, and decades later I still remember some of their comments, but I don't let it impact me anymore because I know their opinions of me are worthless because they had no idea who I was or what I was like, or what my actual struggles were, and they definitely couldn't have predicted where I would have ended up in different points in my life.

Don't let them define you.