I disagree, its not in my opinion a meaningless distinction at all. A difficulty in cognition might prevent a person from reading War And Peace. Thats a direct result of having a learning disability. Someone with a visual disability who cannot access audio books or braille versions of War And Peace is not being affected by their disability but by the fact an accessible version is not available.
You might argue the end result is the same - an inability to read War And Peace - but the point is that for the person with a visual disability the situation is fixable if society is prepared to make the effort.
In regards to your situation you've had terrible experiences but they are not down to the fact youre autistic, they're down to the fact your NT 'friends' weren't really friends at all. I'm sorry they let you down but I'm pretty sure I could find similar stories where nobody in the story was autistic.