https://nces.ed.gov/fastfacts/display.asp?id=38

https://nces.ed.gov/surveys/piaac/2023/national_results.asp

They unfortunately rework the metrics by which they measure this stuff, and have many different ways of trying to measure roughly the same thing... but hey, thats true of NBER and the BLS and FRED over this same time period too, way before Elon/Trump basically fired everyone.
So yeah, its hard to pinpoint exactly, without... doing my own meta analysis of all their data, but basically around ~2012 were the peak of totally averaged literacy and numeracy scores.
Before that, it was climbing, then peaked, and has since been falling, quite rapidly the closer you get to present day.
Your anecdotal observed deviation is indeed biased, you probably do not spend your time around a purely statistically average set of people, how/why exactly that is the case would require me to get some PID from you to attempt to explain though, lol.
Most likely explanation would probably be regional / location variance.
US education systems vary wildly in quality by Zipcode, the average net worth of families in the zipcode you grew up in is still the most statistically significant way to accurately predict overall life outcomes.