I don't think many people realize what a kick-the-can-down-the-road kind of failure the SSN now is as a concept, since it's also being used as a kind of national ID. It's part of the reason why identity theft against US citizens just gets easier and easier.
Before the 1980s, it was common not to even have a SSN before the age of 15 or so, since it was originally only to be used for financial purposes. It was government action against tax fraud which ultimately made it common to get one assigned effectively from birth: eventually the feds required a SSN for any child over 2 years of age named as a dependent for tax purposes.
But you definitely should never be carrying the card around with you.