How would you know that no animal had ever crapped over the clay deposit and had rain drift in durable, long-lasting bacterial spores from the waste whiøe the rain was on its way through the clay to join the groundwater? That a burrowing critter didn’t die just upstream in the ground of where you’re digging? It’s not terribly likely that those would get you sick maybe but ‘absolutely no risk’ is a high bar to clear.
Also there can be other sorts of non-obvious contamination like if there are trace amounts of heavy metals. Kaopectate got sued by California because the clay contained in their popular anti-diarrhea pills had traces of lead, such that the adult version pills would have fifty micrograms of lead a pop or six to twelve micrograms for children. For reference California currently mandates a warning slapped on if a product exposes you to half a microgram of lead per day. It’s difficult to know the provenance and full risks of stuff dug from a hole for a regular person. Even trained people with a lot on the line sometimes screw up and get people hurt.