Hear me out. There's nothing innate to an object that makes it "food". It's an attribute we give to certain things that meet certain qualities, i.e. being digestible, nutritious, perhaps tasty or satisfying in some way, etc. We could really ingest just about anything, but we call the stuff that's edible "food". Does that make it a social construct?
Food is a non-toxic, organic substance that provides nutrition in the form of carbohydrates, protein, fibre, fats/oils, and/or vitamins and minerals. Sure there are some edge cases which you can argue the point in, such as a lump of rock salt maybe, but for the most part it is something which provides sustenance.
Sure you could eat mud or plaster and it won't kill you, but it won't help keep you alive either, so its not food.