I ask the bizfolks: would YOU allow a guy in a suit with no prior engineering knowledge, but with an ai design the structural elements for a bridge for thousands of vehicles a day?

They would allow it. The difference is that our societies decided that liability for builders of bridges is a sensible thing and nonnegotiable, but for reasons unknown to me you can release any piece of crappy software to the public without any liability whatsoever as long as you tack on it some terms and conditions that absolve you of warranty and liability. This is unthinkable with all other commodities. When it comes to software, the people in charge turn off their brains.