Context matters. In science, if you are proposing something which will upend the established consensus, the burden of proof is on the claimant to show that some detail has been missed, or that some other theory fits the evidence better. What Coxon and others are claiming is not a scientific theory, however. How could anyone provide evidence to prove that building an AGI would likely result in the extinction of the human race? What kind of evidence would be acceptable? And if we continue building bigger and smarter models and before we can gather the evidence it turns out the premise is correct, what then? ’Well, we’re all dead, but at least we stuck to our guns regarding that old axiom.’

I think that, given the potential stakes, the burden of proof should be on the AI companies to show that their technology is not harmful, and until they do they should be restricted from pursuing it. By that I mean that they need to show that their existing models are safe and can be controlled, and if they can’t they need to roll back to the Last Safe Model in their roster. I’d even go further, putting the onus on the companies to show that they are not on net damaging our environment, that their models cannot be compromised to provide terrorists information on building bioweapons, etc., that they will support and help strengthen the vulnerable and never (again) encourage people to commit suicide. And that’s just off the top of my head.

The harms already caused by this technology are staggering, and we need laws to put the brakes on development so that we can collectively adapt, and if the leaders of these companies are offering us a way to do that, let’s grab that opportunity.