[continued from parent]

Advanced Artificial Intelligence

This is not ChatGPT, not as we know it in 2026; I see some people complaining about that in this thread.

I'm talking about the kind of thing that can think more-or-less on a human level, solve problems in a general sense. Artificial General Intelligence, maybe. It's hard to draw exact lines on what constitutes a real existential threat to humanity, but I think that it's difficult to argue that we will not be able to create such a thing.

I've written a lot about this in the past, and I'm not going to repeat a lot of it here. It's a hot topic in general, these days. But some highlights:

• AI has tremendous potential benefits, and even more so the more-advanced the forms.

  It will be very hard to prevent humanity from exploring AI. The potential returns are immense. This means that it will not be a technology easy to forswear development on.

• Enforcement of a prohibition on development of some form of AI may be very hard

  If we chose not to develop AI or some forms of AI, it will basically amount to an arms control treaty. These have had a spotty efficacy in the past. Imagine that all countries get together, decide on some set of technological lines that must not be crossed and then develop some sort of treaty to enforce it. Basically, they'd have to agree to go to war against anyone trying to develop such a thing (and actually follow through on this). That means that at minimum, you'd need the world's major military powers onboard with such a total ban. How would you enforce it? Eliezer Yudkowsky proposed prohibitions on the development of AI datacenters, the destruction of "rogue" datacenters, where size and power usage would identify them, and that there be willingness to attack them. I'm not even sure that, setting aside the political challenges in creating such an alliance for all of humanity's future, that that would be enough. My own guess is that you could probably, given the right software, have a human-level intelligence on the sort of computer that one might put on one's desk. The development of AI will surely increase the computational efficiency. The kind of intrusive control that one would need to have to identify anyone working on such a thing on, say, a PC, even given the political will to do so, seems simply Orwellian, and possibly unworkable. Would you accept every computer on the level of a modern PC today having monitoring software on it, worldwide? That might be the kind of thing that would be necessary, if even that were enough.

  Some of the reason that nuclear weapons non-proliferation policy has been even as viable as it has been---and it is certainly not perfect---is that there are very large, unusual, and detectable installations and expertise required to manufacture nuclear weapons. My guess is that this is probably not the case for artificial intelligence.

  If you look at, say, the failures of the Washington Naval Treaty---and it's awfully hard to secretly build, say, a battleship, as it requires a lot of resources, expertise, and can only be done in very specific locations---it makes me awfully pessimistic about the viability of an "AI arms control treaty".

• It may be very hard to counter something as intelligent or more intelligent as we are

  One of humanity's most potent assets has been our intelligence. We have had that overmatch on our side for a long, long time, against anything that we've faced. It's proven to be a very powerful weapon. But...with AI, we face the potential for being on the other side of that equation. That's one point that Yudkowsky hammers on, that we've been so marvelously effective through humanity's history because of the single advantage that we're talking about losing here.

• It may be hard to know exactly what the lines are that constitute an existential threat to humanity.

  Even if we explicitly want to draw lines, limit our development of technology to some sort of "safe" subset, we might not be able to do so, because it's hard to understand the risks well enough to draw "safe" lines. Yudkowsky thinks that we've already crossed them. I'm more-skeptical about that, but I don't really have any sort of a more-permissive set of lines to offer up myself.

• I don't know how to "solve" the risks, or if doing so is even practical

  Humanity has one single great advantage over an advanced artificial intelligence. It gets the first move. We can try to leverage that advantage to control artificial intelligence. We can choose to create whatever structures and rules we want to contain and control the threat. The problem is that those structures and rules have to last and work for...essentially forever. An advanced artificial intelligence itself might develop more-sophisticated forms of artificial intelligence, just as we once developed something that surpassed ourselves, and we'd have to avoid that spiraling out of control in some way dangerous to us, even via our first advanced artificial intelligence making a mistake. We'd have a hard time even understanding the kinds of things that we're trying to control. Imagine at the dawn of life, some primitive bacterium trying to bioengineer life that would never be a risk to it, even in the upcoming billions of years. That's the scale of the problem. It seems to me like a problem that cannot be solved in practical terms.

  Carl Sagan thought---and I'm not sure that he was right---that nuclear weapons posed an existential threat, but that we could "beat" them by becoming a multiplanetary species, that a nuclear war could never be so wide-spanning as to wipe out multiple planets of humanity. Maybe he was right. And I can imagine such a universe, with little-enough interchange, where maybe we could also avoid existential risks from at least an accidentally-created genetically-engineered organism; someone intentionally creating such a threat could place it multiple places ahead-of-time. But AI can potentially replicate across computer systems at the speed that information travels, the speed of light, and it seems likely that we will link up computer systems wherever we go. I think that it is likely that we will not have any such "threshold of safety", ever.