Finally got around to actually subjecting myself to AI 2040. I started writing this up for the stubsack then realized I had accidentally put together a lot of words and our previous discussion on the subject had slipped into last week's sack. That said, let's start from the top.
Having put humanity on this path, the AI companies find it acceptable. But most people don’t. Forget thinking about his legacy—the President is starting to think about what’ll happen to him after he leaves office and the world gets transformed
That's not how American Politics works. I mean it's a bad quote because the dumb is smeared across a vast expanse of boring prose, but the idea that any election can be summed up on one single solitary issue is ridiculously naive. Local variance and immediate context matter a lot and pretending otherwise makes you a hack.
The President announces that the US will pursue international cooperation to avoid an imminent intelligence explosion.
That's not how international cooperation works. I hope I'm overly cynical here but I think that the lessons of international coordination on everything from land mines and chemical munitions to crimes against humanity and nuclear non-proliferation should be relevant here. The efficacy of the various climate change pledges and agreements are also relevant.
Both sides can still use datacenters for running AIs that already exist (i.e. inference), but they will retrofit each other’s datacenters with devices to verify that they aren’t being used for new training runs
That's not how IT infrastructure or independent verification work. First off, if you try and tell any competent IT security department to stick a network tap that sees all their inbound and outbound traffic and directly sends it to their competitors you're gonna get laughed out of the room, basilisk be damned. But let's also not forget that "how do I trust that your validators aren't going to demand to tour unrelated infrastructure to collect unrelated military intelligence" has historically been a massive sticking point in, say, nuclear nonproliferation talks.
The Consortium countries come up with a simple high-level framework: we’ll agree to let each other see all the AI research. Then, if we don’t like something someone is doing, we’ll talk about it and perhaps agree to ban it.
That's not how regulation works. This is a classic rationalist fallacy that we see repeated throughout the piece of assuming that ultimately if you express the argument properly everyone will agree on the right thing, and that the differences in what (e.g.) different countries don't like to see done are somehow going to resolve factually to everyone independently coming to the same conclusion. Otherwise you have what is effectively a race to the bottom where even within "the consortium" everyone just migrates their operations to whoever has the most lax regulations that lets them do what they want even at the cost of breaking the world.
AI workloads can be split into research and development (the process of building new AIs) and inference (the use of the existing AIs). In our proposal, research is almost entirely transparent, while inference is still private.
That's not how network monitoring works. How do you trust that I'm not running training data on inference channels unless you're monitoring everything. Remember, we're not just concerned with actually identifying wrongdoing, we're trying to establish beyond a doubt that wrongdoing isn't happening. Also let's note that this argument assumes that continuous learning is impossible, when "actively learning long-term lessons from your experiences" is a pretty large part of human-level intelligence, much less the kind of super intelligence they're concerned about.
The result will be a competitive market for AI, in which consumers of AI services have many options to choose from and excellent visibility into what they are buying.
That's not how economics works. The entire concept of patent law revolves around making sure that you don't lose the competitive advantage that comes from investing in improved products and processes because if you don't do that then there's no market incentive to improve things. If we're gonna throw out market incentives why aren't we throwing out capitalism altogether again? But even if we set that aside, given how capital-intensive the AI training game (supposedly) is, the big players are going to be able to take advantage of economies of scale and use the resulting efficiency to outcompete smaller shops, leading to concentration of power. We saw this happen with crypto mining pools, and should expect the same to happen basically anywhere that more returns are bought direct with more money.
To accomplish this, they agree to build the datacenters in the third-party countries least secure against their rival’s military intervention
That's not how international agreements work. While it's true that during the Cold War we saw third-party aligned countries hosting nuclear weapons, this was because they directly benefited from being under their ally's nuclear umbrella and the associated improvements in relations. Like, Europe hosted American nukes not because they had an agreement, they had an agreement because hosting American nukes meant Russia was less likely to Leeroy Jenkins their way through the Fulda gap. Imagine telling (West) Germany that they should host Soviet nuclear infrastructure specifically so that in the event of a breakdown in relations they can be targeted by their allies for violence. I know that there's a kind of neo-realism in international relations these days that likes to assume that non-superpowers don't actually have agency, but I promise you this isn't the case.
The attitude towards safety flips. The incidents of AI misbehavior, plus the fact that AI is so deeply deployed into the world economy, plus the transparency into AI research, plus the breathing room to process what’s happening, all combine to dramatically shift the burden of proof.
That's not how public opinion works! At this point we're firmly in science fiction, but I think it's significant that part of the plan is once again "then everyone realizes we were right all along and agrees to go along with our plan with minimal friction".
It escalates to the President. He calls Xi Jinping. They bargain and threaten. They yell at each other. Ultimately Xi agrees to ban this type of thing if the US does too. Details are left to the respective regulators to hash out.
That's not how negotiations work. I've wanted to talk about this before but couldn't find a clear point to throw it in, so here it is: are we gonna assume that literally every other issue or conflict in the world other than AI regulation is going to magically cease to exist? Was this part of everyone magically agreeing with everything Eliezer Yudkowsky - who totally isn't a cult leader - wrote is completely correct? Like, are we expecting the negotiations about chip manufacturing in Taiwan not to run into friction due to the ongoing dispute about Taiwanese sovereignty? And that's just the most obvious example, ignoring the energy crisis brought about by the disruption in Hormuz and a thousand other areas where disagreement is inevitable and can't simply be wallpapered over by two executive leaders yelling at each other about it.
Now that the United States is limited in the number of total robots it can build, it must choose how to allocate this capacity between companies. They decide to use the free market via a cap-and-trade system. Permits to build robots or compute are sold to the highest bidder and can be freely traded.
That's not how the financial system works. Okay the cap-and-trade solution here is cringe neoliberal realism, but I think the kind of numbers they're describing merit a reality check of their own. Like, if that share of the overall economy is entirely driven by AI then from the perspective of a normal family trying to buy food and pay rent I'm not sure how you wouldn't see massively disruptive inflation. Citizen's dividend or not. And while I'm loathe credit mainstream economists, I also can't see how a dramatic increase in that dividend like they talk about several paragraphs later doesn't similarly crater the value of money since nothing that AI is revolutionizing is really impacting the kinds of stuff that people would be buying save that there's more of it and that people are no longer being paid to produce it. Also from a political economy perspective the way this twists things is really dark. Like, if your industry accounts for three-quarters of government revenues then you basically are the government no matter what anyone says. And that's pretty goddamn bleak if you don't own an AI company. Even accepting the science fiction premise the only alternatives are complete capture of the state by AI companies or the nationalization of those companies by the state. Either option, combined with the existing social dislocation, could arguably spark a civil war or revolution. But then I guess we would have to acknowledge that neoliberalism isn't God's perfect economic regime, which isn't possible to do here.
It’s fine for AIs to get better at using valid arguments and evidence to convince people of things for the right reasons. That kind of persuasion is asymmetric: it works much better when the argument pushes towards the truth.
That's not how the postmodern condition works. (Yes I am going to keep pushing this bit past it's limits until this dead horse starts walking and becomes funny again.) Evidence, argument, and truth itself contain an inherently subjective element that means that we can't nearly separate "good" arguments for rationally advancing a series of syllogisms from "bad" arguments rooted in personal charisma or authority or whatever else the "argumentum ad X" crowd is going on about these days. Truth is not a viable value in its own right, and pretending otherwise just serves to reinforce the social and political systems that can decide how truth is understood at the present moment. In their sci fi Dreamland, that would be either the US and Chinese governments or the CEOs of AI companies, and lots of people are already really mad at them now even before all the sci fi nonsense starts kicking in.
Everything after this point is pure science fiction, and of a kind that isn't even interesting. Good science fiction uses the speculative technology or world to interrogate something about reality. What does it mean to have a perfect simulacrum of a human being that nonetheless isn't one? How much power should we afford to corporations in an attempt to make miracles happen? Would you still love the god-king if he was a worm? This instead tries to use it to argue why we should ignore reality entirely. The utopia they outline is not meaningfully connected to the policies and values they advocate for, and those policies themselves are fundamentally based on a fiction about how the world actually functions.
What kills me is that there are so many obvious ways to be less wasteful about this. Nondestructive book scanners exist. They're expensive but it's not like AI companies are averse to throwing money down a fucking hole. Even if that's not an option, it's possible to rebind the pages and return the books to the market. And regardless of what happens to the physical books, the scans they create could be archived in a format that could be available as a digital library rather than just being fed into the statistical meat grinder to keep the trough topped off with slop. Even if they've got to fight with copyright holders it would cost them basically nothing to leave the option open and given the PR battle they've been losing it feels like doing so would be incredibly obvious. Hell, even Google Books was able to navigate this in a way that was less cartoonishly evil than this because they could point to their digitization effort as a public good in ways that Anthropic here just fucking can't.