I've seen this story play out in software engineering: people were very impressed when the AI does unexpectedly well in one out of 50 attempts on an easy task, and so people decided to trust it for everything and turn their codebases into disasters. There was no great wave of new high-quality software. Instead, the only real result was that existing software has become far more buggy and insecure.

Now we have people using AI in science and math because it was impressive in random demonstrations of solving math problems. I now have friends asking me why I'm not using AI, and also saying that AI will be better than all mathematicians in 30 years or whatever. Do you really think I refuse to use AI out of ignorance? No, I know too much about it! I have seen the same story play out in software engineering, and what makes this any different?

I think the main difference here is that breaking RSA now just requires scaling up existing approaches, while breaking LWE or anything like that would need a major conceptual breakthrough. The former possibility is much more likely, and in any case, cryptographers are the most paranoid people on the planet for a reason.

Unfortunately, one can never be sure about much in cryptography until P vs NP is solved (and then some).

(Of course, just because some people say that scaling up is enough doesn't mean it's actually true. For breaking RSA, we know have Shor's algorithm, while the only evidence AI bros have from superintelligence coming from scaling is "trust me bro".)

The article's entire premise is Musk saying some random shit. Remember how Musk said that he would land a man on Mars in 10 years 13 years ago? Honestly, I am incensed that people like Musk and Trump can just say shit and many people will just accept it. I can no longer tolerate it.

Putting aside the very real human ability to screw up such a concept and turn any fair system into an unfair one, ...

He says this after mentioning UBI. He really doesn't want to confront the unfortunate fact that UBI is entirely a political issue. Whatever magical beliefs one may have about how AI can create wealth, the question of how to distribute it is a social arrangement. What exactly stops the wealthy from consolidating all that wealth for themselves? The goodness of their hearts? Or is it political pushback (and violence in the bad old days), as demonstrated in every single example we have in history?

I'd say the problem is even worse now. In previous eras, some wealthy people funded libraries and parks. Nowadays we see them donate to weirdo rationalist nonsense that is completely disconnected from reality.

No getting up early and commuting on public transit. ...

This is followed by four whole paragraphs about how the office sucks and wouldn't it be wonderful if AI got rid of all that. Guess what, we have remote work already! Remember how, during COVID, many software engineering jobs went fully remote, and it turned out that the work was perfectly doable and the workers' lives improved? But then there were so many puff pieces by managers about the wonderful environment of the office, and back to the office they went. Don't worry, when the magical AI is here, they'll change their minds.

Yes, there are "mindless, stupid, inane things" like chores that are unavoidable. There are also other mindless, stupid, inane things that are entirely avoidable but exist anyway because some people base their entire lives around number go up.

I decided to take a look at the bitcoin white paper.

Usually, the introduction of a technical paper is fluff and people quickly move on to the technical parts. However, the casual claims made in the first paragraph of this paper have aged extremely poorly, to say the least. In a better world, Bitcoin would have remained as an obscure academic toy, and this introduction would have remained fluff.

While the system works well enough for most transactions, it still suffers from the inherent weaknesses of the trust based model.

What weaknesses are there in the trust based model? Let's find out!

Completely non-reversible transactions are not really possible, since financial institutions cannot avoid mediating disputes. The cost of mediation increases transaction costs, limiting the minimum practical transaction size and cutting off the possibility for small casual transactions, and there is a broader cost in the loss of ability to make non-reversible payments for non-reversible services. With the possibility of reversal, the need for trust spreads. Merchants must be wary of their customers, hassling them for more information than they would otherwise need.

It seems like this guy really loves non-reversible transactions! But as we've seen with the history of crypto, non-reversible transactions sound really good until you fall victim to a crypto scam and there is no way to appeal to the bank to reverse the charges. Reversibility actually increases trust because you no longer need to be absolutely certain that you're dealing with an honest person.

A certain percentage of fraud is accepted as unavoidable.

Almost like that is a problem of human nature. And it's not like cryptocurrency has a spotless record when dealing with fraud! The problem with fraud is not the third party (the bank), but with the second party (the merchant or customer you're dealing with).

The introduction is not long, and most of the paper concerns the technical details of the construction of Bitcoin. By itself, there really is no way to complain about a pile of definitions. But there are still dumb comments that have aged poorly in retrospect.

A block header with no transactions would be about 80 bytes. If we suppose blocks are generated every 10 minutes, 80 bytes * 6 * 24 * 365 = 4.2MB per year. With computer systems typically selling with 2GB of RAM as of 2008, and Moore's Law predicting current growth of 1.2GB per year, storage should not be a problem even if the block headers must be kept in memory.

But why would you want a block header with no transactions? If you wanted to, I don't know, replace the world's financial system, you would need to handle millions of transactions every 10 minutes. How big would the blocks be then? And remember that many copies of the same blockchain would need to be stored (certainly, every miner would need to store a copy). How many thousands or millions of times would that multiply things?

Businesses that receive frequent payments will probably still want to run their own nodes for more independent security and quicker verification.

Turns out it was a bold assumption to think that businesses would just run their own bitcoin miners.

The proof of security (Section 11) is extremely sketchy by modern standards. (They're assuming that all attackers would follow a certain format to attack and not try something different. I get it, proper proofs of security in cryptography are very subtle and difficult.) There is also a page of fluff making random calculations with the Poisson distribution. In any case, the security of Bitcoin requires that the collective computational power of the defenders exceeds the power of any attacker (so the defenders can make new blocks faster).

Bitcoin is very strange as a cryptographic system in that the defender must have more resources than any possible attacker. In most cryptographic systems, the system should be secure even if the attacker has vastly more resources than the defender. Your phone's cryptography should be secure even if some government agency dedicated their supercomputers to try and break it. This means that Bitcoin must waste tons of energy, since that is required to maintain security. Any more energy dumped into it will only increase security and not make the actual transactions faster, which makes Bitcoin horrendously inefficient.

As a purely academic idea in cryptography, it is an interesting curiosity, but the arguments for why it's useful are sketchy. There are other such curiosities that are much more interesting, like homomorphic encryption or secure multiparty computation. It would be a nice line on a CV, but not "incredible".

The true significance of Bitcoin was the terrible libertarian economic argument for it, and the chain of events that would transform it into nothing more than a speculative fashion trend. It has nothing to do with the technical details of Bitcoin. The technical and economic arguments for Bitcoin turned out to be so weak that nowadays, the only real support for Bitcoin is that maybe you can sell it for a higher price to a greater fool.

“California is, I believe, the only state to give health insurance to people who come into the country illegally,” Kauffman said nervously. “I think we probably should not be providing that.”

“So you’d rather everyone just be sick, and get everyone else sick?” another reporter asked.

“That’s not what I’m saying,” said Kauffman.

“Isn’t that effectively what happens?” the reporter countered. “They don’t have access to health care and they just have to get sick, right?”

Kauffman contemplated that one for a moment. “Then they have to just get sick,” he said. “I mean, it’s unfortunate, but I think that it’s sort of impossible to have both liberal immigration laws and generous government benefits.”

Do I need to comment on this one?

I study complexity theory so this is precisely my wheelhouse. I confess I did not read most of it in detail, because it does spend a ton of space working through tedious examples. This is a huge red flag for math (theoretical computer science is basically a branch of math), because if you truly have a result or idea, you need a precise statement and a mathematical proof. If you're muddling through examples, that generally means you either don't know what your precise statement is or you don't have a proof. I'd say not having a precise statement is much worse, and that is what is happening here.

Wolfram here believes that he can make big progress on stuff like P vs NP by literally just going through all the Turing machines and seeing what they do. It's the equivalent of someone saying, "Hey, I have some ideas about the Collatz conjecture! I worked out all the numbers from 1 to 30 and they all worked." This analogy is still too generous; integers are much easier to work with than Turing machines. After all, not all Turing machines halt, and there is literally no way to decide which ones do. Even the ones that halt can take an absurd amount of time to halt (and again, how much time is literally impossible to decide). Wolfram does reference the halting problem on occasion, but quickly waves it away by saying, "in lots of particular cases ... it may be easy enough to tell what’s going to happen." That is not reassuring.

I am also doubtful that he fully understands what P and NP really are. Complexity classes like P and NP are ultimately about problems, like "find me a solution to this set of linear equations" or "figure out how to pack these boxes in a bin." (The second one is much harder.) Only then do you consider which problems can be solved efficiently by Turing machines. Wolfram focuses on the complexity of Turing machines, but P vs NP is about the complexity of problems. We don't care about the "arbitrary Turing machines 'in the wild'" that have absurd runtimes, because, again, we only care about the machines that solve the problems we want to solve.

Also, for a machine to solve problems, it needs to take input. After all, a linear equation solving machine should work no matter what linear equations I give it. To have some understanding of even a single machine, Wolfram would need to analyze the behavior of the machine on all (infinitely many) inputs. He doesn't even seem to grasp the concept that a machine needs to take input; none of his examples even consider that.

Finally, here are some quibbles about some of the strange terminology he uses. He talks about "ruliology" as some kind of field of science or math, and it seems to mean the study of how systems evolve under simple rules or something. Any field of study can be summarized in this kind of way, but in the end, a field of study needs to have theories in the scientific sense or theorems in the mathematical sense, not just observations. He also talks about "computational irreducibility", which is apparently the concept of thinking about what is the smallest Turing machine that computes a function. This doesn't really help him with his project, but not only that, there is a legitimate subfield of complexity theory called meta-complexity that is productively investigating this idea!

If I considered this in the context of solving P vs NP, I would not disagree if someone called this crank work. I think Wolfram greatly overestimates the effectiveness of just working through a bunch of examples in comparison to having a deeper understanding of the theory. (I could make a joke about LLMs here, but I digress.)

The sad thing is I have some idea of what it's trying to say. One of the many weird habits of the Rationalists is that they fixate on a few obscure mathematical theorems and then come up with their own ideas of what these theorems really mean. Their interpretations may be only loosely inspired by the actual statements of the theorems, but it does feel real good when your ideas feel as solid as math.

One of these theorems is Aumann's agreement theorem. I don't know what the actual theorem says, but the LW interpretation is that any two "rational" people must eventually agree on every issue after enough discussion, whatever rational means. So if you disagree with any LW principles, you just haven't read enough 20k word blog posts. Unfortunately, most people with "bounded levels of compute" ain't got the time, so they can't necessarily converge on the meta level of, never mind, screw this, I'm not explaining this shit. I don't want to figure this out anymore.

It is how professors talk to each other in ... debate halls? What the fuck? Yud really doesn't have any clue how universities work.

I am a PhD student right now so I have a far better idea of how professors talk to each other. The way most professors (in math/CS at least) communicate in a spoken setting is through giving talks at conferences. The cool professors use chalkboards, but most people these days use slides. As it turns out, debates are really fucking stupid for scientific research for so many reasons.

1. Science assumes good faith out of everyone, and debates are needlessly adversarial. This is why everyone just presents and listens to talks.
2. Debates are actually really bad for the kind of deep analysis and thought needed to understand new research. If you want to seriously consider novel ideas, it's not so easy when you're expected to come up with a response in the next few minutes.
3. Debates generally favor people who use good rhetoric and can package their ideas more neatly, not the people who really have more interesting ideas.
4. If you want to justify a scientific claim, you do it with experiments and evidence (or a mathematical proof when applicable). What purpose does a debate serve?

I think Yud's fixation on debates and "winning" reflects what he thinks of intellectualism. For him, it is merely a means to an end. The real goal is to be superior and beat up other people.

In my experience most people just suck at learning new things, and vastly overestimate the depth of expertise. It doesn't take that long to learn how to do a thing. I have never written a song (without AI assistance) in my life, but I am sure I could learn within a week. I don't know how to draw, but I know I could become adequate for any specific task I am trying to achieve within a week. I have never made a 3D prototype in CAD and then used a 3D printer to print it, but I am sure I could learn within a few days.

This reminds me of another tech bro many years ago who also thought that expertise is overrated, and things really aren't that hard, you know? That belief eventually led him to make a public challenge that he could beat Magnus Carlsen in chess after a month of practice. The WSJ picked up on this, and decided to sponsor an actual match with him and Carlsen. They wrote a fawning article about it, but it did little to stop his enormous public humiliation in the chess community. Here's a reddit thread discussing that incident: https://www.reddit.com/r/HobbyDrama/comments/nb5b1k/chess_one_month_to_beat_magnus_how_an_obsessive/

As a sidenote, I found it really funny that he thought his best strategy was literally to train a neural network and ... memorize all the weights and run inference with mental calculations during the game. Of course, on the day of the match, the strategy was not successful because his algorithm "ran out of time calculating". How are so many techbros not even good at tech? Come on, that's the one thing you're supposed to know!

Just had a conversation about AI where I sent a link to Eddy Burback's ChatGPT Made Me Delusional video. They clarified that no, it's only smart people who are more productive with AI since they can filter out all the bad outputs, and only dumb people would suffer all the negative effects. I don't know what to fucking say.

More AI bullshit hype in math. I only saw this just now so this is my hot take. So far, I'm trusting this r/math thread the most as there are some opinions from actual mathematicians: https://www.reddit.com/r/math/comments/1o8xz7t/terence_tao_literature_review_is_the_most/

Context: Paul Erdős was a prolific mathematician who had more of a problem-solving style of math (as opposed to a theory-building style). As you would expect, he proposed over a thousand problems for the math community that he couldn't solve himself, and several hundred of them remain unsolved. With the rise of the internet, someone had the idea to compile and maintain the status of all known Erdős problems in a single website (https://www.erdosproblems.com/). This site is still maintained by this one person, which will be an important fact later.

Terence Tao is a present-day prolific mathematician, and in the past few years, he has really tried to take AI with as much good faith as possible. Recently, some people used AI to search up papers with solutions to some problems listed as unsolved on the Erdős problems website, and Tao points this out as one possible use of AI. (I personally think there should be better algorithms for searching literature. I also think conflating this with general LLM claims and the marketing term of AI is bad-faith argumentation.)

You can see what the reasonable explanation is. Math is such a large field now that no one can keep tabs on all the progress happening at once. The single person maintaining the website missed a few problems that got solved (he didn't see the solutions, and/or the authors never bothered to inform him). But of course, the AI hype machine got going real quick. GPT5 managed to solve 10 unsolved problems in mathematics! (https://xcancel.com/Yuchenj_UW/status/1979422127905476778#m, original is now deleted due to public embarrassment) Turns out GPT5 just searched the web/training data for solutions that have already been found by humans. The math community gets a discussion about how to make literature more accessible, and the rest of the world gets a scary story about how AI is going to be smarter than all of us.

There are a few promising signs that this is getting shut down quickly (even Demis Hassabis, CEO of DeepMind, thought that this hype was blatantly obvious). I hope this is a bigger sign for the AI bubble in general.

EDIT: Turns out it was not some rando spreading the hype, but an employee of OpenAI. He has taken his original claim back, but not without trying to defend what he can by saying AI is still great at literature review. At this point, I am skeptical that this even proves AI is great at that. After all, the issue was that a website maintained by a single person had not updated the status of 10 problems inside a list of over 1000 problems. Do we have any control experiments showing that a conventional literature review would have been much worse?

OpenAI claims that their AI can get a gold medal on the International Mathematical Olympiad. The public models still do poorly even after spending hundreds of dollars in computing costs, but we've got a super secret scary internal model! No, you cannot see it, it lives in Canada, but we're gonna release it in a few months, along with GPT5 and Half-Life 3. The solutions are also written in an atrociously unreadable manner, which just shows how our model is so advanced and experimental, and definitely not to let a generous grader give a high score. (It would be real interesting if OpenAI had a tool that could rewrite something with better grammar, hmmm....) I definitely trust OpenAI's major announcements here, they haven't lied about anything involving math before and certainly wouldn't have every incentive in the world to continue lying!

It does feel a little unfortunate that some critics like Gary Marcus are somewhat taking OpenAI's claims at face value, when in my opinion, the entire problem is that nobody can independently verify any of their claims. If a tobacco company released a study about the effects of smoking on lung cancer and neglected to provide any experimental methodology, my main concern would not be the results of that study.

Edit: A really funny observation that I just thought of: in the OpenAI guy's thread, he talks about how former IMO medalists graded the solutions in message #6 (presumably to show that they were graded impartially), but then in message #11 he is proud to have many past IMO participants working at OpenAI. Hope nobody puts two and two together!