A fairly good and nuanced guide. No magic silver-bullet shibboleths for us. I particularly like the observation that in reality LLMs are less likely to use unusual words, because they are statistically less likely.

Consequently, the LLM tends to omit specific, unusual, nuanced facts (which are statistically rare) and replace them with more generic, positive descriptions (which are statistically common). Thus the highly specific "inventor of the first train-coupling device" might become "a revolutionary titan of industry." It is like shouting louder and louder that a portrait shows a uniquely important person, while the portrait itself is fading from a sharp photograph into a blurry, generic sketch. The subject becomes simultaneously less specific and more exaggerated.

Some juicy extracts:

Soon enough then the appointed day came to pass, that Mr. Assi began playing some of the town's players, defeating them all without exception. Mr. Assi did sometimes let some of the youngest children take a piece or two, of his, and get very excited about that, but he did not go so far as to let them win. It wasn't even so much that Mr. Assi had his pride, although he did, but that he also had his honesty; Mr. Assi would have felt bad about deceiving anyone in that way, even a child, almost as if children were people.

Yud: "Woe is me, a child who was lied to!"

Tessa sighed performatively. "It really is a classic midwit trap, Mr. Humman, to be smart enough to spout out words about possible complications, until you've counterargued any truth you don't want to hear. But not smart enough to know how to think through those complications, and see how the unpleasant truth is true anyways, after all the realistic details are taken into account." [...] "Why, of course it's the same," said Mr. Humman. "You'd know that for yourself, if you were a top-tier chess-player. The thing you're not realizing, young lady, is that no matter how many fancy words you use, they won't be as complicated as real reality, which is infinitely complicated. And therefore, all these things you are saying, which are less than infinitely complicated, must be wrong."

Your flaw dear Yud isn't that your thoughts cannot out-compete the complexity of reality, it's that it's a new complexity untethered from the original. Retorts to you wild sci-fi speculations are just minor complications brought by midwits, you very often get the science critically wrong, but expect to still be taken seriously! (One might say you share a lot of Humman misquoting and misapplying "econ 101". )

"Look, Mr. Humman. You may not be the best chess-player in the world, but you are above average. [... Blah blah IQ blah blah ...] You ought to be smart enough to understand this idea."

Funilly enough the very best chess players like Nakamura or Carlsen will readily call themselves dumbasses outside of chess.

"Well, by coincidence, that is sort of the topic of the book I'm reading now," said Tessa. "It's about Artificial Intelligence -- artificial super-intelligence, rather. The authors say that if anyone on Earth builds anything like that, everyone everywhere will die. All at the same time, they obviously mean. And that book is a few years old, now! I'm a little worried about all the things the news is saying, about AI and AI companies, and I think everyone else should be a little worried too."

Of course this a meandering plug to his book!

"The authors don't mean it as a joke, and I don't think everyone dying is actually funny," said the woman, allowing just enough emotion into her voice to make it clear that the early death of her and her family and everyone she knew was not a socially acceptable thing to find funny. "Why is it obviously wrong?"

They aren't laughing at everyone dying, they're laughing at you. I would be more charitable with you if the religion you cultivate was not so dangerous, most of your anguish is self-inflicted.

"So there's no sense in which you're smarter than a squirrel?" she said. "Because by default, any vaguely plausible sequence of words that sounds it can prove that machine superintelligence can't possibly be smarter than a human, will prove too much, and will also argue that a human can't be smarter than a squirrel."

Importantly you often portray ASI as being able to manipulate humans into doing any number of random shit, and you have an unhealthy association of intelligence with manipulation. I'm quite certain I couldn't get at squirrel to do anything I wanted.

"You're not worried about how an ASI [...] beyond what humans have in the way of vision and hearing and spatial visualization of 3D rotating shapes.

Is that... an incel shape-rotator reference?

It can't be that stupid, you haven't read the sequences hard enough.

The best answer will be unsettling to both the hard skeptics of AI and the true believers.

I do love a good middle ground fallacy.

EDIT:

Why did the artist paint the sky blue in this landscape painting? […] when really, the answer is simply: Because the sky is blue!

I do abhor a "Because the curtains were blue" take.

EDIT^2:

In humans, a lot of problem-solving capabilities are highly correlated with each other.

Of course "Jagged intelligence" is also—stealthily?—believing in the "g-factor".

I was also a Elon skeptic back-then, but I'll admit I did get a kick out of the "don't panic" dashboard.

But golly does he read H2G2 completely wrong (transcript):

I think and it highlighted an important point which is that a lot of times the question is harder than the answer. And if you can properly phrase the question, then the answer is the easy part. So, to the degree that we can better understand the universe, then we can better know what questions to ask. Then whatever the question is that most approximates: what’s the meaning of life? That’s the question we can ultimately get closer to understanding. And so I thought to the degree that we can expand the scope and scale of consciousness and knowledge, then that would be a good thing.

It's backwards! It misses the joke! It took thousands of years and they got a nonsensical answer before any question! It took a thousand more and they got a nonsensical—incompatible—question! It has been theorized that should someone understand the universe it would be replaced by something more complicated! It has also been theorized this has already happened! Also regarding scale of knowledge, Trin Tragula definetly showed that the One thing you can't afford to have in this universe, is a sense of perspective!

Surely his reading comprehension isn't actually this bad, and he only got a bad meme-cliffnotes version of the radio-series/books/movies!?!

How nice it must be to never ponder how large humanity is, and how each and every person you see outside has a full and rich interior and exterior world, and you that only see a tiny fraction of the people outside.

Personally one of my "oh other people are real!" moment, was when our parents (along with my sisters) took us on a surprise ferry trip to England (from France) and our grandparents that—at least as far as kid me remembered—we only ever saw in their home city, were waiting for us in Portsmouth, and we visited the city together (Portsmouth Historic Dockyard is quite nice btw).

I knew they were real, but realizing that they weren't geo-locked, made me more fully internalize that they had full and independent lives, and therefore that everyone had.

How about people here? When did you realize people are real?

No no no it's fine! You get the word shuffler to deshuffle the—eloquently—shuffled paragraphs back into nice and tidy bullet points. And I have an idea! You could get an LLM to add metadata to the email to preserve the original bullet points, so the recipient LLM has extra interpolation room to choose to ignore the original list, but keep the—much more correct and eloquent, and with much better emphasis—hallucinated ones.

SaaS = ~~Storage as a Service~~ Sneer as a Service

Quinn enters the dark and cold forest, crossing the threshold, an omnipresent sense of foreboding permeates the air, before being killed by a grue.

Meanwhile some of the comments are downright terrifying, also the whole "research" output is overly-detailed yet lacking any substance, and deeply deeply in fantasy land, but all the comments a debating in favour of or against what is perceived as "real work", and in terms of presentation "vibes".

I mean my parents always said that fascist/cultish movements have issues distinguishing signified and signifier, but good grief. (Yes too much Lacan in the household)

And yet they can spit out copyrighted material verbatim, or near-verbatim, how strange and peculiar.

Not every rationalist I've met has been nice or smart ^^.

I think it's hard to grow up in our society, without harboring a kernel of fascism in our hearts, it's easy to fall into the constantly sold "everything would work better if we just put the right people in charge". With varying definitions of who the "right people" are:

• Racism
• Eugenics
• Benevolent AI
• Fellow tribe,
• The enlightened who can read "the will of the people" or who are able to "carve reality at the joints"
• Some brands of "sovereign citizen" or corporate libertarianism (I'm the best person in charge of me!).
• The positivist invokers of ScientificProgress™

Do they deserve better? Absolutely, but you can't remove their agency, they ultimately chose this. The world is messy and broken, it's fine not to make too much peace with that, but you have to ponder your ends and your means more thoughtfully than a lot of EAs/Rationalists do. Falling prey to magical thinking is a choice, and/or a bias you can overcome (Which I find extremely ironic given the bias correction advertising in Rationalists spheres)