This is a rough excerpt from a quintet of essays I've intended to write for a
few years and am just now getting around to drafting. Let me know if more from
this series would be okay to share; the full topic is:
Power Relations
- Category of Responsibilities
- The Reputation Problem
- Greater Internet Fuckwad Theory (GIFT), Special Internet Fuckwad Theory
(SIFT), & Special Fuckwittery
- System 3 & Unified Fuckwittery
- Algorithmic Courtesy
This would clarify and expand upon ideas that I've stated here and also on
Lobsters (Reputation
Problem, System
3 (this
post!)) The main idea is to understand how folks exchange power and responsibilities.
As always, I did not use any generative language-modeling tools. I did use
vim's spell-checker.
Humans are not rational actors according to any economic theory of the past
few centuries. Rather than admit that economics might be flawed, psychologists
have explored a series of models wherein humans have at least two modes of
thinking: a natural mode and an economically-rational mode. The latest of
these is the amorphous concept of System 1 and System 2; System 1 is an older
system that humans share with a wide clade of distant relatives and System 2
is a more recently-developed system that evolved for humans specifically. This
position does not agree with evolutionary theories of the human brain and
should be viewed with extreme skepticism.
When pressed, adherents will quickly retreat to a simpler position. They will
argue that there are two modes of physical signaling. First, there are
external stimuli, including light, food, hormones, and the traditional senses.
For example, a lack of nutrition in blood and a preparedness of the intestines
for food will trigger a release of the hormone ghrelin from the stomach,
triggering the vagus nerve to incorporate a signal of hunger into the
brain's conceptual sensorium. Thus, when somebody says that they are hungry,
they are engaged by a System 1 process. Some elements of System 1 are
validated by this setup, particularly the claims that System 1 is autonomous,
automatic, uninterruptible, and tied to organs which evolved before the
neocortex. System 2 is everything else, particularly rumination and
introspection; by excluded middle, System 2 also is how most ordinary
cognitive processes would be classified.
We can do better than that. After all, if System 2 is supposed to host all of
the economic rationality, then why do people spend so much time thinking and
still come to irrational conclusions? Also, in popular-science accounts of
System 1, why aren't emotions and actions completely aligned with hormones and
sensory input? Perhaps there is a third system whose processes are confused
with System 1 and System 2 somehow.
So, let's consider System 3. Reasoning in System 3 is driven by memes: units
of cultural expression which derive semantics via chunking and associative
composition. This is not how System 1 works, given that operant conditioning
works in non-humans but priming doesn't reliably replicate. The contrast with
System 2 is more nebulous since System 2 does not have a clear boundary, but a
central idea is that System 2 is not about the associations between chunks as
much as the computation encoded by the processing of the chunks. A System 2
process applies axioms, rules, and reasoning; a System 3 process is strictly
associative.
I'm giving away my best example here because I want you to be convinced.
First, consider this scenario: a car crash has just happened outside! Bodies
are piled up! We're still pulling bodies from the wreckage. Fifty-seven people
are confirmed dead and over two hundred are injured. Stop and think: how does
System 1 react to this? What emotions are activated? How does System 2 react
to this? What conclusions might be drawn? What questions might be asked to
clarify understanding?
Now, let's learn about System 3. Click, please!
Update to the scenario: we have a complete
tally of casualties. We have two hundred eleven injuries and sixty-nine dead.
When reading that sentence, many Anglophones and Francophones carry an ancient
meme, first attested in the 1700s, which causes them to react in a way that
wasn't congruent with their previous expressions of System 1 and System 2,
despite the scenario not really changing much at all. A particular syntactic
detail was memetically associated to another hunk of syntax. They will also
shrug off the experience rather than considering the possibility that they
might be memetically influenced. This is the experience of System 3:
automatic, associative, and fast like System 1; but quickly rationalizing,
smoothed by left-brain interpretation, and conjugated for the context at hand
like System 2.
An important class of System 3 memes are the thought-terminating clichés
(TTCs), which interrupt social contexts with a rhetorical escape that provides
easy victory. Another important class are various moral rules, from those
governing interpersonal relations to those computing arithmetic. A
sufficiently rich memeplex can permanently ensnare a person's mind by
replacing their reasoning tools; since people have trouble distinguishing
between System 2 and System 3, they have trouble distinguishing between
genuine syllogism and TTCs which support pseudo-logical reasoning.
We can also refine System 1 further. When we talk of training a human, we
ought to distinguish between repetitive muscle movements and operant
conditioning, even though both concepts are founded upon
"wire together, fire together." In the former, we are creating so-called "muscle
memory" by entraining neurons to rapidly simulate System 2 movements; by
following the principle "slow is smooth, smooth is fast", System 2 can chunk
its outputs to muscles in a way analogous to the chunking of inputs in the
visual cortex, and wire those inputs and outputs together too, coordinating
the eye and hand. A particularly crisp example is given by the arcuate
fasciculus connecting Broca's area and Wernicke's area, coordinating the
decoding and encoding of speech. In contrast, in the latter, we are creating a
"conditioned response" or "post-hypnotic suggestion" by attaching System 2
memory recall to System 1 signals, such that when the signal activates, the
attached memory will also activate. Over long periods of time, such responses
can wire System 1 to System 1, creating many cross-organ behaviors which are
mediated by the nervous system.
This is enough to explain what I think is justifiably called "unified
fuckwittery," but first I need to make one aside. Folks get creeped out by
neuroscience. That's okay! You don't need to think about brains much here. The
main point that I want to rigorously make and defend is that there are roughly
three reasons that somebody can lose their temper, break their focus, or
generally take themselves out of a situation, losing the colloquial "flow
state." I'm going to call this situation "tilt" and the human suffering it is
"tilted." The three ways of being tilted are to have an emotional response to
a change in body chemistry (System 1), to act emotional as a conclusion of
some inner reasoning (System 2), or to act out a recently-activated meme which
happens to appear like an emotional response (System 3). No more brain talk.
I'm making a second aside for a persistent cultural issue that probably is not
going away. About a century ago, philosophers and computer scientists asked
about the "Turing test": can a computer program imitate a human so well that
another human cannot distinguish between humans and imitations? About a
half-century ago, the answer was the surprising "ELIZA effect": relatively
simple computer programs can not only imitate humans well enough to pass a
Turing test, but humans prefer the imitations to each other. Put in more
biological terms, such programs are "supernormal stimuli"; they appear "more
human than human." Also, because such programs only have a finite history,
they can only generate long interactions in real time by being
"memoryless" or "Markov", which means that the upcoming parts of an
interaction are wholly determined by a probability distribution of the prior
parts, each of which are associated to a possible future. Since programs don't
have System 1 or System 2, and these programs only emit learned associations,
I think it's fair to characterize them as simulating System 3 at best. On one
hand, this is somewhat worrying; humans not only cannot tell the difference
between a human and System 3 alone, but prefer System 3 alone. On the other
hand, I could see a silver lining once humans start to understand how much of
their surrounding civilization is an associative fiction. We'll return to this
later.
It's important to understand that the book's premise is fairly hollow. Yudkowsky's rhetoric really only gets going once we agree that (1) intelligence is comparable, (2) humans have a lot of intelligence, (3) AGIs can exist, (4) AGIs can be more intelligent than humans, and finally (5) an AGI can exist which has more intelligence than any human. They conclude from those premises that AGIs can command and control humans with their intelligence.
However, what if we analogize AGIs and humans to humans and housecats? Cats have a lot of intelligence, humans can exist, humans can be more intelligent than housecats, and many folks might believe that there is a human who is more intelligent than any housecat. Assuming intelligence is comparable, does it follow that that human can command and control any housecat? Nope, not in the least. Cats often ignore humans; moreover, they appear to be able to choose to ignore humans. This is in spite of the fact that cats appear to have some sort of empathy for humans and perceive us as large slow unintuitive cats. A traditional example in philosophy is to imagine that Stephen Hawking owns a housecat; since Hawking is incredibly smart and capable of spoken words, does it follow that Hawking is capable of e.g. talking the cat into climbing into a cat carrier? (Aside: I recall seeing this example in one of Sean Carroll's papers, but it's also popularized by Cegłowski's 2016 talk on superintelligence. I'm not sure who originated it, but I'd be unsurprised if it were Hawking himself; he had had that sort of humor.)