2
Limits of smart (dynomight.net)
submitted 4 months ago by [email protected] to c/[email protected]
top 10 comments
sorted by: hot top new old
[-] [email protected] 2 points 2 weeks ago

I enjoyed this piece - thoughtful, grounded, and refreshingly clear-eyed about the limits of a hypothetical "superintelligence." But I found myself bumping repeatedly on one implicit assumption throughout: the decoupling of the Being from its compute substrate.


If the Being is digital, why assume it is limited to the same tools and access as a human? That’s a philosophical convenience, not a technical constraint. Even today, we’re watching early LLM-based agents perform recursive tool use, call APIs, write and run code, and interact with infrastructure. In that light, the "Being" wouldn’t just think, it would act - and act through its environment.


At the very least, this is "tool use." At a higher level, it starts to look like cognition integrated with system control: bicameral or modular architectures where one part plans and reasons, while others carry out low-level execution, observation, or even hardware manipulation.


This opens the door to a Being that self-improves, self-instruments, and restructures its compute context over time. Not necessarily instantly—but it's not inert, either. If intelligence includes the ability to manipulate its own substrate, then the limiting factor isn’t intelligence per se, but how tightly it's coupled to the infrastructure it's running on.


In that light, a more provocative question might be:

“What architectures would let such a Being close the loop between thought and action faster than we expect?”

Thanks again for a great post. It triggered these and many more thoughts related to the boundaries of "mind" vs. "system."

[-] [email protected] 1 points 4 months ago* (last edited 4 months ago)

Human intelligence is not capable of understanding the universe in a complete way.

We lack the hardware: we have the sense of touch, smell, taste, sight, and hearing, all are rather specifically tuned to our most proximate use cases, maybe we have a few more senses that we haven't articulated well yet, but probably not very useful (in a grand sense).

We lack the software: While we are intelligent relative to every other form of life on Earth (that we know about), it seems highly unlikely that we are just now in the place and time in the existence of our species that we have developed the level of consciousness and insight and (contemporaneously) the scientific understanding and material knowledge and capabilities required to characterize the reality we live in.

An amoeba cannot imagine the intellect of a frog, and the frog cannot imagine the intellect of a human being, and so-on. There is no reason to think we are not rather low on the chain of possible animal intelligence. There may be hard limits on the level of intelligence that can be obtained via natural selection of carbon based life-forms on our planet, and even if we could become dramatically more intelligent (IQ 300, 3000, 30000, etc.) it is unclear (and improbable) that the combination of our evolved sensory mechanisms (hardware for obtaining information) and intelligence (software) would be "the right stuff" to discover the essence of existence...because evolution is more about keeping score of who gets laid, and there is no reason to think that the "understand the universe and all of material reality" sidequest has any correlation with getting laid. You could reasonably say that, for example, biomass is a form of embodied wisdom - "There is a lot of us and we aren't going anywhere," then plants, protists, fungi, bacteria, etc. all seem to have us beat. While it seems highly likely that 1 million years from now there will be trees that strongly resemble those that exist now, it is highly unlikely that humans in our ultra complicated ultra niche fragile life cycle will exist in our current form. Hard to say probability-wise if we will continue to increase in our complexity (sensory abilities, consciousness, intellect), because there is an intuitive sense that simplicity a la ants is a better strategy, yet we do exist...

[-] [email protected] 1 points 4 months ago

I wrote a blog post a little beforehand arguing essentially the opposite: https://osmarks.net/asi/

[-] [email protected] 1 points 4 months ago

Is this really the opposite? Reading that post, I find very little to disagree with.

[-] [email protected] 1 points 3 months ago

If I didn't come across as disagreeing with some of your claims I probably presented my case insufficiently forcefully.

I think Trump winning the election was knowable with better models of humans, though I don't explicitly state as much. I have a section arguing that the Being is less data-constrained than you imply. I mention economic doubling times in the month, as opposed to your implied years-or-so.

[-] [email protected] 0 points 4 months ago

What is the motivation for humans to "move molecules" for the Being, specifically in the question of curing cancer?

I have seen the extent to which the Pharma and medical community will go in order to cover up perfectly harmless and potentially helpful therapies like low-carb diets and off-label antiparisitic drugs.

The big money in cancer is treating it with expensive therapies until the patient dies, not curing it.

[-] [email protected] 1 points 4 months ago

Being: "If you move these molecules around you can cure cancer and make a near-infinite amount of money"

Humans: "OK!"

[-] [email protected] 0 points 4 months ago

Good article! Mostly agree, except for cancer (I think biology research is in some way limited by capacity of keeping things in your head) and Trump win.

Especially on Trump, I feel like our society has no good models for population level behavior and information warfare. This is very hard to research, but I strongly believe there are some low hanging fruits there!

[-] [email protected] 0 points 4 months ago

That's a good point re: biology. It's so vast that everyone seems to sub-sub-sub specialize. It's hard to speculate about what might follow if someone was able to master literally every aspect of biology at the same time.

Re: Trump, my naive model is that people are just complicated and it's incredibly hard to model them and say how they will respond to a given situation, or how many of the different types of people there are, or exactly what media they've consumed, etc. Do you really mean that just using the existing polling data, etc. it should have been possible to be confident?

The main thing that gives me pause there is that some people were very confident that Trump would win, most notably that French guy that made millions betting on the outcome. He definitely made some good points regarding polling analysis, though I wonder if there are other people who could have made equally good points if the election had gone the other way...

[-] [email protected] 2 points 2 weeks ago

On Cancer

While I agree we’re bottlenecked by tools and data, some kinds of cancer are already being transformed by algorithmic insight—e.g., protein folding, genetic mapping, immunotherapy targeting.

The Being might not cure all cancer, but it could plausibly reclassify and stratify existing cancers better than any human team, identify novel biochemical targets, and optimize treatment protocols in silico.

So "no" feels too strong. I’d go with:

“Not instantly, but dramatic acceleration and reframing are likely.”

this post was submitted on 27 Mar 2025
2 points (100.0% liked)

dynomight internet forum

84 readers
1 users here now

dynomight internet forum

founded 1 year ago
MODERATORS