305
‘Too little, too late’: critics perplexed and suspicious of AI leaders’ call for a slowdown (www.theguardian.com)
submitted 1 week ago by to c/technology@lemmy.world
50 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[+] -11 points 1 week ago* (last edited 1 week ago) (14 children)

The pace of advancement is still going strong, especially in math and "reasoning". For example the frontier math benchmark is showing big strides for the frontier models. A year ago they were only getting <10% of questions, now sol from openai is scoring 89% and they're reporting the unreleased astra is at 98% .

They even had to create a new frontier math benchmark called erdos that all previous models scored 0% on and astra got 3% , which may not look like much but is a huge relative jump.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 14 child comments
    •
  • [–] 19 points 1 week ago* (11 children)

    I'm aware of what the pace is. You might want to check up on the current controversy surrounding OpenAI's math "achievements".

    By my reckoning, the difference between Mythos and Opus is smaller than the difference between Opus and Sonnet. Same with the difference between GPT 5.5 to 5.6 is smaller than the difference between GPT 4 to GPT 5.

    The size of improvements over time is diminishing. We're not in "big bang" territory anymore and we're about two years into the "incremental refinement" period. We're about to enter the next AI Winter unless somebody comes up with a new architectural component as revolutionary as transformers have been for ML models.

    The core problem is that LLMs do not create. Full stop. All creativity is borne by the human inputs. Until that changes - until the model gains the capability to truly create new information, we've hit the limits in raw capability.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 11 child comments
    •
  • [–] 5 points 1 week ago*

    The size of improvements over time is diminishing. We're not in "big bang" territory anymore and we're about two years into the "incremental refinement" period. We're about to enter the next AI Winter unless somebody comes up with a new architectural component as revolutionary as transformers have been for ML models.

    The models are also getting extremely big as well, since the big improvement currently seems to largely be stuffing the model with more parameters, and making that work.

    I can only imagine that the training cost has also been skyrocketing.

    The core problem is that LLMs do not create. Full stop. All creativity is borne by the human inputs. Until that changes - until the model gains the capability to truly create new information, we've hit the limits in raw capability.

    The models suppress outliers by design. Statistically speaking, the most novel thing is a garbled mess of random words, but random noise is useless, so it ends up being suppressed. You can see by fiddling the samplers, or increasing the temperature.

    The Library of Babel is the most creative thing in the world, containing every possible combination of English words and letters. You can basically act like an LLM by trying to find a new coherent sentence in it, but also one that hasn't been said before. It's basically impossible.

    But that is what an improvement is supposed to be. Compare that to finding a sentence that has been said, or something close to it.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 1 week ago (8 children)

    You might want to check up on the current controversy surrounding OpenAI's math "achievements".

    I'm aware of the controversies, this benchmark isn't about making new proofs on previously unsolved problems, it's whether it can answer complex math problems, which it's getting better at.

    By my reckoning, the difference between Mythos and Opus is smaller than the difference between Opus and Sonnet. Same with the difference between GPT 5.5 to 5.6 is smaller than the difference between GPT 4 to GPT 5.

    Do you have any benchmarks or data to back this "reckoning"

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 5 points 1 week ago (7 children)

    Do you have any benchmarks or data to back this “reckoning”

    I work with LLMs daily. I read papers as they hit arxiv. Also daily. You clearly don't.

    I'm not interested in convincing anyone, which is why I'm speaking non-technically.

    The benchmarks being cited aren't as interesting as you appear to believe they are. You've not fully grasped the fact that solving pre-made problems where the solutions are known or knowable isn't anywhere close to the same thing as asking truly novel research questions independent of a human prompt. For OpenAI to also be embroiled in allegations of plagiarism only serves to underscore the gap between the two concepts.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] -1 points 1 week ago (6 children)

    fact that solving pre-made problems where the solutions are known or knowable isn't anywhere close to the same thing as asking truly novel research questions independent of a human prompt.

    I understand that, but the original statement was about the models stalling out in progress.

    Would you say a student stalled out in progress if they could barely do 2 + 2 a couple years ago and is now able to consistently solve some of the most complex math problems known just because that student isn't creating novel research?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 1 point 1 week ago (5 children)

    You're trying to analogize your way into a subject you clearly haven't studied.

    There's pre-existing research here. Godel's Incompleteness Theorem holds, plus others.

    There's already a known upper bound here that you're clearly unaware of.

    There's as yet been zero LLM-based architectures that have created new information. Everything they produce is somewhere within the training data. LLMs are a very specialized data compression algorithm, in a fashion.

    The stall is around whether Recursive Self-Improvement is achievable. Recent papers out of China are trying to chart a course to it. But, until someone succeeds, The current pace of improvement is already slowing signs of slowing. It's not about where the finish line is placed, it's about how fast they get there.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 1 week ago (4 children)

    How are you defining "new information"? If I ask Claude "what is 235 x 567 + 57,899? " There is a high likelihood that that question was not in its training data, but it still gets it right and generates new information. It will be using existing methods from its training data to "reason" to get the answer, and it hasn't shown the ability to devise new methods. But 99.99% of the problems in the world can be solved by using existing methods.

    Open AI isn't going to go bust because it can't make novel math techniques, almost all of the knowledge work requiring math doesn't require new methods. If it's able to master every existing technique used by knowledge workers without creating new methods, ie. your finish line, then they'll have automated a huge chunk of labor and probably become a hugely profitable company.

    Recursive self improvement may not be panning out currently so we won't see any singularity style exponential take off, but we're still seeing steady linear improvement in tasks such as math and reasoning through other techniques, like the opaque recurrence that astra is using.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 1 week ago (3 children)

    As has been stated, this isn't new territory. If you're unfamiliar with information theory, you have some reading to do.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 1 week ago* (2 children)

    Bringing up information theory and godels incompleteness theorem in regards to ai advancement is like saying there are hard physical limits on shrinking the dye in the 1980s. Yes hard axiomatic limits exist, but within those limits is a huge problem space. Even if we limit it to solving problems on a screen using existing techniques, that problem space is huge and occupying a large amount of time for the knowledge workers who make up ~25% of the labor force.

    Unlike shrinking the dye though we are not near that limit, we still have a lot of improvement to make before AI can solve all the problems in that space. I think we can both agree the current LLMs aren't able to replace all screen work. But it is making big strides in that problem space, such as going from failing basic arithmetic to solving the hardest math problems in a couple years.

    Like with shrinking the dye there's not going to be a singular breakthrough, like recursive self improvement, that'll throw us up against those limits almost immediately. Progress is done incrementally over years with a bunch of different smaller improvements, like opaque recurrence. This does have some acceleration as the technology helps with the design of the technology, computers help chip architects design better chips, which make better computers, but a human is still required. Same with AI, every lab is probably heavily relying on coding agents to assist them with developing and deploying new techniques. The AI may not be creating these techniques, but it is accelerating the creation of these techniques.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 0 points 1 week ago* (1 child)

    Once again, you're trying to analogize your way into something you haven't studied.

    What exactly is your purpose here? You still can't even articulate the fundamental flaws LLMs have with tasks not fully within the training data. Right now, today, there is no frontier model that can operate on tasks without resorting to reward hacking once its outside the small class of problems its training data covers. The solution for this is to invest significant time from domain experts to meticulously define how to solve tasks in other domains.

    Here's a trivial example: Try getting Claude to generate coherent COBOL. Or TCL. Or even Powershell. Any language that has low representation on StackOverflow is a language that Claude can't speak until someone teaches it how. Even the Python it generates has limited expressiveness or extensibility.

    Everywhere you look, the AI is limited by the fact that it can't generate its own new information. Navier-Stokes and statements in pure mathematics like it are the absolute best case scenario for agentic work against rigorous specification. The theorem statement itself is already a rigorous specification. It has undergone decades of auditing by the mathematical community and its rendering in Lean is a straightforward translation defined in terms of battle-tested mathematical objects from mathlib. The verifier, the Lean theorem prover, has been extensively audited and specifically designed to avoid the types of unsoundness that would make it vulnerable to reward hacks.

    No other domains outside of mathematics have such rigorous specifications. Yet, somehow, you believe there's magic pixie dust somewhere within the LLM that will help it achieve something without human interventions and that, somehow, we're "close" to that accomplishment. Don't quit your day job.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 week ago* (last edited 1 week ago)

    Explain to me how the analogy is wrong then.

    I have accepted that llms have hard limits, like you said with regards to information theory and godels incompleteness theorem. I don't have enough knowledge to articulate those limits, just like I don't know the physics that prevents you from shrinking the dye further but I accept that they exist.

    So we agree there are hard limits, what I am saying is:

    • within those limits is a huge problem space
    • millions of people are putting labor in that problem space today
    • LLMs have not saturated that problem space yet
    • LLMs are making gains in that problem space, like with math and reasoning.

    I'm not arguing where that "finish line" is because I know I don't know enough about that. I do understand that 99.999% of problems aren't passed that finish line and there's a lot to go before LLMs reach that line.

    Almost all use cases for llms don't require creating novel math techniques, so why are you focusing on them? If "can't create genuinely new techniques" disqualifies something from being economically transformative or intellectually significant, that standard would disqualify most human knowledge labor too.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 0 points 1 week ago

    I would say that AI is becoming much more effective for the tokens produced. I have been running Qwen 3.8 Flash-Next on my new PC. This AI model weighs in at 120b parameters, with another 50b added via an Engram table that costs about an extra 20% in RAM for its parameter count. This is a major jump in world knowledge, and the model in question has 262k context length.

    While experimental and with rough edges, this model has been able to use Lingua Gacha to undertake a game translation project - understanding and converting a Chinese game into English.

    Agentic AI has become practical on local hardware over this last year, and there looks to be much more refinement to come.

    00000000

    What we are seeing with Anthropic and the other American companies, is a scramble to protect themselves from the consequences of badly planned data centers and poor practices in AI development. Their Chinese competition has produced much more efficient models, while delivering almost the same level of quality as American offerings. This means that local AI is becoming "good enough", which makes data centres and AI subscriptions unnecessary for many businesses if they distrust the cloud.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 0 points 1 week ago

    We use AI as an umbrella term for so many things. Literally the type of machine learning you’re talking about has been in use in science and mathematics for quite some time and unrelated to all this buzz. It is decidedly not what OpenAI, Anthropic and Meta’s product are. Those are LLMs friend. The promised harbinger of AGI via agenetic AI. So far, all it’s been is an embarrassingly expensive and destructive waste of resources.

  • permalink
  • fedilink
  • source
  • parent