606
Adult life in 2026, so beautiful (thelemmy.club)
submitted 2 months ago by to c/memes@lemmy.world
142 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 153 points 2 months ago (15 children)

I know this is a meme, but for the record math is one of the things LLMs are famously worst at

  • permalink
  • fedilink
  • source
  • hideshow 15 child comments
    •
  • [–] 64 points 2 months ago (4 children)

    Can confirm; I teach statistics and allow take home exams. I don't even really need guardrails on the math-- if you cheat via LLM, your answer is almost always hilariously wrong.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 8 points 2 months ago (3 children)

    I've not used gpt for quite a while, but can't you ask to python script the evaluation?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 25 points 2 months ago* (last edited 2 months ago) (2 children)

    For simple math that could work, and as long as the question is close enough to an exact match with plenty of published examples to copy.

    A good rule of thumb is that the script it will come up with is about as likely to be correct as blindly taking the highest voted answer to the most similar question on Stack Overflow.

    If the question is simple and common, the odds are quite good. If the question is nuanced or rare, the odds of a correct result drop off aggressively.

    Edit: Your mileage may vary - providing an API to these LLMs that can do math correctly for them is pretty easy. Getting the LLM to consistently detect when to use that API is more challenging.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 months ago* (1 child)

    cannot confirm. have used chatgpt many times to check my physics homework (after doing it myself first) and it was basically correct about 90% of the time.

    these were not standard exercises (i think)

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 2 months ago*

    Thanks. I think every data point helps people.

    I can't say that particularly shocks me. I imagine that Chat GPT has probably swallowed dozens of Physics textbooks.

    The bigger part of the problem is knowing where we are on the human knowledge novelty / training data available curve, or rather, when we need to get off of the ride.

    Edit: Any chance your chosen AI has access to Wolfram Alpha as an MCP Tool? Because that would be very different, too.

    If it's successful at math, there's likely much more than a learning model in play.

    Which brings us back to the opacity problem. :(

    If the answer machine has been cleverly rigged (and weighted) to pass off math questions to more capable software, things can be great.

    But a person who just hears that AI can do math now can have a very bad time if they go blindly use a brute force pleasant answer machine on a math problem. :(

  • permalink
  • fedilink
  • source
  • parent
  • [–] 46 points 2 months ago (4 children)

    Yesterday I literally had to count 14 thousand pounds of fire extinguishers (as in count each extinguisher in a bunch of crates), one of my coworkers came over and said "Why don't you just use AI to count them." They then pulled out their phone, took a picture, asked AI to count them, and was off buy almost 60% on the first crate. Then went "oh, well that's just one example,"

    They then tried 4 more times, being between 75% and 50% off each time. They then went "Oh, well, ummm, nevermind." and wandered off again.

    (I had to count the extinguishers to make a Bill of Lading for them for shipping them out, you need an accurate count otherwise it will be held by the shipping company until you provide a correct BoL with the correct count due to legal reasons. I couldn't just weigh them and divide the total wieght by the weight of a single extinguisher as they were a bunch of different sizes mixed together. NGL it was the easiest money I've ever made at the at job.)

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 10 points 2 months ago (1 child)

    NGL it was the easiest money I've ever made at the at job

    Easy money for you. For me that work would have been torture.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 17 points 2 months ago

    Oh it sucked for sure, I kept getting interrupted which meant whatever crate I was counting at the time had to be started over.

    But the biggest perk was not having to deal with the other idiots on our loading dock for the entire time I was counting.

    Though I've joked about it in the past I really should get a shirt that says "I'm trying to focus on my job, please don't distract me" or something like that.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 6 points 2 months ago* (last edited 2 months ago) (3 children)

    Yes, but most of the paid "LLMs" are not actually just LLMs, they have specific aritimetic modules that end up being activated for most math questions that actual LLMs suck at, however are pretty easy for arithmetic modules, like simple calculations, unit conversions, etc...

    So yeah, any edge "LLM" tool currently will always get the original question asked here CORRECT even on the free tiers, you can test it yourself on Google Search itself in "AI mode", on ChatGPT, or DeepSeek...

    So you are technically correctly in saying that LLMs suck and cannot be trusted for math in almost any case. However you are wrong on implying that ChatGPT, or other commercial models are just LLMs, they are not.

    P.S.: I'm not condoning the use of AI to resolve math related questiona, nor saying that their replies should be trusted, just stating the fact that actual "AI" products use more than just a large language model to generate their responses, and for math related questions most "AI" today will get arithmetic question answered correctly NOT DEPENDING on having training for the exact questiona being asked since they can identify a math arithmetic question being asked and instead of trying to fully generate the answer with the model itself, which would lack the training data to be able to process such wide range of possible question, they generate placeholders internally and let a arithmetic like module do the calculations, which take negligible processing power to be cacoulated by CPUs.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 2 months ago* (2 children)

    The question I have to ask is, computers have been doing math since the very beginning, and now we are wasting a shit ton of resources (and giving every shred of info to billionaires so they can hold it hostage) to do the same math even worse when that is what the computer was designed for. Has everyone forgotten the purpose of the computer or something? I feel like I'm taking Crazy pills.

    Almost everything llm supercomputers do has already been done. It is the least innovative thing. If you take ANYTHING and shove a trillion dollar supercomputer at it, of course its going to be surface level impressive to simpletons. I just do not understand any of the hype. Its not exciting or new. Hell, vr was more interesting than this and that had already been done decades before the oculus and index popularized it. I'm real sick of these bandwagoners pretending they know tech when all they know is corporate overlord advertising.

    Not ranting at you ruan but just screaming into the void to be ignored by the mass populace, as per usual.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 week ago (1 child)

    I completely get you, man.

    During a "AI traning" my job enlisted me, the instrutor, someone with apparently high technical knowledge of software systems and development, design patterns, with years of experience and supposedly familiar with using CLIs and Linux shell commands wanted to create a symlink to a file and instead of typing up the command, or if he did not remember it, just quickly looking up on a usual search or even a LLM powered search which command to use to archieve it, he wrote up on his Claude Code terminal "create a symbolic link from X to Y"...

    The Claude Code agent of course performed the action correctly, however the amount of processing power wasted to execute some basic action of his job, which is executing a command correctly, was all left for the "AI" agent to do because it was "easier".

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •