Thought today's smbc comic was too funny not to share. Source

you are viewing a single comment's thread
view the rest of the comments
[–] 14 points 1 month ago (16 children)

I think you grasp the point, it is exactly what an LLM does

  • source
  • parent
  • hideshow 16 child comments
  • [–] 2 points 1 month ago (15 children)

    An LLM doesn't transcribe what someone else writes for a given input.

  • source
  • parent
  • hideshow 15 child comments
  • [–] 15 points 1 month ago (14 children)

    Technically right in that it doesn't necessarily translate audio into text, but that's hardly the point. The point is someone gives the LLM a giant table that directs its response.

  • source
  • parent
  • hideshow 14 child comments
  • [–] -3 points 1 month ago (3 children)

    If you are giving the weight matrix to the model yourself, you are doing ML wrong. Machine's supposed to "learn" the weights itself. That's the entire point?

  • source
  • parent
  • hideshow 3 child comments
  • [–] 7 points 1 month ago (1 child)

    Dude... Why do you think the whole point is to have properly tagged data? Or why there were thousands of people working at Amazon Turk categorizing images and files for cents per document?

    No. You still have to give it a starting point and the starting point is manually configured tables basically.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 1 month ago*

    And a lot of how it "learns" is by adding more of these into the tables. Every time someone make a video that "brokes" a LLM, these companies will put put those trick questions and screw in the correct answer to train the next generation manually.

    Which is why the messing up counting stuffs stayed broken for so long, there's infinite amount of variations of things that can be counted and validated by a human extremely easily.

  • source
  • parent
  • [–] 6 points 1 month ago

    That's a distinction without a difference. The machine doing the inference is not the same machine doing the learning. From the perspective of the machine doing LLM inference, it could not tell you if the table was hand rolled by a human, fitted using ML or just a table of random noise.

  • source
  • parent
  • [–] -4 points 1 month ago (9 children)

    ? A neural network is not a lookup table.

  • source
  • parent
  • hideshow 9 child comments
  • [–] 12 points 1 month ago* (2 children)

    it's a compressed lookup table. rather than there being one response for every input, the input is used as a seed to decompress relevant parts of the dataset, with some added randomness. you can even do it with gzip itself: https://nathan.rs/posts/gzip-lm

    the linked paper is very good: https://arxiv.org/pdf/2309.10668

  • source
  • parent
  • hideshow 2 child comments
  • [–] 6 points 1 month ago (5 children)

    It could be represented with a series of lookup tables, especially quantized LLMs. A series of inputs results in a specific output that gets passed to the next set of nodes. Repeat 7 billion times, and the final output from the last set of nodes is a set of token probabilities.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 1 point 1 month ago (4 children)

    I don't understand how a network of weighted nodes could be represented by a lookup table.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 1 month ago (3 children)

    There are a finite number of possible inputs if the network is quantized. (And even if the network isn't quantized, it just makes the possible input space much larger.) Especially if there's been connection pruning for input nodes that don't have a significant impact on the output.

    Let's say you have five inputs. For each possible input from node 1, you have a volume. Input is 3? Go grab volume 3. Node 2 relates to chapter; value is 10? Flip to chapter 10. Each chapter is a series of tables. Node 3 is for which table to look at, node 4 is row, node 5 is column. Voila, your five inputs have become a single output value.

    Anything that is immutable and has a finite number of inputs and outputs can be represented by a lookup table.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 1 month ago (2 children)

    That doesn't sound anything like a neural network. Did you know that the chance of having a particular order of a deck of 52 cards shuffled 7 times is 8x10^67. What do you think the number of possible configurations is for a neural network of hundreds of millions of nodes?

  • source
  • parent
  • hideshow 2 child comments