687
No real output. Lots of layoffs though (thelemmy.club)
submitted 3 months ago by to c/fuck_ai@lemmy.world
69 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 17 points 3 months ago (3 children)

Tokens are an AI context of units of text. Could be a character or a word. Why use the term tokens instead of words? Who knows. Maybe because some units aren’t words? The tech industry delights in creating stupid terms for things that already had terms.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 4 points 3 months ago

    There are to many words to have each be it's own input, but bundling frequently used snippets is more efficient than using characters.
    Then the investors came and decided to measure success by wasted energy and the rest is history.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 3 months ago

    A token can range from individual letters to several words in a stock phrase, from what little I understand. It's a piece of whatever they're doing behind the scenes leaking into the public-facing terminology.

    According to my SO, it's safe enough to just say "a token is a word" as long as you remember this is only an approximation.

  • permalink
  • fedilink
  • source
  • parent