you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 3 months ago

A token can range from individual letters to several words in a stock phrase, from what little I understand. It's a piece of whatever they're doing behind the scenes leaking into the public-facing terminology.

According to my SO, it's safe enough to just say "a token is a word" as long as you remember this is only an approximation.

  • source
  • parent