you are viewing a single comment's thread
view the rest of the comments
[–] 10 points 2 months ago

That doesn't make much sense. When Anthropic moved to Sonnet 5 they introduced a new tokenizer which increased token use up to 35%. If these would be unrelated kinds of tokens why would the usage go up when the process of tokenization changes?

  • source
  • parent