mercano comments on d wha

[–] mercano@lemmy.world 13 points 4 months ago (5 children)

AI doesn’t see a word as a sequence of letters, they just see it as a pointer to an entry in Words table.

permalink

fedilink

source

hideshow 5 child comments

[–] Viceversa@lemmy.world 10 points 4 months ago* (last edited 4 months ago) (2 children)

Semantic Vectors don't work that way.

permalink

fedilink

source

parent

hideshow 2 child comments

[–] lambdabeta@lemmy.ca 12 points 4 months ago (1 child)

Yeah, if words were actually encoded as 1-hot vectors this would be pretty trivial, but the rest of LLM training would be somewhere between infeasible and impossible. The actual embedding vectors obscure spelling even more.

Side note: last time I checked, current embedding vectors were approximately 40 dimensional... Has that gone up significantly in the last couple of years?

permalink

fedilink

source

parent

hideshow 1 child comment

[–] Meron35@lemmy.world 5 points 4 months ago

A fair bit. EmbeddingGemma is open weights and allows for 128-768 dimensions.

It's not as simple as more dimensions = better, due to size, efficiency, and context rot limitations though.

Introducing EmbeddingGemma: The Best-in-Class Open Model for On-Device Embeddings - Google Developers Blog - https://developers.googleblog.com/en/introducing-embeddinggemma/

permalink

fedilink

source

parent

[–] echodot@feddit.uk 3 points 4 months ago

Oh thank God. I was worried that I was really stupid.

permalink

fedilink

source

parent

[–] chicken@lemmy.dbzer0.com 3 points 4 months ago

Shouldn't it help that it separated them out with underlines? How does this text break down in terms of tokens?

permalink

fedilink

source

parent