▲ 342 ▼ d wha (thelemmy.club) submitted 4 months ago by Abrinoxus@lemmy.today to c/aneurysmposting@sopuli.xyz 21 comments fedilink hide all child comments d
[–] Viceversa@lemmy.world 10 points 4 months ago* (last edited 4 months ago) (2 children) Semantic Vectors don't work that way. permalink fedilink source parent hideshow 2 child comments replies: [–] lambdabeta@lemmy.ca 12 points 4 months ago (1 child) Yeah, if words were actually encoded as 1-hot vectors this would be pretty trivial, but the rest of LLM training would be somewhere between infeasible and impossible. The actual embedding vectors obscure spelling even more. Side note: last time I checked, current embedding vectors were approximately 40 dimensional... Has that gone up significantly in the last couple of years? permalink fedilink source parent hideshow 1 child comment replies: [–] Meron35@lemmy.world 5 points 4 months ago A fair bit. EmbeddingGemma is open weights and allows for 128-768 dimensions. It's not as simple as more dimensions = better, due to size, efficiency, and context rot limitations though. Introducing EmbeddingGemma: The Best-in-Class Open Model for On-Device Embeddings - Google Developers Blog - https://developers.googleblog.com/en/introducing-embeddinggemma/ permalink fedilink source parent
[–] lambdabeta@lemmy.ca 12 points 4 months ago (1 child) Yeah, if words were actually encoded as 1-hot vectors this would be pretty trivial, but the rest of LLM training would be somewhere between infeasible and impossible. The actual embedding vectors obscure spelling even more. Side note: last time I checked, current embedding vectors were approximately 40 dimensional... Has that gone up significantly in the last couple of years? permalink fedilink source parent hideshow 1 child comment replies: [–] Meron35@lemmy.world 5 points 4 months ago A fair bit. EmbeddingGemma is open weights and allows for 128-768 dimensions. It's not as simple as more dimensions = better, due to size, efficiency, and context rot limitations though. Introducing EmbeddingGemma: The Best-in-Class Open Model for On-Device Embeddings - Google Developers Blog - https://developers.googleblog.com/en/introducing-embeddinggemma/ permalink fedilink source parent
[–] Meron35@lemmy.world 5 points 4 months ago A fair bit. EmbeddingGemma is open weights and allows for 128-768 dimensions. It's not as simple as more dimensions = better, due to size, efficiency, and context rot limitations though. Introducing EmbeddingGemma: The Best-in-Class Open Model for On-Device Embeddings - Google Developers Blog - https://developers.googleblog.com/en/introducing-embeddinggemma/ permalink fedilink source parent