this post was submitted on 28 Mar 2025
33 points (100.0% liked)
Programming
19290 readers
148 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities [email protected]
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Put them into an opensearch database. It is the open source fork of elasticsearch. It has an sql plugin so you can retrieve the raw data the usual way. And there is probably also an integeation/library for it if you use any major framework/language in the backend.
But on top of it you get a very performant full text search. This might come in handy for example when you remember a sentence from a story, or if you want to find all stories with a specific character name or word for whatever reason.
Opensearch will be the most performant. Anything sql will likely start to stumble with lots of stories or really long stories where this is exactly what lucene based search engines (solr, elastic, opensearch) are designed to do. Could an SQL solution solve your problem, yes, but it may be a bit on the slow side as your amount of stories and size grows.
I do like the sound of that.
I'm not too worried about performance, since, once everything is running, most of the operations will only be ran every few weeks or so. Don't want it slowing to a crawl for sure though.
The text search looks promising. I've had the idea of automating "likely tags" that look for keywords (sword = fantasy while spaceship = sci-fi). It's not perfect, but it could be useful to roughly categorize all the stories that are missing tags.
An alternative could be to use something like postgres with the pgvector extension to do semantic searches instead of just text-based searches. You can generate embeddings for the text content of the story, then do the same for "sci-fi" or something, and see if searching that way gets you most of the way there.
Generating embeddings locally might take some time though if you don't have hardware suitable for it.
Is there anything Postgres doesn’t do?
I would say run Doom, but I'm not confident in that. At the very least, Skyrim hasn't been rereleased on it yet.