Programming

19290 readers

148 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]

founded 2 years ago

MODERATORS

[email protected]

YAML, SQL, or something else? Looking for recommendations for making a database of stories. (lemm.ee)

submitted 5 days ago by [email protected] to c/[email protected]

39 comments fedilink hide all child comments

Short version of the situation is that I have an old site I frequent for user written stories. The site is ancient (think early 2000's), and has terrible tools for sorting and searching the stories. Half of the time, stories disappear from author profiles. Thousands of stories and you can only sort by top, new, and 30-day top.

I'm in the process of programming a scraper tool so I can archive the stories and give myself a library to better find forgotten stories on the site. I'll be storing tags, dates, authors, etc, as well as the full body of the text.

Concerning the data, there are a few thousand stories- ascii only, and various data points for each story with the body of many stores reaching several pages long.

Currently, I'm using Python to compile the data and would like to know what storage solution is ideal for my situation. I have a little familiarity with SQL, json, and yaml, but not enough to know what might be best. I am also open to any other solutions that work well with Python.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 3 points 5 days ago (2 children)

What's your reasoning for that?

At this point, I think I'll only use yaml as the scraper output and then create a database tool to convert that into whatever data format I end up using.

[–] [email protected] 2 points 4 days ago* (last edited 4 days ago) (1 children)

https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell

JSON is a much simpler (and consequently safer) format. It's also more universally supported.

YAML (or TOML) is decent for a manually read and written configuration. But for a scraper output for storage and follow-up workflows being through code parsing anyway, I would go for JSON.

[–] [email protected] 1 points 4 days ago

That's an interesting read. I'll definitely give json a try too.

[–] [email protected] 2 points 5 days ago (1 children)

Very wise idea. And if you want to up your game, can validate the yaml against a schema.

Check out strictyaml

The author is ahead of his time. Uses validated yaml to build stories and weave those into web sites.

Unfortunately the author also does the same with strictyaml tests. Can get frustrating cause the tests are too simple.

[–] [email protected] 1 points 5 days ago

Gonna be honest, I'll need to research a bit more what validating against a schema is, but I get the general idea, and I like it.

For initial testing and prototypes, I probably won't worry about validation, but once I get to the point of refining the system, validation like that would be a good idea.