Does lemmy have any communities dedicated to archiving/hoarding data?

top 50 comments

sorted by: hot top controversial new old
[–] 210 points 10 months ago (11 children)

For wikipedia you'll want to use Kiwix. A full backup of wikipedia is only like 100GB, and I think that includes pictures too.

  • source
  • hideshow 13 child comments
  • [–] 50 points 10 months ago* (10 children)

    Last time I updated it was closer to 120GB but if you're not sweating 100 GB then an extra 20 isn't going to bother anyone these days.

    Also, thanks for reminding me that I need to check my dates and update.

    EDIT: you can also easily configure a SBC like a Raspberry Pi (or any of the clones) that will boot, set the Wi-Fi to access point mode, and serve kiwix as a website that anyone (on the local AP wifi network) can connect to and query... And it'll run off a USB battery pack. I have one kicking around the house somewhere

  • source
  • parent
  • hideshow 10 child comments
  • load more comments (10 replies)
  • load more comments (9 replies)
    [–] 114 points 10 months ago (2 children)

    The English Language Wikipedia probably wouldn't be hard, or Debian Stable.

    All of Debian's packages might be a tad more expensive, though.

  • source
  • hideshow 4 child comments
  • [–] 29 points 10 months ago

    It depends if you want the images or previous versions of wikipedia too. The current version is about 25Gb compressed, the dump with all versions is aparently multiple terabytes. They don't say how much media they have, but I'm guessing it's roughly "lots".

  • source
  • parent
  • [+] 89 points 10 months ago* (1 child)
  • [–] 15 points 10 months ago (1 child)

    "backups"? Pray tell, fine sir and or madam, what is that?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 54 points 10 months ago (4 children)

    How would one go about making an offline copy of the repos? Asking for a friend.

  • source
  • hideshow 5 child comments
  • [–] 45 points 10 months ago (1 child)
  • load more comments (3 replies)
    [–] 53 points 10 months ago (7 children)

    I stumbled across this sort of fascinating area of doomsday prepping a few weeks back.

    https://prepperpress.com/usb/

    A nice addition to that, don't just make it a USB, but a raspberry pi. So you'd have a reasonably low-powered computer you could easily take with you.

    Not suggesting this one as it seems a bit expensive to me, but https://www.prepperdisk.com/products/prepper-disk-premium-over-512gb-of-survival-content?view=sl-8978CA41

  • source
  • hideshow 8 child comments
  • load more comments (6 replies)
    [–] [S] 47 points 10 months ago (3 children)

    Curious about the mindset of the one (so far) person who has downvoted this post. What is there to dislike about archiving Linux and Wikipedia? πŸ€”

  • source
  • hideshow 5 child comments
  • [–] 82 points 10 months ago (1 child)

    They are probably using a phone app which allows you to swipe sideways to downvote and also using screen gestures to 'go back'. I've accidentally downvoted things this way.

  • source
  • parent
  • hideshow 2 child comments
  • load more comments (1 reply)
    [–] 45 points 10 months ago (4 children)

    I have been archiving Linux builds for the last 20 years so I could effectively install Linux on almost any hardware since 1998-ish.

    I have been archiving docker images to my locally hosted gitlab server for the past 3-5 years (not sure when I started tbh). I've got around 100gb of images ranging from core images like OS to full app images like Plex, ffmpeg, etc.

    I also have been archiving foss projects into my gitlab and have been using pipelines to ensure they remain up-to-date.

    the only thing I lack are packages from package managers like pip, bundler, npm, yum/dnf, apt. there's just so much to cache it's nigh impossible to get everything archived.

    I have even set up my own local CDN for JS imports on HTML. I use rewrite rules in nginx to redirect them to my local sources.

    my goal is to be as self-sustaining on local hosting as possible.

  • source
  • hideshow 5 child comments
  • load more comments (3 replies)
    [–] 44 points 10 months ago* (1 child)

    Neither are that bad honestly. I have jigdo scripts I run with every point release of Debian and have a copy of English Wikipedia on a Kiwix mirror I also host. Wikipedia is a tad over 100 GB. The source, arm64 and amd64 complete repos (DVD images) for Debian Trixie, including the network installer and a couple live boot images, are 353 GB.

    Kiwix has copies of a LOT of stuff, including Wikipedia on their website. You can view their zim files with a desktop application or host your own web version. Their website is: https://kiwix.org/

    If you want (or if Wikipedia is censored for you) you can also look at my mirror to see what a web hosted version looks like: https://kiwix.marcusadams.me/

    Note: I use Anubis to help block scrapers. You should have no issues as a human other than you may see a little anime girl for a second on first load, but every once and a while Brave has a disagreement with her and a page won't load correctly. I've only seen it in Brave, and only rarely, but I've seen it once or twice so thought I'd mention it.

  • source
  • hideshow 1 child comment
  • load more comments (1 reply)
    [–] 35 points 10 months ago (4 children)

    I also recommend downloading β€œFlashpoint archive” to have flash games and animations to stay entertained.

    There is a 4gb version and a 2.3TB version.

  • source
  • hideshow 4 child comments
  • load more comments (4 replies)
    [–] 34 points 10 months ago (14 children)

    FWIW :

    fabien@debian2080ti:/media/fabien/slowdisk$ ls -lhS offline_prep/
    total 341G
    -rw-r--r-- 1 fabien fabien 103G Jul  6  2024 wikipedia_en_all_maxi_2024-01.zim
    -rw-r--r-- 1 fabien fabien  81G Apr 22  2023 gutenberg_mul_all_2023-04.zim
    -rw-r--r-- 1 fabien fabien  75G Jul  7  2024 stackoverflow.com_en_all_2023-11.zim
    -rw-r--r-- 1 fabien fabien  74G Mar 10  2024 planet-240304.osm.pbf
    -rw-r--r-- 1 fabien fabien 3.8G Oct 18 06:55 debian-13.1.0-amd64-DVD-1.iso
    -rw-r--r-- 1 fabien fabien 2.6G May  7  2023 ifixit_en_all_2023-04.zim
    -rw-r--r-- 1 fabien fabien 1.6G May  7  2023 developer.mozilla.org_en_all_2023-02.zim
    -rw-r--r-- 1 fabien fabien 931M May  7  2023 diy.stackexchange.com_en_all_2023-03.zim
    -rw-r--r-- 1 fabien fabien 808M Jun  5  2023 wikivoyage_en_all_maxi_2023-05.zim
    -rw-r--r-- 1 fabien fabien 296M Apr 30  2023 raspberrypi.stackexchange.com_en_all_2022-11.zim
    -rw-r--r-- 1 fabien fabien 131M May  7  2023 rapsberry_pi_docs_2023-01.zim
    -rw-r--r-- 1 fabien fabien 100M May  7  2023 100r-off-the-grid_en_2022-06.zim
    -rw-r--r-- 1 fabien fabien  61M May  7  2023 quantumcomputing.stackexchange.com_en_all_2022-11.zim
    -rw-r--r-- 1 fabien fabien  45M May  7  2023 computergraphics.stackexchange.com_en_all_2022-11.zim
    -rw-r--r-- 1 fabien fabien  37M May  7  2023 wordnet_en_all_2023-04.zim
    -rw-r--r-- 1 fabien fabien  23M Jul 17  2023 kiwix-tools_linux-armv6-3.5.0-1.tar.gz
    -rw-r--r-- 1 fabien fabien  16M Oct  6 21:32 be-stib-gtfs.zip
    -rw-r--r-- 1 fabien fabien 3.8M Oct  6 21:32 be-sncb-gtfs.zip
    -rw-r--r-- 1 fabien fabien 2.3M May  7  2023 termux_en_all_maxi_2022-12.zim
    -rw-r--r-- 1 fabien fabien 1.9M May  7  2023 kiwix-firefox_3.8.0.xpi
    
    

    but if you want the easier version just get Kiwix on whatever device in front of you right now (yes, even mobile phone assuming you have the space) then get whatever content you need.

    If need a bit of help I recorded TechSovereignty at home, episode 11 - Offline Wikipedia, Kiwix and checksums with a friend just 3 weeks ago.

    I also wrote randomly update https://fabien.benetou.fr/Content/Vademecum and coded https://git.benetou.fr/utopiah/offline-octopus but tbh KDE-Connect is much better now.

    The point though is having such a repository takes minutes. If you don't have the space, buy a 512Go microSD for 50EUR then put that on, stuff it in a drawer then move on. If you want to every 3 months or whenever you feel like it, updated it.

    TL;DR: takes longer to write such a meme than actually do it.

  • source
  • hideshow 14 child comments
  • load more comments (14 replies)
    [+] 33 points 10 months ago (15 children)
    load more comments (15 replies)
    [–] 31 points 10 months ago (10 children)

    Yeah not gonna lie, i think i heard someone in a youtube video a while back talk about how the entirety of wikipedia takes up like 200 gigs or something like that, and it got me seriously considering to actually make that offline backup. Shit is scary when countries like the uk are basically blocking you from having easy access to knowledge.

  • source
  • hideshow 10 child comments
  • load more comments (10 replies)
    [–] 26 points 10 months ago (2 children)

    If you do this please share your IP so I can use your backup too

  • source
  • hideshow 2 child comments
  • load more comments (2 replies)
    [–] 26 points 10 months ago (2 children)

    Is there a context to this or just random thought?

  • source
  • hideshow 4 child comments
  • [–] 23 points 10 months ago

    I can answer one part of your question. Yes, it's not as big as you think it is.

  • source
  • [–] 22 points 10 months ago (7 children)

    Sorry, I'm out of the loop. Is there something particular that triggered this that I missed?

  • source
  • hideshow 9 child comments
  • [–] 25 points 10 months ago* (last edited 10 months ago) (2 children)

    The broad censorship of government data in the US, combined with the recent political attacks on Wikipedia caused me to download the whole English Wikipedia earlier this year. Guessing OP is similar

    Not sure why they'd download Debian with all packages though

    Edit: I should mention it's less about a potential loss of Wikipedia as it is a personal source of truth on politically sensitive topics that get censored, or turned to propaganda by bots

    For example the Wounded Knee Massacre. Pete Hegseth has recently been calling it the, "Battle of Wounded Knee". I wouldn't be surprised if the current administration went to war with Wikipedia and forced them to 1) Change articles they disagree with, and 2) Hide those changes from history

  • source
  • parent
  • hideshow 2 child comments
  • load more comments (2 replies)
  • load more comments (5 replies)
    [–] 20 points 10 months ago (1 child)

    Wait, isn't there an offline copy of a part of Wikipedia? The article Just by yourself a nice printer with enough ink and do it yourself ;)

  • source
  • hideshow 1 child comment
  • load more comments (1 reply)
    [–] 20 points 10 months ago (5 children)

    Last year I bought a hard copy of my favorite webcomic in case the website goes down.

  • source
  • hideshow 5 child comments
  • load more comments (5 replies)
    [–] 18 points 10 months ago (3 children)

    we need all repos to be stored offline, and documentations to troubleshoot.

    the 1st i have no idea how much space we will need. Most linux packages are prerry light, no? But there is A LOT of them...

    the 2nd is easy. Heard someone say the entire of wikipedia is 200GB, should be doable. Dont forget the technical wikis too: Debian, Gentoo, Arch.

  • source
  • hideshow 3 child comments
  • load more comments (3 replies)
    [–] 17 points 10 months ago (1 child)

    Did I miss something? Whats happening to debian stable?

  • source
  • hideshow 1 child comment
  • load more comments (1 reply)
    [–] 17 points 10 months ago

    Years ago I bought a physical encyclopedia. I remember having one as a kid and using it for school reports. Also just looking through it can be cool. Learning about something you never knew existed is just a unique experience and doing it through a physical book just deepens the whole experience.

    I also learned the practice of printing a physical encyclopedia is going out of fashion. I think there is only one company the still prints a yearly encyclopedia and it's not Encyclopedia Britannica of all things. Might have change since I bought my copy but go give some physical media some love if you can.

  • source
  • [–] 17 points 10 months ago

    Get out of my mind.

  • source
  • [+] 16 points 10 months ago* (last edited 1 month ago) (5 children)
  • load more comments (3 replies)
    [–] 16 points 10 months ago (3 children)

    I would add in some rom collections and book repositories as well. The whole library of Nintendo games is under a gig and would go a long way for entertaining people.

  • source
  • hideshow 3 child comments
  • load more comments (3 replies)
    [–] 15 points 10 months ago (5 children)

    I keep a wiki copy as well as Reddit pre-fuckuspez. A Debian archive copy sounds like a good idea.

  • source
  • hideshow 5 child comments
  • load more comments (5 replies)
    load more comments
    view more: next β€Ί