this post was submitted on 29 Apr 2025
558 points (97.4% liked)
Technology
69600 readers
3279 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I believe he's returning a gzip HTTP response stream, not just a file payload that the requester then downloads and decompresses.
Bzip isn't used in HTTP compression.
For scrapers that not just implementing HTTP, but are trying to extract zip files, you can possibly drive them insane with zip quines: https://github.com/ruvmello/zip-quine-generator or otherwise compressed files that contain themselves at some level of nesting, possibly with other data so that they recursively expand to an unbounded ("infinite") size.
Brotli is an option, and it's comparable to Bzip. Brotli works in most browsers, so hopefully these bots would support it.
I just tested it, and a 10G file full of zeroes is only 8.3K compressed. That's pretty good, though a little bigger than BZip.