this post was submitted on 27 Sep 2024
787 points (98.4% liked)
Technology
60047 readers
2981 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Pro-tip for webscrapers: using AI to solve captchas is a massive waste of effort and resources. Aim to not be presented with a captcha in the first place.
I think thats much more difficult than it seems, because usually only residential IPs are the ones that don't get those. And if you start to use a residential proxy too much then that IP can also get flagged.
That's why companies like Oxylabs exist
That's when you rotate the proxy. By default most residential proxies will give you a new proxy for each request unless you specify.
Depends on the case, sometimes its unavoidable.