I’m trying to understand the bot problem in the internet and finding more ways to defend myself. One thing that I can’t seem to understand is why most bots, scrapers and crawlers seem to have residential IPs.
- Is it that ISPs are being paid by tech-bros to assign them these IPs?
- Is it that residential devices have been hacked /contain malware that does this?
- Is it trivial for companies to assign themselves residential IPs?
- Paid volunteers are doing this for AI companies?
Or is there is some other reason for this?
Obviously this is a problem because one can rotate / cycle through residential IPs and if I aggressively block each offender in my logs permanently, then the next person assigned this IP who may be a legitimate user will be unable to access my site.
Ok, this I can answer personally as we did multiple cases of this happening (CSAM, bomb threats, etc) at work.
So, anonymity on the Internet is not actually a thing. Whether its an IP address or telecom switch or whatever, there is a path back to you even if only for either billing or connectivity purposes. So, for IP, we would receive a subpoena signed by a judge to hand over any and all information regarding the identify of the a given IP address (they include a long list of things whether applicable or not in the order so every potential base is covered). Once legal was able to review and handed it off to us, we take that and look at the DHCP logs to see that on a given date at a given time that the IP address was assigned as part of shelf A / slot B / port C. That shelf/slot/port combination is tied physically to an address/account. We provide the relevant logs and personal information of that user to law enforcement.
For bomb threats over the phone, telecom switches love to tell every other telecom switch who they are (again, connectivity purposes). So, when you make a call to a business/school doing that, their PBX is going to log to the millisecond when that call occurred and who the switch was. Again, subpoena and we pull the SIP logs. We can even provide the RTP/RTCP packets and reconstruct the phone call audio if the subpoena asks for that.
I can pay for internet in cash and the only details I gave them are fake and a random username. 4G internet isn’t even tied to the same location as I can move freely.
As I’m not bothered about doing cybercrimes I don’t bother doing that much about security though.