I’m using CloudFlare to hide my home IP and to reduce traffic from clankers. However, I’m using the free tier, so how am I the product? What am I sacrificing? Is there another way to do the above without selling my digital soul?

  • mesa@piefed.socialEnglish
    4·
    1 day ago

    I have something similar with fail2ban + hidden buttons. If the requester goes and clicks on the hidden buttons on the main site, it gets into a rabbit hole. After 3 requests, it gets banned for a bit. Usually stops the worst offenders. OpenAI and some of the scrapers are the worst.

    Google/bing, I do actually see them hit robots.txt then jump off, which is what they should be going.

    • Admiral Patrick@dubvee.orgEnglish
      1·
      1 day ago

      Oooooh. That’s smart. I mostly host apps, but in theory, I should be able to dynamically modify the response body and tack on some HTML for a hidden button and do that.

      I used to disallow everything in robots.txt but the worst crawlers just ignored it. Now my robots.txt says all are welcome and every bot gets shunted to the tarpit 😈

      • mesa@piefed.socialEnglish
        1·
        1 day ago

        Nice! Thats another way to do it. 😀

        I know others use Arabis(?) I think thats what it called. The anime girl one that does a calc on top. Ive never had good luck with it. I think bot are using something to get around and it messes with my requests. Might also be my own fiddling.