Admiral Patrick

I’m surprisingly level-headed for being a walking knot of anxiety.

Ask me anything.

Special skills include: Knowing all the “na na na nah nah nah na” parts of the Three’s Company theme.

I also develop Tesseract UI for Lemmy/Sublinks

Avatar by @SatyrSack@feddit.org

  • 23 Posts
  • 251 Comments
Joined 3 years ago
cake
Cake day: June 6th, 2023

help-circle
  • So, I set this up recently and agree with all of your points about the actual integration being glossed over.

    I already had bot detection setup in my Nginx config, so adding Nepenthes was just changing the behavior of that. Previously, I had just returned either 404 or 444 to those requests but now it redirects them to Nepenthes.

    Rather than trying to do rewrites and pretend the Nepenthes content is under my app’s URL namespace, I just do a redirect which the bot crawlers tend to follow just fine.

    There’s several parts to this to keep my config sane. Each of those are in include files.

    • An include file that looks at the user agent, compares it to a list of bot UA regexes, and sets a variable to either 0 or 1. By itself, that include file doesn’t do anything more than set that variable. This allows me to have it as a global config without having it apply to every virtual host.

    • An include file that performs the action if a variable is set to true. This has to be included in the server portion of each virtual host where I want the bot traffic to go to Nepenthes. If this isn’t included in a virtual host’s server block, then bot traffic is allowed.

    • A virtual host where the Nepenthes content is presented. I run a subdomain (content.mydomain.xyz). You could also do this as a path off of your protected domain, but this works for me and keeps my already complex config from getting any worse. Plus, it was easier to integrate into my existing bot config. Had I not already had that, I would have run it off of a path (and may go back and do that when I have time to mess with it again).

    The map-bot-user-agents.conf is included in the http section of Nginx and applies to all virtual hosts. You can either include this in the main nginx.conf or at the top (above the server section) in your individual virtual host config file(s).

    The deny-disallowed.conf is included individually in each virtual hosts’s server section. Even though the bot detection is global, if the virtual host’s server section does not include the action file, then nothing is done.

    Files

    map-bot-user-agents.conf

    Note that I’m treating Google’s crawler the same as an AI bot because…well, it is. They’re abusing their search position by double-dipping on the crawler so you can’t opt out of being crawled for AI training without also preventing it from crawling you for search engine indexing. Depending on your needs, you may need to comment that out. I’ve also commented out the Python requests user agent. And forgive the mess at the bottom of the file. I inherited the seed list of user agents and haven’t cleaned up that massive regex one-liner.

    # Map bot user agents
    ## Sets the $ua_disallowed variable to 0 or 1 depending on the user agent. Non-bot UAs are 0, bots are 1
    
    map $http_user_agent $ua_disallowed {
        default 		0;
        "~PerplexityBot"	1;
        "~PetalBot"		1;
        "~applebot"		1;
        "~compatible; zot"	1;
        "~Meta"		1;
        "~SurdotlyBot"	1;
        "~zgrab"		1;
        "~OAI-SearchBot"	1;
        "~Protopage"	1;
        "~Google-Test"	1;
        "~BacklinksExtendedBot" 1;
        "~microsoft-for-startups" 1;
        "~CCBot"		1;
        "~ClaudeBot"	1;
        "~VelenPublicWebCrawler"	1;
        "~WellKnownBot"	1;
        #"~python-requests"	1;
        "~bitdiscovery"	1;
        "~bingbot"		1;
        "~SemrushBot" 	1;
        "~Bytespider" 	1;
        "~AhrefsBot" 	1;
        "~AwarioBot"	1;
    #    "~Poduptime" 	1;
        "~GPTBot" 		1;
        "~DotBot"	 	1;
        "~ImagesiftBot"	1;
        "~Amazonbot"	1;
        "~GuzzleHttp" 	1;
        "~DataForSeoBot" 	1;
        "~StractBot"	1;
        "~Googlebot"	1;
        "~Barkrowler"	1;
        "~SeznamBot"	1;
        "~FriendlyCrawler"	1;
        "~facebookexternalhit" 1;
        "~*(?i)(80legs|360Spider|Aboundex|Abonti|Acunetix|^AIBOT|^Alexibot|Alligator|AllSubmitter|Apexoo|^asterias|^attach|^BackDoorBot|^BackStreet|^BackWeb|Badass|Bandit|Baid|Baiduspider|^BatchFTP|^Bigfoot|^Black.Hole|^BlackWidow|BlackWidow|^BlowFish|Blow|^BotALot|Buddy|^BuiltBotTough|
    ^Bullseye|^BunnySlippers|BBBike|^Cegbfeieh|^CheeseBot|^CherryPicker|^ChinaClaw|^Cogentbot|CPython|Collector|cognitiveseo|Copier|^CopyRightCheck|^cosmos|^Crescent|CSHttp|^Custo|^Demon|^Devil|^DISCo|^DIIbot|discobot|^DittoSpyder|Download.Demon|Download.Devil|Download.Wonder|^dragonfl
    y|^Drip|^eCatch|^EasyDL|^ebingbong|^EirGrabber|^EmailCollector|^EmailSiphon|^EmailWolf|^EroCrawler|^Exabot|^Express|Extractor|^EyeNetIE|FHscan|^FHscan|^flunky|^Foobot|^FrontPage|GalaxyBot|^gotit|Grabber|^GrabNet|^Grafula|^Harvest|^HEADMasterSEO|^hloader|^HMView|^HTTrack|httrack|HTT
    rack|htmlparser|^humanlinks|^IlseBot|Image.Stripper|Image.Sucker|imagefetch|^InfoNaviRobot|^InfoTekies|^Intelliseek|^InterGET|^Iria|^Jakarta|^JennyBot|^JetCar|JikeSpider|^JOC|^JustView|^Jyxobot|^Kenjin.Spider|^Keyword.Density|libwww|^larbin|LeechFTP|LeechGet|^LexiBot|^lftp|^libWeb|
    ^likse|^LinkextractorPro|^LinkScan|^LNSpiderguy|^LinkWalker|msnbot|MSIECrawler|MJ12bot|MegaIndex|^Magnet|^Mag-Net|^MarkWatch|Mass.Downloader|masscan|^Mata.Hari|^Memo|^MIIxpc|^NAMEPROTECT|^Navroad|^NearSite|^NetAnts|^Netcraft|^NetMechanic|^NetSpider|^NetZIP|^NextGenSearchBot|^NICErs
    PRO|^niki-bot|^NimbleCrawler|^Nimbostratus-Bot|^Ninja|^Nmap|nmap|^NPbot|Offline.Explorer|Offline.Navigator|OpenLinkProfiler|^Octopus|^Openfind|^OutfoxBot|Pixray|probethenet|proximic|^PageGrabber|^pavuk|^pcBrowser|^Pockey|^ProPowerBot|^ProWebWalker|^psbot|^Pump|python-requests\/|^Qu
    eryN.Metasearch|^RealDownload|Reaper|^Reaper|^Ripper|Ripper|Recorder|^ReGet|^RepoMonkey|^RMA|scanbot|SEOkicks-Robot|seoscanners|^Stripper|^Sucker|Siphon|Siteimprove|^SiteSnagger|SiteSucker|^SlySearch|^SmartDownload|^Snake|^Snapbot|^Snoopy|Sosospider|^sogou|spbot|^SpaceBison|^spanne
    r|^SpankBot|Spinn4r|^Sqworm|Sqworm|Stripper|Sucker|^SuperBot|SuperHTTP|^SuperHTTP|^Surfbot|^suzuran|^Szukacz|^tAkeOut|^Teleport|^Telesoft|^TurnitinBot|^The.Intraformant|^TheNomad|^TightTwatBot|^Titan|^True_Robot|^turingos|^TurnitinBot|^URLy.Warning|^Vacuum|^VCI|VidibleScraper|^Void
    EYE|^WebAuto|^WebBandit|^WebCopier|^WebEnhancer|^WebFetch|^Web.Image.Collector|^WebLeacher|^WebmasterWorldForumBot|WebPix|^WebReaper|^WebSauger|Website.eXtractor|^Webster|WebShag|^WebStripper|WebSucker|^WebWhacker|^WebZIP|Whack|Whacker|^Widow|Widow|WinHTTrack|^WISENutbot|WWWOFFLE|^
    WWWOFFLE|^WWW-Collector-E|^Xaldon|^Xenu|^Zade|^Zeus|ZmEu|^Zyborg|SemrushBot|^WebFuck|^MJ12bot|^majestic12|^WallpapersHD)" 1;
    
    }
    
    
    deny-disallowed.conf
    # Deny disallowed user agents
    if ($ua_disallowed) { 
        # This redirects them to the Nepenthes domain. So far, pretty much all the bot crawlers have been happy to accept the redirect and crawl the tarpit continuously 
    	return 301 https://content.mydomain.xyz/;
    }
    









  • Because ban evasion is an everyday occurrence here. Someone gets banned for being a jackass, and 9 times out of 10, they come right back with a brand new account and come in hot with inflammatory posts and a chip on their shoulder.

    Or trolls just troll - spin up new accounts over and over and throw out rage bait or whatever their shtick is just to be a dick.

    Combined with the fact that the fediverse’s growth has been a bit stagnant lately, when a bunch of new accounts suddenly pop up, they’re rightly met with suspicion until they’ve established themselves as being here with good intentions.

    Not saying new accounts should be ostracized, but given the context of the current state of the fediverse, being suspicious of them is warranted.



  • Not sure about Android, but on iOS, when one scans a QR code it shows the web address on the screen that the user then taps on. For the average user, I doubt that they are going to question what the URL is before following through to the website.

    Android does the same. The problem is most of those QR codes are encoded short links which tells you nothing about where they’re taking you.

    https://short.link/au1034gha could take you to a PDF on the restaurant’s Wordpress site or it could take you to malware or somewhere else you really don’t want to go.

    In that case, I blame the people generating the codes for using URL shorteners. My org uses them in flyers for the public, and I always have to chastise them and re-create the QR codes because they run the URL to our website through bit [dot] ly. 😡



  • I’ve had pretty good experience with Nextcloud’s instant upload. The only time I’ve had it shit the bed was ages ago when it would occasionally get stuck on a conflict, but that hasn’t happened in a long time. Pretty much all of my image folders (camera/DCIM, Screenshots, Downloads) get synced. The only annoying thing was when apps would suddenly change where they download to and I’d have to reconfigure yet another sync folder, but I can’t really fault NC for that.

    Mine is set to upload and keep a local copy and only do a one way sync (phone to NC). Not sure if that causes less issues than a 2 way sync or deleting the local copy after upload?


  • I tried a true dumb phone but was breaking out my laptop too much for everyday tasks (dumb phones these days can do hotspot).

    The flip phone I have runs Android 11, so I have the bare minimum necessary chat apps, email, GPS maps, and such. The main draw is that those work well enough, but anything more than that is possible but very frustrating. That’s kind of what the Minimal is about: e.g. yeah, you can watch YouTube videos on it, but you won’t want to.

    Then, when the detox period is up and you’re fully off the addiction, you can get the standard phone back.

    That’s kinda what I did. I used my flip like a true dumb phone for 30 days as a challenge and then un-dumbed it a little bit back to where only my basic needs were met and nothing more. I assumed I’d have rushed back to my old smartphone, but after breaking a bunch of habits, I found I didn’t really want to. Plus, I really missed T9 texting as weird as that sounds lol.


  • Yeah the marketing for it was lost on me. I already digitally detoxed last year when I switched to the flip phone I’m currently using, so I ignored the sales pitch and just looked at it from the cool hardware perspective and mostly reasonable price.

    Credit where it’s due, though: I tried unsuccessfully to just uninstall the time sink apps from my regular smartphone and always ended up just reinstalling them. It took using a device that couldn’t feasibly run those (plus a weaning-off period) for me to fully let go. Seems like that is what the marketing is trying to target.





  • Mostly the e-ink display and the QWERTY keyboard.

    By its nature, it’s not great for doom scrolling, TikTok, or videos (the major time sinks with most phones). Those aren’t my use-cases anyway; I’m more of a reader than a watcher so I figured this would be like a supercharged version of my Kobo (which I love) that has a physical keyboard (which I have missed terribly in smartphones).

    Edit: Plus, its 4:3 e-ink display and keyboard just scream “install Termux on me!” Also planned to use it as a nice portable SSH terminal like I used to have back when smartphones had slide-out keyboards.

    I’m rocking an aging Cat S22 Flip right now and have been trying to figure out what to replace it with. The Minimal Phone seemed like exactly what I wanted. Sad that it arrived DOA and am probably not going to bother replacing it; waiting weeks for something that arrives DOA is a hard thing to recover from.