TechTakes

It's a constant cat and mouse atm. Every week or so, we get another flood of scraping bots, which force us to triangulate which fucking DC IP range we need to start blocking now. If they ever start using residential proxies, we're fucked.

jwz gave the game away, so i'll reveal:

the One Weird Trick for this week is that the bots pretend to be an old version of Chrome. So you can block on useragent

so I blocked old Chrome from hitting the expensive mediawiki call on rationalwiki and took our load average from 35 (unusable) to 0.8 (schweeet)

caution! this also blocks the archive sites, which pretend to be old chrome. I refined it to only block the expensive query on mediawiki, vary as appropriate.

nginx code:

        # block some bot UAs for complex requests
        # nginx doesn't do nested if, so we set a test variable
        # if $BOT is both Complex and Old, block as bot
        set $BOT "";
        if ($uri ~* (/w/index.php)) {
            set $BOT "C"; }

            if ($http_user_agent ~* (Chrome/[2-9])) {
                set $BOT "${BOT}O";}
            if ($http_user_agent ~* (Chrome/1[012])) {
                set $BOT "${BOT}O";}
            if ($http_user_agent ~* (Firefox/3)) {
                set $BOT "${BOT}O";}
            if ($http_user_agent ~* (MSIE)) {
                set $BOT "${BOT}O";}

            if ($BOT = "CO") {
                return 503;}

you always return "503" not "403", because 403 says "fuck off" but the scrapers are used to seeing 503 from servers they've flattened.

I give this trick at least another week.

In my experience with bots, a portion of them obey robots.txt, but it's tricky to find the user agent string that some bots react to.

So I recommend having a robots.txt that not only target specific bots, but also tell all bots to avoid specific paths/queries.

Example for dokuwiki

User-agent: *
Noindex: /lib/
Disallow: /_export/
Disallow: /user/
Disallow: /*?do=
Disallow: /*&do=
Disallow: /*?rev=
Disallow: /*&rev=

Re the blocking of fake useragents, what people could try is see if there are things older useagents do (or do wrong) which these do not. I heard of some companies doing that. (Long ago I also heard of somebody using that to catch mmo bots in a specific game. There was a packet that if the server send it to a legit client, the client crashed, a bot did not). I'd assume the specifics are treated as secret just because you don't want the scrapers to find out.