rbits

Ok but they do respect it, we know that https://searchengineland.com/microsoft-confirms-reddit-blocked-bing-search-444385

They even have a page telling you how to use it https://blogs.bing.com/webmaster/June-2008/Robots-Exclusion-Protocol-joining-together-to-pro/

Yeah but it stops bing and a bunch of AI scrapers that want to act like they're following the rules

I think spoiler tag support is the most important thing missing from Boost. @rmayayo@lemmy.world please fix this

I don't think they actually block malicious bots, the change they've made is just to the robots.txt, they don't have to do anything.

Yeah that's because lemmy is way more of an echo chamber so everyone agrees with each other more.

Damn, the US needs to get their shit together

Right, so similar to locks? Usually can be easily bypassed if you know how, but it at least filters out the people who aren't determined enough to put in the effort.

I mean, I would've said the same about the mobile apps, but here we are.

Yeah for tech stuff stack exchange is the way to go. But if I'm looking for info on non-tech stuff, there's not really a site that I know of that has a bunch of general user submitted q&a. There's quora, but that's absolutely horrible.

Also even with stack exchange, the rules are a bit more strict there, low effort posts are uncommon which can mean you can't find some stuff there. Although usually there's some other forum you can find your answer on.

Right, but having a centralised search index thingy is better than none at all. Maybe there could be something where it's a joint effort from admins from many of the biggest servers, idk if that would work.

I thought people like 3