this post was submitted on 03 Sep 2026
-5 points (14.3% liked)
Text-Based User Interfaces (TUI; CLI) 💻🖮
128 readers
5 users here now
Forum for advanced users who grok the power of text-based apps, the advantage of tmux/GNU screen, the keyboard and who often find the mouse a hinderance to a fast workflow. A text-based UI is also a decent escape from enshitified resources.
This forum broadly covers tools, hacks, and advocacy of text-based environments.
Slightly marginally kind of related:
- !sustainabletech@lemmy.sdf.org
- !permacomputing@lemmy.sdf.org
- !uiux@lemmy.sdf.org
- !mindful_tech@lemmy.sdf.org
- !keyboards@lemmy.sdf.org
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
A lot of the world runs on gentlemen's agreements and handshake deals. There's no real way to have a robots.txt for a public facing internet without starting an arms race with the server having rate limiting, metrics to check and see if someone's going alphabetically (for example), captchas on every page to verify it's a human, which the scraper will counter with a randomized scrape list, LLMs to pass captchas, etc. The server could institute some sort of login, but that defeats the purpose of an open internet. So robots.txt can't be enforced from a technological standpoint. One could glean information from who's ignoring it and obviously scraping, but that's about it.
There's no technological difference between these if the scraper wants. I was going to deep dive htrack because I've never heard of it before, but stopped when I saw some of it's options. It's straight up a website scraper. It not only has options to ignore robots.txt, but it also has a setting that apparently uses robots.txt to know where to scrape! Wrap a systemd timer around it and start working on your own archive.org.
Agreed with everything you said, I just wanted to modify this
It’s important to note that there’s no difference at all from the server’s side, other than an implication of how often scraping happens or how the data is being used.
Which is why the GUI restriction was added to robots.txt in the first place. From the server’s perspective, most command-line applications have historically been some kind of “robot” instead of something more interesting from the server owner’s perspective.
This is hilarious.
I thought about maybe checking server logs for what files a scraper might skip just for bandwidth or whatever, but I don't know much about that world.
I had no idea there was a GUI restriction since I stopped digging pretty quickly about it. Upon further reading it's apparently been used in several lawsuits successfully as a sort of "no tresspassing" sign with some court precedent to it. Neat.
^ THIS
Bingo.
Not on to advocate for internet open just for people privileged to have access in their homes.
Are you calling Firefox a scraper then?
Consider there is:
Why would you call “2” a scraper and “1” otherwise?
Nonsense. The author was sloppy in their phrasing. That is not what he means by “follow robots.txt”. By “follow”, he means “obey”, not visit. I was also initially fooled by the same ambiguous wording when I first saw it.
Real question, what exactly are you doing there? Why not access the site with a browser, CTRL+S, and move to the next page? You're getting an internet connection off the library, so it's either a wifi or through a computer of theirs.
I enter the library with a laptop, and list of tasks and URLs. Tasks, meaning I have to e.g. search for a PDF manual for a 2nd-hand appliance I either bought for pulled from a dumpster. Or research something. For URLs that I just need to save for later reading, I open them in FF (many tabs) and use the SingleFile extension to save them one by one. Of course that robs me of human time that I need for tasks that must be interactive. My time would be more wisely managed to have HTtrack fetching what I need in the background while I do interactive things. I sometimes stay until I get kicked out because the library is closing, in which case my needs were not all satisfied.
For Lemmy, I save posts in advance as text files and copy-paste the text into a Lemmy web client. This is also not a good use of my time but the only offline lemmy client is broken. But if there were a non-broken lemmy client for offline access, it would probably face the same discrimination by this reckless and obnoxious anti-bot movement.
Gotcha.
I'm sure there's a way to wire it up that way. It's bound to have a Turing complete language in it somewhere and by definition it has all the tools to copy down files from the internet. If I were going into the scraping business I'd personally start with a scraper and give it a Firefox user agent. In short, scraping is about intent as much as tooling, and there's no way to convey and enforce intent over HTTP. From a practical standpoint there's a spectrum of activities that approximately go from Browsing to Scraping to DDoS in that order that really only differ in terms of scale.
If I'm hosting content then if #2 is a scraper and #1 is a browser, and they don't cost me anything different then I don't really care (someone else who's done that kind of stuff can weigh in if I'm wrong). What a lot of sites seem to be reporting is that they have a small, steady hosting cost per month until WHOOM! an AI company scrapes their whole site and their hosting cost jumps that month, then the next month and the next as well because companies that would destroy copies of rare books to feed to their training don't care. Facebook got called out sort of recently for scraping some entire pornsite's content. It sounds like what you're describing is just timeshifted browsing. I doubt anyone notices unless you're really describing what you're trying to do incorrectly.
You'll often hear/read "follow" used by itself to imply "follow the rules," in this case "follow the rules specified by robots.txt."
From a technical standpoint an LLM recommends trying this code snippet to make httrack ignore the robots.txt file:
I haven't tried it personally.
Yes, you simply turn off images. Then Firefox is a scraper from the server standpoint. I use FF that way when operating under a bandwidth quota. Scraping a page is far less of a burden on the server than fetching the whole page (images, js, and all) for GUI rendering.
Turing completeness is wholly irrelevant. A fetch operation does not absolutely need a turing complete language. But you will struggle to render the content for a graphical presentation without a turing complete language.
Not “in that order”. Browsing graphically causes every image to be fetched. It’s much heavier than the HTML text that a scraper pulls. 10,000 gui users each fetching a page will strain your webserver faster than 10,000 html text pages being scraped.
Non-graphical users suffer from marginalization driven by the shitty attitude that “bots are bad and humans should alter their behavior and fetch images to look less like a bot as we don’t give a shit about oppressing bots.”
Timeshifted browsing is purpose of HTtrack. An AI operation doesn’t use HTtrack which is designed to organize the browsing session into interactive projects which launch a local GUI browser for offline viewing. An AI op has zero interest in that. A scraping operation uses curl, wget, beautiful soup/python, etc.
It’s this kind of commentary that distorts public perception.
Scraper bots themselves serve many users, often more than the original site could afford to host. A scraper bot that grabs just raw text builds a database and makes the information more useful to more people than the original site. E.g. instead of each traveler manually visiting the website of every single airline (a costly image-rich bloat high in CO₂ emissions), they can go to a site that has scraped Ryanair and others. The aggregated text info is more useful. Consumers then only have to visit /one/ airline website after seeing all the info side-by-side. This is much less strain on resources.
That’s what I was saying. Hence why the claim that HTtrack follows locations listed in
robots.txtis nonsense.I appreciate the tip. Indeed
-s0is what I worked out that I needed. You seem bent against AI so it’s a bit unexpected that you would be using it. I avoid AI because I generally boycott the corporations pushing the LLMs. But I’m not generally opposed to LLM tech. I have a local language translator and probably need to look into other LLMs that can run locally.I'll stop then. Have a nice day.