Text-Based User Interfaces (TUI; CLI) 💻🖮

128 readers

5 users here now

Forum for advanced users who grok the power of text-based apps, the advantage of tmux/GNU screen, the keyboard and who often find the mouse a hinderance to a fast workflow. A text-based UI is also a decent escape from enshitified resources.

This forum broadly covers tools, hacks, and advocacy of text-based environments.

Slightly marginally kind of related:

founded 2 years ago

MODERATORS

evenwicht@lemmy.sdf.org

-5

Mildly infuriating: GUI browsers can ignore robots.txt, but CLI browsers cannot. WTF, can someone rationalize why GUI users should have higher privilege? (lemmy.sdf.org)

submitted 3 days ago* (last edited 2 days ago) by evenwicht@lemmy.sdf.org to c/text_ui@lemmy.sdf.org

14 comments fedilink hide all child comments

cross-posted from: https://lemmy.sdf.org/post/58474758

Someone brought the HTtrack app to my attention in this thread. Superficially it’s a great concept. But it did not work for me.

The author of HTtrack says they respect the robots.txt files. I’m not sure if this is my problem. But what an absurd line to draw. Why should a realtime GUI user have more privilege to view a website than a CLI user?

If anything, it should be the other way around.

People who lack the priviledge of having Internet at home face this nasty discrimination of being treated like a bot after they make the extra effort of commuting to a public library just to fetch a website for offline viewing later. This 2nd-classing of a demographic who is already marginalized is quite despicable.

People with Internet at home can schedule HTtrack to run at an off-peak time with a narrow bandwidth put less burden on the server than the realtime GUI users who obviously hit the site mostly during peak times.

(update) Some people are on measured rate Internet connections that give tiny daytime quotas and generous late night quotas. Tools like HTtrack are needed to manage this. Which ultimately benefits the more privileged Internet users who have no constraints.

Update

There is a poorly worded -s0 option to ignore robots.txt. Fooled some people into thinking the tool uses robots.txt to direct the fetches.

you are viewing a single comment's thread
view the rest of the comments

[–] calliope@retrolemmy.com 5 points 3 days ago* (last edited 3 days ago) (2 children)

Agreed with everything you said, I just wanted to modify this

It’s not scraping. It’s fetching.

There's no technological difference between these if the scraper wants.

It’s important to note that there’s no difference at all from the server’s side, other than an implication of how often scraping happens or how the data is being used.

Which is why the GUI restriction was added to robots.txt in the first place. From the server’s perspective, most command-line applications have historically been some kind of “robot” instead of something more interesting from the server owner’s perspective.

it not only has options to ignore robots.txt

This is hilarious.

[–] historicaldocuments@lemmy.world 2 points 2 days ago

It’s important to note that there’s no difference at all from the server’s side, other than an implication of how often scraping happens or how the data is being used.

I thought about maybe checking server logs for what files a scraper might skip just for bandwidth or whatever, but I don't know much about that world.

Which is why the GUI restriction was added to robots.txt in the first place.

I had no idea there was a GUI restriction since I stopped digging pretty quickly about it. Upon further reading it's apparently been used in several lawsuits successfully as a sort of "no tresspassing" sign with some court precedent to it. Neat.

[–] evenwicht@lemmy.sdf.org 1 points 2 days ago

It’s important to note that there’s no difference at all from the server’s side

^ THIS