Ask Experienced Devs

1509 readers

1 users here now

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 3 years ago

MODERATORS

Ategon@programming.dev

OAI university endpoints (kbin.earth)

submitted 6 days ago by obbeel@kbin.earth to c/ask_experienced_devs@programming.dev

2 comments fedilink hide all child comments

I’m using university journal portal OAI endpoints for the third time in a row to fix a typo and handle DOI lookups.

Does anyone know how disrespectful or intrusive this is considered to be?

A few years ago, I scraped arXiv and they blocked my IP almost immediately; perhaps if I had used arXiv’s OAI interface, they wouldn't have blocked me.

But I really have no idea just how disrespectful or intrusive this is!

top 2 comments

sorted by: hot top controversial new old

[–] xianjam@programming.dev 2 points 4 days ago* (last edited 4 days ago) (1 children)

Basic etiquette, you'd probably want to ask the university portal maintainer.

Scraping is generally considered bad manners, and yes you would be immediately blocked if you ignore robots.txt. I assume by OAI you mean OpenAI? I'm unsure what kind of integration your university has, but like with many AI integrations, there is little oversight, which leads to... oversights.

[–] obbeel@kbin.earth 1 points 4 days ago

OAI are API endpoints for metadata access of articles. I imagine API access is more respectful in terms of bandwidth since it's organized specifically for access by the universities?