Data Hoarder

221 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 2 years ago

MODERATORS

communick@selfhosted.forum

What's a reasonable download speed for ripping a database? (alien.top)

submitted 2 years ago by bomba1749@alien.top to c/datahoarder@selfhosted.forum

1 comments fedilink hide all child comments

Basically, I've found a ~100gb website that I need to have on hand in order to use it's data for a research project. However, the website is really old (started in the 90s and still looks like it), and I need to download it so I don't get fucked if it goes down at some point. Atm I've just had HTTrack running at it's 25kb/s default limit 24/7 but if I don't increase the bandwidth it would take like 2 months before it's all downloaded, which doesn't work for my timeframe. So, what's a reasonable bandwidth to put it at so I don't bother the admins or DOS the website but also don't have to wait absolute ages? Oh and I need all the images, videos, etc, so I can't rely on archive.org.

top 1 comments

sorted by: hot top controversial new old

[–] bobbarker4444@alien.top 1 points 2 years ago

Have you considered contacting the admins and seeing if they'd be able and willing to provide a dump for you?