A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
buying tech products by brand allows brands to sell you a shitty product at premium price points. companies will always shit out bad eggs at times, and its your job to know which product line are bad and not let brand loyalty bypass that.
at the bare minimum, if you are buying by brand, buy it solely based on customer support, as some companies are significantly better at that than others, which is an objective trait.
I learned a long time ago that the manufacturer doesn't matter much on the long run. They all have a bad model occasionally. I have 500GB Seagate drives that still work, and some 1TB drives that died within a year. I've had good luck with recent WD Red 4TB drives, but my 2TB Green drives have all died on me. I had a some of the Hitachi Deskstar drives that worked perfectly for years when no one would touch them because of a bad production run. I currently have a Toshiba 8TB that I had never heard of before, but seems to be rock solid for the last year.
Pick a size that you want, look at what's available, and research the reasonably priced ones to see if anyone is complaining about them. Review sites can be useful, but raw complaints in user forums will give you a better idea of which ones to avoid.
Can confirm the statistics: I recently consolidated about a dozen old hard disks of various ages and quite a few of them had a couple of back blocks and 2 actually failed. One disk was especially noteworthy in that it was still fast, error-less and without complaints. That one was a Seagate ST3000DM001. A model so notoriously bad that it's got its own Wikipedia entry: https://en.wikipedia.org/wiki/ST3000DM001
Other "better" HDDs were entirely unresponsive.
Statistics only really matter if you have many, many samples. Most people (even enthusiasts with a homelab) won't be buying hundreds of HDDs in their life.
With spinning disks, I preferred Seagate over Western Digital. And then move to HGST.
Back in those days, Western Digital had the best warranty. And I used it on every Western Digital. But that was still several days without a drive, and I still needed a backup drive.
So it was better to buy two drives at 1.3 x the price of one Western Digital. And then I realized that none of the Seagate or HGST drives failed on me.
For SATA SSDs, I just get a 1TB to maximize the cache and wear leveling, and pick a brand where the name can be pronounced.
For NVME, for a work performance drive, I pick a 2TB drive with the best write cache and sustainable write speed at second tier pricing.
For a general NVME drive, I pick at least a 1 TB from anyone who has been around long enough to have reviews written about them.
Why does 1TB help with the wear leveling?
An analogy is writing everything on one piece of paper with a pencil. When you need to change or remove something, you cross it out, instead of erasing, and write the new data to a clean part of the paper. When there are no more clean areas, you use the eraser to erase a crossed off section.
The larger the paper, the less frequent you come back to the same area again with the eraser.
Using an eraser on paper slowly degrades the paper until that section tears and never gets used again.
In general and simplifying, my understanding is:
There is the area where data is written, and there is the File Allocation Table that keeps track of where files are placed.
When part of a file needs to be overwritten (either because it inserted or there is new data) the data is really written to a new area and the old data is left as is. The File Allocation Table is updated to point to the new area.
Eventually, as the disk gets used, that new area eventually comes back to a space that was previously written to, but is not being used. And that data gets physically overwritten.
Each time a spot is physically overwritten, it very very slightly degrades.
With a larger disk, it takes longer to come back to a spot that has already been written to.
Oversimplifying, previously written data that is no longer part of a file is effectively lost, in the way that shredding a paper effectively loses whatever is written, and in a more secure way than as happens in a spinning disk.
Afaik, the wear and tear on SSDs these days is handled under the hood by the firmware.
Concepts like Files and FATs and Copy-on-Write are format-specific. I believe that even if a filesystem were to deliberately write to the same location repeatedly to intentionally degrade an SSD, the firmware will intelligently shift its block mapping around under the hood so as to spread out the wear. If the SSD detects a block is producing errors (bad parity bits), it will mark it as bad and map in a new block. To the filesystem, there's still perfectly good storage at that address, albeit with a potential one-off read error.
The larger sizes SSD just gives the firmware more extra blocks to pull from.
Does that mean that manually attempting to overprovision SSDs isn’t necessary for maximising endurance? Eg. partition a 1TB SSD as 500GB.
That would be called under-provisioning.
I haven't read anything about how an SSD deals with partitions, so I don't know for sure.
Since the controller intercepts the calls for specific locations, I'm inclined to believe that the controller does not care about the concept of partitions and does not segregate any chips, thus it would spread all writes across all of the chips.
Isn’t it overprovisioning because you’re artificially limiting the usable capacity of a volume?
https://www.techtarget.com/searchstorage/definition/overprovisioning-SSD-overprovisioning
As the other person said, I don't think the SSD knows about partitions or makes any assumptions based on partitioning, it just knows if you've written data to a certain location, and it could be smart enough to know how often you're writing data to that location. So if you keep writing data to a single location, it could decide to logically remap that location in logical memory to different physical memory so that you don't wear it out.
I say "could" because it really depends on the vendor. This is where one brand could be smart and spend the time writing smart software to extend the life of their drive, while another could cheap out and skip straight to selling you a drive that will die sooner.
It's also worth noting that drives have an unreported space of "spare sectors" that it can use if it detects one has gone bad. I don't know if you can see the total remaining spare sectors, but it typically scales with the size of a drive. You can at least see how many bad sectors have been reallocated using S.M.A.R.T.
I thought you meant 1 TB as a sort of peak performer (better than 2+ TB) in this area. From the description, it's more like 1 TB is kinda the minimum durability you want with a drive, but larger drives are better?
From the drives I have seen, usually there are 3 write-cache sizes.
Usually the smallest write-cache is for drives 128GB or smaller. Sometimes the 256GB is also here.
Usually the middle size write-cache is for 512GB and sometimes 256GB drives.
Usually the largest write-cache is only in 1TB and bigger drives.
Performance-wise for writes, you want the biggest write cache, so you want at least a 1TB drive.
For the best wear leveling, you want the drive as big as you can afford, while also looking at the makeup of the memory chips. In order of longest lasting listed first: Single Level, Multi Level, Triple Level, Quad Level.
This is great, thank you! My next drive is going to be fast and durable.
Yup, knock on wood, I've had lots of Seagate drives over the decades and I've never had any of them go bad. I've had two WD drives and they both failed
Assume your hard drives will fail. Any time I get a new NAS drive, I do a burn-in test (using a simple badblocks run, can take a few days depending on the size of the drive, but you can run multiple drives in parallel) to get them past the first ledge of the bathtub curve, and then I put them in a RaidZ2 pool and assume it will fail one day.
Therefore, it's not about buying the best drives so they never fail, because they will fail. It's about buying the most cost effective drive for your purpose (price vs avg lifespan vs size). For this part, definitely refer to the Backblaze report someone else linked.
When buying disks do some research for the exact model to ensure they are not SMR drives if you plan on using them in RAID. Some manufacturers will not tell you if they are SMR drives and this can do anything from tank write performance to make the RAID reject the drive entirely.
2 of my 6 disks are failing thanks to WD's EFAX line
Bastards
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
| Fewer Letters | More Letters |
|---|---|
| NAS | Network-Attached Storage |
| RAID | Redundant Array of Independent Disks for mass storage |
| SATA | Serial AT Attachment interface for mass storage |
| SSD | Solid State Drive mass storage |
| ZFS | Solaris/Linux filesystem focusing on data integrity |
5 acronyms in this thread; the most compressed thread commented on today has 6 acronyms.
[Thread #555 for this sub, first seen 29th Feb 2024, 00:35] [FAQ] [Full list] [Contact] [Source code]
HGST personally, because my failure count over time for those drives has been in the single digits through ~60 drives in around 15 years, though every manufacturer is going to have missteps or failures. I can say I've had bad experiences with Toshiba, but I'm sure you can find someone who swears by them also. Ultimately my anecdotal evidence in either direction is an unreliable crystal ball you should take with a grain of salt.
The suggestion to check Backblaze reports is great, but I'd also recommend to vary your manufacturers if you're able and instead build your storage solution with the assumption that drives are "wear units" and will fail. If you have some redundancy built in where you're able to tolerate the failure of one (or ideally multiple) drive failures without losing data, then even though the question still matters, it matters a bit less.
I swear allegiance to the only one true storage vendor, Micropolis. The Micropolis 1323A being the embodiment of perfection in storage basked in the glow of the only holy storage interconnect, MFM.
I wait patiently for the return of Micropolis so that I may serve as their humble servant.
For hard drives Toshiba, though SeaGate would be my second pick. Fuck WD.
On SSDs I go on Wikipedia and look at a list of flash + controller manufacturers and pick one of those. (Samsung, Kioxia (I think), Sandisk)
Toshiba oddly enough. I've been burnt by the big names like Seagate a few times now.
Hard disks, WD/HGST.
I've had good luck with EMC and NetApp for enterprise solutions, Synology for SMB class NAS storage, and rely on TrueNAS/ZFS on supermicro hardware at home, which has been rock solid for years and years.
None of them, because every manufacturer has made good and bad products. Seagate had really bad 3TB drives which gave them a lot of that reputation.
I just buy whatever fits my budget for HDDs and have proper backups in place. I think almost all of my HDDs are 'refurbished' ones.
For SSDs I look for one with a good TBW rating with a cache in it. Typically I'll go for used enterprise SSDs as well.
With the very limited number of drives one may use at home, just get the cheapest ones (*), use RAID and assume some drive may fail.
(*) whose performances meet your needs and from reputable enough sources
You can look at the backblaze stats if you like stats, but if you have ten drives 3% failure rate is exactly the same as 1% or .5% (they all just mean "use RAID and assume some drive may fail").
Also, IDK how good a reliabiliy predictor the manufacturer would be (as in every sector, reliabiliy varies from model to model), plus you would basically go by price even if you need a quantity of drives so great that stats make sense on them (wouldn't backblaze use 100% one manufacturer otherwise?)
SSD? Crucial.
They officially supported a non-standard setup I had, and they were totally there for me in a jam. Spinning rust I'll go other names, but SSD is crucial-first. I don't even care the cost.
Definetly Western Digital for used drives, some time ago I sold like 3 of these IDE old drives from like 15 or 20 years ago and they was working perfectly. IDK of nowadays WD drives but used to be very good at least for me.
If you want real data instead of feels, look at the BackBlaze report they publish every year. Even within a particular manufacturer varying models can have widely different failure stats.
https://www.backblaze.com/blog/backblaze-drive-stats-for-2023/
HGST does trend towards being a winner, and now with the largest Western Digital drives. But you definitely should pay attention to specific models like you said.
I think you are good as long as you avoid the cheap brands.