The original post: /r/datahoarder by /u/vff on 2025-06-03 14:18:52.
You may be familiar with the book A Million Random Digits with 100,000 Normal Deviates from the RAND corporation that was used throughout the 20th century as essentially the canonical source of random numbers.
I’m working towards putting together a similar collection, not of one million random decimal digits, but of at least one quadrillion random binary digits (so 128 terabytes). Truly random numbers, not pseudorandom ones. As an example, one source I’ve been using is video noise from an old USB webcam (a Raspberry Pi Zero with a Pi NoIR camera) in a black box, with every two bits fed into a Von Neumann extractor.
I want to save everything because randomness is by its very nature ephemeral. By storing randomness, this gives permanence to ephemerality.
What I’m wondering is how people sort, store, and organize random numbers.
Current organization
I’m trying to keep this all neatly organized rather than just having one big 128TB file. What I’ve been doing is saving them in 128KB chunks (1 million bits) and naming them “random-values/000/000/000.random” (in a zfs dataset “random-values”) and increasing that number each time I generate a new chunk (so each folder level has at most 1,000 files/subdirectories). I’ve found 1,000 is a decent limit that works across different filesystems; much larger and I’ve seen performance problems. I want this to be usable on a variety of platforms.
Then, in separate zfs dataset, “random-metadata,” I also store metadata as the same filename but with different extensions, such as “random-metadata/000/000/000.sha512” (and 000.gen-info.txt and so on). Yes, I know this could go in a database instead. But that makes sharing this all hugely more difficult. To share a SQL database properly requires the same software, replication, etc. So there’s a pragmatic aspect here. I can import the text data into a database at any time if I want to analyze things.
I am open to suggestions if anyone has any better ideas on this. There is an implied ordering to the blocks, by numbering them in this way, but since I’m storying them in generated order at least it should be random. (Emphasis on should.)
Other ideas I explored
Just as an example of another way to organize this, an idea I had but decided against was to randomly generate a numeric filename instead, using a large enough number of truly random bits to minimize the chances of collisions. In the end, I didn’t see any advantage to this over temporal ordering, since such random names could always be applied after-the-fact instead by taking any chunk as a master index and “renaming” the files based on the values in that chunk. Alternatively, if I wanted to select chunks at random, I could always choose one chunk as an “index”, take each N bits of that as a number, and look up whatever chunk has that index.
What I do want to do in the naming is avoid accidentally introducing bias in the organizational structure. As an example, breaking the random numbers into chunks, then sorting those chunks by the values of the chunks as binary numbers, would be a bad idea. So any kind of sorting is out, and to that end even naming files with their SHA-512 hash introduces an implied order, as they become “sorted” by the properties of the hash. We think of SHA-512 as being cryptographically secure, but it’s not truly “random.”
Validation
Now, as an aside, there is also the question of how to validate the randomness, although this is outside the scope of data hoarding. I’ve been validating the data, as it comes in, in those 128KB chunks. Basically, I take the last 1,048,576 bits as a 128KB binary string and use various functions from the TestU01 library to validate its randomness, always going once forwards and once backwards, as TestU01 is more sensitive to the lower bits in each 32-bit chunk. I then store the results as metadata for each chunk, 000.testu01.txt.
An earlier thought was to try compressing the data with zstd, and reject data that compressed, figuring that meant it wasn’t random. I realized that was naive since random data may in fact have a big string of 0’s or some repeating pattern occasionally, so I switched to TestU01.
Questions
I am not married to how I am doing any of this. It works, but I am pretty sure I’m not doing it optimally. Even 1,000 files in a folder is a lot, although it seems OK so far with zfs. But storing as one big 128TB file would make it far too hard to manage.
I’d love feedback. I am open to new ideas.
For those of you who store random numbers, how do you organize them? And, if you have more random numbers than you have space, how do you decide which random numbers to get rid of? Obviously, none of this can be compressed, so deletion is the only way, but the problem is that once these numbers are deleted, they really are gone forever. There is absolutely no way to ever get them back.
(I’m also open to thoughts on the other aspects of this outside of the data hoarding and organizational aspects, although those may not exactly be on-topic for this subreddit and would probably make more sense to be discussed elsewhere.)
TLDR
I’m generating and hoarding ~128TB of (hopefully) truly random bits. I chunk them into 128KB files and use hierarchical naming to keep things organized and portable. I store per-chunk metadata in a parallel ZFS dataset. I am open to critiques on my organizational structure, metadata handling, efficiency, validation, and strategies for deletion when space runs out.
