← Back to context

Comment by strenholme

5 days ago

This is why I use, in security critical contents of my software (where the numbers have to be computationally infeasible to produce), a type of random number generator called an XOF (extendable-output function).

It takes entropy from multiple different sources, makes it all input to the XOF, then the XOF uses cryptography to output a stream that has as much entropy as the combined entropy of all of its sources of randomness. So if an XOF, for example, takes 100 runs of rdrand16, along with the system time in microseconds and the number of milliseconds between receiving 100 packets over the network, the XOF will output a completely random stream without artifacts like never returning 0x0000, even if rdrand16 never outputs 0x0000.

Isn’t this effectively what systems like /dev/(u)rand do? Pool multiple random sources together to hedge against these things?

I fail to see why one should either rely on a single random source nor roll their own.

  • Yes, on any modern system you should use the kernel provided random number sources.

    The only legitimate reason to roll your own is when you're developing for an embedded system or a bootloader or something like that where there is no kernel API available.

    • The code I wrote has been used by embedded developers in embedded spaces; I remember getting a bug report from someone in China because they used my code in an embedded system before the timestamp was correctly set on said system.

  • Yes, /dev/(u)random is supposed to do that, but what if there’s a bug in a kernel (e.g. some embedded system which may not even be running Linux) which causes /dev/(u)ramdom to be less than secure? There’s also issues where, for example, it may no longer be possible to read /dev/(u)random after putting the process in a chroot() sandbox (chroot() isn’t defined in POSIX so its behavior is not guaranteed to be consistent across multiple operating systems).

    getrandom() is often times suggested, but alas isn’t a standardized function, i.e. it’s not part of the POSIX specification. Considering how the C23 changes to the C specification caused a lot of perfectly good C code to no longer compile, I’m very anal about sticking to specs; I use '-std=C99' for my code these days (even though it can compile as C23 code) and stick to POSIX functions (except chroot() and setgroups(), but both of those predate POSIX, and even here I have a compile-time option to compile my code without those non-POSIX syscalls).

    The code using a secure XOF (the algorithm was developed by the same team which later on made SHA-3, and includes people who helped make AES) has been around for nearly two decades (the code where I roll my own RNG to make secure random numbers has been around for over 25 years, but used AES before XOFs existed) and not one security problem has found with the RNG code has ever been found. [1] “Don’t roll your own RNG” is a suggestion, but it is possible to do so securely if one knows what they are doing (i.e. they have read Applied Cryptography and keep current with cryptographic developments).

    For anything vibe coded (my code is 100% human written, for the record), rolling one’s own RNG is a really bad idea.

    [1] There was a theoretical issue with cache timing attacks over two decades ago, so I put mitigations in place, and then chose to use an XOF for newer code.

    [2] There was an issue where a separate implementation I made of this XOF would generate incorrect test vectors in clang, but only at some optimization levels. I now test the XOF in both GCC and clang at multiple optimization levels to make sure it acts correctly.

    • > getrandom() is often times suggested, but alas isn’t a standardized function

      The POSIX standard function is getentropy(), which internally calls getrandom() on Linux.

      > what if there’s a bug in the kernel which causes /dev/(u)ramdom to be less than secure?

      It's often the other way around: the Linux kernel contains thousands of workarounds for buggy hardware, while the buggy hardware itself doesn't always get patched. Linux developers take this stuff very seriously. As a result it's often safer to rely on kernel APIs than to access the hardware directly.

      The kernel code involving random number generation receives an exceptionally high amount of scrutiny because of its security implications, so I'd trust it to do the right thing over a naked call to RDRAND which nobody knows how exactly it's implemented in proprietary hardware or a handrolled solution to mix the RDRAND output with other entropy sources.

      Remember the Debian openssl disaster from 2008? That happened exactly because someone had handrolled their entropy mixing solution, then someone else broke it.

      13 replies →

    • > but what if there’s a bug in the kernel which causes /dev/(u)ramdom to be less than secure?

      so instead you suggest trusting your own untested unlooked at implementation more?

      12 replies →

    • > Yes, /dev/(u)random is supposed to do that, [...] getrandom() is often times suggested, but alas isn’t a standardized function, i.e. it’s not part of the POSIX specification.

      Is /dev/random or /dev/urandom part of the POSIX specification?

      1 reply →

    • Refusing the platform's CSPRNG for such nonsense reasons is perhaps the dumbest form of POSIX worship. This is obviously an area where platform feature detection makes sense, there's no reason to follow a religion of standards adherence when it directly leads you into harm's way

      1 reply →

    • i remember some linux kernel dev got ousted by the community because he/she wanted to not implement a backdoor that would compromise the results of /dev/urandom.

      13 replies →

Well that very similar to how the Linux kernel does it. The linux kernel does it a little differently in that it uses the chacha8 stream cipher instead of a XOF. The chacha8 stream key is frequently reseeded by hashing the entropy pool with blake2b over the collected randomness from all sources but a lot comes from the nanosecond timing of hardware interrupts. Depending on configuration the blocking rng does not return unless 256 bits of trusted randomness are mixed into the entropy pool.

If anyone is interested in this topic please just read the code[0]. It has a lot of interesting tricks that you would not have just rolling your own.

[0]https://github.com/torvalds/linux/blob/master/drivers/char/r...

If you're building a userland XOF RNG to extend the kernel's RNG (that has the same security properties) you are reducing security, not improving it. The kernel has advantages for managing and securing a secret "entropy" pool that you won't replicate in userland.

But if you're using a custom kernel that has a custom KRNG based on an XOF, sure, whatever, I guess.

Nifty! Out of curiosity, how much different is that from taking several partly-random streams and XORing them together? I always assumed what was going on was essentially a fancier version of that.

Oh, I guess you have to ensure the inputs aren’t correlated, or they’ll cancel out?

  • The advantage of a secure XOF is that a malicious source of entropy needs to do a good deal more work than a simple XOR to generate controlled PRNG output (the attacker needs to do 2^n XOF operations to generate n bits of PRNG output, and that’s only if the attacker knows the output of all other sources of entropy—someone with that level of access can do far more effective attacks).

    The sources of entropy can be correlated and won’t cancel out with a well designed secure XOF. SHAKE-256 is an example of a secure XOF.

You can effectively achieve the same result with this simple operation:

  hash = sha256(current_time());
  for i := 0; i < n; i++ {
      hash = sha256(hash.append(current_time()))
  }

This is because the number of nanoseconds between hashes is actually itself variable, and this is true for physics reasons that are basically beyond the control of any attacker trying to manipulate your entropy. If your time() function has a resolution of nanoseconds, you only need your loop to iterate about 50 times to get a cryptographically secure amount of entropy. If your time() function has a resolution of milliseconds, you need to let this run for more like 20 milliseconds, and if your time() function has a resolution of seconds you need to let it run for more like 5 seconds.

The reason I like doing it this way is that it happens entirely in userspace, it's genuinely a secure method of generating entropy, and it has no dependencies on potentially buggy firmware or microcode outside of the time() call, which is both fairly narrow, fairly heavily used (meaning a bug is likely to be discovered during testing, as the implementation is likely heavily scrutinized), and also fairly easy to test independently - just look at the number of nanoseconds that elapse at each consecutive call to sha256(current_time()) and verify that there's some statistical variance. The above suggestions are assuming about 2.5 bits of variance between calls, meaning there should be a range of at least 20 nanoseconds between your slowest and fastest hash call. This has been true on every CPU I've ever measured, including microcontrollers.

  • This comment demonstrates everything that's wrong with people trying to be clever and rolling their own crypto.

    The security of your system depends on time() providing enough entropy, even though that's not what it's designed to do. It's built on top of the wrong primitive from the start.

    > The reason I like doing it this way is that it happens entirely in userspace

    On Linux this is often true, but there is no portable way to get the current time that is _guaranteed_ not to do any system calls.

    > If your time() function has a resolution of nanoseconds, you only need your loop to iterate about 50 times to get a cryptographically secure amount of entropy.

    You haven't proven that at all. It's easy to imagine that on a CPU running at a fixed frequency the interval between reads is constant, so if anyone knows (or can guess) the start time the resulting seed is entirely predictable.

    This is completely independent of timer resolution. You seem to realize that as you were writing that:

    > just look at the number of nanoseconds that elapse at each consecutive call to sha256(current_time()) and verify that there's some statistical variance

    Oh yes, because evaluating the quality of a random number generator is such a trivial thing to do, it's not like there is decades of research behind it or anything.

    And assuming you are able to verify the statistical variance: are you going to put that logic in the loop, making it significantly more complex?

    Or are you going to do this test on your machine and then ship your code on the assumption that if it works on your machine, it will work everywhere else, too?

    > if your time() function has a resolution of seconds you need to let it run for more like 5 seconds.

    So not only is it insecure, it's agonizingly slow by design. Why do a system call that takes milliseconds at best, when we can run a loop in userspace for 5 seconds?

    All this just so you can avoid writing the obviously correct oneliner:

        if (getentropy(&seed, sizeof(seed)) != 0) abort();

    • And to show my objections are not just theoretical I wrote a little program to check:

          #include <time.h>
          #include <stdio.h>
          
          static int estimate_entropy(long l) {
              int bits = 1; /* for the sign bit */
              if (l < 0) l = -l;
              while (l > 0) {
                  ++bits;
                  l >>= 1;
              }
              return bits;
          }
          
          int main() {
              struct timespec ts;
              if (clock_getres(CLOCK_REALTIME, &ts) != 0) {
                  perror("clock_getres");
                  return 1;
              }
              printf("Clock resolution: %ld.%09ld\n", (long) ts.tv_sec, (long) ts.tv_nsec);
              
              #define N 50  /* number of samples */
              struct timespec samples[N];
              for (int i = 0; i < N; ++i) {
                  clock_gettime(CLOCK_REALTIME, &samples[i]);
              }
          
              printf("Deltas (ns):");
              long deltas[N - 1];
              for (int i = 0; i < N - 1; ++i) {
                  deltas[i] = 
                      (samples[i + 1].tv_sec - samples[i].tv_sec)*1000000000L
                      + (samples[i + 1].tv_nsec - samples[i].tv_nsec);
                  printf(" %4ld", deltas[i]);
              }
              printf("\n");
              long entropy = 0;
              printf("Deltas of deltas: ");
              for (int i = 0; i < N - 2; ++i) {
                  long dd = deltas[i + 1] - deltas[i];
                  printf(" %4ld", dd);
                  entropy += estimate_entropy(dd);
              }
              printf("\n");
              printf("Maximum entropy: %lld\n", entropy);
          }
      

      On my system this prints:

          Clock resolution: 0.000000001
          Deltas (ns):   55   51   23   23   25   24   24   24   24   24   25   25   24   24   24   24   24   25   24   24   24   25   25   24   24   23   25   24   24   25   24   23   25   25   26   23   25   24   24   25   26   24   23   25   25   26   24   25   24
          Deltas of deltas:    -4  -28    0    2   -1    0    0    0    0    1    0   -1    0    0    0    0    1   -1    0    0    1    0   -1    0   -1    2   -1    0    1   -1   -1    2    0    1   -3    2   -1    0    1    1   -2   -1    2    0    1   -2    1   -1
          Maximum entropy: 92
      

      So no, 50 iterations of that loop does not provide 256 bits of entropy due to random fluctuations in nanontime between calls.

      9 replies →

    • Depends in what trust do you have over your hardware/OS. If you assume the hardware is potentially backdoored, and the OS is proprietary, or even if open could have malware/rootkits that can thinker around the random number generator, the solution of using a sole implementation inside the program (assuming the sha256 function is inside the program itself) maybe better.

      Sure an infected system may as well fake time values, but that is much more difficult and it's possible to detect from a userspace program. For example you mention to use getentroy, but on a compromised system you know how easy it is to change something that is implemented in a system library (e.g. libc) or even if you read /dev/random directly without passing from the libc how easy it's to make it read whatever you want?

      To me that is not that bad implementation, in fact it's an implementation that is used in a lot of security software (including GPG, not as the sole source of course but as one of many).

      7 replies →

    • Any good crypto library will have a solid secure random source that usually combines entropy from multiple sources with a provably secure hash based mixing scheme.

      Hardware RNGs can be one source, but no single source is trusted, and they're all combined in a way where even an intentionally malicious source is lost in noise and cannot actually determine output.

      5 replies →

    • The reason I roll entropy in userspace is because there's a very long history of "cryptographic" libraries getting it wrong (see the parent article for an example). Crypto tokens stolen because the underlying call to the web browser entropy only had 32 bits of actual randomness. Crypto tokens stolen because the underlying embedded system (like cold card) turned off some security critical features to improve performance and power.

      Pretty much the only thing you can control when shipping software to many devices is that it runs on a physical CPU and has a timer. Every other RNG assumption over the decades has shown that sometimes someone upstream gets something catastrophically incorrect.

  • I wouldn’t trust it as a sole source of entropy, but it can be one of multiple entropy sources to feed in to an XOF to get secure numbers.

    The nice thing about using multiple entropy sources with a secure XOF is that the resulting entropy is at least as strong as the most secure entropy source given to the XOF.

    • Unfortunately you are not correct, and djb explains it quite well here:

      https://blog.cr.yp.to/20140205-entropy.html

      TL;DR adding a compromised source of entropy to a pool of already secure sources of entropy can catastrophically compromise the final result.

      It's better to source entropy from a smaller number of harder-to-compromise sources. That's why I like the iterated hashes method; the security surface area is both very small and highly likely to be well tested.

      2 replies →

  • I know that there's a really strong culture in the software world around downvoting anything that looks or smells like "hand-rolled cryptography", but this is my actual profession and specialization within the software world, and most of what I'm seeing in this thread is knee-jerk reactions to an unexpected technique rather than careful intellectual commentary and consideration of the merits of the technique.

    I am happy to have a discussion with you at the deepest technical levels of applied cryptography, this is not something I blindly made up on my own. I'm well studied in the field and can readily defend this technique.