Comment by trebligdivad

Any use of sum types would have prevented this immediately.

The whole thing exists because APIs that return a NULL value that "nobody checks", plus a string that you shouldn't use if you actually did check the null value.

In languages with sum types, this cannot happen.

Either you get your hash, or no result at all. You couldn't get eg NULL and *.

C's lack of all of

- sum/union types

- multiple return values/tuple types

- exceptions

means that most ways of returning an error from a function are not available in C. Because of this, many C APIs, including `crypt` in this case, resort to in-band error signalling by using a sentinel value (like NULL, a negative integer, "*" etc.).

Most other languages lack one or two of the above, but very few lack all three, and so most have developed a different culture around error reporting (Result in Rust, multiple return values in Go, exceptions in Java, Python etc.).

For the other part of the issue (the truncation in SHA512), in Rust you could've had:

    struct Input<'a>(&'a [u8]);
    

    impl<'a> Input<'a> {
      fn new(input: &'a [u8]) -> Result<Self, Error> {
        if input.len() >= 256 {
           return Err(Errro...);
        }
        
        Ok(Self(input))
      }
    }

Now the truncation is a terrible idea in the first place, but at least with this it would have been entirely obvious to the caller that it was there.

Wouldn't Rust's Result or Go's error checking prevent that?

While not mandatory, a person would probably use it instead of the * hack.

Simply because the calling side wouldn't have an option to forget the check like with the NULL case