Comment by consp
9 hours ago
As long as you don't statically link you can easily replace malloc (LD_PRELOAD). Many debug libraries do. Why is this so special in openssl? (I don't know if there is some special reason, though openssl is a weird one to begin with)
Using OpenSSL's malloc may bypass protections of hardened libc mallocs like OpenBSD's.
If memory crosses the boundary between OpenSSL and your app, or some other library, freeing it with a different allocator than the one it was allocated with is undefined behavior.
OpenSSL's allocator doesn't free in in the same ways other mallocs do, which prevents memory sanitization tools like valgrind from finding memory bugs.
OpenSSL has a completely separate idea of a secure heap, with it's own additional malloc implementation, which can lead to state leakage or other issues if not used perfectly at the (non-existent because the entire library surface is exposed) security boundary and is accidentally intermingled with calls to the (insecure?) malloc.
It's just a big can of security worms which may have been useful on odd platforms like VMS, though that's questionable, and only serves to add additional layers of inscrutability and obfuscation to an already messy codebase today. It's not enough to know what malloc does, one must familiarize themselves with all the quirks of both(!) of OpenSSL's custom implementations, which are used precisely nowhere else, to judge the security or code correctness implications of virtually anything in the codebase. There's no good reason for it.