Comment by bkallus

ABISan. Think of it like UBSan, but for assembly.

It's a custom assembler built on top of the LLVM assembler (llvm-mc) that emits instrumentation code to catch ABI violations at runtime. Stuff like clobbering nonvolatile registers, misaligning the stack pointer, misusing the redzone, assuming volatile registers don't change across a function call, etc.

Hoping to finish up basic x86_64 support within the next few days. I can now reliably assemble and run unoptimized gcc output without hitting false positives, but I still have to iron out some false positives triggered by OpenSSL's handwritten assembly routines.

TODO items for the near future include porting the runtime support library into a kernel module so I can instrument Linux, and beginning ports other architectures (ideally something semi-obscure like POWER or RISC-V). I also need to figure out how to support dynamic linking, because the tool currently needs static linking to access its thread-local variables.

https://github.com/kenballus/llvm-project/tree/abisan/llvm/t...