Comment by actionfromafar

Right, so this is the crux of the vulnerability. Firstly, note that the `MAX` macro is defined as:

    #define MAX(a, b) ((a) > (b) ? (a) : (b))

This is important because it doesn't cast either arg to any particular type. You can use it with `float`s, or `int`s, or `u_int`s... or a combination.

Referring back to the implementation, the first use of `MAX` (inside the `for` loop) is this:

    ...
    for (i = 0; i < nsyscalls; i++)
        npins = MAX(npins, syscalls[i].sysno);
    ...

`npins` is an `int`, but `sysno` is a `u_int`. C integer promotion rules means that we'll actually be implicitly casting `npins` to a `u_int` here; it's as if we did this:

    ...
     npins = MAX((u_int)npins, syscalls[i].sysno);
    ...

This means that `npins` can end up as any value we like -- even up to `0xffffffff`. But remember that `npins` is _actually_ a signed int, so once it comes out of `MAX`, it'll be signed again. Thus we can use this to make `npins` negative.

Once we're out of the loop, `MAX` is used again here:

    ...
    npins = MAX(npins, SYS_kbind);
    ...

Where `SYS_kbind` is just:

    ...
    #define SYS_kbind       86
    ...

Integer literals in C are signed, so now this use of `MAX` is actually dealing with two signed integers. If we used the loop to make `npins` negative (as described just before) then this line will now take 86 as the maximum of the two values.

With `npins = 86`, an array of 86+1 will be allocated, but the `syscalls[i].sysno` in the next loop could of course easily be greater than 86 -- thus leading to out-of-bounds array access.