Right, so this is the crux of the vulnerability. Firstly, note that the `MAX` macro is defined as:
#define MAX(a, b) ((a) > (b) ? (a) : (b))
This is important because it doesn't cast either arg to any particular type. You can use it with `float`s, or `int`s, or `u_int`s... or a combination.
Referring back to the implementation, the first use of `MAX` (inside the `for` loop) is this:
...
for (i = 0; i < nsyscalls; i++)
npins = MAX(npins, syscalls[i].sysno);
...
`npins` is an `int`, but `sysno` is a `u_int`. C integer promotion rules means that we'll actually be implicitly casting `npins` to a `u_int` here; it's as if we did this:
...
npins = MAX((u_int)npins, syscalls[i].sysno);
...
This means that `npins` can end up as any value we like -- even up to `0xffffffff`. But remember that `npins` is _actually_ a signed int, so once it comes out of `MAX`, it'll be signed again. Thus we can use this to make `npins` negative.
Once we're out of the loop, `MAX` is used again here:
...
npins = MAX(npins, SYS_kbind);
...
Where `SYS_kbind` is just:
...
#define SYS_kbind 86
...
Integer literals in C are signed, so now this use of `MAX` is actually dealing with two signed integers. If we used the loop to make `npins` negative (as described just before) then this line will now take 86 as the maximum of the two values.
With `npins = 86`, an array of 86+1 will be allocated, but the `syscalls[i].sysno` in the next loop could of course easily be greater than 86 -- thus leading to out-of-bounds array access.