For awhile, a lot of them came from fuzzers; if you're interested in how that works, strong recommend for Zalewski's blogs on his open-source "american fuzzy lop" fuzzer.
Browsers are huge attack surfaces, as complex as operating systems, with different modules of wildly different quality. There are lot of bugs to find.
The process makes more sense when you grok how much effort goes into getting privileged RCE from a hardened browser. Nobody can sit down with that code and pull one of them out of thin air on the spot.
The software is a fuzzer.
I think much of it is just a bit of luck that you're looking in the right place.
Many of those were pretty simplistic... fortunately for almost everyone at the time, the browser wasn't a widely used method of exploit, and to my knowledge compromising distributed computer networks via such compromises and ad networks wasn't thought of either. Though by 2002/2003, I had started blocking Flash, Java and Adobe Reader at home when I saw what could be done in the browser.
More concrete answers of your questions guessing :
1. By brute forcing with popular techniques for buffers overflows ;
2. By tracking what the browser is doing via debuggers and trying to match a kernel bug with it ;
3. By reading a lot about latest exploits on the internet and connecting the dots.
What I think nobody from them is doing :
1. Analysing open-source code ( basically try to find the exploit by reading the code ) ;
2. Studying 5 years CS before doing this
4. By using various forms static/dynamic analysis (taint analysis, fault propagation, fault injection).
5. By fuzz-testing the software.
Taint analysis is looking at which registers/memory locations you have access to. You inject data into a process and then check where in memory that data shows up, which execution branches are taken, which registers are used with the data, what the stack does, etc.
Fault propagation looks at where errors and faults are handled and how they are handled. The idea is to get a good idea of how a process manages errors and then find an error that is not handled (correctly).
Fault injection is injecting errors into a process and then watch the fireworks. Think of this as the brute-force approach to fault propagation analysis.
Fuzz testing is bombarding the process with semi-valid/semi-random input and watching what happens. There are many tools with this. Example output for browser-fuzzing could look like this:
<html>>style="\>>overflow:none"></body><html>
Obviously none of today's browsers would be exploited by that, but with mutli-threaded rendering, memory management, javascript reading/writing html and all that stuff going on at the same time it's not surprising that fuzz testing can turn up lots of errors (though not all exploitable).
Looks like some code analysis tools helped for one researcher.