IIRC, the mercurial project had already started when Linus started working on Git. (In fact, very early versions of Xen used BitKeeper, following suit with Linux; when the license changed, Xen moved over to mercurial because git wasn't ready yet, and stuck with it for a number of years.)
The main reason Linus wasn't happy with mercurial was the performance -- Linux just has far far more commits and files to deal with than nearly any other project on the planet, and even at the time, operations in mercurial took just a bit too long for Linus.