I was under the impression (though I may be wrong) most standard video types are converted by some in-house closed source software developed by Google, but then if it's an odd/obscure format their software can't process they fall back on FFMPEG.
Even if they do use FFMPEG I'd guess it's heavily modified to suit their needs/infastructure