HandBrake uses some parts of FFmpeg libraries: libavformat, libavcodec, and libavfilter. But even so, it's an entire different app, the decoders, some demuxers and some filters are the same, but the way they are connected together is entirely different than FFmpeg cli app.