Try compiling it on Windows using cygwin with x264 enabled... I literally spent an entire workday on it. It was not fun.
On the other hand, once I compiled the libraries, I had no issues interfacing it with code in Visual C++ \C#. The source files are fairly clean and documented enough that I could mostly figure out what to do by looking at the code.