I really appreciate your detail comment, it helps.

>working memory sizes, buffering of input and output data, etc.

A confusion I have had regarding buffering is, I don't buffer video/audio frames at all before sending them to encoder (I mean, I just grab a frame and send it to the encoder and hope it is buffering them), maybe I should implement a buffer of 10 frames or so, what would you suggest?