I think a recurrent neural network is more suitable here as the size of context for each example might be different, instead of fixing the frame size.