Could you share your experience with FFMPEG+ Lambda ? I ran into trouble with this when dealing with large files, especially when some of the files were being pulled off non S3 sources. Also what EC2 cores were you using. ?
* The maximum timeout window for a function invocation is 300 seconds
* The maximum available temp disk space per instance is 500MB
* Memory is maxed at 1.5GB
In the function invocation time window you need to:
* retrieve the file (to memory or disk)
* transcode the file (outputting to memory or disk)
* upload the file (as the disk is not persistent)
This along with the following facts make it infeasible:
* transfers from S3 are fast, but non-s3 sources are likely to be much slower.
* assuming you mean large as in GB - you have nowhere to put the files (disk too small, memory too small).
* transfers to S3 are fast, but uploading the transcoded video to a non-s3 source will likely me much slower.
Hope that helps.
You can process several GBs in the 5 minute window by piping your S3 download stream through your transformation steps then directly into an S3 upload stream. Nothing ever persists to disk, so your only worry if anything is managing your stream buffers so you don't run out of memory.
As long as any single step of your pipeline doesn't exceed the time limit, you can make really nifty pipelines for large file processing by using the S3 upload as "temp space" then an S3 event to automatically trigger the next step of your pipeline.