Benchmarking cloud video encoding: Zencoder, Encoding.com, Sorenson, & Panda | Hacker News Reader

Benchmarking cloud video encoding: Zencoder, Encoding.com, Sorenson, & Panda

blog.cloudharmony.com

43 pointsjon_dahl25 comments

Bit of a shameless plug, but we wrote our own EC2 + ffmpeg encoder, since we didn't want to pay out the ears for encoding:

http://media-nommer.readthedocs.org/

Open source (BSD), beta'ish, but very much functional. Save some money! Pull requests are welcome and greatly appreciated, as is feedback on the docs.

Very cool. How long did it take you to build that?

December 21, 2010 was our start date, so approaching a year now.

A shameless plug on top of your shameless plugin: We also wrote our own encoding system, called Tootsie (think Dustin Hoffman), which is in production use:

https://github.com/alexstaubo/tootsie

Written in Ruby, based on FFmpeg + ImageMagick, handles both video, audio and photos (we use it for photo/thumbnail scaling). JSON API, easy to use. It's designed to run anywhere, EC2 in particular. MIT license.

Looks cool, but are there any real central docs to look through (aside from that README)?

No. The README covers pretty much everything you need to know. I would be happy to explain (and rectify the readme) if there is anything that is unclear, however.

This only measures the speed, not the quality. Encodings may vary enormously in their complexity, you can't treat 2 encoders that produce the same "resolution" as being equal.

That's much harder to do objectively, but I'd love to see some 3rd party evaluations.

They (almost) all use the same encoder.

Albeit most likely with different settings, for a different speed/quality tradeoff. And some might be using ancient versions from the dawn of the Bronze Age.

Where's the shout out to ffmpeg? Don't tell me that almost all services involved aren't built on top of it.

Here you go: ffmpeg rules. Zencoder, Panda, and Encoding.com use it. Sorenson doesn't. That's part of the reason why our video quality > Sorenson. Also here: http://zencoder.com/opensource.

Big shout outs to x264, libavcodec, gpac, mediainfo, lftp, curl, mkvtoolnix, oggz, sox, lame, Ubuntu Linux, C, C++, Ruby, Rake, Postgresql, Redis, Dennis Ritchie, Steve Jobs, vim, emacs, and a hundred others. We couldn't have done it without all of you.

Big shout out to ffmpeg from Panda.

As far as I know, we are the only ones to give you full access to the command-line options in case you want to do funny stuff. The disadvantage is that we have been stuck with ffmpeg 0.6 on custom commands to not break backward-compatibility (which is what they used).

If they had used the presets, the encoding would have run with the latest ffmpeg which is also a lot faster.

Ouch! (regarding getting fsck-ed with command line options)

Before current job I built a fairly large more general media management system which included tons of video related work (and encoding ;) ) using ffmpeg for some fairly prominent (something something geographic something / etc) so I definitely appreciate how powerfully awesome ffmpeg is once you get over the initial overwhelming hump of having to learn all the more general video concepts involved outside of ffmpeg.

Which is why seeing a "performance comparison" blog post was immediately extremely suspicious sounding for the very valid point that everyone has brought up that speed isn't all that matters. From my experience speed is one of the lowest concerns anyone doing anything professionally with video has. Quality / reliability / tweak-ability are all much more important.

If you're writing software you intend to sell for money for anything other than cat videos uploaded to youtube - do yourself a favor and make sure you're informed about all the core concepts before randomly picking someone. You'd be surprised at all the hiccups and encoding issues you run across once you have a large enough sample set of "problem videos" to test against.

Sponsored by Zencoder.

As much as I want to trust this benchmark, I'm doubtful.

If Zencorer knew which account you used, they could've assigned the best dedicated servers to your jobs.

Sure, it could be a scam. But for what it's worth, it's not - we absolutely didn't cheat. Cheating in a benchmark might work some places, but it certainly doesn't work for a company that sells to hackers.

If you want to verify the results yourself, we're going to post the benchmarking source code to Github this afternoon. You can review the methodology that CloudHarmony used and run the tests yourself, with your own account, without us having any knowledge that you're testing us. That way, we have no way of cheating. You can even change up the files and use different encoding profiles to be sure that we didn't pick specific files or settings that happen to be faster at Zencoder.

Would be interesting if someone independent could corroborate the results. The sponsorship disclosure is appreciated, but I agree with the original comment - raises red flags.

Here's the source code that anyone can use to independently test and corroborate the results: https://github.com/zencoder/cloudharmony-benchmark. We chose Cloud Harmony to run the benchmark on the basis of their independent reputation. However, we'd happily sponsor (pay for encoding fees) another third-party to run the tests as well. Feel free to suggest an additional, neutral person/company that could do this. Though the results will be the same, no matter who runs it. :)

And how did you measure the quality of the outputted file? This matters a huge amount, it doesn't matter if you are 4 times as quick if the output is not up to the standards needed by the client.

Definitely - this is important as a control.

Every test was done using the same core settings (codecs, profiles, resolution, frame rate, bitrates, two-pass encoding). Beyond that, CloudHarmony used the default settings at each provider. We'll try to publish each video for visual inspection, but suffice it to say that they all look pretty close, especially three of the four, which use the same core encoder (x264) and the same settings. There may be microversion differences and minor variations in settings, but not much. The fourth uses MainConcept, which isn't a bad encoder itself. You'll probably see a bigger spread in audio quality - Encoding.com and Panda rely on faac, I think, which is an abandoned AAC encoder that isn't very good.

Thanks for taking the time to do these tests (@panda here). I checked your benchmarks and I can confirm that they are legit on our side.

To be fair our customers are not the same type. We mainly have people who trans-code a lot of videos (like DVD libraries) and in that case we are a lot cheaper than zencoder.

Our presets also use a much more recent version of ffmpeg since we are not stuck with command-line compatibility issues

My service relies on Zencoder. before we ended up with them we tested most of these services and build own stuff. result? we are very happy with zencoder as we found similar results. its really fast(you will notice difference once you switch from their competitors) and works all the time.

disc: I am not affiliated with zencoder in any way.

Same experience here. My only affiliation is they sent me a t-shirt. Lovely service.

If you want to prove you're the better service what's your options? Hope someone does the benchmarking for you?

Generally you design a benchmark and pay an "independent" lab to run it and publish the results.

For what it's worth, I ran the tests and wrote this blog post and can assure you that we made every effort to be fair with the methodology and honest about the results. Take a look at our blog and website. Since 2009, we've released many benchmarks publicly and have a good reputation for providing unbiased, objective analysis.

Reply on news.ycombinator.com