Luckily, the team developing the open-source Daala video codec have a solution. "AreWeCompressedYet.com", an online resource that allows you to compare the performance of popular codecs.
How it works:
Codec developers submit "jobs" to the site. Each job involves coding and decoding a set of common test videos at a range of bitrates. This can be a very time consuming process, but the site collects and stores the results from many previous test runs. The results of each job can be viewed as a rate-distortion curve, with bitrate on the x-axis and "quality" on the y-axis:
This example compares the performance of five video codecs, under the following conditions:
- Test video set: "ntt-short-1", a set of 1-second video clips with varying characteristics.
- Video resolution: 720p, 24 frames per second
- Picture quality metric: Peak SNR (PSNR)
In this example, HEVC and VP9 have the "best" performance (highest curves), followed by Thor, H.264 and Daala. There are a few things you should be aware of:
- Measuring picture quality is a notoriously inexact science. Changing the metric from PSNR to a different metric such as SSim may - and often will - change the ranking of the codecs.
- Changing the video set or resolution may affect the results, since each codec's performance depends on the actual video content.
- The online tool is comparing specific implementations of each codec. For example, different software implementation of HEVC will probably perform differently.
- There is probably significant scope for further improvements to the HEVC, VP9, Thor and Daala codecs, since these are all relatively new (or still-developing) formats.
With all this in mind, arewecompressedyet.com provides a quick and easy way to compare codec technologies under identical experimental conditions. Will it resolve the endless arguments about which codec is best? Maybe!