Second Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison

For professional users and codec authors

Second H.264/AVC comparison home page!

Part 4. Additional information and conclusions

2-pass mode and High profile

Additionally to one-pass mode in Main profile of standard H.264 we tested two-pass mode and compression using possibilities of High profile. Only x264, Elecard, Ateme and Fraunhofer IIS codecs took part in this test. This test was held on “foreman” and “battle” sequences and without speed measuring. As it is seen on picture below, it is difficult to analyze results using PSNR/Bitrate graphs. That is why we use Delta Y-PSNR graphs instead of absolute Y-PSNR graphs in this part. In all graphs reference is Main Profile one pass results for current codec.


Picture 1. Different presets of codec x264, Absolute Y-PSNR

 

Sequence “foreman”, preset “Best quality”


Picture 2. Different presets of codec x264, Delta Y-PSNR

 


Picture 3. Different presets of codec Elecard H.264, Delta Y-PSNR

 


Picture 4. Different presets of codec Ateme H.264, Delta Y-PSNR

 


Picture 5. Different presets of codec Ateme H.264, Delta Y-VQM

 


Picture 6. Different presets of codec Ateme H.264, Delta-Y-SSIM

 


Picture 7. Different presets of codec Fraunhofer IIS H.264, Delta Y-PSNR

 

Sequence “battle”, preset “Best quality”


Picture 8. Different presets of codec x264, Delta Y-PSNR

 


Picture 9. Different presets of codec Elecard H.264, Delta Y-PSNR

 


Picture 10. Different presets of codec Ateme H.264, Delta Y-PSNR

 


Picture 11. Different presets of codec Ateme H.264, Delta Y-VQM

 


Picture 12. Different presets of codec Ateme H.264, Delta Y-SSIM

 


Picture 13. Different presets of codec Fraunhofer IIS H.264, Delta Y-PSNR

Note:

Fraunhofer IIS H.264 codec showed some not very adequate results during this part of testing because of presets that were provided for us: Main profile with two passes has quality parameter equal to 4 (good quality) Main profile with one pass and High profile have quality parameter equal to 6 (best quality)

Per frame comparison

In addition to average metrics’ values we measured metrics’ values for separate frames. Basing on this data, Rate Control of each codec could be analyzed. As a rule, on such graphs it is clearly seen whether codec uses B-frames or not (“striation” of graph), location of I-frames (abrupt peaks or collapses on graph), scene changing reaction, quality of R-D models, etc.

On the next graphs there are frames on abscissa axis and bit rates on axis of ordinates (low bit rates are below). Color shows metric’s value: the more red, the lesser is value. Thus, for Y-PSNR metric red color means bad quality, blue - good quality. All graphs of one sequence have the same colors (every color corresponds to only one value of metric on all graphs).

Analyzing these graphics we should take into account bit rate handling of codecs since these graphs do not consider it.

Sequence “foreman”, Y-PSNR, Preset “Best quality”

 


Picture 14. Codec DivX. Sequence "foreman".

 


Picture 15. Codec x264. Sequence "foreman".

 


Picture 16. Codec ArcSoft H.264. Sequence "foreman".

 


Picture 17. Codec Elecard H.264. Sequence "foreman".

 


Picture 18. Codec Ateme H.264. Sequence "foreman".

 


Picture 19. Codec Fraunhofer IIS H.264. Sequence "foreman".

   

Sequence “bbc”, Y-PSNR, Preset “Best quality”


Picture 20. Codec DivX. Sequence "bbc".

On this graph compression quality does not significantly depend from bit rate, because DivX codec does not maintain low bit rates well, and works badly on high ones.


Picture 21. Codec x264. Sequence "bbc".

 


Picture 22. Codec ArcSoft H.264. Sequence "bbc".

 


Picture 23. Codec VSS H.264. Sequence "bbc".

 


Picture 24. Codec Elecard H.264. Sequence "bbc".

 


Picture 25. Codec Ateme H.264. Sequence "bbc".

 


Picture 26. Codec Fraunhofer IIS H.264. Sequence "bbc".

Remarks: These graphs show that x264, Elecard, Ateme and VSS codecs use B-frames. DivX and Elecard codecs have very interesting B-frames placement algorithm. They change positions of each type of frames depending on bit rate value. This is shown on graphs as vertical wavy lines.

“bbc” sequence clearly shows problems in bit rate control of ArcSoft codec. It is represented as presence of “holes” on average bit rates. Considering complex circular motion in this sequence, one could suppose presence of problems with motion estimation in this codec.

Large quality overstating of I-frames in x264 codec (it is also typical for Ateme and Fraunhofer IIS codecs, but at smaller degree) may be caused by inaccurate prediction of current frame type. It is represented as distinct blue lines that appear several times in sequence.

Graph type Total number of graph Inserted in this document
Per frame metrics 1414 13 (0.9%)

Visual comparison

Even in spite of appearance of new metrics, that more adequately reflect human perception of video, subjective video tests still do not lose their importance.

We set ourselves a problem to compare video codecs using only existing objective metrics. But in addition to various graphs that impartially evaluate quality of obtained results, below we give several frames from “bbc” and “battle” sequences, compressed with different codecs.

Sequence “bbc”, frame 170, bitrate 1140 kbps

Picture 27. Original Picture 28. DivX Picture 29. x264
Picture 30. Original. Image differences. Y-PSNR Picture 31. DivX. Image differences. Y-PSNR Picture 32. x264. Image differences. Y-PSNR

 

Sequence “bbc”, frame 250, bitrate 1140 kbps

Picture 33. Original Picture 34. DivX Picture 35. x264
Picture 36. Original. Image differences. Y-PSNR Picture 37. DivX. Image differences. Y-PSNR Picture 38. x264. Image differences. Y-PSNR

 

Sequence “battle”, frame 527, bitrate 700 kbps


Picture 39. Original

 


Picture 40. DivX

 


Picture 41. x264

 


Picture 42. ArcSoft

 


Picture 43. VSS

 


Picture 44. Elecard

 


Picture 45. Ateme

 


Picture 46. Fraunhofer IIS


Sequence “battle”, frame 527, bitrate 700 kbps, Y-PSNR

Picture 47. Y-PSNR ArcSoft Picture 48. Y-PSNR Ateme
Picture 49. Y-PSNR DivX Picture 50. Y-PSNR Elecard
Picture 51. Y-PSNR Fraunhofer IIS Picture 52. Y-PSNR VSS
Picture 53. Y-PSNR Original Picture 54. Y-PSNR x264

Informal codec comparison

  As it is clearly seen on the graphs, codecs show significantly different results on different sequences. Selected sequences have different types in terms of motion and noise, and this allows checking which codecs are tuned to all types of sequences, and which one - to only one. But also it is very interesting to evaluate codec behavior on all test data.

We considered averaging of results between different sequences as incorrect, and used a method of assigning points to codecs for different measurements. Points are united in arbitrary informal rating in the end.

We tried to reflect real situation on test data as much as possible. We are still developing methodology of our informal comparison. Therefore it is better to consider following results as subjective opinion of comparison authors.

Informal comparison rules

Separate comparisons were carried out for “Best Quality” and “Best Speed” presets. Several graphs were chosen for each preset, on which comparison was held. Obtained marks were summed with some predefined weights.

Codecs received points in a following way:

Following graphs were used for “Best Quality” preset with corresponding coefficients:

Following graphs were used for “Best Speed” preset with corresponding coefficients:

Average values were computed as arithmetic average of all successfully encoded sequences (i.e. codecs’ errors were not taken into account).

Informal comparison results

Preset “Best quality

Sequence “foreman”

  DivX x264 ArcSoft VSS Elecard Ateme Fraunhofer IIS
Y-PSNR 1 2 1 0 2 3 2
U-PSNR 1 3 1 0 3 3 2
V-PSNR 1 3 2 0 3 3 2
SSIM 1 3 1 0 2 2 2
VQM 1 2 1 0 2 3 2
BH 1 2 2 0 2 2 2
Time 2 2 3 0 2 3 1
Total 13 27 18 0 26 32 23

  Sequence “susi”

  DivX x264 ArcSoft VSS Elecard Ateme Fraunhofer IIS
Y-PSNR 2 2 1 2 2 3 2
U-PSNR 2 3 1 2 2 2 1
V-PSNR 2 3 1 3 2 2 1
SSIM 2 2 1 2 2 3 2
VQM 2 2 1 2 1 3 2
BH 1 2 3 2 2 2 1
Time 2 2 3 2 2 2 1
Total 21 26 20 25 23 30 18

  Sequence “bbc”

  DivX x264 ArcSoft VSS Elecard Ateme Fraunhofer IIS
Y-PSNR 1 2 1 2 2 3 3
U-PSNR 1 3 1 2 2 2 3
V-PSNR 2 3 1 2 2 3 3
SSIM 2 2 1 3 2 3 2
VQM 2 2 1 2 2 3 3
BH 1 3 1 2 2 2 3
Time 2 2 3 2 2 2 1
Total 16 29 14 25 24 31 33

  Sequence “battle”

  DivX x264 ArcSoft VSS Elecard Ateme Fraunhofer IIS
Y-PSNR 2 2 1 1 2 3 2
U-PSNR 1 3 1 2 3 2 3
V-PSNR 1 3 1 2 3 2 3
SSIM 2 2 1 1 2 3 2
VQM 2 2 1 2 2 3 3
BH 1 3 2 2 2 2 3
Time 2 2 3 2 2 2 1
Total 19 29 17 19 26 30 29

  Sequence “simpsons”

  DivX x264 ArcSoft VSS Elecard Ateme Fraunhofer IIS
Y-PSNR 2 3 1 3 2 3 2
U-PSNR 1 3 1 3 2 2 2
V-PSNR 1 3 1 3 3 2 2
SSIM 2 3 1 3 2 2 2
VQM 2 3 1 3 2 2 2
BH 1 2 2 2 2 2 3
Time 2 2 3 2 2 2 1
Total 19 32 17 32 25 28 26

  Sequence “matrix”

  DivX x264 ArcSoft VSS Elecard Ateme Fraunhofer IIS
Y-PSNR 2 2 1 2 2 3 1
U-PSNR 2 3 1 3 2 2 1
V-PSNR 2 3 1 2 3 2 1
SSIM 2 2 1 2 2 3 1
VQM 2 2 1 2 2 3 2
BH 1 1 2 3 2 2 2
Time 2 2 3 2 2 2 1
Total 21 23 17 28 25 30 16

  Sequence “concert”

  DivX x264 ArcSoft VSS Elecard Ateme Fraunhofer IIS
Y-PSNR 0 2 1 3 1 3 3
U-PSNR 0 3 3 2 2 2 1
V-PSNR 0 3 3 2 2 2 1
SSIM 0 2 1 3 2 1 3
VQM 0 2 1 3 1 2 3
BH 0 2 2 2 1 2 3
Time 0 2 2 2 3 2 1
Total 0 26 20 30 17 27 30

  Results

Codec Average points Place
Ateme 29.71 1
x264 27.43 2
VSS 26.5 3
Fraunhofer IIS 25 4
Elecard 23.71 5
DivX 18.17 6
ArcSoft 17.57 7

Preset “Best speed”

Sequence “foreman”

  DivX x264 ATI ArcSoft VSS Elecard Fraunhofer IIS
Y-PSNR 1 2 1 2 0 3 2
BH 1 2 3 2 0 2 1
Time 2 2 3 2 0 2 1
Total 11 14 19 14 0 15 8

 

Sequence “susi”

  DivX x264 ATI ArcSoft VSS Elecard Fraunhofer IIS
Y-PSNR 2 2 1 2 2 3 2
BH 1 2 2 3 3 2 1
Time 2 2 3 2 2 2 1
Total 12 14 17 16 16 15 8

 

Sequence “bbc”

  DivX x264 ATI ArcSoft VSS Elecard Fraunhofer IIS
Y-PSNR 1 2 2 1 2 3 3
BH 1 2 2 1 2 2 2
Time 2 2 3 2 2 2 1
Total 11 14 18 11 14 15 11

 

Sequence “battle”

  DivX x264 ATI ArcSoft VSS Elecard Fraunhofer IIS
Y-PSNR 2 3 1 2 1 3 2
BH 1 2 2 2 2 2 3
Time 2 2 3 2 2 2 1
Total 12 15 17 14 13 15 12

 

Sequence “simpsons’

  DivX x264 ATI ArcSoft VSS Elecard Fraunhofer IIS
Y-PSNR 2 2 1 2 1 3 3
BH 1 2 2 2 2 2 3
Time 2 2 3 2 2 2 1
Total 12 14 17 14 13 15 13

 

Sequence “matrix”

  DivX x264 ATI ArcSoft VSS Elecard Fraunhofer IIS
Y-PSNR 3 2 2 2 2 2 2
BH 1 2 2 3 3 2 2
Time 2 2 3 2 2 2 1
Total 13 14 18 16 16 14 10

 

Sequence “concert”

  DivX x264 ATI ArcSoft VSS Elecard Fraunhofer IIS
Y-PSNR 0 3 1 2 2 3 3
BH 0 2 1 2 2 1 3
Time 0 2 3 2 2 2 1
Total 0 15 15 14 14 13 13

 

Results

Codec Average points Place
ATI 17.29 1
Elecard 14.57 2
VSS 14.33 3
x264 14.29 4
ArcSoft 14.14 5
DivX 11.83 6
Fraunhofer IIS 10.71 7

 

General conclusions

Comparing to previous year comparison, the level of H.264 standard codecs has increased significantly. One year ago best codecs of old MPEG4-ASP standard were comparable to best codecs of H.264 standard, now this situation greatly changed. DivX codec is inferior to most codecs of new standard, even despite considerable growth of its quality comparing to previous version.

It is rather difficult to choose absolute leader by quality among codecs of new standard. x264 and Ateme codecs showed approximately equal best results.

Obvious leader by speed is codec from ATI company. It was significantly faster than the others while not always showing worst results.

10 Mar 2011
See Also
Parallax range estimation in S3D video
Parallax determines the depth of S3D movies. The range of parallaxes should be both comfortable and entertaining for spectators.
Call for HEVC codecs 2019
Fourteen modern video codec comparison
HEVC Video Codecs Comparison 2018 (Thirteen MSU Video Codec Comparison)
13th MSU video codecs comparison
HEVC Video Codecs Comparison 2017 (Twelfth MSU Video Codec Comparison)
12th MSU video codecs comparison
MSU Video Codec Comparisons (6 test of lossless, MPEG-4 and MPEG-4 AVC)
Call for HEVC codecs 2018
Site structure