# MSU Quality Measurement Tool: Metrics information

## PSNR

This metric, which is used often in practice, called peak-to-peak signal-to-noise ratio – PSNR.

where *MaxErr* — maximum possible absolute value of color components
difference, *w* — video width, *h* — video height. Generally, this
metric is equivalent to Mean Square Error, but it is more convenient to
use because of logarithmic scale. It has the same disadvantages as the
MSE metric.

In MSU VQMT you can calculate PSNR for all YUV and RGB components and
for L component of LUV color space.

In MSU VQMT there are four PSNR implementations. “PSNR” and “APSNR” use
the correct way of PSNR calculation and take maximum possible absolute
value of color difference as *MaxErr*. However, this way of calculation
gives an unpleasant effect after color depth conversion. If color depth
is simply increased from 8 to 16 bits, the “PSNR” and “APSNR” will
change, because *MaxErr* should change according to maximum possible
absolute value of color difference (255 for 8 bit components and 255 +
255/256 for 16 bit components). Thus, “PSNR (256)” and “APSNR (256)” are
implemented. They would not change because they use upper boundary of
color difference as *MaxErr*. The upper boundary is 256. This approach
is less correct, but it is used often because it is fast. Here are the
rules of *MaxErr* definition:

- “PSNR” and “APSNR” —
*MaxErr*varies on color components bits usage:- 255 for 8 bit components
- 255 + 3/4 for 10 bit components
- 255 + 63/64 for 14 bit components
- 255 + 255/256 for 16 bit components
- 100 for L component of LUV color space

**Notes**- If bits depth differs for two compared videos, then maximum bits
usage is taken to select
*MaxErr*. - All color space conversions are assumed to lead to 8 bit images.
It means that if, for example, you are measuring R-RGB PSNR for
14 bit YUV file, then 255 will be taken as
*MaxErr*.

- “PSNR (256)” and “APSNR (256)” —
*MaxErr*is selected according to the next rules:- 256 for YUV and RGB color spaces
- 100 for L component of LUV color space

The difference between “PSNR” and “APSNR” is the same as between “PSNR (256)” and “APSNR (256)” and is in the way of average PSNR calculation for a sequence. The correct way to calculate average PSNR for a sequence is to calculate average MSE for all frames (average MSE is arithmetic mean of the MSE values for frames) and after that to calculate PSNR using ordinary equation for PSNR:

This way of average PSNR calculation is used in “PSNR” and “PSNR (256)”.
However, sometimes it is needed to take simple average of all the per
frame PSNR values. “APSNR” and “APSNR (256)” are implemented for this
case and calculate average PSNR by simply averaging per frame PSNR
values.

The next table summarizes the differences:

Metric |
MaxErr calculation |
Average PSNR calculation |

PSNR |
correct |
correct |

PSNR (256) |
256 (fast, inexact) |
correct |

APSNR |
correct |
averaging |

APSNR (256) |
256 (fast, inexact) |
averaging |

“PSNR” metric is recommended for PSNR calculation since it is implemented according to the original PSNR definition.

Source | Processed | Y-YUV PSNR |

Colors, in order of PSNR growing: red, yellow, green, blue, black (Note: larger PSNR means smaller the difference)

## MSAD

The value of this metric is the mean absolute difference of the color components in the correspondent points of image. This metric is used for testing codecs and filters.

Source | Processed | MSAD |

## Delta

The value of this metric is the mean difference of the color components in the correspondent points of image. This metric is used for testing codecs and filters.

Source | Processed | Delta |

Note: Red color X_{ij} >
Y_{ij}, green color X_{ij} < Y_{ij}

## MSU Blurring Metric

This metric allows you to compare power of blurring of two images. If value of the metric for first picture is greater than for second, it means that second picture is more blurred than first.

Source | Processed | MSU Blurring Metric |

Note: Red color — first image is more sharper, than second. Green color — second image is sharper than first.

## MSU Blocking Metric

This metric was created to measure subjective blocking effect in video sequence. For example, in contrast areas of the frame blocking is not appreciable, but in smooth areas these edges are conspicuous. This metric also contains heuristic method for detecting objects edges, which are placed to the edge of the block. In this case, metric value is pulled down, allowing to measure blocking more precisely. We use information from previous frames to achieve better accuracy.

Source | MSU Blocking Metric |

## SSIM Index

SSIM Index is based on measuring of three components (luminance similarity, contrast similarity and structural similarity) and combining them into result value.

Original | Processed |

SSIM (fast) | SSIM (precise) |

Note: Brighter areas correspond to greater difference.

There are 2 implementations of SSIM in our program: fast and precise. The fast one is equal to our previous SSIM implementation. The difference is that the fast one uses box filter, while the precise one uses Gaussian blur.

Notes:

- Fast implementation visualization seems to be shifted. This effect is caused by the sum calculation algorithm for the box filter. The sum is calculated over the block to the bottom-left or up-left of the pixel (depending on if the image is bottom-up or top-down).
- SSIM metric has two coefficients. They depend on the maximum value
of the image color component. They are calculated using the
following equations:
*C1 = 0.01 * 0.01 * video1Max * video2Max**C2 = 0.03 * 0.03 * video1Max * video2Max*

where

*video1Max*is the maximum value of a given color component for the first video,*video2Max*is the maximum value of the same color component for the second video. Maximum value of a color component is calculated in the same way as for PSNR:*videoMax*= 255 for 8 bit color components*videoMax*= 255 + 3/4 for 10 bit color components*videoMax*= 255 + 63/64 for 14 bit color components*videoMax*= 255 + 255/256 for 16 bit color components

## MultiScale SSIM INDEX

MultiScale SSIM INDEX based on SSIM metric of several downscaled levels of original images. Result is weighted average of those metrics.

Original | Processed |

MSSSIM (fast) | MSSSIM (precise) |

Note: Brighter areas correspond to greater difference.

Two algorithms are implemented for MultiScale SSIM — fast and precise,
as for SSIM metric. The difference is that the fast one uses box filter,
while the precise one uses Gaussian blur.

**Notes:**

- Because result metric is calculated as multiplication of several metric values below 1.0 visualization seems to be dark. Fast implementation visualization seems to be shifted. This effect is caused by the sum calculation algorithm for the box filter. The sum is calculated over the block to the bottom-left or up-left of the pixel (depending on if the image is bottom-up or top-down).
- Levels weights (0 corresponds to original frame, while 4 corresponds
to higher level):
*WEIGHTS[0] = 0.0448;**WEIGHTS[1] = 0.2856;**WEIGHTS[2] = 0.3001;**WEIGHTS[3] = 0.2363;**WEIGHTS[4] = 0.1333;*

## 3-Component SSIM INDEX

3-Component SSIM Index based on region division of source frames. There are 3 types of regions — edges, textures and smooth regions. Result metric calculated as weighted average of SSIM metric for those regions. In fact, human eye can see difference more precisely on textured or edge regions than on smooth regions. Division based on gradient magnitude is presented in every pixel of images.

Original | Processed |

3-SSIM regions division | 3-SSIM regions metric |

Note: More bright areas corresponds to greater difference.

## Spatio-Temporal SSIM

The idea of this algorithm is to use motion-oriented weighted windows for SSIM Index. MSU Motion Estimation algorithm is used to retrieve this information. Based on the ME results, weighting window is constructed for every pixel. This window can use up to 33 consecutive frames (16 + current frame + 16). Then SSIM Index is calculated for every window to take into account temporal distortions as well. In addition, another spooling technique is used in this implementation. We use only lower 6% of metric values for the frame to calculate frame metric value. This causes larger metric values difference for difference files.

Note: Brighter blocks correspond to greater difference.

## VQM

VQM uses DCT to correspond to human perception.

Source | Processed | VQM |

Note: Brighter blocks correspond to greater difference.

## MSE

Source | Processed | Y-YUV MSE |

## VMAF

VMAF (Video Multimethod Assessment Fusion) is modern reference metric developed by Netflix in cooperation with the University of Southern California. VQMT has full support of VMAF, which multiple configuration switches. You can use default model (vmaf_v061), see elementary features or load custom model from .pkl file. Note: you should have .model near .pkl in case of using custom model.

“Model preset” setting (in command line you should use “-set model_preset=…” after metric) describes what features or model you want to use. If you want to use model, select “vmaf_v061” or “custom”. If you want to view elementary vmaf features, select “basic_features” or “all_features”. If you want to see both: vmaf_v061 and all elementary features, select “all”.

Specify path to custom model file in field “Custom model (*.pkl)” (in command line you should use “-set custom_model_file=…”). To activate this setting, you should specify “custom” in “Model preset” setting.

VMAF can’t perform visualization in case of sophisticated metric algorithm. But you can visualize some of basic VMAF components. If you turned visualization on, specify demanded algorithm in field “Visualize algorithm”.

“Use phone model” (“-set phone_model=on/off/both” in command line) switch allows you to apply postprocessing, developed by Netflix for smartphones. If you select “both” in this field, you will see results without this postprocessing and with it in one output. This settings affects only on model output (not on basic features).

“Disable clipping values” (“-set disable_clip=false/true” in command line) can turn off clipping result value to 0..100 (or other range, specified in model setting). This value affects only model output.

“Use multithreading” (“-set multithreaded=true/false” in command line) to turn off multithreading. If “true”, basic features will be calculated asynchronously. If you already have parallel workprocess, disable multithreading to achieve better performance.

See also: https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

## NIQE

NIQE (Naturalness Image Quality Evaluator) is first VQMT no-reference image quality metric, developed in The University of Texas at Austin by Mittal, A., R. Soundararajan, and A. C. Bovik. This version uses reference model from authors. Visualization is currently not supported by this metric.

To know more about this metric refer to [1] Mittal, A., R. Soundararajan, and A. C. Bovik. “Making a Completely Blind Image Quality Analyzer.” IEEE Signal Processing Letters. Vol. 22, Number 3, March 2013, pp. 209-212.

See also: http://gfx.cs.princeton.edu/pubs/Liu_2013_ANM/sa13.pdf

## Contacts

E-mail: video-measure@compression.ru