# MSU Video Quality Measurement tools: subjective quality comparison

## Subjective quality metric comparison

There are a lot of full reference objective quality metrics and each of
them somehow represents difference between two video sequences. We
performed comparisons with subjective quality for each metrics to
understand which metric is better. Such comparison provided for the most
popular metrics: PSNR, SSIM, 3-SSIM, MS-SSIM and new stSSIM. Subjective
quality of video sequence is represented as **MOS** value (Mean Opinion
Score) - mean quality score of a sequence estimated by a group of
people. More about standards of subjective quality measurement you can
read on the VQEG website or in the description of our
Perceptual Video Quality
Tool.
Two indices of similarity between subjective quality and objective
metric values for these metrics provided:

## Pearson Linear Correlation Coefficient (LCC)

Pearson product-moment correlation coefficient is a measure of the correlation (linear dependence) between two variables X and Y, giving a value between +1 and -1 inclusive. It is widely used in the sciences as a measure of the strength of linear dependence between two variables. Wikipedia

So, correlation value of +1 means that the metric values have a perfect
linear dependence on a subjective quality. Graphically it means that
pairs of (MOS, metric value) for all of the videos in the database lie
on the strict line. LCC index can answer the question: how well metric
represents **how much** one video is visually better than another?
Automatically this means that higher metric values means better
subjective visual quality. Also we perform non-linear regression of
metric values with logistic function over MOS values, to fit metric
model to the MOS scores. We use following four-parameters function for
the regression:

where Q - metric values and initial parameters values are:

*β*= max(Q)_{1}*β*= min(Q)_{2}*β*= mean(MOS)_{3}*β*= 1.0_{4}

To calculate LCC, perform the next evaluation for all of the pairs (MOS,
fitted metric value) as (*x _{i}, y_{j}*):

## Spearmen Rank Order Correlation Coefficient (SROCC)

Spearman’s rank correlation coefficient is a non-parametric measure of statistical dependence between two variables. It assesses how well the relationship between two variables can be described using a monotonic function. If there are no repeated data values, a perfect Spearman correlation of +1 or -1 occurs when each of the variables is a perfect monotone function of the other. (Wikipedia)

So, correlation value of +1 means that metric values are the perfect
monotone function of subjective quality: two metric values X_{1}
and X_{2} where X_{1} is greater than X_{2}
means that average human will rate first video higher than other.
Otherwise for correltion value -1. Correlation value 0 means there is no
dependency at all. Usually objective quality metrics are non-linear
functions of subjective quality. In this case SROCC index can answer the
question: how well metric values correspond that one video is visually
better than another? (higher metric value corresponds th the higher
subjective quality?)

To calculate SROCC you need to convert pairs of metric values and MOS
(*X _{i}, Y_{j}*) for every video in the video database
to the ranks (

*x*) - both values from a pair are assigned a rank equal to the average of their positions in the ascending order of the values separately. Then you need to perform Pearson Linear Correlation calculation over the results.

_{i}, y_{j}## Comparison results

Following coefficients provided for public video bases with subjective quality values:

- Laboratory for Image & Video Engineering Video Quality Database
- Video Quality Experts Group Phase I video sequences database

Correlation coefficients for LIVE video quality database |

Correlation coefficients for VQEG Phase-I video quality database |

## Contacts

E-mail: video-measure@graphics.cs.msu.ru