MSU Video Quality Measurement tools: subjective quality comparison

Subjective quality metric comparison

There are a lot of full reference objective quality metrics and each of them somehow represents difference between two video sequences. We performed comparisons with subjective quality for each metrics to understand which metric is better. Such comparison provided for the most popular metrics: PSNR, SSIM, 3-SSIM, MS-SSIM and new stSSIM. Subjective quality of video sequence is represented as MOS value (Mean Opinion Score) - mean quality score of a sequence estimated by a group of people. More about standards of subjective quality measurement you can read on the VQEG website or in the description of our Perceptual Video Quality Tool. Two indices of similarity between subjective quality and objective metric values for these metrics provided:

Pearson Linear Correlation Coefficient (LCC)

Pearson product-moment correlation coefficient is a measure of the correlation (linear dependence) between two variables X and Y, giving a value between +1 and -1 inclusive. It is widely used in the sciences as a measure of the strength of linear dependence between two variables. Wikipedia

So, correlation value of +1 means that the metric values have a perfect linear dependence on a subjective quality. Graphically it means that pairs of (MOS, metric value) for all of the videos in the database lie on the strict line. LCC index can answer the question: how well metric represents how much one video is visually better than another? Automatically this means that higher metric values means better subjective visual quality. Also we perform non-linear regression of metric values with logistic function over MOS values, to fit metric model to the MOS scores. We use following four-parameters function for the regression:

where Q - metric values and initial parameters values are:

To calculate LCC, perform the next evaluation for all of the pairs (MOS, fitted metric value) as (xi, yj):

Spearmen Rank Order Correlation Coefficient (SROCC)

Spearman’s rank correlation coefficient is a non-parametric measure of statistical dependence between two variables. It assesses how well the relationship between two variables can be described using a monotonic function. If there are no repeated data values, a perfect Spearman correlation of +1 or -1 occurs when each of the variables is a perfect monotone function of the other. (Wikipedia)

So, correlation value of +1 means that metric values are the perfect monotone function of subjective quality: two metric values X1 and X2 where X1 is greater than X2 means that average human will rate first video higher than other. Otherwise for correltion value -1. Correlation value 0 means there is no dependency at all. Usually objective quality metrics are non-linear functions of subjective quality. In this case SROCC index can answer the question: how well metric values correspond that one video is visually better than another? (higher metric value corresponds th the higher subjective quality?)

To calculate SROCC you need to convert pairs of metric values and MOS (Xi, Yj) for every video in the video database to the ranks (xi, yj) - both values from a pair are assigned a rank equal to the average of their positions in the ascending order of the values separately. Then you need to perform Pearson Linear Correlation calculation over the results.

Comparison results

Following coefficients provided for public video bases with subjective quality values:


Correlation coefficients for LIVE video quality database

Correlation coefficients for VQEG Phase-I video quality database

Contacts

E-mail: video-measure@graphics.cs.msu.ru

21 Feb 2019
See Also
Parallax range estimation in S3D video
Parallax determines the depth of S3D movies. The range of parallaxes should be both comfortable and entertaining for spectators.
MSU Video Quality Measurement Tool: Why upgrade?
MSU Video Quality Measurement Tool (PSNR, MSE, VQM, SSIM)
MSU Video Quality Measurement Tool (VQMT) is a program for objective video quality assessment.
MSU Video Quality Measurement Tool: SDK
MSU Perceptual Video Quality: Subjective video quality methods information
MSU Perceptual Video Quality Tool: FAQ
Site structure