Swapped views detection in S3D movies

Introduction

Channel mismatch is one of the most painful-for-viewer problems in stereo 3D movies. This problem is very hard to detect and simply to eliminate: just swap the views.

Our technique provides an opportunity for automatic detection of scenes with channel mismatch.

Details of the algorithm

Our algorithm uses five baselines.

Firstly, it is checking on three basic heuristics (i.e certain assumptions have to work for the scene):

  1. Objects with less depth are often at the bottom of the frame, and objects with a greater depth are often at the top of the frame.
  2. Usually, the objects that are most interesting to the viewer are “falling out”, that is, they have the smallest depth among other objects and are closer to the center of the screen.
  3. Usually, the areas in front of the screen plane occupy a third of the screen space, and the areas behind the screen plane - two thirds.

Secondly, two occlusion-analysis based algorithms are performing.

In the end, the weighted sum of all baselines results in one factor indicating whether the scene has a channel mismatch

Results

Here listed some results of channel mismatch detection in S3D movies. The analysis showed the swapped views occurred even in the high-budget movies (e.g. Avatar).

How often channel mismatch occurs

Movie name Release year Budget, $M Scenes with CM Total CM duration, sec Film duration, sec CM percentage
The Child’s Eye 2010 $4.5 15 57.5 5823 0.9875%
The Nutcracker in 3D 2010 $90 9 28,9 6480 0,447%
3D Sex and Zen: Extreme Ecstasy 2011 $2.5 9 23,1 6775 0,341%
Spy Kids 3D — Game Over 2003 $39 5 10,3 5063 0,203%
Sharks 3D 2004 $5 1 8,9 3073 0,290%
Avatar 2009 $237 1 3,3 9702 0,034%

Comparison with other algorithms

Due to the fact that there were no other channel mismatch detection algorithms, a comparison was conducted to our previous version of this algorithm.

Algorithm AUC ROC Time for a scene, sec
Proposed method 0,986945 15,4
Previous algorithm 0,837557 138,6

Instead of conclusion

Channel mismatch perceptibility

A subjective study of perceptibility of channel mismatch was conducted.

We composed a test sequence of 56 scenes with swapped views. In addition, the sequence included scenes preceding and following each scene. 59 people took part in the experiment, each of them rated perceptibility of swapped views for every scene from 1 to 5.

Two graphs show dependencies between channel mismatch perceptibility, movie budget, and release date.

Due to a large number of clear films, the trends are improving, but the “outliers” (even in 2D-3D conversion) are still encountered.

The swapped views scenes more often occur in low budget films.

28 May 2019
See Also
Call for HEVC codecs 2019
Fourteen modern video codec comparison
Automatic local color correction in S3D video
Stereo video may contain a huge color discrepancy. Most of the problems are hard to eliminate because of possible different distortions in each area in the frame.
Automatic sharpness mismatch detection and compensation in stereo
Algorithm works frame-wise: for a pair of source image's views it outputs whether it contains significant sharpness mismatch or no.
MSU 3D-video Quality Analysis (MSU Video Quality Measurement Tool 3D Project: VQMT3D)
MSU 3D-video Quality Analysis. Report 9
MSU 3D-video Quality Analysis. Report 8
Site structure