Method for region of interest selection with noticeable stereoscopic distortions in S3D videos

Introduction

Shooting 3D with two cameras without proper calibration causes geometric and sharpness distortions. The search of such distortions is manual and time-consuming. So, the special algorithm has been developed. It automates the process of fragments’ selection in stereoscopic frame containing the most noticeable geometric distortions and inconsistency of views in terms of sharpness.


Types of distortions

Types of distortions


Example of sharpness mismatch

Sharpness mismatch example


Example of color mismatch
Spy Kids 3D: Game Over, 0:14:19

Color mismatch example


Example of rotation mismatch
Drive Angry, #6167

Rotation mismatch example Rotation mismatch example


Example of scale mismatch

Scale mismatch example


Example of vertical disparity
Journey to the Center of the Earth 3D, #33964

Vertical disparity example Vertical disparity example

Proposed method

Algorithm scheme

Algorithm scheme

The algorithms of region selection for frames, containing scale, rotation and/or sharpness mismatch, were improved through machine learning methods.

Machine learning

Experiments

A dataset was created to train the model, which would predict the correctness of the detected region. The dataset was made by human experts who selected regions of interest, and consists of:

The results of classifiers (cross-validation and 95% confidence interval)

The results of classifiers C — regularization weight 𝛄 — kernel parameter

For each type of distortion, we chose the model that showed the best results. The model predicts the region that would likely be selected by an expert.

Results

To decide whether the machine learning model is better than the baseline algorithm, we marked 100 additional frames and conducted an expert comparison. Two regions with distortions were shown to each participant: one area from the baseline algorithm and one from the machine learning model. The participants were asked to choose which region was better.

Comparison of the base algorithm and machine learning model

Comparison


The most important features for scale mismatch

The most important features for scale mismatch


The most important features for rotation mismatch

The most important features for rotation mismatch


The most important features for sharpness mismatch

The most important features for sharpness mismatch

05 May 2020
See Also
Call for HEVC codecs 2019
Fourteen modern video codec comparison
Parallax range estimation in S3D video
Parallax determines the depth of S3D movies. The range of parallaxes should be both comfortable and entertaining for spectators.
Geometric distortions analysis and correction
Production of low-budget movies is prone to errors. Our method automatically corrects rotation and scale mismatch.
Automatic detection of artifacts in converted S3D videos
Our set of algorithms detects edge sharpness mismatch, cardboard effect, and crosstalk noticeability.
Temporal shift estimation for stereoscopic videos
How to take into account geometric distortions in the estimation of the temporal shift?
Neural network-based algorithm for classification of stereoscopic video by the production method
What method was used to create the 3D scene?
Site structure