MSU Shot Boundary Detection Benchmark 2020

Comprehensive analysis of shot boundary detection methods

This benchmark designed to evaluate different algorithms of the Shot Boundary Detection task. The benchmark provides an opportunity to test your own algorithm for detection of cuts, dissolves, wipes and fades scene transition.

What’s new

Key features of the Benchmark

To participate in our benchmark, please follow instructions in the How to submit section.

We appreciate new ideas. Please, write us an e-mail to sbd-benchmark@videoprocessing.ml

Leaderboard

The table below shows a comparison of SBD algorithms by F1 score and speed.

Click on the labels to sort the table.

Performance Charts


Per-video F1 score on different datasets

Choose dataset to see detailed perfomance

Dataset:


Speed/F1 score — resolution dependency

Methodology

Definitions

Video Dataset

Our dataset is constantly being updated. Now it consists of 120+ videos with a total duration about 1010 minutes with approximately 10500 scene transitions.

Our collection contains videos from popular RAI dataset, videos from MSU codecs comparison 2019 and 2020 test sets, and also videos collected from different sources (youtube.com, vimeo.com etc.). Our analysis has shown that groud truth in RAI contains imperfections, which we fixed in our collections. Dataset were extended by replacing every cut scene transition in these datasets with fades and dissolves.

Final dataset contains videos with different resolutions from 360×288 to 1920×1080 in MP4 and MKV formats. Videos include samples in rgb scale or in grayscale with FPS from 23 to 60.

Click to open spatio-temporal distribution and examples of video dataset
Detailed SI-TI of each scene in dataset

Select the area to zoom in

SI-TI metric:

Dataset examples

Evaluation Steps

1. Launches
We launch your algorithm with given options (or tune them if you would like to) and measure running time.
Important notes:
  • We expect the output of your algorithm to be in the following format:
    json or csv file that contains three keys or columns ー cuts, dissolves and fades.
    Each of these three must contain a list of frames with scene transitions according to the Definition Section
    Output example
  • If your algorithm depends on randomness (KMeans, for example) we will launch it several times
    to get the average of the outcomes.
  • We have time limitation so your algorithm must process more than 5 frames per second.


2. Calculation With given shot boundaries we calculate F1 score, Precision and Recall.

Here are the formulas for these metrics:



3. Verification We send you final results of your algorithm including metrics and visualisation on our video dataset.


4. Publication If you give us permission we add results to the main of our benchmark alongside with others algorithms.

How to submit

Send us an email to sbd-benchmark@videoprocessing.ml with the following:

If you have any suggestions or ideas how to improve our benchmark, please write us an e-mail on video@compression.ru.

Contacts

For questions and propositions, please contact us: deinterlacer-benchmark@videoprocessing.ml

28 Dec 2020
See Also
MSU 3D-video Quality Analysis. Report 10
MSU SBD Benchmark 2020
MSU Deinterlacer Benchmark 2020
MSU Deinterlacer Benchmark 2020
Detection of stereo window violation
How to find objects that are present only in one view?
Depth continuity estimation in S3D video
How smooth is the depth transition between scenes?
Site structure