Nvidia Creates Surprisingly Smooth Super Slomo Video Technology

Slow-motion video is a little difficult to achieve on most common cameras on the market. Some flagship phones offer this feature, but are usually limited in size, resolution or frame rate, and are also hampered by the limited storage of  mobile devices for such large files.  

On the other hand, applying slow-motion effects to previously recorded videos will generally produce unpleasant results with unnatural movements as a result of the software trying to fill in the frames of the original video. However, Nvidia, along with researchers from the University of Massachusetts and the University of California, has created a solution that could make it possible to turn any video into a slow-motion video without sacrificing the smoothness of playback.

The technology, which will be presented at this year's Computer Vision and Pattern Referencing conference - which takes place this week - features two convoluted neural networks (CNNs) that work together to determine where objects are moving between frames and position. in which they will be in the intermediate frames. VentureBeat describes how the two CNNs work together:

A Convolutional Neural Network (CNN) estimates the optical flux - the pattern of movement of objects, surfaces, and edges in the scene - both forward and backward in the timeline between the two input frames. It then predicts how the pixels will move from one frame to another, generating what is known as a flow field - a predicted motion 2D vector - for each frame, which merges to approximate a field of flux to the frame intermediary.

A Second CNN then interpolates the optical stream by refining the approximate flow field and predicting visibility maps to exclude occluded pixels by objects in the frame and subsequently reduce artifacts in and around moving objects. Finally, the visibility map is applied to the two input images, and the intermediate optical flow field is used to distort them in such a way that one frame smoothly transitions to the next.

The researchers used the Nvidia Tesla V100 GPUs and the PyTorch deep learning framework accelerated by the cuDNN to train the system with 11,000 videos recorded at 240 frames per second, after which it could fill in the missing frames in slow motion video.

The technology produces the results you see in the video above, which looks surprisingly smooth for an artificially generated effect, even on videos with only 30 frames per second. The company also worked with the YouTube The Slow Mo Guys channel to test technology on high frame rate videos such as 240 frames per second. In addition, the technology can be used to slow down videos at any time, although, presumably, decelerating videos takes longer to populate all the intermediate frames of the video.

No matter how promising the technology is, Nvidia does not believe it's ready for the consumer market, since it needs a lot of optimization before it can run in real time and even if it reaches consumers, most of the processing will have to be made in the cloud. With that being said, the technology is certainly interesting and could bring slow motion video to a lot more people sometime in the future.