茄子直播

'Experience what characters are feeling': 茄子直播 researchers use AI to add '4D' effects to movies

Photo of researchers Makarand Tapaswi and Yuhao Zhou
Fourth-year computer engineering undergraduate Yuhao Zhou (right) and 茄子直播 postdoctoral fellow Makarand Tapaswi are using neural networks to automate older films into 4D movie experiences (photo by Ryan Perez)

James Cameron's 3D film Avatar sought to revolutionize the movie-going experience when it was first released in 2009, creating an immersive world for viewers. But what if you also wanted to feel the heat and the wind, while flying on a banshee, direct from your cinema seat?

While a small number of so-called "4D" movies that add a physical element already exist, researchers from the University of Toronto are working on a way to apply the feature more broadly. 

鈥淯sually the chair will shake, there can be splashing or some other kind of interaction while watching the film,鈥 says Yuhao Zhou, a fourth-year undergraduate in the Edward S. Rogers Sr. department of electrical & computer engineering, of the emerging entertainment. 鈥淩ight now all these effects are created from the first phase of production. We鈥檇 like to automate this kind of process for movies that were not originally created for 4D cinemas.鈥

Zhou is working with Makarand Tapaswi, a 茄子直播 postdoctoral fellow of computer science, and Sanja Fidler, an assistant professor at 茄子直播 Mississauga鈥檚 department of mathematical and computational sciences and the tri-campus graduate department of computer science. They recently had their work, , featured in a spotlight presentation at the Computer Vision and Pattern Recognition (CVPR) conference in Salt Lake City, Utah.

Zhou says a 4D movie is usually perceived from the first-person viewpoint, or camera. If Will Turner in Pirates of the Caribbean is feeling the wind blowing in his face, and the moviegoer wants to experience being Turner, then they, too, would have to experience wind in their face.

鈥淲e want to have a feature where you can just flip a switch and experience what characters are feeling,鈥 Zhou says.

To take a regular or 3D movie to 4D, the researchers used a freelance website to annotate the film鈥檚 effects for their 4D prediction model.

鈥淔or example, [in Lord of the Rings: The Fellowship of the Ring] Frodo pulls Sam out of the water, but there are several effects happening simultaneously,鈥 says Zhou, who began working with Fidler during his third-year of undergraduate studies. 鈥淔irst, he pulls him 鈥 there's a physical interaction with the hand. When Sam goes back down into the water, he pulls Frodo, and the boat shakes. 

鈥淭he camera is your input,鈥 adds Tapaswi. 鈥淏ut in this case you want to experience not only what the camera sees, but also one of the characters 鈥 relive how the characters felt shaking and so on.鈥 

While 4D technology is still out of the range of physical interactions 鈥 that is, a hand pulling 鈥 Tapaswi envisions pressure sensors to simulate touch as the technology advances. The model could prove useful in other areas such as virtual reality or augmented reality.

鈥淲e're collecting these types of annotations for future studies,鈥 Zhou says.

For their dataset, they applied both effect classification and detection. For effect classification, Zhou says their neural network, a function of machine learning that allows deep analysis and learning of data, extracted features from a short clip, including movement and audio. For detection, he says, the neural net can predict what the effects are, and where they occur, in a long video clip.

鈥淵ou don't only want to know what happens to a character in a particular shot. You want to be able to say, '[the effect] is wind now, not only because I see the wind right now, but [because] it was probably windy before,鈥欌 Tapaswi says.

The researchers found certain genres of movies tended to share similar effects 鈥 for example, movies set in space like Interstellar or Gravity. This can be seen as a novel way to cluster, says Tapaswi.

鈥淯sually with 3D movies, film-goers wear glasses and sit in a chair,鈥 says Zhou. 鈥淲ith automatic 4D cinema, the neural network would process 2D and 3D movie information, feed it into the chair, and simulate the effects.鈥

Topics

The Bulletin Brief logo

Subscribe to The Bulletin Brief

Computer Science