Disney Research Creates Keying Algorithm, AI See 'n Say

Math erases green-screen details

LOS ANGELES and ZURICH—Disney researchers say they have come up with mathematic methodologies to key out green-screen remnants and to associate sound with images.

Disney research scientist Tunç Aydin noted that software compositing tools have a hard time discerning the green-screen when foreground colors mix because of motion blur, “intricate object boundaries such as hair, or colors cast onto an object from the green screen itself.” Such instances require “painstaking” touch-up work, he said.

Disney’s methodology relies on a “color-unmixing” algorithm that extracts multiple colors as layers using a color model of the image. The model is created through an interface in which an artist “scribbles” over distinct colors. The algorithm picks up the color and computes the underlying colors and mixing ratios for each area. A subset of these colors is then used to produce a final keying result.

The method requires one-tenth the time of manual processing, and can also be applied to non-green-screen backgrounds. “Interactive High-Quality Green-Screen Keying via Color Unmixing” will be presented at SIGGRAPH next summer in Los Angeles, and was said to be “published in the August issue of ‘ACM Transactions on Graphics’ journal,” (although no August issue appears in the journal’s archive).

On the sound side, Disney researchers have come up with an artificial intelligence version of the venerable See ‘n Say from Fischer Price.

“Given a picture of a car, for instance, their system can automatically return the sound of a car engine,” the related press release said. A system that can identify the sound of a vehicle, breaking glass, or a slamming door could be used in multiple applications such as sound effects or audio feedback for people with visual disabilities, said Jean-Charles Bazin, associate research scientist at Disney Research.

Researchers took video with audio tracks, filtered out extraneous or uncorrelated sounds, and developed a sound-to-object association algorithm.