Background Removal

Computer vision is about teaching computers to see. Trying to make a machine that sees – like a human or even a dog sees – is a challenge (see LiveObject demo), but some interesting technologies get invented along the way. Background Removal, just released in Microsoft Office 2010, allows an Office user very simply to peel away the foreground of an image, such as a girl standing in front of a curtain, ready to insert into a new scene. It is simple to use, thanks to the power of probability working inside the software.

 

Inside the computer, each picture is represented as an array of pixels, each of which has its own colour, and these dots of colour blend to form the picture that you see. Part of the principle of separating foreground from background is to group the pixels by colour, and then deduce what palettes of colours belong to the foreground and to the background. It turns out though that this principle of separation of colours is not strong enough to give really reliable results. The problem is at its most acute when the object is camouflaged against the background, as with a lizard lying on a stone. The foreground and background palettes are too similar in that case. An additional principle must be applied to obtain reliable  results. The principle of the coherence of matter, that objects tend to consist of contiguous clumps, means that foreground pixels tend to stick together. The principle is applied in the Background Removal tool ,using probabilistic reasoning, and results in a tendency to give solidity to the background and to the foreground.

 

For a broader view of how new ideas in vision and machine learning are being used in image and video editing systems, see i3l. For additional details of some of the science behind the demonstrations, see Discrete Optimization for Computer Vision.