The algorithm used to determine the vector displacement needs to be reasonably sophisticated since it must correct for camera shake, while preserving camera panning and/or the motion of objects within the frame. Fortunately, camera shake is a relatively high-frequency jitter compared with panning and object motion. Therefore, objects can be distinguished by the application of filters (Fig. 1). Other frame-to-frame artefacts like dynamic range, saturation, and color balance, as well as visual effects like diffusion and sepia tones, can be corrected or applied in the same way.

Far more challenging to correct are the aforementioned person-related defects. The processing limitations of mobile platforms make some shortcuts necessary. Take, for example, the fact that humans are very face-centric. If an image contains a face, we lock onto that face first. Studies show that provided the face is well presented—in focus, and properly exposed and colored—then we’re easily satisfied with the whole image. This means only the pixels occupied by the principal faces in the image require processing, which is obviously far less intensive than the complete frame.

The first challenge of face-based imaging is to identify the faces in the scene.  Mathematically this isn’t a trivial exercise, not least because of the diversity of faces, further complicated by profile, glasses, hats, earrings, and other factors. In this case, the shortcut is not to try and identify the faces in each frame, but rather track the location of faces from frame to frame, which reduces the data handling to a simple vector.

In a hardware-accelerated algorithm, core sub-routines common to multiple software algorithms are implemented in hardware and then made available to the overlying software. This combination achieves fast execution with low memory requirements while retaining the flexibility and adaptability of a full software algorithm.

Once the faces are located, many interesting features become possible. Skin tones can be smoothed. Flaws like wrinkles, spots, and freckles eliminated or toned down. And, on the whimsical side, faces can be morphed and warped (Fig. 2).

Continue on next page