Cell-phone manufacturers are seeking new functionality to differentiate their mobile handsets and bring about a host of newer and richer user experiences. Motion processing is emerging as the only technology that can deliver such promise.
Motion processing technology can bring about a next-generation gesture-based user interface (UI) for menu navigation. It also can enable mobile authentication, enhance location-based services, deliver an immersive gaming experience, and improve camera image stabilization (IS).
But delivering this credibly isn’t possible in existing solutions based purely on three-axis accelerometer sensors or even by adding a compass. However, a six-axis motion processing solution using microelectromechanical-systems (MEMS) technology can be used to enable emerging motion-based applications in the new generation of smart phones.
Mobile Phones Sense Motion Today
Apple’s introduction of the iPhone demonstrating tilt control enabled by MEMS accelerometers in 2007 raised the bar for motion-based applications in mobile devices across all platforms. According to iSuppli, 27% of the cell phones produced in 2009 feature a MEMS accelerometer, due to their inclusion in the iPhone and other popular smart phones.1
The accelerometer detects motion when the phone is turned from portrait to landscape and changes the display accordingly. The three-axis MEMS accelerometer is also used to control some rudimentary games, perform power management, shuffle music, “undo” actions, and enable pedometers and other context-aware apps.
While accelerometers provide basic motion sensing for simple orientation and tilt applications, they have limitations affecting their operation and performance in more complex applications, such as handwriting recognition or image stabilization where up to 20 Hz of hand jitters must be detected.
Continue to next page
Accelerometers can only deliver the sum of linear and centripetal acceleration, gravity, and vibration. They are at their best when movements are static or close to 1 Hz. Extracting a single element of linear motion information from the accelerometer is not feasible without an additional sensor, such as a gyroscope.
Apple’s iPhone 3GS and the Google Nexus One adopted the digital compass to provide yaw, and the accelerometer detects tilt to compensate the compass. The digital compasses used in these handsets use a low-cost Hall-Effect sensor architecture. But these sensors have magnetic sensitivity that is so low, it severely impacts their performance.
Other compass architectures may be better, but they are larger and cost more. In addition, a good compass is limited in how well it can measure heading due to interference from magnetic disturbances in the device and in the outside environment. With resolutions measured in a few degrees and slow response, they can only provide general information about rotation and direction.
Also called angular rate sensors, gyroscopes measure how quickly an object rotates. Gyroscopes are the only inertial sensors that provide accurate, latency-free measurement of rotations without being affected by any external forces, including magnetic, gravitational, or other environmental factors. This rate of rotation can be measured along any of the three axes: X (roll), Y (pitch), and Z (yaw) (Fig. 1).
Adding gyroscopes to the sensor mix allows algorithm designers to take advantage of a pure measurement of angular velocity that cannot be delivered by compasses or accelerometers. This measurement allows for accurate pitch and roll measurements when combined with an accelerometer, and these measurements can be used to more accurately tilt-compensate the compass.
The introduction of silicon MEMS-based technology has allowed the development of new MEMS gyroscopes that are suitable for achieving a challenging industry cost target of less than one dollar per axis, while meeting the package size and the appropriate sensing accuracy to become suitable for mobile phones, game controllers, remote controls, and portable navigation devices.
Continue to next page
Improving The User Experience
In addition to tilt control from accelerometers, the smart-phone user experience has been improved with a touchscreen UI that lets users flip menu pages by sliding a finger or zoom in and out of the viewing area by “pinching” on the screen. The touch-panel UI eliminates the need for cumbersome and tiny keypads on the phone, but it creates inconveniences such as the need to use two fingers to operate (e.g., zoom by pinching) and the presence of fingerprint smudges on the screen, affecting viewing quality.
The combination of the accelerometer and gyroscope creates a more user-friendly UI based on gesture recognition. Gesture recognition implemented on motion processing-enabled devices permits users to input commands or data by moving, shaking, or touching the device (Fig. 2). Additionally, picture zoom in and out can be accomplished with a single hand using gesture recognition UI to detect slight tilt movement where the gyroscope plays a critical role.
Smart phones use the accelerometer as a tilt sensor by isolating the signal due to gravity to reference orientation. But this signal can only be isolated when the accelerometer is motionless. When there is movement, linear and centripetal accelerations will corrupt the output. This makes any tilt-based UIs or applications extremely unstable.
A common compromise is to low-pass filter the accelerometer to isolate tilt. This makes a correct assumption that, in human movement, linear accelerations will be transient and can be filtered out. Unfortunately, it also makes the incorrect assumption that tilting movements are all slow. As a result, faster changes in orientation will also be filtered out.
A low-pass filtered accelerometer makes an extremely poor tilt sensor for a UI. Combining the accelerometer with a gyroscope produces a rapid, accurate, high-bandwidth tilt sensor. Gyroscopes can be used without accelerometers in applications that require dynamic tilt but not the fixed reference of gravity (e.g., the zoom control).
Continue to next page
Gesture recognition will enable other more sophisticated cell-phone use cases. A cube-based menu can be created and accessed by hand motion (Fig. 2, right). Or, the phone can be answered by two or three shakes. Short cuts can be created by recognizing numbers or characters such as hand-waving a letter “C” to turn on the camera (Fig. 2, left) or a number “2” to initiate speed dial. Even the hand-waving “signatures” can be captured in high fidelity for identity authentication.2 Basically, gesture recognition UI gives users the freedom to easily interact with their smart phones with only one hand.
In 2006, the Nintendo Wii revolutionized mainstream console games by adding a three-axis accelerometer to the game controller. This new generation of console games introduced primitive motion sensing without controller tapping or shaking.
To overcome the limitations of accelerometer-only motion sensing, Nintendo introduced the Wii MotionPlus accessory with integrated three-axis gyroscopes in 2009. According to Wii game developers, the improvement of motion sensing brought by the addition of a gyroscope is analogous to “the leap from the fidelity of VHS to that of Blu-ray.”3
A similar paradigm shift is expected in the mobile-phone handset market. In 2007, Apple included an accelerometer in the iPhone and opened up a new world of mobile gaming, introducing fun, engaging, and interactive applications such as virtual golf, racing games, and music instrument players.
The deficiency of motion sensing by accelerometers, however, is more noticeable in mobile gaming than in console gaming. For instance, when the player is walking or in a moving vehicle, the linear acceleration will degrade the system’s ability to sense the player’s real motion. To further improve gaming quality, adapting gyroscopes to mobile phones will be a natural future trend.
Location-Based Services (LBS)
Now that GPS is becoming a standard function in mobile phone handsets, the anticipated trend in location-based services and the next killer application for smart phones will be turn-by-turn car and pedestrian navigation. With LBS, users will be able to use their phone to quickly locate the desired store, or particular goods in the store (mobile commerce), or friends in the neighborhood (mobile social networking).
Also, store managers (or mall managers) will be able to post advertisements on the phone to attract customers or even collect fees for advertisements. For example, real estate sales can become mobile transactions with real-time location-based data easily downloaded to the handset.
Continue to next page
LBS software applications first appeared on the market in 2009. Wikitude4 is one of the pioneers, identifying the user’s location by GPS and the heading by compass, presenting users with data about their surroundings, nearby landmarks, and other points of interest via “augmented reality” (i.e., overlaying information on the real-time camera view of a smart phone). Wikitude was first launched in the Android Marketplace and is now available in the iPhone App Store. Yelp5 has similar software that shows the location of a restaurant or bar, for example, and displays online reviews of the business.
Several activities are in progress to deliver a more complete LBS solution. To improve location resolution and the accuracy of conventional GPS, augmented technologies such as differential GPS (DGPS), assisted GPS (A-GPS), and enhanced GPS (E-GPS) have been proposed. Companies are working on digital maps for pedestrian navigation.
New technologies such as Wi-Fi triangulation have also been developed. Signals from gyroscopes and accelerometers can integrate with a compass sensor to detect a user’s nine-DOF motion and provide precise alignment of augmented reality information with the camera view to deliver improved pedestrian navigation performance.
Variety Of Applications
As motion processing technology keeps advancing, smart phones will be used in more situations that are beyond current common phone use cases. In September 2009, Fujitsu announced a mobile phone application called ETGA Swing Lesson that records and analyzes the user’s golf swing movement by reading the accelerometer and magnetometer data from the mobile phone clipped on the user’s waist.6
Gyroscopes could improve the accuracy of the captured body movement and detect subtle actions such as the turn of the wrist. The user can hold and wave the handset and practice the moves anywhere. And the benefit that motion processing brings to learning and education is not limited to sports. Once the augmented reality technology is deployed to school classrooms, students will be able to use their camera phones to view virtual 3D models in science, art, biology, or interior design classes.
Health care is another area where motion processing can contribute to advanced applications. With high-sensitivity gyroscopes, the mobile phone will be able to detect minute body vibration and thus measure heartbeats or electrocardiograms when users are simply holding the device in their hand or putting it in a chest pocket.
Motion processing also will improve personal safety. For elder care and emergency 911, pedestrian dead reckoning technology can be used to track the user’s location. The phone also can be set to automatically dial a hospital or 911 when elderly users walk to dangerous areas or stay motionless after a period of time.
Continue to next page
Engineers adopting six- or nine-axis motion processing functionality within handset applications face the choice of either assembling gyroscopes and accelerometers from multiple sources or selecting a fully integrated solution from a vertically integrated motion processing supplier.7 There are merits and challenges to each approach.
Before selecting a motion processing product, designers should consider seven interoperability points to maximize the value of motion processing functionality when the design includes multiple applications such as image stabilization, gesture recognition, mobile gaming, authentication, and navigation:
• Each application requires different gyroscope full-scale output and data sampling rates.
• Anti-aliasing measures must be employed to ensure motion data accuracy through the use of low-pass filtering that is specific to the particular app.
• Accurate timing data is essential to determine the angular data calculations of the gyro through mathematical integration.
• The drive, sense, and harmonic frequencies of gyroscopes should be designed so they don’t interfere with each other or any other frequencies within the system.
• Synchronous sampling of the accelerometer, gyroscope, and digital compass data will ensure high-quality position coordinate information.
• To alleviate the computation and communication loading of a host system, motion sensors should have enough processing power to integrate multiple sensors and to extract higher-level motion information (e.g., shake, tap).
• Handset designs are very sensitive to size, power consumption, and cost. The motion processing solution that provides the smallest IC footprint, least power consumption, and lowest cost will be the most advantageous.
The InvenSense MPU-3000 three-axis digital gyro offers on-chip intelligent motion processing capability (Fig. 3). Measuring 4 by 4 by 0.9 mm, the gyroscope easily fits into any cell-phone platform.
The digital motion processor (DMP) is a custom-designed digital signal processing unit embedded inside the MPU device. It receives gyro signals (internal) and accelerometer signals via a secondary I2C interface to perform advanced sensor fusion and motion processing.
Today, there is no gyroscope IC that integrates with other types of inertial sensors. But as MEMS technology progresses, it is expected that there will be a single-chip six- or nine-axis IMU for the handset market.
Continue to next page
Advances in MEMS technology have significantly improved the size, cost, and power consumption of inertial sensors, enabling the handset market to create motion-aware applications. With the combination of the gyroscope, accelerometer, and compass, a highly accurate, robust, and latency-free motion processing solution can be established to revolutionize the way people use mobile phones.
A gesture recognition-based user interface will enable users to access their phones in a very easy and intuitive way most of the time with only a single hand. The inclusion of gyroscopes has raised the accuracy of motion capture to the degree that air signature authentication is possible. It also has brought the user experience of motion-based mobile gaming to a whole new level.
Further, motion processing will play an important role in the booming mobile commerce and LBS service application market. Higher-precision position estimation can be achieved, and better augmented reality experiences will be provided.
1. “MEMS Market Brief, December 2009,” iSuppli
2. D. Sachs, et al., “Intelligent Computer System and Techniques for Character and Command Recognition Related to Human Movements,” USPTO 61/259,288
3. “Hands on: Nintendo’s Wii MotionPlus,” techRadar.com, www.techradar.com/news/gaming/hands-on-nintendos-wii-motionplus-587634
4. “Wikitude, world browser,” www.wikitude.org/world_browser
5. “Yelp for iPhone,” download.cnet.com/Yelp-for-iPhone/3000-2379_4-10863636.html
6. “Fujitsu Develops Golf-Swing Analyzer Featuring Latest Sensing Technology,” www.fujitsu.com/global/news/pr/archives/month/2009/20090928-01.html
7. S. Nasiri, D. Sachs, and M. Maia, “Selection and integration of MEMS-based motion processing in consumer apps,” www.planetanalog.com/features/signal/showArticle.jhtml?articleID=218401148
Steve Nasiri, CEO, founded InvenSense in 2003. He has an MBA from Santa Clara University, an MSME from San Jose State University, and a BSME from the University of California, Berkeley. He also has more than 50 patents, issued and pending.
Shang-Hung Lin, system engineering director in the Handheld Business Unit,obtained his BS from National Taiwan University and a PhD from Princeton University. He currently holds 20 patents in the fields of image processing and pattern recognition.
David Sachs, senior advanced application development engineer, has designed many gyroscope-based systems. He has an MS from the Massachusetts Institute of Technology and a BA from Oberlin College.
Joseph Jiang, vice president, leads the Handheld Business Unit and the Imaging & Custom Business Unit. He has a BS in electronics engineering from National Chiao Tung University in HsinChu, Taiwan, and an MS in electrical engineering from San Jose State University.