Roberto Pieraccini, chief technology officer of SpeechCycle, has been at the leading edge of spoken dialog technology for more than 25 years, both in research as well as in the development of commercial applications. He has a doctorate in engineering from the Universita degli Studi di Pisa, Italy. Also, he holds several patents and has authored more than 120 publications on the subjects of speech recognition, language modeling, language understanding, and dialog. He is a fellow of ISCA and IEEE as well.

We love using our smart-phone applications. It seems there’s an app for everything. Even my apartment building in New York City has an app that enables me to leave instructions with the doorman, check whether my laundry has been delivered, and submit an urgent repair request. Phone, wireless, and cable service providers have their own branded apps that let you check a bill or manage mobile service features anywhere and anytime you want.

Article Tools

• 

As mobile services are becoming more complex, so are smart-phone apps. We’re reaching a point where we need a better way to understand how to use them and learn the extent of their capabilities. Equally important, we must know how to find which part of an application or which resource shared on the Web, such as a specific entry in an FAQ list, would answer a particular question without having to check a manual or help section. This is especially true for service provider apps, considering we only use them occasionally.

A well-designed hierarchical menu can handle the complexity behind a customer care application on a smart phone. But depending on the complexity of the provided services and features, there may be two problems when taking this route. First, there’s—again—the complexity. A three-level, five-choice menu, say, could lead to 125 different choices. But often the number of choices may be higher than that, which would require increasing the number of menu levels, or number of choices at each menu, or both. 

This increase in complexity—it also frustrated users of interactive voice response (IVR) systems—leads inevitably to the second problem: the arbitrariness of certain choices due to the increased complexity of the number of choices. For example, in a department store app, the total number of possible choices depends on the total number of different items carried (easily in the tens of thousands). 

For users to select an appropriate item, they would have to follow a categories menu. While the menu choices may be obvious for certain items, such as “men’s shoes,” others have ambiguous placement. If you’re looking for a “satellite car radio docking kit,” you may have to search in either automotive, appliances, or electronics. The placement in those categories is quite arbitrary, since the item may belong to each. You may end up navigating through a complex menu to find the item.

Here, the burden of locating something is the user’s, but it would be desirable to have the system assume that burden. Search engines are designed precisely for that purpose. (By the way, categorization systems, like those attempted by Yahoo and other portals a number of years ago, inevitably failed.) But users won’t always know how to describe what they’re looking for. Take, for example, their own mobile service. They might be able to spell out this: “I don’t know how to activate international roaming on my telephone,” or “My battery charge does not last more than six hours. What can I do?” but they wouldn’t necessarily know where to look for the answer. Fortunately, there’s “semantic search,” which allows the transformation of language sentences into one or more symbolic elements that can be used to locate what’s needed to solve a problem. Semantic search allows the transfer of additional data or asks users for other missing information, helping them clarify better. From a user’s point of view, semantic search may not look different from a traditional Web search.

In fact, a search box that can be filled with free-form text would be the iconic symbol for such a tool. Additionally, with the recent advent of mobile voice search capability, the search text may entered—with the click of a microphone icon—by speaking it. However, when it comes to things like mobile service, a traditional hierarchical menu will always be available for easy-to-locate items, like the current bill or account information.

T3, as in Touch-Text-Talk, is how we refer to this new form of multi-modal interaction suited for smart phones. It will enable users of smart phones to interact with a semantic search engine using the modality they prefer; they can type, text, or talk, or navigate a traditional menu by clicking on the items. The availability of sophisticated mobile devices makes T3 a reality that can become a new interface for applications. Overall, as consumers get increasingly dependent on mobile devices, developers need to ensure they’re delivering apps that will utilize all of these capabilities.