Touchscreen technology makes today’s mobile devices much easier to use. It began with simple taps on a display. Now, most devices can decipher seven kinds of gestures for input: flicking, tapping, double tapping, pinching, pressing, tapping and sliding, and rotating.
There has been some recent controversy, though, as companies attempt to patent these gestures. Do these claims have merit? We believe these gestures precede any existing or valid and broad patent claims. We also believe that such fundamental gestures are in the public domain and are free to be used by all device manufacturers.
Optical mice, optical finger navigation devices, magnetic sensing input devices, and capacitive touchscreen controllers all rely on gestures for navigation and input. In particular, capacitive and optical touchscreen devices track the movement of one or more fingers or a stylus. Computers, cell phones, MP3 players, and similar devices all can use non-camillian pointers for user-input command.
Of course, it is possible that there are many patents combining autonomous gestures with touchscreens that have novel structures or processes. Still, significant freedom of action exists with respect to autonomous gestures used in combination with commonly known touchscreen architectures.
For a gesture to be most valuable and to have meaning, in the context of communicating with a computer or other computing device, it needs to have a commonality of meaning. That is, one product manufacturer cannot use a common gesture, such as tapping, differently than another product manufacturer since it would be overly cumbersome for consumers to use the same gesture differently for those devices. Inconsistent gestures also would frustrate users and slow down the adoption of gesture-controlled devices.
Fortunately, several gestures have become autonomous due to their early and frequent use. Some gestures go back to the early input device, the mouse, such as tapping the left button to select an item on the screen, tapping and holding the left button to select and move an item on the screen, or double tapping the left button to issue a command. How unfortunate would it be, then, if one product manufacturer could claim to have invented a common gesture and the intention of the user to communicate with a device, at the exclusion of all other companies?
Beyond the user-related issues of patenting gestures, the gestures themselves (when not in combination with an apparatus) are not patentable. Section 101 of the United States Patent Statutes does not mention gestures or motions as a type of patentable subject matter, only stating that “whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent.”
Furthermore, patent examiners at the United States Patent Office are required to follow the Manual of Patent Examining Procedures (MPEP) in which MPEP 2100, coinciding with Section 101, states that inventions that are broadly interpreted as encompassing a human being are not patentable. Thus, a gesture performed by a human being is not patentable.
An early gesture document, published in 2000, was a doctoral thesis written by Caroline Hummels for Delft University of the Netherlands. In her thesis, “Gesture Design Tools: Prototypes, Experiments, and Scenarios,” Hummels describes a substantial number of gestures used with a computer in the context of the use of gestures for interacting with a computer. She describes a user’s intent to use gestures to communicate to manipulate data in the computer or be displayed on the computer.
However, the seven autonomous gestures in combination with common sensing and input devices had been described in public documents many years prior to Hummels’ thesis. Therefore, this predates many claims that they have been allegedly patented for use with an input device. What is patentable, on the other hand, is the common gesture in combination with a novel improvement to a sensing device apparatus or a novel method for sensing such a gesture.
Unfortunately, news stories, press releases, and other claims have led many manufacturers and would-be manufacturers to believe that they cannot use autonomous gestures in their devices because, even in combination with a well-known sensing device, they are patented. They, therefore, erroneously conclude that a patent-infringement lawsuit is awaiting them when they introduce a product that uses basic input gestures.
Prior Art On Gestures
Avago Technologies defines prior art to describe gestures in combination with a touchscreen that do not extend to additional elements that combine gestures with input devices. Algorithms are generally stored in a touchscreen controller for recognizing the gestures at the touchscreen. After the gesture is detected, an operating system will typically send the gesture data to the application on the device that has the touchscreen. The seven autonomous finger gestures include:
|• Tapping gestures: quickly touching the touchscreen display to start an application or to activate an icon (Fig. 1)|
|• Double-tapping gestures: tapping the touchscreen twice in relatively quick succession to open or close an application or document (Fig. 2)|
|• Tapping and sliding gestures: bringing a finger onto the touchscreen and sliding the finger for moving an item relative to the display, or to pan through a page, map, picture, etc. (Fig. 3)|
|• Pressing gestures: placing and temporarily holding a finger on the touchscreen to open a document, application, or other function (Fig. 4)|
|• Flicking gestures: quickly moving a finger up, down, left, or right on a touchscreen to scroll or flip through lists of items or through pages of material (Fig. 5)|
|• Pinching gestures: setting two fingers on the touchscreen and either bringing them toward each other or away from each other to zoom in or out of a displayed item (Fig. 6)|
|• Rotating gestures: placing two fingers on the touchscreen and rotating the fingers clockwise or counterclockwise for similarly rotating an object in the display (Fig. 7)|
A rich source of information that describes these technologies has been available for many years in patent databases, and the prior description of gestures is no exception. Some gestures described in patents were recognized on a touch-sensitive screen well over 20 years ago, and the patents are now expired.
The tapping gesture is a simple one and may be the gesture known and described before other gestures. Several patents that are now expired describe the tapping gesture. For example, U.S. Patent No. 3,696,409 to Braaten describes the tapping gesture as recognizing the touching of a pad and transmitting that touch information to a computer so new information can be presented as a result of the touch.
This Braaten patent was issued in 1972 and expired more than a decade ago. Braaten did not specifically identify the tapping gesture as patentable, but instead described a system for the detection of the capacitance that results from the touch on the touch plate (using transparent conductive pads disposed on the surface of the display screen).
A second patent, U.S. Patent No. 4,550,221 to Mabusth issued in 1985, teaches the use of a touch-sensitive screen (capacitive) to generate signals identifying the location of the touch to control the location of a cursor or to make commands or provide control signals to a computer or other electrical equipment. Mabusth likewise recognized the use of transparent sensing pads, in addition to non-transparent. Even at this early time, multiple finger touches (up to four as described by Mabusth for the rotating gesture described above) and multi-tapping of a single finger were described.
U.S. Patent No. 4,686,332 to Greanias, et al., describes the use of transparent conductors arranged in a grid and adapted to a display for simultaneously or alternately sensing finger touching (tapping gesture) and stylus touching. A processor responds to stored program instructions to detect the touches. Already by this time, the simple tapping gesture was being described in the background section of the patent.
Two more patents that have also expired describe the tapping gesture include U.S. Patent Nos. 4,698,460 to Krein, et al, and 4,698,461 to Meadows, et al. These patents are directed to a device having a touch-sensitive surface that determines the location, via capacitance changes, where the surface is touched for responding to information on the screen or selecting a displayed item.
Thus, no one today can rightfully claim to have an active patent on the tapping gesture in combination with a general touchscreen. A protectable invention today, as in the case of all the fundamental gestures, would have to be based on specific improvements, for example, to the touchscreen technology.
U.S. Patent No. 4,455,452 to Schuyler describes a tapping and sliding gesture whereby a touch-activated system is responsive to information associated with a finger moving along a plurality of axes. The capacitance of the user’s finger is used to detect the sliding gesture, providing both location and speed information. Issued in 1984, this patent has been expired for nearly a decade.
Additionally, U.K. Patent Application No. GB 2 139 762, issued to Prosenko and also expired, was published in 1984 and describes a touch-sensitive surface for providing signals representing a user’s finger touching the surface for tap and tap and slide gestures by the user. A patent that expired in 2007, U.S. Patent No. 4,899,138 to Araki, et al., describes seven gestures (for ultimate use on an audio product) that could be detected on an optical touch panel. Here, the tap and slide gesture is described as tracking the finger moving up, down, left, or right, as well as pressing (described as KEEP), and double tapping (described as 2HIT).
U.S. Patent No. 5,347,295 was issued on September 13, 1994, to Rainey, et al., and titled “Control of a Computer through a Position-Sensed Stylus.” This patent is significant because it describes numerous gestures with detailed examples on a laptop computer and was published as early as May 1992 based on a PCT Publication Number WO 92/08183.
More specifically, the publication disclosed a notebook computer equipped with a capacitive touchscreen display that receives input from an electronic stylus. The notebook computer is “completely controllable through gestures and printed characters drawn on the display with the electronic stylus.”
In addition to disclosing the detection of different types of gestures or touches made at particular locations on or near the touchscreen display by the stylus (see claims 27 and 28 of the 92/08183 patent publication), numerous other gestures that are not included here, such as the seven fundamental gestures, are described.
The PCT patent publication explicitly discloses opening and closing applications with tap gestures, press-and-hold gestures to drag or magnify, double-tap gestures for opening and closing applications, flick-to-scroll gestures, rotation gestures, and the detection or recognition of such gestures by a computer system. Other portions of the specification and drawings of the 92/08183 patent publication disclose further details concerning gestures and gesture detection.
Another patent that teaches multiple gestures is U.S. Patent No. 5,157,384, also issued to Greanias, et al, at IBM in 1992 and, therefore, is now expired. In addition to giving a very thorough background regarding multiple input devices, including capacitive touchscreens, this patent also details how the gestures are detected and then communicated to the host computer.
Besides teaching the use of detecting tapping or press gestures to activate pop-ups, this patent is important because it teaches that the touchscreen can recognize a circle gesture for calling up a pop-up keyboard. Being able to recognize a circle is significantly different than recognizing taps and presses, and the rotate gesture is a simple variation thereof.
Non-Patent Prior Art On Gestures
Patents are an excellent source of prior art information for gestures, due in part to the requirement that claims must be “enabled” and significantly detailed in the specification. However, non-patent literature is also very rich in describing the seven autonomous gestures. Non-patent prior art includes presentations and videos that can present information in a manner that patents are unable to achieve.
A video presented by Jeff Han in February 2006 demonstrates a multi-touch sensor screen where Han demonstrates the tapping, pressing, tapping and sliding, pinching, and rotating gestures. Likewise, an article, “Artificial Reality II,” by Myron W. Kruger, was published in 1991 and discloses several autonomous gestures in the context of a touchscreen or within proximity of a screen.
In this article, a clear illustration is given of the tap gesture where a user is described as being able to touch graphic buttons to issue commands. Even at this early date, Kruger describes work known as VideoDesk where a user’s hands are on or proximate a work surface (a camera above the work surface monitors motions) and where it was known all gestures used could previously be accomplished with a mouse, joystick, or data tablet. Even more impressive was that the pinch gesture was described at this early date.
Bill Buxton, who is well known for his longtime work at Microsoft, presented a paper in the Society for Information Display (SID) Symposium Digest of Technical Papers, Volume 41(1) in May 2010 titled, “A Touching Story: A Personal Perspective on the History of Touch Interfaces Past and Future.”
In this paper, Buxton points to examples of the first touch input work that went all the way back to the 1960s. Here, a Casio watch was described as being available in 1984 and having a touchscreen for sensing input from a finger or stylus. Buxton is also able to point to multi-touch sensing going back as far as 1984.
Autonomous gestures have been reported on for more than 20 years. Likewise, multi-touch and proximity to the screen as well as the combination of autonomous gestures with touch or multi-touch sensing have been described for many years. Because gestures are so well known and because many original patents have expired, they cannot be patented, except in combination with some new gesture or some novel architecture.It is important for manufacturers who make and sell navigation input devices using touchscreens to recognize this fact about gestures so they can be free to act on the design of devices with autonomous gestures. The industry will only benefit if the common language of gestures can be used from device to device and by one manufacturer to another. Therefore, manufacturers should be wary of claims with broad patent coverage of any autonomous gestures in combination with a generally known touchscreen or multi-touchscreen.