According to a recent 30 page US patent filing, Apple has been tinkering with a multi-touch system that will integrate visual and voice data into the control systems for its devices.
Filed by Wayne Westerman and John Elias, co-founders of Fingerworks, a firm that had been acquired by Apple prior to the launch of the first generation iPhone, the patent purposes a variety of secondary inputs to incorporate with existing touch gestures under the title, “Multi-Touch Data Fusion.”
Pairing data from sensing adaptations with fingertip gestures could provide significantly enhanced user functionality. There is no doubt that some operations are best accomplished with senses other than touch, as the filing suggests that in certain circumstances, voice can be used to permute certain properties of an object. A visual editor would allow a user to freely speak out commands that can effect the item that is being touched. For example, a word processor could allow a user to select and drag text by touch and speak out font changes, or voice commands could alter an objects size, rotation and color.
While today’s multi-touch technology is efficient at detecting human fingers, it is limited in determining which fingers are making contact. In some instances, knowing precisely which fingers are performing certain actions can be imperative.
Wayne Westerman notes, “Finger painting, where each finger has an assigned color, stroke, or other characteristic, is a simple example of an application that would be significantly enhanced compared to the state-of-the-art by using finger identification with multi-touch data fusion,” adding “For example, if the index finger of the left hand is assigned the color red and the other fingers are assigned different colors the application must be able to determine when the index finger of the left hand is in contact with the surface in order to paint red. Conversely, the application must be able to determine when the red-assigned finger is not in contact with the surface. The fusion of finger identification data with multi-touch movement data allows the application to function without error.”
The filing also notes that this technology would lend itself to permit an iSight camera or visual sensor to recognize facial intentions, such as a users gaze at a certain object on screen to determine what operation to perform. Even the users facial expressions can be recognized as a user’s frustration could tell a computer that it’s misinterpreting a command. For example, an inexperienced user may become frustrated with with the learning process involved in performing certain tasks.
Westerman writes, “Say that the user is trying to scroll through a document using a two-finger vertical movement (gesture). Scrolling, however, is not working for him because he is unknowingly touching the surface with three fingers instead of the required two…He becomes frustrated with the ‘failure’ of the device. However, in this case, the system recognizes the frustration and upon analyzing the multi-touch movement data concludes he is trying to scroll with three fingers. At this point, the device could bring the extra-finger problem to the attention of the user or it could decide to ignore the extra finger and commence scrolling. Subsequent emotional data via facial recognition would confirm to the system that the correct remedial action was taken.”
A users grip on the device could also generate data that would provide additional information. This could include modifying an object that is being touched by the angle of the device itself, as well as hand and finger size, and even data from from body temperature and heart rate.
Such biometric fusions suggest “typical applications that might benefit from the fusion of biometric data with multi-touch movement data would include games, security, and fitness related activities,” Westerman wrote. “Hand characteristics such as size, shape, and general morphology can be used to identify an individual for the purpose of allowing access to secured areas, including computer systems. While hand characteristics alone would not provide a sufficient level of identity verification, it could be the first door through which a user must pass before other security measures are applied.”
[via AppleInsider] [via Electronista]
