By now, probably everyone who cares, knows that Nokia is working on a next generation, Apple Multi-Touch like user interface - Nokia S60 Touch. But beyond pretty video pictures and generic headings like ” new sensor framework”, “UI accelerator kit”, etc; Nokia is pretty tight lipped about how the new S60 Touch will actually work.
And then there’s the issue of iPhone and Apple’s patents, which seam to cover lots of Multi-Touch UI related things.
Well, today I’ve got my hands on one interesting Nokia Touch User Interface patent, filed in June 2007, about six months after the iPhone announcement. And I’m impressed.
Not only Nokia has found a way around most of the things that Apple may get IP protection on, they seem to be on the way to develop user interface technology, that goes well beyond touch.
How about the multi-touch like mobile handset UI, that does not actually require touch, can track your hand movements in three dimensional space and even recognize 3D hand gestures?
Traditional Nokia S60 multi-touch features
But lets start with the traditional multi-touch implementation by Nokia.
After the iPhone announcement, instead of trying to reinvent the wheel, Nokia engineers went shopping for the alternatives to Apple’s multi-touch technology.
Since iPhone relies so heavily on a capacitive multi-touch display and covered a lot of ground here with public and not yet public patent applications, that was out. One of the touch technologies that Nokia settled on, was Active Matrix LCD with Integrated Optical Touch Screen.
They have licensed AMLCD technology developed by Planar inc., covered by a couple of patents, threw in some electronic perception capabilities covered by these three patents from Canesta Inc., and went on to develop their own multi-touch device.
What they’ve got is AMLCD technology based touchscreen that is able to recognize multi-touch gestures quite well. In this implementation, the device has optical sensors embedded throughout the display, that can identify fingers placed on it and record their movement.
These recorded movements are then translated into control gestures using pattern recognition methods. Various one, two or multiple finger gestures can be used to perform different operations. Among the examples provided in patent application are:
Single finger based Gestures/commands
- Clockwise, counter clockwise circular rotations - browsing, scrolling listing applications
- Subsequent typing by a single finger (Tap1-Tap2…) - activate device/phone, run/execute pre-selected option
- Finger stays motionless (over certain time treshold) above some object/icon - select object/icon
- Finger stays above some item/object/icon, followed by slow movement - select object till end of the move
- Crossed perpendicular lines (X mark) - Delete
- Perpendicular moving breach (Check mark) - Accept/Verify
Two finger based Gestures/Commands
- Linear approaching/digression (fingers approach, then move apart, and vice versa) - Zoom In/Out, size adjustment
- Simultaneous touching of an icon/object by two fingers - Select the icon/object ready for size adjustment
- Simultaneous tapping by two fingers (Tap 1&2, Tap 1&2, repeated) - High level importance Acceptance & Verification
- One finger stays above an icon/object then object specific menu appears; the other finger performs circular rotations and toggles through the menu options, lifting simultaneously both fingers up selects and executes menu option
Compound Gestures/Commands
- COPY=S
- CUT= S
- PAST
These are just a few gestures that can be used with the new Nokia multi-touch UI. But that’s just a repetition and incremental extension of what Apple did a year ago.
What Nokia engineers thought of next, is much more interesting. It’s a device control interface, that can track and recognize hand and finger movements in three dimensional space.
Beyond Multi-Touch. Nokia S60 3D Touchless device control interface
To go beyond touch and create touchless 3D device control interface, Nokia decided to use Ultrasonic Transducers (USTs), arrayed around the perimeter of the display.
Ultrasonic transducers emit ultrasonic sound waves that propagate through air. When finger gets into the path of these waves, the signal is reflected back and detected by the transducer.
Three or more transducers (14 A-B-C in picture above) arrayed around a perimeter of the mobile device create a 10-20 cm working volume of space above the display, where user finger ( 20A and B) locations and their movements in real time can be detected using triangulation techniques. These movements can then be interpreted as various three dimensional gestures.
To ease the operation of the device, visual finger pointers (12A, B) are displayed on the screen of mobile device.
3D gestures are translated into various object oriented, or gesture/browsing oriented commands. The patent application gives several examples.
Object oriented commands:
- Select: - Picking up gesture - Finger 1 at a display corner or some reserved area, Finger 2 moves slowly under a displayed object to be selected
- Copy: - when selected, click by single finger on the object
- Paste: - fast double click by a single finger
- Move: - move slowly two fingers located on moving objects
- Delete: - double (fast) click by two fingers on previously selected object
- Switch: - switching (on/off) is based on change in directions of fingers movement or, alternatively, on a change in finger acceleration
Gesture/browsing oriented commands:
- Select object attributed to the pointer position: Open/closed hand
- Forward/Backward Browsing: anticlockwise/clockwise cyclic rotation by a single finger
- Zoom in/Out: expand/close two fingers
- Run/
In addition to operation of the device, 3D tracking can be used for text entry. In this case virtual keyboard is presented on the screen of the device, with visual and audio effects providing the necessary feedback to the user, e.g. symbols change color when a finger moves above them, become bigger and a click sound is heard when a key pressed and accepted.
Different finger combinations can be used also as a shortcuts to various applications. Make a letter C with you fingers and your calendar application launches, use a “V” sign to launch an address book. And if the operation of the device grinds to a halt and starts to really piss you off, just give him a “finger” and it resets to start anew 
Mini projector with USTs
And what about all the talk about backwards compatibility? What about all these older S60 smartphones, that could be upgraded to S60 Touch feature pack, but lack the necessary sensors to detect finger movements?
Nokia guys thought of that too. How about a small mini-projector, like the one Macrovision is showing off, with a system of 3 Ultrasonic Transducers and coupled to your smartphone via Bluetooth or UWB connection?
Yep. That’s all it takes.
And with an amount of memory, storage and computing power that is coming to mobile handsets, you may soon be carrying a full featured computing device in your pocket.
“It’s what computers may become”, indeed 
Is it real or not?
So, just how real and close to implementation the features described in this Nokia patent application are?
I think that the Multi-touch features using optical touchscreen are real enough to be the basis of the upcoming S60 Touch interface. We should see first devices with them within a few months.
Ultrasonic transducers and 3D gestures? I don’t know. Sounds pretty far-off and may not make it to the first generation S60 Touch.
On the other hand, during S60 introduction in London, Lee Williams, Senior Nokia VP, dropped enough hints about “… this being so much more then about an orientation and double taps, and even more then a Wii gaming type of options…”.
He was probably talking about the combination of touch sensors with accelerator and GPS chips. But then again, he might have been hinting at the upcoming UST 3D gesture capabilities as well.
