The Future, as it Looked from 1987

An oldie but goodie has resurfaced on YouTube, at least for people who were in Boston in 1987 and attended MacWorld Expo (or were followers of Apple Computer, as it was called back then). At that event, then CEO John Sculley showed a ‘The Knowledge Navigator’, a short film produced by The Kenwood Group for Apple that communicated his (and Apple’s) vision of what it would be like to work with their products in the future. He didn’t say how far in the future it was, but it was clearly a time we would be able to relate to.

Besides the 100% correct prediction that we would be concerned with the deforestation of the Amazon area and its effects on the level of CO2 in the atmosphere (although it didn’t go the next step to foretell that this and other human activities would wreak havoc through climate change*), The Knowledge Navigator also demonstrated some aspects of computing that did in fact come to pass, others that are not quite there yet, and a few things that we probably won’t see for a long time to come. For those who weren’t lucky enough to see it in 1987 (or would like to take a brief walk down memory lane), here it is:

Here’s what they got absolutely right:

Touch Screen Interface
This year seems to be the year of the touch screen, whether it’s Apple’s own iPhone or Microsoft’s Surface computer (see previous posting). Tablet computers that use a stylus for input have been around for quite awhile, but the intimacy and immediacy of a finger as an input device seems to have truly taken hold in 2007.

Voice Synthesis
Despite the fact that the demo computer’s synthetic voice is a bit smoother than today’s synthetic voices, we are getting pretty close to this. (I myself went the extra mile to purchase a voice that sounds a bit like a Butler with a British accent that my computer uses for alerts and other notifications.) There are times when many have called for some service over the phone and mistaken the synthetic voice on the other end of the line for a human.

Here the film hits a home run and almost exactly the way it was shown. In fact, in a few hours I’m going to be using it to talk to my parents on the other end of the continent, back in the US. Next year, the example of the other professor sharing a screen with the local computer will be quite common, as Apple’s iChat AV will then include not just videoconferencing, but presentation and screen-sharing.

Telephony Integration
While not widespread, talking on the telephone through the computer is growing by leaps and bounds. There are several packages for integrating voice mail using the computer as the processor and storage medium for messages, and the widespread adoption of VOIP (voice over Internet protocol) will only hasten this trend. The iPhone’s ‘visual voicemail’ is surprising in that it does exactly the opposite of the demo (lists voice mails as messages, like an email), rather than announcing calls missed or the caller ID of an incoming call.

Intermingling of Professional and Personal Data
While this may be seen as a trivial detail, it’s worth remembering that in 1987, only true geeks kept their calendar and to-do lists in their computers, and everyone else continued to use day planners, faxes, snail mail and post-it notes for much of their personal organization. Only with better screens, faster processors, and software that got smarter and better designed about these tasks (and much of it has quite a ways to go), did the marginalia of personal lives find their way onto personal computers in offices. Still, this trend isn’t finished yet, and one can still find plenty of Daytimers and postits in most homes and offices.

Here’s what they were a little too optimistic on (although not completely missing):

Speech Recognition
The demo goes to great lengths to show that the professor is mumbling, giving partial information, and not talking to his device as a computer, but as another human. At one point during his telephone conversation, he even pauses and the computer fills in with an appointment time, as if it had been following in on the conversation! While speech recognition on computers has reached the point where it is common to be able to dictate an email, letter, article, or even a book to your computer and have it type the entire thing out as you speak it, one still has to speak clearly and the computer still does make some mistakes (although a lot fewer than it used to). I once saw a great T-Shirt slogan on someone who was on the team at Apple working on this technology: “I helped Apple Wreck a Nice Beach” (If you didn’t get it, say it out loud).

Full Colour Mapping of Geological Data
Google Maps and Google Earth, along with mashups of data from a variety of sources has started, but there are few simple, easy ways of doing a geo-plot with any data source without some massaging of the data (although this may not be as much the case in certain circles). There are hundreds of companies in the field of GIS (Geographical Information Systems), and it’s possible that some standards for interoperability will start to arise, especially with the rise of XML-based data sources. RSS feeds of Real Estate listings have already been mashed up, and traffic management and weather forecasting have both employed computer-based mapping for years. It’s only a matter of time before geophysical data is also available to the general public in an easy-to-assemble format.

Form Factors and other Hardware details
While there has been a move by most computer users to laptops from desktops, there are still a number of things in the demo (Speech recognition, perhaps) that are doable on a desktop but that most laptops still don’t quite have the horsepower for. This will change in the next 2 to 3 years (if not sooner). It’s surprising that there is no keyboard visible anywhere, but that is probably more to make a point (as is also the case with Microsoft’s Surface demos). Also, there is the conspicuous (at least for me) absence of wires for the connection to the phone, and no clear microphone. While wi-fi connections to the Internet are a reality, a wireless (voice) telephone interface built into a laptop would probably only work for Skype or some other VOIP solution. Still, perhaps there is an interesting potential product for people who want to use their computer as a cell speakerphone with omni-directional microphone.

Here’s what we are still a long way from:

Artificial Intelligence
Wow, did they ever miss on this one! AI is the flying car of computing. The idea that there would be an ‘agent’ as they referred to him, living in your computer, responding to you intelligently and with knowledge of both you and the outside world… well I suspect we are still a few decades away from this. There are many points in the demo where I said ‘Nice try’, imagining how today’s technology would make a mess of searches, taking calls, connecting you to data, etc.

Still, that’s not a bad hit rate for 20 years. John Sculley should give himself a little pat on the back for some impressively accurate vision, and the computer industry (and Apple in particular) should be pleased that a lot of this has come true. Now, it will be interesting to see when the next generation of college professors (and others) use these tools as much and as easily as the actor in the demo did.

*At the very least, one has to give Sculley credit for being one of the very first people in public life to sound the alarm about this ecological issue. It’s very appropriate that 20 years later we find Al Gore on Apple’s Board of Directors. It’s also perhaps a little ironic that Al Gore’s office setup — 3 huge screens plus a dedicated flatscreen TV, mountains of paper and bookcases filled to overflowing) is absolutely nothing like the climatology professor in the video.

Surface: Microsoft’s Sexy Coffee Table

I’ll bet that there are days at the Microsoft campus when some groups are working on a super-secret project that they just wish they could tell the world about it. In fact, keeping their secret must drive them crazy. Especially when some other company comes out with a similar project to the one they were working on, or shows off some feature in one of their demos that seriously steals some of their thunder. Time to market is part of the game, and when you lose the race, it hurts. That’s probably how the group who were working on the project called ‘Surface‘ felt some months ago when Steve Jobs made his spectacular iPhone demo, particularly when he showed that product’s new User Interface called Multi-touch. Multi-touch, at least the way that Apple defined it, means that a device can tell when you have one or two fingers touching a touch screen, and behaves differently depending on how those fingers interact. For the iPhone, this means that you can tap and double tap for some behaviors, tap and fling to move a screen or scroll a page, or touch with two fingers and move them toward or away from each other in order to zoom in or out on an image. It’s that last one that Microsoft’s new product has, and I’ll bet they were gnashing their teeth and grimacing with each ooh and ah from the crowd as they reacted to Jobs’ demo at MacWorld last January.*

But I’m getting ahead of myself here. Surface, besides all of that multi-touch stuff, is a strangely exotic and futuristic product. Most folks would call it ‘bleeding edge’. It’s frankly not something I expect from Microsoft. When I think of Microsoft, I think of Windows™. Windows is not bleeding edge. It’s market-tested, well worn, doesn’t take chances, and is definitely not exotic and futuristic. It runs on hardware that is getting cheaper by the day, and most of the time that hardware is, well, ugly (with a few exceptions from Sony and maybe one or two others).

Surface is none of that. It’s a 30 inch acrylic display with touchscreen built into a rather austere-looking coffee table that’s 22 inches high, 21 inches deep and 42 inches wide. There’s no keyboard and no mouse, although it does have Wi-Fi, Bluetooth and wired Ethernet connectivity. The main way that you interact with it is by touching the screen. Actually, Surface is designed to work not only with more than one finger touching at once, but with more than one person interacting with it at once. Oh, and yes, in case you forget, it’s a computer running Microsoft’s own Vista OS (Windows is still in there somewhere!) and will probably cost somewhere between $3,000 and $5,000. (Update: I just found out that these numbers are $5,000 to $10,000. No big surprise there.) What remains is the question of what you actually do with a $10,000 coffee table touchscreen computer with Internet and wireless connectivity.

Here are some examples of what Microsoft has in mind: (click on these thumbnails to see a larger image).

Multi-touch in Action Multi-User Computing

In the Fact Sheet on their web site, Microsoft says that they will ship Surface (and yes, this is a shipping product) “… to partners with a portfolio of basic applications, including photos, music, virtual concierge and games, that can be customized to provide their customers with unique experiences.” The web site for showing off Vista has several video demos, ranging from happy-smiling-people (a term I learned from my days at Fidelity that refers to those models you see in business ads who always seem to be having a better day than you are) arranging digital photographs, planning a trip, playing cards, and interacting with their cell phones as they place them on the glass table-top display. One of the demos that seems just a little unrealistic has a little girl using Surface to paint a cute picture. I’m sorry, but even though the top 1% of the very wealthy in the US are getting wealthier, it’s hard to imagine anyone but the Gates families and a few others that can afford a 10K electronic paint toy for their kid, much less place it in the living room. What did look the most interesting, however, were the instances where someone was interacting with one or more cell phones, allowing people to download trip information into them by dragging the information into a box that stood for the phone on the screen or moving music tracks from one cell phone/audio player (iPhone? Zune?) to another. The one that made me think ‘Hmm, I don’t think they’ve thought through the social dynamics of this one…’ was an example where a group of diners in a restaurant split the bill and dragged their portions to their cell phones. It might be nice to be able to split a bill with perfect math accounting for each dish ordered, but somehow having that appear on the screen felt a bit…awkward? I suppose it’s no worse than the waitress splitting up the bill, but can you imagine the tug-of-war that might ensue when one or more diners tries to pay for the other?

The fact is, this is a product that is probably going to be seen in casinos and some upscale restaurants and hotels first. The living room will have to wait. I have to say that I like that Microsoft is thinking outside the box (or rather, the desktop or laptop), and some of the applications do look fun. Will this catch on? I’m not sure. For a long time, people were hot on touchscreen kiosks for some of this activity, and they never really took off, and I can’t really see Surface working as a real restaurant table:

Oh, I spilled my Curry sauce all over the computer screen…
Don’t worry sir, I’ll just wipe it up, oh, whoops, sorry to order that round of champagne…
Oh, how cute, little Dylan is playing Blackjack between courses…Oh, don’t stab at the table with your fork, sweetheart…

Damn that messy physical world, full of food, klutzes and kids getting in the way of our cool software technology!

I used and designed interfaces for a touchscreen years ago, and I was struck then by the intimacy of the interactions. Rather than move a cursor via a physical proxy (the mouse, trackball or trackpad), you can touch a part of the screen, and sometimes that piece of the screen would change, just like in the physical world. The examples that Microsoft showed had this happening most of the time. You touch something, it either ripples, moves or highlights. The Surface UI is meant to be more than simply a new display, form factor and method of input, it’s a different style of interactivity that looks like the iPhone writ large (and for two or more people). I’ll bet it just kills the Surface team to hear that.

*It’s worth mentioning that Jeff Han demoed a surface interface at TED in 2006, and much of what he demonstrated is reflected in Microsoft’s new product. Han’s demo also got oohs and ahs, but it has only been seen by a relative few, where the hype for the iPhone definitely went farther into the mainstream media.

UIs in the Movies


Forbidden Planet

A recent paper by Michael Schmitz, a student at Saarland University (a fascinating institution near the borders of Germany, France, Luxembourg and Belgium), surveys several different user interfaces from science fiction movies throughout film history. Human Computer Interaction in Science Fiction Movies cites several films, including Metropolis, Johnny Mnemonic, The Matrix, Logan’s Run, Forbidden Planet, Gattaca, Minority Report, Total Recall, Star Trek: The Next Generation (one of the movies, I assume), X-Men, 2001: A Space Odyssey, Dark Star, the 2002 remake of The Time Machine, and even Galaxy Quest.

It is intriguing the way Schmitz has organized the UIs he is taking note of:

The movie clips in the main section of his survey are categorized according to their area of real-life applications and research:
Neuro Technology: Technologies that connect to brains are introduced here.
Identification: Electronical identification of individuals
Displays: Various kinds of displays as an output medium
Speech: Including speech synthesis and recognition as well as intelligent assistants/avatars as special subgroup.
Other I/O technologies: All technologies that are more specialised and that were difficult to categorize according to the fields above, for example gesture recognition or tangible user interfaces

I think these categories arose because of the scenes that he chose to include. There are perhaps some other movies that were not chosen that may have brought in other areas of classification (for instance, Tron, Blade Runner, Serenity, any of the Star Wars films, The Fifth Element, Ghost in the Shell, any of the James Bond films, 12 Monkeys or Contact).

The fact is, the UIs that we often see in the movies are not always accurate because they aren’t necessarily built for good usability, but instead for good dramatic effect. Sure, Tom Cruise could have done some of his research in Minority Report using a mouse and an LCD screen, but it was so much more dramatic for him to don VR gloves, and perform the exhausting 3D manipulation in virtual space in front of him. Even everyday, mundane events like receiving an email get full-screen 3D animations in the movie Disclosure with Michael Douglas and Demi Moore in 1994.

Nevertheless, movies do have a way of placing imagery and expectations in the mind of many movie viewers. In fact, because a surprisingly large number of people can’t tell the difference between movies and reality, some of these UI inventions may be adopted sooner after they are developed for real, everyday use. What James Bond sees on his computer screen may very well be built by some small startup determined to bring just such a tool to market, at whatever price the public wants to pay. They know they want it, though, because they saw it work just fine at the theatre.

Navigating Data Quickly and Easily

Infosthetics points out another great UI demo. Moritz Stefaner, a User Interface Design researcher at the University of Potsdam has created what he calls an ‘Elastic List’, which is a way of browsing data that has multiple facets or parts to it. In this example, you can quickly located winners of any Nobel Prize by zeroing in on the Prize type, the gender of the winner, their country, the decade they won the award, and once you’ve narrowed down to the decade, the year that they won. In a few clicks I was able to find the two Male Canadian winners in the 1990s. Not only does this demo have a nice way of showing the data (note that the ‘pay dirt’ for a search is photos of the people you are trying to find), but it also has a marvelous bit of Mac OS X-style animation that shows the lists expanding or contracting as you make choices:

Elastic List - Partial View

It’s done using Flash, but I bet with a little work it could be done in AJAX.

Besides allowing you to zoom in on the data quickly, it encourages you to explore it to look for what might be unusual aspects that you might not have known. For instance, I learned that Germany has not had a Nobel prize winner in Medicine since 1908, which indeed, surprised me.

What’s the Best UI for Twitter?

The Twitter Logo

I’ve become fascinated with the new web application called Twitter. It’s a way of simply and tersely updating others on your status: This makes it kind of like a cross between a blog, instant messaging and perhaps SMS (cell phone text messaging). Twitter is like a blog, because it broadcasts your thoughts, moods, impressions, and other personal mumblings to the world at large. It’s like instant messaging because it consists of short messages, no more than 140 characters in length. It’s like SMS messaging because twitters (or is the singular a ‘tweet’?) can be recieved on a cell phone (and I believe you can update your own status via a cell phone, although this is much easier and cheaper in the US than Canada or overseas). Oh, and your twitter status updates only go to the people who choose to ‘follow’ you (and likewise, you only get updates from those people who you know and want to be updated about. Your posse is your update audience, and you get their updates as well.

To use twitter, you bring up the Twitter web site, and if you have an account and have left a cookie that logs you in automatically, you make a twitter/tweet by typing your message into the form. To help you stay under the 140 character limit, there is a countdown javascript that tells you the number of characters you have left. It’s not that inconvenient or confusing, but I keep wondering if there’s a better way.

I suppose that using an SMS cell phone to update my status would be best, but even then, typing is a problem. Perhaps there could be a special twitter application for phones and PDAs (or both, like my Treo or a Blackberry), with set phrases that you could use by pressing buttons or the option to insert your location, if it knew you were near a particular place that you are often located at, like Work, Home or a friend’s residence. The key here is that twitter has a different User Model than the web, or Instant messaging, or even perhaps SMS.

A User Model, as I define it, is everything about a given situation that a user experiences when they are accessing or operating software or hardware, including their posture, how much time they have, their level of comfort (or discomfort), what else they might be busy doing, the amount of attention they can/want to dedicate to the activity, etc.. A lot of software assumes that you are sitting in a task chair, have a keyboard, mouse or trackpad and a monitor of decent size, that you have a block of time to dedicate to the activity you are engaged in and you can devote nearly your full attention to the task at hand. Some software assumes that you are connected to the Internet, but other (non-web, of course) packages don’t. In contrast, Instant Messaging always assumes that you are connected continuously to the Internet, are a pretty fast typist, and can devote all or perhaps part of your attention to the conversation you are having. A web application called ‘Do I Need a Jacket’ (or is designed to be used just as you are just leaving your home or office. You look at it for a second, perhaps from a standing position and peering over at your screen, and it remembers the last place you put in for it’s ‘setting’ (you can also change the thresholds for cold, chilly or wind speed that trigger a yes or no answer for the question of whether you need a jacket or not). One click is all it needs once this is set up. SMS assumes only that you have your phone or Blackberry with you and that it is turned on. It also assumes that you don’t have a full keyboard (although the remarkably good keyboard on the Blackberry has begun to change this a bit).

Twitter needs only a little bit of attention (like IM), but it is required sporadically, like an incoming SMS or IM. Like a blog, it requires that you think about yourself, or at least what you want to say, but unlike a blog, you don’t have to be a writer (or even be able to write a full sentence!) Twitter can serve some very useful purposes, like letting a bunch of friends know where you are if you are meeting up later, or receiving constant status reports (even from RSS feeds, like the local weather), or perhaps even some applications we haven’t thought of yet.

At any rate, this is a new kind of application, with some very interesting challenges related to the User Model as well as the role that online messaging can have in our lives. For a version 1.0 it’s crude, but then again, IRC (Internet Relay Chat) was also pretty crude as well, and look what that gave rise to (AOL chats, Minitel, AIM, Gtalk, iChat AV, Meebo, and who knows what else?!)