How the net has become multi-sensory

Abstract

Today, people and computers inhabit parallel universes: People live in a sensory-rich, physical, analog world; computers live in a deaf, dumb and blind digital world.

However this situation is changing even as we speak. We're putting eyes, ears and sensory organs on our computers and our Inter-networks in absolutely unprecedented ways. We are now asking them to observe and manipulate the physical world on our behalf.

The Internet in bringing about such devices needed to come in multiple flavours depending on the application required. In essence these interfaces stimulate one of our 5 senses; usually sight, hearing, taste, touch, and smell.

We are seeing our computer mouse give the on-line shopper the ability to feel the texture of linen before buying, or allow a blind person "see" famous paintings for themselves. We can smell the fragrance of perfume attached to e-mail or use the salinity in our bodies to act as a cable connection to our integrated voice mail messaging system.

Internet Wireless technologies send signals directly into the body, augmenting the senses of the blind and the deaf.

Much of what I have written is not from the position of an expert. I have simply collected from better minds than mine. Some of the knowledge here is gathered from speeches, but most are the ideas from the web sites and books of people I consider to be visionaries.

What, a better mousetrap?

Mice have feelings too, thanks to new computer accessories that have made their debuts on our Internet computer systems. The first mainstream mice to transmit vibrations when a person scrolls over a button, onto a link or a picture on a Web page or passes the cursor over a pull-down menu have now been developed.

This is an entirely new category of pointing device. Up until now, mice have been used to provide input only, just button clicking and scrolling. These new devices go beyond that with a mouse that can provide the user with a "feel-able" feedback with tactile vibrations."

Whenever the cursor hits a link or pull-down commands a signal is sent to the motor, which then activates a vibrator, the motor and vibrator turn off when the cursor is in a neutral area. Vibrations can be disabled or minimised if they become annoying.

These mice are sparking new sensory developments for computers. Several E-commerce companies are developing software that tells a mouse to "feel" fabrics or materials, seen as a potential bonus for on-line retailers.

One of the biggest complaints of customers at on-line clothing stores, for instance, is that they can't touch items such as silk scarves or feel the weight of fabric such as fleece or denim before they buy. These new systems transmit combinations of vibrations that simulate the bumps of rough corduroy or the smoothness of a silk scarf.

Why would we want a computer mouse that vibrates?

As Web pages become information dense and jammed with hyperlinks in relatively small font sizes, it is becoming increasingly difficult to know exactly when to click the mouse.

Research indicates a ready audience already exists for a vibrating mouse with older people, who don't have an innate understanding of the mouse; children, who will like the novelty of the vibration and those with limited mobility.

Rehabilitation experts say these new gadgets could especially help people who have motor skill problems or sight impairment. People who have tremendous difficulty manoeuvring the mouse-driven Web.

How will it benefit the visually or hearing impaired?

Apart from the basic necessities, communication is probably the single most important thing there is to a human being. We use our eyes and ears to absorb both text and speech. But how does someone get along in the information-driven world of today if they cannot see or hear effectively?

The blind and visually impaired absorb information 60% by touch. Compared to sighted people this is their strength and advantage. Modern computer screens are based on complex graphic representations. The tool of choice to work in these environments has been the mouse. This left the blind and visually impaired with little or no access to the graphical computer environment.

This may have all changed recently when one manufacturer designed a mouse so complete that it allowed blind people to "see" the screen by touch and feel.

This device integrates to the ordinary computer mouse and tactile display functions in one device. This mouse is a special screen scanner-mouse. It contains three tactile displays each incorporating 32 rounded pins arranged in a four by eight matrix. These pins give feedback to the user via the screen cursor on the computer graphics, pixel by pixel.

By using three fingers, the blind can understand the curves and shading of the scanned screen. The tiny dots (pixels) that make up screen images are presented to the user through the differing pin heights. Each pin moves up and down on several height levels presenting four shades: White, Light Gray, Dark Gray and Black.

Other buttons on the top and side of the device provide user interaction with the computer through screen navigation, the sending of commands and the changing of device settings

One blind person stated that "This device has opened my eyes. Never before had I realised that the lines drawn by Picasso in a picture of a young girl could be as emotionally exciting to me as real physical beauty." Another, a soccer fan says "I have been a football fan for twenty years and now, for the first time, I can see the real picture of a football field and the positions of the players. I finally understand what the announcer means by ‘Upfield’ and ‘Penalty Box’."

Having a 32-pin matrix for each of three fingers makes it possible for the device to provide access to all standard alphabets rather than Braille. Feeling the patterns of pins aligned exactly the same way as on-screen pixels, makes it easy to learn and read a regular alphabet when compared to Braille code translations. Additional text software immediately translates both reading/writing systems and integrates the use of voice-mediation in the background.

This technology is now available and estimated to be around the US$5000.00 mark.

If we can see it and touch it, shouldn't we be able to smell it?

One company is patenting a system of electronic sensors that will be able to smell an environment and digitise the results. It is called Digital Scent Technology"

What is "Digital Scent Technology"?

Imagine being able to smell things using a device that connects to your computer. Like a new language, this technology is a new set of tools for self-expression, communication, and commerce. It includes software components and a peripheral device called the Personal Scent Synthesiser.

That being the case is Scentography the same?

Yes and no, Scentography is a new art form that integrates scent with other forms of digital media, such as games, DVDs, and web sites. Scentography adds an entirely new dimension to the web and other forms of media by letting you communicate with smells.

How does it work?

It is a speaker-sized computer peripheral device that attaches to the serial or USB port of the computer and plugs into a standard electrical socket. It emits naturally based vapours into the user's personal space. The device is triggered either by user activation (such as a mouse click) or a timed response.

Humans have thousands of receptors somewhere between the eyebrows that capture the chemical characteristics of particles that enter the nose. The brain then identifies the odour. In contrast this system uses 32 sensors to classify molecules within a half-inch of its airspace by shape, electrical charge, polarisation, etc. A microprocessor then logs the scent "fingerprint" and pattern recognition software assesses its similarity to previously detected smells.

Apparently it can not only sniff out a grain of rice from other materials, but can tell you where that grain of rice was grown and it's type. In place of the human brain, it catalogues its odours locally or connects to the company's Internet scent database to identify a smell.

Markets for the device are said to include food quality monitoring, chemical identification, food processing, manufacturing quality control, and non-invasive medical applications. It is expected to hit the market at about five dollars. Current comparable technologies known as gas chromatography currently cost about $70,000

Why would anyone want digital scent technology?

The device produces odours-on-demand in response to users' actions or situation. For example killing an alien in the computer game Quake will send a request to a scents database where the smell is indexed by chemical make-up and its location in the smell "spectrum." The "smell code" of alien carnage is retrieved from the database and sent back to the device, which "plays" it for the pleasure and sensation of the game players.

Perhaps you are a Movie junkie and are watching a car-racing event on the Internet. Expect to get the full compliment of burning rubber and high-octane racing fuel wafting by your nostrils.

The manufacturers claim their technology will let doctors monitor their patients remotely for certain diseases (particles in your breath can reveal maladies besides halitosis) and allow refrigerators to detect when foods go bad.

If you are one of those folks who gets pleasure out of receiving letters from their beloved with half-bottle of perfume doused on them then you're in luck. In a blast of back to the future you may now bring scents of love to your E-mails as well.

The company also expects Web sites will begin adding branded smells to home pages and banner ads. Eventually individuals will create their own odours and register them in the first-ever database of smells. Imagine the subliminal advertising possibilities! Banner ads for fast food etc

How are all these scents created for the user?

Much like the palette for mixing and printing colours, this method is using a scent palette for designing scents and incorporating them into media presentations. The software instructs a device to emit one or a combination of basic scents.

Is this really something people will be interested in?

Well, it is said that every time we breathe, we smell. The sense of taste is almost 100% smell. That's why when we have the flu we can't taste our food. Most cosmetics, soaps, and cleaning products contain fragrances. Human beings have forever used spices to improve the taste of food. Men and women have sought to improve their allure with perfumes.

Are there any adverse effects?

If you are allergic to something that also has a particular odour (say grass pollen), it is possible you may associate that odour with the unpleasant allergic reaction (usually hay fever or asthma) and thus develop an aversion to that smell (e.g. freshly cut grass).

What are some of the less obvious benefits?

The inability to smell is a condition called anosmia. People with anosmia have lost the ability to smell or were born without it. They may have lost their ability to smell from head injuries or the aging process. Research has shown that for these people life is less stimulating and can lead to depression

Losing your ability to smell can signal the onset of worse diseases, including diabetes, hypertension, malnutrition, and some degenerative diseases of the nervous system such as Parkinson's disease or Alzheimer's disease.

Researchers are even looking at loss of smell as a predictor of developing AIDS in HIV positive patients.

Explain Personal Area Network (PAN) and Intrabody Signalling.

The Personal Area Network or PAN, in effect, transforms the human body into a copper cable, taking advantage of the natural salinity of the human body, which makes it an excellent conductor of electrical current. Having a data transmission rate of a 2400-baud modem (compared to a reasonable 33600-baud), it may not be enough to carry audio or video but more than enough to carry identification, financial or medical information.

While it is increasingly easy to send data into satellite orbit, getting the data the last few feet to where it is needed has proved cumbersome. By changing the average voltage of a body by a tiny amount, it is possible to send data through a person, creating a Personal Area Network (PAN). This lets familiar gestures be associated with logical meaning: exchanging business cards through a handshake, authenticating an identity by touching a doorknob or downloading messages by picking up a telephone.

By shaking hands or identifying another individual in the same region a computer could then pull up names and backgrounds of people. It then feeds the information to the user. Gone is the embarrassment of forgetting a name or important details about someone's position in a company. The system needs to access the Internet quickly, in two seconds or less, a sort of unspoken limit beyond which people don't want to wait for information.

Wearable computers

A definition:

The concept of a wearable computer is a device that is always attached to a person (ie. that can constantly be carried, not like a personal stereo), is comfortable and easy to keep and use, and is as "unobtrusive as clothing".

Another thing we should realise is that everyone will use wearable computers in different ways. For some it will be a necessity to give "hearing" to those completely deaf or a new way to "see" or at least move without colliding into walls.

This same ability can amplify a human in ways we can only imagine, perhaps a form of constant wireless communication or super database-like memory. Some will retreat inwardly in a virtual world fantasy that will scare those who visit.

We're discovering how to do things like sew circuitry using electrically conducting threads, so we can put the chips right into fabric and it looks and feels like ordinary fabric, but is actually a computer. We can embed displays in what look like ordinary glasses

In the medical arena, one company offers a shirt with embedded sensors that can send vital-sign readings back to a computer in a doctor's office. The system can be adapted to track life signs in infants, the elderly, firefighters, scuba divers or anyone whose health requires monitoring.

Jogging shorts are coming equipped with tiny computers that monitor your heart rate and notify you if you're starting to overdo it (jogging that is).

Airline mechanics are starting to wear computers linked to display glasses so they can read from a repair manual while keeping both hands free.

The future.

Under the trend of miniaturisation, we are seeing systems become more embedded into the body rather than actually being worn on the body. Interfaces developed for the disabled could make their way into consumer electronics: More than just a hearing aid; Ear Implants could be used instead of earphones and Prosthetic retinas could be used to replace HMDs for instance.

Cochlear implants are being developed to help the deaf as they attempt to replace the lost functionality of damaged or missing hair cells. Here wearable electronics attempts help by going into the root of the problem, where the senses system receptors are damaged but the underlying nerve structure is still intact

More than just integrating some of the electronic devices into clothing, the next-generation smart clothing will act as an interface between the body and the external world. Clothing will perform functions according to the body's needs, requirements and which adapt to the environment.

These could be relatively 'simple functions' like Identifying and memorising the different objects people carry with them everyday. So you might expect clothes can give reminders to take with them your keys and wallet etc. Or your clothes will perform temperature control mechanisms in a smart jacket that adjusts its temperature according to body temperature or outside temperature.

Integrating the wearable medical into a bio-sensor layer to monitor body factors such as heartbeat, blood pressure, temperature will lead to a "virtual doctor" layer, which would assess the overall health of the individual, and, make emergency phone calls on the wearers behalf should things get out of control.

A shirt that reminds us that we forgot such and such, that remember our feelings, emotions and memories or that perform first aid in times of emergency

Such clothing in the future may adapt to a user feelings, moods, aches and pains etc by monitoring brain activity.

Arthur Hissey
Computer Research & Technology
www.crt.net.au