NPAC Interface Lab Activities 1996

In early 1996, _Eyal Sherman_ joined our research team and worked with us in developing and testing a number of expressional interface approaches based on technologies from the _I3_, including two electronic interface packs, called TNG-1 and TNG-2 (pron. Thing-1 and Thing-2), that connect to the serial port of a PC. TNG-1, which is battery operated, processes myolectric (muscle-electric) signals using analog and digital filters. TNG-2, more generally, accommodates variable voltages from a wide range of sensors (potentiometers, photocells, strain gauges, thermistors, Hall Effect transducers, etc.); TNG-2 uses the 5 volt line from the serial port as a power supply for its internal circuitry and for the transducers. The TNG devices were designed and built by Salamo ________ and Dave Warner.

The associated software consisted of a mature DOS-based program called Neat Software and a partial Java prototype called _NeatTools_. Both of these were written by Joe Johannsen in close consultation with Dave Warner (NeatTools was developed under an NPAC contract).

During the Spring semester of 1996, using TNG-1 and self-adhesive electrodes, we (Dave Warner, Edward Lipson, and Adrian Blanarovitch) provided Eyal with resources that enabled him at home to produce some music and to operate a remote-controlled car by means of his facial expressions. The car was operated via the parallel port of the PC and a modified Radio Shack (Tyco) hand controller (acknowledge Patrick Keller, Markus _______, and Salamo). Eyal and family used a Pentium PC provided by NPAC and communicated with us via e-mail using an NPAC account we set up for the family. This also provided them free access to Web resources. Together we created the _Eyal's home page_.

In the summer of 1996, the project made significant advances in both hardware and software with the formation of a team of four REU students (Anna B, Adrian B, Rahul P, and Shital S) as part of an NSF funded program at NPAC. The students were co-supervised by DW and EL and assisted by two graduate students, Jiangang Guo and Myeong-Jin Lee. Technical support was provided by Lou Buda from the Physics Dept. using the resources of the physics shop.

The hardware part of the work used five types of _headpieces_ (http://www.npac.syr.edu/REU/reu96/www/sshah/hardware.html) and associated sensors for Eyal. With these, Eyal was able successfully to play PacMan (using Neat software and a modifed Sega game controller) and Tetris (Java version via NeatTools), and to do some text typing using a novel text triangle design by which any letter could be typed by three successive gestures (Java/NeatTools). The sensors that we used included self-adhesive electrodes (for TNG-1) and for TNG-2: a) cadmium-sulfide photocells (depending on expressional changes in reflected light from Eyal's cheeks and chin; these used either ambient light or small light bulbs) and b) a Hall Effect transducer that sensitively detected displacement of a small magnet. We also did some exploratory work with bifurcated fiber-optic light guides.

A full description of this work is contained in the _final report_ by the REU team. This report also describes some prototype work on a Java-based mouse-cursor-control algorithm.

This work with Eyal and others will be continued during the 1996-97 academic year at NPAC and other sites and will be featured at _SC'96_ in Pittsburgh (Nov. 96) and _MedVR (MMVR)_ in San Diego (Jan '97).

Among the refinements needed for this project are:

• quick-connect hardware (it currently takes about 30 minutes to rig up the various headpieces to the wheelchair headrest and to Eyal)
• less obtrusive hardware (we'd like to miniaturize the sensors and holders so that they could be would be less visible and conspicuous)
• more sophisticated signal processing (currently the facial gestures are recognized in a simple binary fashion by means of threshold detection for each channel; we want to access analog and timing information that would allow many more channels of information to be detected for each sensor)
• a wider range of capabilities for Eyal and other users (mouse-cursor control; Web surfing; rapid typing; speech generation from typed text, as done by _Stephen Hawking_)
• simplicity, convenience, modularity and standardization so that users and their families can easily manage these low-cost technologies
• (_Pulsar_ philosophy)

We want particularly to acknowledge the major contributions and diligent effort by Eyal Sherman and his family in making this project a success.