Emerging methods for human-computer interaction [HCI; 1] offer revolutionary opportunities to advance healthcare and quality of life, particularly as the power, functionality, and affordability of computers continues to soar. In particular, the advent of wearable computers calls for new types of interfaces, since the users are typically not desk-bound. Further, for people with disabilities who are unable to use a keyboard and/or mouse, the need for alternative interfaces is compelling. Clinical environments can enjoy improved efficiencies and outcomes, as new ways evolve to interface patients, caregivers, and instruments to computers and networks.
Our group has been developing powerful, low-cost technologies combining modular software and hardware that accommodate expressional gestures and perceptual modalities as essential parts of the interface. These systems allow for adaptive rapid prototyping in which practically any input to the computer can be mapped to appropriate actions and outputs.
The NeatTools visual-programming environment allows rapid prototyping and implementation of HCI and other dataflow applications, in conjunction with custom sensors, mounting hardware, computer interface boxes (TNGs), and clinical/scientific instruments.
- NeatTools Software
- Interface Devices
- Sensors
NeatTools constitutes a visual-programming and runtime environment that produces fine-grain dataflow networks for data acquisition and processing, gesture recognition, external device control, virtual world control, remote collaboration, and perceptual modulation. The design goals of NeatTools have been to make it simple, object-oriented, network-ready, robust, secure, architecture neutral, portable, high-performance, multithreaded, and dynamic. The program and representative applications are downloadable from www.pulsar.org. NeatTools can readily accommodate custom interface devices, or commercial devices including clinical instruments. Figure 1 shows two simple NeatTools programs. For a full-fledged application program, see the section below on the JoyMouse Network.
NeatTools is written in C++ but built on top of a thin-layer Java-like cross-platform C++ application programming interface (API), which operates presently on Windows 95/NT, Unix (Sun), Irix (SGI), and Linux. In due course, Macintosh will be supported, once its multitasking, multithreaded operating system is released (note that it can run provisionally on a Mac-based PC-simulator, such as Connectix Virtual PCÔ ), along with appropriate C++ development tools.
Currently, NeatTools includes serial, parallel, and joystick port interfaces; multimedia sound; MIDI (Musical Instrument Device Interface) controls; recording and playback; Internet connectivity (sockets, telephony, etc.); various display modalities including for time signals; time generation functions; mathematical and logic functions (including a state machine module); character generation; and a visual relational database system including multimedia functionality. Keyboard and mouse events can be received or generated via Keyboard and Mouse modules. This allows, among other things, the user to control a graphical user interface by alternative input devices that in effect simulate keyboard and mouse events. Data types in NeatTools include integer, real, string, block, byte array, MIDI event, and audio or video streams. NeatTools allows the visual programmer to package a dataflow network inside a container module that constitutes a reusable "complex module" with simple overt appearance. This procedure can be iterated to accommodate several layers of hidden complexity.
NeatTools modules provide multithreaded, real-time support. Editing and execution are active concurrently, without need for compilation steps. This generally accelerates system design, and facilitates rapid prototyping and debugging. To construct a dataflow network, the user drags and drops modules (objects) from toolboxes to the desktop and then interconnects them with input/output and control/parametric lines. Properties of the desktop and many of the modules are set via a right-mouse-click. In this way, users are in effect developing elaborate interface programs without having to know C++ or the fundamental structure of NeatTools, or indeed having to write any textual program code at all. On the other hand, the system is open, so that experienced programmers can develop external modules by following instructions in an online developer’s kit. External modules can be loaded into the system at runtime, or arranged to preload automatically. The NeatTools executable development program, while massive in terms of source code (~40,000 lines of C++), is compact; the downloadable compressed archive file is about 600 kilobytes in size, so it easily fits on a diskette along with a compressed archive (under 100 kilobytes) of representative "*.ntl" files.
The system hardware consists of mounting components, sensors, serial interface boxes, computer, and optional output interfaces and devices. Our current electronic interface module (TNG-3; www.mindtel.com/mindtel/anywear.html) accommodates up to 8 analog and 8 digital (switch) sensors and streams the data at 19,200 bits per second to the serial port of a computer. Connections are made via standard stereo and mono plugs. The heart of TNG-3 is a programmable microcontroller integrated circuit [2], a type of computer-on-a-chip commonly used in industrial and office automation, and in automotive, communication, and consumer electronics under the general rubric of embedded control systems. The microcontroller in TNG-3 is programmed in assembly language for optimal performance. TNG-3 requires no batteries or wall transformer, as it derives 5 volt power for the onboard circuitry and sensors (requiring only modest power) by exploiting some of the unused serial-port lines—a technique commonly used to power a serial mouse on a PC.
Among the sensors we have used are switches, cadmium-sulfide (CdS) photocells, Hall Effect transducers (magnetic sensors), rotary and linear-displacement potentiometers, bend sensors, piezo film sensors, strain gauges, and custom electroconductive-plastic pressure sensors. Most of these sensors are inexpensive, some costing under a dollar and some costing but a few dollars. Certain types (Hall Effect and capacitive) require preamplifiers and/or signal processing electronics, which increase the cost, but not unduly.
The most substantive technical result of our work to date is the development of the NeatTools system along with the TNG interfaces and sensors, as described above. We have begun to apply the core technologies in a number of key application areas.
- Disabilities Applications
- JoyMouse Network
- Education, Rehabilitation, Telemedicine, and Defense Applications
- Education
- Rehabilitation
- Telemedicine
- Defense
For illustration, we describe the types of systems we developed for Eyal Sherman, a member of our team who, is a brainstem quadriplegic, unable to move his head or to vocalize. He is currently a senior at Nottingham High School in Syracuse. We have enabled Eyal to precisely control mouse motion, and thereby control graphical user interfaces, such as Windows 95. Eyal and his family have achieved independence in using this system; his mother is able to set up the hardware and software routinely in a matter of minutes.
The primary interface device is a chin joystick, extracted from an inexpensive game controller, mounted to a curved support rod, which is clamped in turn to the wheelchair headrest post, thereby allowing the device to be rotated away when not in use. To allow easy mounting and adjustment of sensors near Eyal’s expressive facial regions—mainly cheeks and forehead—an industrial designer on our team, Michael Konieczny, built lightweight adjustable mounts that attach to eyeglasses. Currently we are using small switches as the expressional sensors, but we have also used Hall Effect transducers (together with tiny rare-earth magnets) and photocells to detect facial gestures.
An application program demonstrating the considerable power of NeatTools is the JoyMouse dataflow network (Fig. 2), which Eyal and other youngsters with quadriplegia have been using with good results. For details, manual, images, and downloads, see http://www.pulsar.org/neattools/edl/joymouse_docs/JoyMouseManual.html. This uses a modest fraction of the channel capacity of TNG-3 (2 of the 8 analog inputs; and currently 3 of the 8 digital inputs). The JoyMouse application uses advanced features of NeatTools including logic gates, multiplexers and demultiplexers, encoders and decoders, various timing and mathematical operations, and sockets (here in "localhost" mode so that two windows on the same platform can communicate). The network is shown here both in developer mode and in user mode, wherein editing is blocked and only essential regions of the network are visible.
Figure 2 includes a graph of the available relationships between mouse-cursor velocity and analog-joystick displacement. For all three functions, there is a dead band, or free-play zone, near the origin so that the mouse cursor is not subject to jitter when the joystick is physically at rest. The linear relation offers essentially proportional control. The nonlinear relations—quadratic (necessarily inverted for negative displacement) and cubic—offer fine control for up to about half-maximal displacement, and rapid travel with larger displacements. In most applications, the cubic function offers the best performance. Various parameters (pertaining to gain, resolution, etc.) can be set or modified using sliders while remaining in user mode.
The network also accommodates input from three switches: a) a left cheek switch for left-mouse-button; b) a variable-use right-cheek switch for right-mouse-button, enter-key or backspace-key; and c) a forehead switch to dynamically select action mode of the right-cheek switch. Alternatively, the switches could be replaced with analog sensors for which thresholds would be set with sliders within the JoyMouse program. Calibrator modules are included in the JoyMouse program, as in many other NeatTools programs, to automatically adjust to the signal range for analog inputs.
The network as shown can be minimized, once the Enable button has been activated, so that the operating system desktop becomes fully available to other application programs while the JoyMouse runs in background. An optional small satellite window (Fig. 2), a related NeatTools application, can remain visible to display the state of essential options that are under dynamic control of the user; this is made possible by using socket modules to communicate locally between the JoyMouse main window and satellite window. The user can toggle, for example, between mouse click and drag modes by a "smile" gesture (both cheek switches activated for 1 second).
By using this the JoyMouse in conjunction with low-cost commercial utility programs, including an onscreen keyboard (Fitaly™ from Textware Solutions, sometimes with their InstantText™ program for word/phrase prediction and abbreviation expansion), Eyal has been able to a) type text, b) generate speech, c) dial in to a server, d) invoke and use Web browsers and other application programs, e) compose and send e-mail messages, f) play video games alone or with others, g) operate remote controlled cars, h) draw sketches, and i) participate in science experiments and data analysis at school. Other people with severe disabilities have also successfully used the JoyMouse system and other applications with good success, for example children with cerebral palsy.
NeatTools has many possible applications and roles in the education arena. We have mentioned the use of NeatTools to allow students with disabilities to participate actively in science laboratory activities. More generally, NeatTools lends itself well to student projects in the classroom, laboratory, science fairs, etc. Moreover, NeatTools can be used for training and prototyping in an industrial or community college setting. Because NeatTools can accommodate diverse external modules, the environment can be adapted to a wide range of simulation applications, notably in medicine. With the increasing use of sophisticated technology in healthcare, environments like NeatTools can be expected to play an increasing role in practice and in training of healthcare practitioners. Medical students, interns, and residents can benefit from the rapid prototyping capability and flexibility of NeatTools. While prior programming experience is clearly of benefit for those who wish to write applications in NeatTools, it can serve, on the other hand, as a training ground for practitioners and others who want to get their feet wet in programming before learning conventional languages like C and C++. The immediacy of the results in this visual programming/runtime environment, without need to cycle through edit/compile/execute cycles is clearly an advantage.
In limited testing, we have observed that schoolchildren are often able to grasp the essentials of NeatTools programming quite rapidly. For example, at SIGGRAPH 98 in Orlando, a number of schoolchildren came to our exhibit in the sigKIDS area. Typically, after the first daytime session, they downloaded NeatTools at home the first evening, proceeded to develop applications of their own, and then returned to our site the following morning to continue their programming and obtain more advanced training. Some of the programs they wrote were quite remarkable.
In the rehabilitation field, our devices have been used for monitoring range of motion, for example at an elbow or knee joint, during exercises, and other aspects of human performance. Our systems are currently in use at two rehabilitation centers, namely the Sister Kenny Institute at Abbott Northwestern Hospital in Minneapolis and at East Carolina University Medical Center. They are currently being implemented at the Extended Care Facility of Oneida City Hospitals in Oneida, NY in a context focused more specifically on monitoring of care of residents.
Development of external modules for digital signal processing, digital image processing, and a host of other advanced modalities will expand the scope of NeatTools for clinical applications, basic research, and education and training. Areas of telemedicine that we anticipate would be well served by NeatTools included telerehabilitation, teleradiology, and general remote patient monitoring including home healthcare, particularly for the elderly still living at home but in need of continual observation. NeatTools already includes a module for the Welch Allyn Vital Signs MonitorÔ . The Internet socket feature of NeatTools, in conjunction with its audio (and soon video) codec, recording, and database functions, already provide base functionality for telemedicine applications.