This phase of the project will allow students with disabilities to participate in the various phases of the work. At Syracuse, we have been developing core software and hardware technology for human computer interfacing, with special focus on disabilities. The centerpiece of this work is NeatTools, a powerful visual-programming and runtime environment that allows both rapid prototyping and full-fledged applications with great potential for helping students with disabilities. This software operates in conjunction with our serial interface boxes (TNGs) that accommodate analog and digital inputs from low-cost sensors that are adapted to individual needs. This overall system has been under development over the past several years by a team consisting of a medical school graduate, computer scientists, physicists, industrial designers, and others. Our companion SmartDesk system offers an instrumented interactive learning environment, especially for students with cognitive (and physical) disabilities. Both NeatTools and SmartDesk allow quantitative event recording and data analysis by use of relational databases and allow protocol changes in real time. These projects have already succeeded in helping several students with quadriplegia, including independent setup and usage by families. This work has received national media attention (AP wire service and CNN Headline News). For further project information, including downloads of NeatTools and application examples, see www.pulsar.org.

In connection with these activities, we have established alliances with several centers that focus on universal design and on outreach activities for people with disabilities. These include the Trace Research and Development Center (trace.wisc.edu) at the University of Wisconsin in Madison; DO-IT (Disabilities, Opportunities, Internetworking, and Technology; http://weber.u.washington.edu/~doit/) at the University of Washington in Seattle; and CAST (Center for Applied Special Technology; www.cast.org) in the Boston area.

For illustration, we describe the types of systems we developed for Eyal Sherman, a member of our team who, is a brainstem quadriplegic, unable to move his head or to vocalize. He is currently a senior at Nottingham High School in Syracuse. We have enabled him to precisely control mouse motion, and thereby control graphical user interfaces, such as Windows 95. Eyal and his family have achieved independence in using this system; his mother is able to set up the hardware and software routinely in a matter of minutes. The primary interface device is a chin joystick, extracted from an inexpensive game controller, mounted to a curved support rod, which is clamped in turn to the wheelchair headrest post, thereby allowing the device to be rotated away when not in use. To allow easy support and adjustment of sensors near Eyal's expressive facial regions--mainly cheeks and forehead--we built lightweight adjustable mounts that attach to eyeglasses. Currently we are using small switches as the expressional sensors, but we have also used Hall Effect transducers (together with tiny rare-earth magnets) and photocells to detect facial gestures. The NeatTools application program that operates this system is the JoyMouse dataflow network. See http://www.pulsar.org/neattools/edl/joymouse_docs/JoyMouseManual.html written by Edward Lipson, who has led the project with Eyal. Starting in the fall of 1999, Eyal will be enrolled at Syracuse University and will continue as a part-time research participant on the project.

NeatTools constitutes a visual-programming and runtime environment that produces fine-grain dataflow networks for data acquisition and processing, gesture recognition, external device control, virtual world control, remote collaboration, and perceptual modulation. The design goals of NeatTools have been to make it simple, object-oriented, network-ready, robust, secure, architecture neutral, portable, high-performance, multithreaded, and dynamic. The program and representative applications are downloadable from www.pulsar.org. NeatTools can readily accommodate custom interface devices, or commercial devices including clinical instruments.

NeatTools is written in C++ but built on top of a thin-layer Java-like cross-platform C++ application programming interface (API), which operates presently on Windows 95/NT, Unix (Sun), Irix (SGI), and Linux. In due course, Macintosh will be supported, once its multitasking, multithreaded operating system is released (note that it can run provisionally on a Mac-based PC-simulator, such as Connectix Virtual PCÔ ), along with appropriate C++ development tools.

Currently, NeatTools includes serial, parallel, and joystick port interfaces; multimedia sound; MIDI (Musical Instrument Device Interface) controls; recording and playback; Internet connectivity (sockets, telephony, etc.); various display modalities including for time signals; time generation functions; mathematical and logic functions (including a state machine module); character generation; and a visual relational database system including multimedia functionality. Keyboard and mouse events can be received or generated via Keyboard and Mouse modules. This allows, among other things, the user to control a graphical user interface by alternative input devices that in effect simulate keyboard and mouse events. Data types in NeatTools include integer, real, string, block, byte array, MIDI event, and audio or video streams. NeatTools allows the visual programmer to package a dataflow network inside a container module that constitutes a reusable "complex module" with simple overt appearance. This procedure can be iterated to accommodate several layers of hidden complexity.

NeatTools modules provide multithreaded, real-time support. Editing and execution are active concurrently, without need for compilation steps. This generally accelerates system design, and facilitates rapid prototyping and debugging. To construct a dataflow network, the user drags and drops modules (objects) from toolboxes to the desktop and then interconnects them with input/output and control/parametric lines. Properties of the desktop and many of the modules are set via a right-mouse-click. In this way, users are in effect developing elaborate interface programs without having to know C++ or the fundamental structure of NeatTools, or indeed having to write any textual program code at all. On the other hand, the system is open, so that experienced programmers can develop external modules by following instructions in an online developer’s kit. External modules can be loaded into the system at runtime, or arranged to preload automatically. The NeatTools executable development program, while massive in terms of source code (~40,000 lines of C++), is compact; the downloadable compressed archive file is about 600 kilobytes in size, so it easily fits on a diskette along with a compressed archive (under 100 kilobytes) of representative "*.ntl" files.

The system hardware consists of mounting components, sensors, serial interface boxes, computer, and optional output interfaces and devices. Our current electronic interface module (TNG-3B; www.mindtel.com/mindtel/anywear.html) accommodates up to 8 analog and 8 digital (switch) sensors and streams the data at 19,200 bits per second to the serial port of a computer. Connections are made via standard stereo and mono plugs. The heart of TNG-3 is a programmable microcontroller integrated circuit, a type of computer-on-a-chip commonly used in industrial and office automation, and in automotive, communication, and consumer electronics under the general rubric of embedded control systems. The microcontroller in TNG-3 is programmed in assembly language for optimal performance. TNG-3 requires no batteries or wall transformer, as it derives 5 volt power for the onboard circuitry and sensors (requiring only modest power) by exploiting some of the unused serial-port lines—a technique commonly used to power a serial mouse on a PC.

Among the sensors we have used are switches, cadmium-sulfide (CdS) photocells, Hall Effect transducers (magnetic sensors), rotary and linear-displacement potentiometers, bend sensors, piezo film sensors, strain gauges, and custom electroconductive-plastic pressure sensors. Most of these sensors are inexpensive, some costing under a dollar and some costing but a few dollars. Certain types (Hall Effect and capacitive) require preamplifiers and/or signal processing electronics, which increase the cost, but not unduly.

SmartDesk constitutes an instrumented learning environment, meaning that the interface is embedded within an array of monitoring systems that keep track of how students perform specific tasks. This feature allows educators to monitor the progress of children with respect to specific learning goals. Like NeatTools, SmartDesk employs a variety of physical interfaces (touch-screen, pressure sensors, joystick, etc.), so it can be used by children with various types of physical disabilities.

At present, SmartDesk is primarily aimed at young children with cognitive and developmental disabilities. However, it could be extended so that older students and students with other disabilities could use it. Indeed, SmartDesk already employs some of the same sensor and interface hardware as NeatTools. Following our work with children with cognitive and developmental disabilities, we are now making SmartDesk into a more generic learning environment. Our ultimate goal is to transform SmartDesk into a tool that can be used to help any student learn science, mathematics, and other subjects that require mastery of knowledge-based skills.