Real Time Training and Integration of Simulation and Planning using the TangoInteractive Collaborative System

Marek Podgorny and Geoffrey C. Fox

{marek,gcf}@npac.syr.edu

Northeast Parallel Architectures Center

Syracuse University

111 College Place

Syracuse, NY 13244-4100

Phone 315 443 2163 and FAX 3154434741

Abstract

We describe TangoInteractive, a Web-based collaborative system which has shown promise in several prototype activities and which can be deployed in a large-scale fashion over the next two years. We present it in terms of a general taxonomy of collaborative or groupware systems represented by other commercial and research activities including Lotus Notes, NetMeeting from Microsoft and Habanero from NCSA. We then describe two applications – command and control and distance learning where we have implemented successful prototype and which represent areas similar to those of interest to the Test and Evaluation Community. We give lessons from this initial work.

1. Introduction

There has been a growing understanding of the importance of technologies that support collaboration and training in DOD. In this paper, we describe the TangoInteractive Web-based collaborative system which has shown promise in several prototype activities and which can be deployed in a large-scale fashion over the next two years. We suggest that such technologies could be very important to the T and E community and extrapolate current uses to this area. We include a general discussion of the TangoInteractive system and its base capabilities in sec. 3. We present this in sec. 2 in terms of a general taxonomy of collaborative or groupware systems and contrast TangoInteractive with several other commercial and research activities including NetMeeting from Microsoft and Habanero from NCSA. In a computational model based on distributed objects, TangoInteractive is used to share access to and views of them. We then illustrate TangoInteractive and the capabilities of such systems in terms of two applications, which have particular relevance to the training and evaluation community. TangoInteractive already includes basic collaborative tools, such as audio/video conferencing, shared whiteboard, chat tool, etc. Firstly we illustrate how these can be exploited and extended use in "computational steering" -- real time interactive control of networked systems by a distributed team. Secondly we show how it can be used in training where simulations can be augmented and annotated by interactive training material.

As described in sec. 4, originally TangoInteractive was developed for Rome Laboratory as the software integration framework for Command and Control and we show that this can be extended to T and E simulations. Note that software like TangoInteractive can run on modern wearable computers with wireless networks and so it is quite practical to imagine large-scale collaborations between all parties in a T and E scenario involving people in field as well as central command centers. As well as specialized planning aids, TangoInteractive provides a rich environment for shared geographical information systems providing 2D and 3D scenes on which one can overlay events and results of a weather simulations which are shared among participants. More generally TangoInteractive can share any simulations with a well-defined API.

In the fall of 1997, Syracuse University and Jackson State University undertook a pilot project for the delivery of "distance training" to DOD Modernization Program users. In sec. 5, we describe how the pilot involved the use of the TangoInteractive collaboratory system to deliver a for-credit, full-semester academic course from instructors located in Syracuse, New York to students located in Jackson, Mississippi using web-based tools. The effort was highly successful, and the course was taught a second time in the spring 1998 term; moreover, the lessons learned from this experience naturally can be used in other applications. We note natural extension to T and E community, where for instance simulations can be shared between teachers and students with both the user input and resultant actions shared.

2. Collaboration

In DOD, industry, academia and society, there is a growing interest in geographically distributed linked activities. These are enabled by communication infrastructure such as the Internet and illustrated by the World Wide Web and its growing acceptance as a premier information resource. Distributed collaboration is not a revolutionary new concept but today’s high performance communication and computing technologies are accelerating the creation of what some call knowledge networks. These link people, computers and instruments exchanging data (or if possible wisdom) interactively and asynchronously. The computers are used for rather mundane storage and dissemination tasks but also for predictive simulations and aids to intelligent planning.

There are several base technologies of importance; for the physical network, ATM and optical fiber illustrate them; however our interest is the higher level distributed system software where the emerging "object web" provides a powerful set of capabilities. The latter include both "web technologies" (from Java to HTML) and distributed object models such as CORBA and COM. Databases are an example of a more complex but critical building block. One builds many high level sophisticated distributed services on top of such base technologies. In this article, we will discuss collaborative capabilities using the system TangoInteractive as an exemplar.

Collaboration has an intuitive meaning – human society is built around this concept with examples at many different emotional and technical levels. Used in distributed (computer-based) systems, collaboration implies technology to share distributed objects. In the simplest such example, two individual collaborate electronically by reading each other’s web pages or by exchanging e-mail with each other. These are perhaps the simplest asynchronous collaboration mechanisms. At another extreme, we have individuals interacting through avatars in a virtual environment. This could be a detailed representation of a battlefield for command and control or perhaps a peaceful summer meadow for a kindergarten virtual field trip. These are examples of synchronous collaboration in which class we can also put audio-video conferencing, chat rooms and shared whiteboards. We know from general experience in society that not only do essentially all activities involve collaboration but also they involve it in both asynchronous and synchronous forms.

Thus collaboration systems involve the sharing of distributed objects – these can be web-pages, databases, visualizations, sensors, instruments, simulations or indeed general computer programs. The latter could include design packages such as AutoCAD or CATIA, planning aids, word processors, spreadsheets and any authoring package. These shared objects can be client or server side. We need tools to allow two or more individuals to share dynamically any number of these distributed objects and interact with them such that state changes are properly seen by all participants. We can also get shared objects with just one participant as in the case that a doctor gets a replicated copy of the readings of a remote medical instrument with no other individuals involved.

There are many examples of collaboration systems available today – each of which supports just a subset of all the features one might wish to use. Let us briefly discuss three examples: Lotus Notes, Microsoft NetMeeting and Habanero/TangoInteractive. Lotus Notes is a rich system which focuses on support of asynchronous collaboration built around a shared document database. It offers many features including shared calendars (event lists) and threaded discussion lists with a good security model. It is designed to support enterprise administration. Microsoft NetMeeting is an impressive product, which has good support for the basic collaborative tools (audio-video conferencing, chat rooms and whiteboard). In a high performance computing language, we can consider SIMD and MIMD collaboration. In the shared display or SIMD model of collaboration, we have only one copy of a given application and the output (display) of this is shared between each client. NetMeeting supports the shared display model for general PC applications but this means that if one shares, say Microsoft Word, only one individual can interact with the application and the others just get a copy of the output frame-buffer and input event queues. NetMeeting uses the more powerful shared event model for its shared whiteboard and chat where each participant runs a separate copy of the application and the collaboration shares the "change-events" – this is for instance the text of messages for chat rooms. NCSA’s Habanero and NPAC’s TangoInteractive use this MIMD (in SPMD Single Program Multiple Data) model for all applications. This provides a much more powerful collaborative model where all participants can interact with each shared object. This interaction is controlled by application-dependent logic that could limit the action of some individuals. For instance when teaching, only the lecturer can choose the web page holding the next set of notes. However these could contain applets or multimedia data. We could set up the sharing rules to allow each student to separately control the invocation of these features from the given curricula page. A more detailed description of this model is given below using our TangoInteractive as an example.

3. TangoInteractive

TangoInteractive is a pragmatic collaboration system, which does not always use the most elegant solution but uses a modern architecture and is designed to support a wide range of client and server applications in essentially any language. The overall structure of this system is shown below in fig. 1. TangoInteractive and the associated shared application objects are invoked from a browser which gives them convenient dissemination and user access. This design choice has brought with it a set of difficult software development problems for today’s browsers are a moving target with still erratically changing features and quite serious bugs. TangoInteractive’s object sharing model has the following characteristics:

  1. Each client (users) runs a small control daemon, which is currently structured as a plug-in but this feature is only necessary in Netscape 3 with its limited security model. The signed applets and JavaScript modules used by TangoInteractive allow the daemon to be moved outside the plug-in in the latest browsers. The daemon is responsible for coordinating messages with the main server and between the linked client-side applications. These use Netscape’s LiveConnect for applets and web pages invoked in the browser but socket communication for standalone applications outside the browser. For Internet Explorer, a functionally equivalent Java/COM communication model replaces the LiveConnect mechanism.
  2. Each client has one special applet called the CA or control applet. The CA provides many useful functions including information on all connected users, applications available for sharing and those that are currently activated.
  3. A group of users can share any one of the applications and this is termed a session. A given application can be launched multiple times with different subsets of the logged-in users involved. Thus many sessions can correspond to the same core application with perhaps multiple chat rooms and whiteboards used with various collaborations.
  4. Each client executes a separate copy of the application in a session. This is usually invoked using the standard web mechanism with the clients fetching the code and data from a web server. The web server can in fact be different for each client and we use such replication of source material to minimize communication on Internet connections in our distance education use of TangoInteractive. As always, one can also invoke client applications outside the browser from client side file systems, in the same way that helper application are used by web browsers. This mechanism is for instance used to share our C++ (Open Inventor) 3-dimensional geographical information system (GIS) and NPAC’s Buena Vista audio video conferencing system, as well as Xemacs (written in LISP) with its powerful set of GNU utilities.
  5. The application developer is responsible for implementing event sharing by supplying routines that send or receive appropriate tagged messages when properties of an application change. These messages are received by TangoInteractive’s Java server and then multicast to the daemon and thence to all other involved client applications. We have an elegant interface – the TangoBean -- under development which assumes the shared objects are built as JavaBeans and then allows one to use visual software development tools to specify which of the Bean’s properties and events are to be shared.
  6. TangoInteractive keeps track of a simple flag indicating the "master" or "slave" status of a client for each session. However the developers can interpret this as they please. For chat rooms and whiteboards all users can send messages or modify the shared view respectively. In most of shared simulations, only the master can set parameters and start the simulation running. In this case one can use the CA to exchange the master token in a flexible fashion among involved clients.
  7. Note that TangoInteractive does not require the developer to share all the change events. In a collaborative visualization, one could choose to share the image but allow the sharing of viewpoint to be an option which can be separately set by each participating client. In fact TangoInteractive does not even require the linked applications to be identical and taking our 3D GIS again, we use this feature to link 2D and 3D client GIS applications. Each client has a separate configuration file specifying the web server and program file associated with each application button in CA. Thus, the system provides support for the independent data views. This is an important concept that enables further work on adaptive session management.
  8. It appears that TangoInteractive’s mechanism only allows one to share client side applications but this is not true. The shared client can invoke a server side object (e.g. as a servlet or CGI script) and in this way we share databases and web-linked simulations. This simple mechanism can share any server side distributed object whose properties can be exported to the web. We intend to link Lotus Notes to TangoInteractive using this approach which is more general than appears at first sight. We expect the new XML standard to be very helpful in specifying object properties in a web compatible fashion.
  9. What objects can be shared? Essentially, one can share any object where it is known which properties are needed to specify its relevant state and for which there is an API (with get and set methods) to access and change the values of these methods. As this is often true for quite complex applications including those from third parties, one can in fact often share objects without changing the application code. A good example here is JDBC (Java Database Connectivity framework), which allows one to easily share any database with a wrapper that can be developed outside the commercial software.
  10. The flexibility of the event-sharing model is illustrated by TangoInteractive’s support of NetMeeting, which can be invoked from TangoInteractive with the latter supplying session and user control. TangoInteractive allows the slave clients to just have the shared display versions of the invoked application.

4. TangoInteractive in Command and Control

Our prototype command and control structure is shown below in fig. 2. It was designed to show in a Rome Laboratory project that web technology was suitable for building a system that typically was constructed with expensive custom software. We added to the architecture of fig. 1, an event driven simulator and server side filters for events shown in fig. 2 which allow a scenario to drive some or all of the events defining the collaboration. Clearly this is relevant in Test and Evaluation applications where this allows personnel to include either simulations or "hardware in the loop" into a rich collaboration environment for the involved personnel.

Command and Control is the military description of a general real time decision (or judgment) support environment involving a complex set of people, datasets, and computational resources. A critical characteristic is the need to make the ``best possible,'' as opposed to ``optimal,'' choice. In a civilian context, crisis management has essentially similar needs with particular interest in the cheap deployable software model underlying TangoInteractive.[See the XII project at http://www.niusr.org ] Again, most application areas have a component that links computers, information, and one or more people to make decisions. Correspondingly, command and control has system requirements that are generally applicable to relatively unstructured decision support. Telemedicine or more generally interventional informatics needs similar support for doctors, patients, and auxiliary care providers interacting with medical instruments and information resources containing relevant medical wisdom.

Our prototype implemented a simple scenario corresponding to an unmanned aircraft flying into U.S. airspace carrying a bacterial agent. We implement planning, execute (does one intercept or not?), and finally response phases to health and life threatening consequences of the downed aircraft. This scenario involves military and civilian radar sensors and personnel tracking aircraft; NORAD as military command in execution stage; federal and state leaders at highest level (President and Governor); weather simulation to assess intercept possibilities, and in response stage, the dispersal of bacterial agents; FEMA as civilian command in response stage; and finally, medical authorities for expertise and treatment.

Let us see how some of the key capabilities of TangoInteractive are used in those various parts of this scenario. Generally as in general collaboratories, digital video conferencing, text-based chat rooms, and shared white boards are used by the participants to interact in a typical unstructured collaborative fashion.

One of the simplest shared applications is the common information source that, in our case, is a shared Web page, but in general, can be information obtained from any database. This is, for instance, used by the tracking officer to search the Web for information on the identified aircraft, and display characteristics and photos with other participants. Both the Web search process, and resultant pages are shared. Note the master generates this information, and broadcasts it to other participants. We can use the filter capability of TangoInteractive to ensure that each participant only received appropriate information. For instance, the President and Governor would be spared a lot of the detailed shared displays used by those lower down the command chain.

More generally, one wishes to present a given object in different ways to different participants. Thus, the radar and weather officers could use complex three-dimensional geographical information systems to study the crisis in detail. The expected weather and tracking data would be presented to others less involved in those areas, as simpler two-dimensional summaries. This flexibility is easily implemented by passing all status messages from the master for each application through a client or server side filter that is controlled by a dynamic script. Note that one uses TangoInteractive’s capability to select different masters for different applications, and allow the "master role'' be passed from one participant to another as the collaboration evolves.

Training is an important component of many decision-making processes, including command and control, crisis management, and health care. Here, the discrete event simulation capability of TangoInteractive is critical as it allows one to "script'' (simulate) any of the participants, and enable general training sessions focusing on any role. Note that in the simple TangoInteractive collaborative mode, ``time'' is the real wall clock time. This mode will support scripting, which can be used to present information at particular times to given clients. For instance, this is a valuable capability in the training mode for carousel-like presentations. In the more general simulation mode, one differentiates real and virtual time as is traditional in discrete event simulators.

Command and control, and crisis management both make decisions about events in the real world, and so particularly critical applications are two- and three-dimensional geographical information (GIS) systems. We originally implemented our three-dimensional GIS in VRML (Virtual Reality Modeling Language), but found this unacceptably slow. We re-implemented the VRML version in terms of its Open Inventor basis, which led to a much faster C++ application. This we can still integrate into TangoInteractive, which supports any client application language. As usual in GIS, these applications support overlays - in the case of the three-dimensional GIS, these include weather simulations with volume rendering of the clouds, and other weather phenomena. As mentioned above, the 3D GIS can interact with powerful 2D Java mappers where the digital map information is layered on a shared whiteboard.

5. TangoInteractive in Distance Learning

5.1 The Jackson State Course CSC 499 offered synchronously over the Internet

Many academic courses at Syracuse University and elsewhere use technology to enhance the learning experience. For example, commercial textbooks often include CD-ROMs complete with databases, online resources, utilities, and hot links to related material on the World Wide Web. Mailing lists and bulletin board systems (such as USENET) are commonly used to make timely course-related announcements and as a medium to facilitate asynchronous group discussions and administer technical forums. Course material is often published on web servers, which students are encouraged to access at their leisure (that is, asynchronously).

There appears to be no shortage of such course material on the web (especially in the technical fields), although the quality of such material varies greatly. What is clearly missing, however, is a vehicle to deliver educational material synchronously, that is, in real time. Although some "distance learning" courses use chat tools to communicate synchronously, there appears to be few (if any) systems that deliver real-time multimedia content in an authentic, two-way interactive format.

The computational science education group at the Northeast Parallel Architectures Center (NPAC) has developed a huge repository of online course material, including lectures, tutorials, and programming examples in various languages. We believe, however, that a significant majority of students require regular and sustained interaction (i.e., synchronous learning activities) involving teachers and other learners, in addition to asynchronous learning materials. This motivated us to use TangoInteractive to deliver CSC 499 over the Internet.

The first offering of Jackson State University (JSU) course CSC 499, Programming for the Web, began on August 18, 1997 and ended December 10, 1997. The same course was taught again the following spring semester from January 8, 1998 to May 5, 1998. CSC 499 is equivalent to Syracuse University course CPS 406, which is also taught as a graduate course (CPS 606).

The goal of CSC 499 is to provide the basic programming skills, such as Java, needed to develop World Wide Web applications as developed for the CPS 406/606 course which is taught at Syracuse in traditional lecture-based format. Students are provided access to a wealth of online material and evaluated on a number of projects, which are assigned, submitted, and graded over the web. See http://www.npac.syr.edu/projects/cps606spring98/index.html for a recent offering of CPS 606. Students in CSC 499 also had access to asynchronous learning material on the web. From the course home page http://www.npac.syr.edu/projects/jsuspring98/index.html, students link to resources covering HTML, CGI (including Perl), and Java programming. These pages contain links to lectures, tutorials, documentation, and numerous examples.

CSC 499, offered as a JSU credit course, included regularly scheduled lectures delivered via TangoInteractive as shown in fig. 3. During these lectures, an instructor would show lecture slides on a workstation in Syracuse, while the students attended class in a lab at JSU. Each lecture slide would appear on the students' workstations as the instructor displayed it. The instructor would deliver the lecture via an audio link, and the students would ask questions either through a chat tool or the audio link. This was implemented by a collaborative tool WebWisdom , which essentially allowed the instructor a sophisticated interface to a curricula database while the students just receive the chosen "educational object". Other tools used extensively was the similar but less structured shared web-page, the whiteboard and a custom "raise hands" applet allowing students to alert the instructor.

5.2 Conclusions and Lessons Learnt from the CSC499 Experiment

Jackson State is a major historically black University (HBCU) with many computer science graduates where the faculty have heavy teaching loads and so find it harder to incorporate latest research and commercial developments into the curricula. Thus our aim was not to compete with base courses but rather offer add-on courses with "leading edge" material (Web Technology) which give JSU graduates skills that are important in their career but not currently offered through standard JSU courses. The students appreciated the course content, which improved their job prospects. We can generalize this to a scenario where universities "specialize" and only deliver courses in areas of expertise to students from many collaborating institutions. Note that there was a JSU Professor assigned to the course, who in this model acts as a mentor to the students. In our case, these mentors mastered the material and technology so well that JSU hopes to offer such Web Technology courses to other HBCU institutions using TangoInteractive. We also note the need for good technical support at both receiving and teaching sites.

A major requirement was good network quality of service with modest bandwidth; assuming course material mirrored at JSU, one needs a guaranteed 30 (audio) to 100 (video) kilobits per second bandwidth link. Note the shared web pages are implemented by just exchanging a URL and there are no needs for large bandwidth transfer. The TangoInteractive Server only transmits control data and those events specified by modest size messages. The actual digital audio-video information is transmitted over separate channels after the session has been established through the TangoInteractive server and client interactions. Next semester we will replace Mirrored Web Server by a CDROM, which would allow us of offer homes, courses using 28.8 kbaud phone links. Note that this course was supported by a large asynchronous resource -- 250 megabytes on the mirrored servers and this illustrates that synchronous and asynchronous collaboration are mutually supportive and in some sense that synchronous delivery is "just a guided tour" of the asynchronous resource.

Our experiments were successful but conservative -- we deliberately tried to reproduce "time honored approach" -- we can expect there to be different optimal approach in distance case than in face to face case -- find by more experiments! Most interesting are new instructional modes such as those with graduate assistant manning ancillary tools such as chat rooms to enhance richness of interactive sessions. Note that professors cannot easily lecture and monitor other tools at the same time. We plan several obvious technology improvements for future such projects. These include a better complete database backend; use of two-way video on reliable link; interactive annotatable lecture material; archiving of audio-video conferencing, use of curricula material and annotations; chat rooms for asynchronous replay; better link of asynchronous and synchronous tools.

We believe that whereas say K-12 education is mainly interested in synchronous collaboration, DOD's main thrust is asynchronous electronic training. Our work shows how this can be integrated with synchronous activities to lead to a very powerful new distance learning paradigm.

Acknowledgements: This distancing learning activity was sponsored by the U.S. Army Corps of Engineers Waterways Experiment Station (CEWES) Major Shared Resource Center (MSRC) at Vicksburg, Mississippi under the DoD HPC Modernization Program, Programming Environment & Training (PET). Program partners include the Information Technology Laboratory at WES and the Nichols Research Corporation.