There has been a growing understanding of the importance of technologies that support collaboration and training in both commercial and academic circles. In this appendix, we describe the TangoInteractive Web-based collaborative system which has shown promise in several prototype activities and which can be deployed in a large-scale fashion over the next two years. We include a general discussion of the TangoInteractive system and its base capabilities in sec. A3. We present this in sec. A2 in terms of a general taxonomy of collaborative or groupware systems and contrast TangoInteractive with several other commercial and research activities including products from startups Centra and Placeware as well as the well known NetMeeting from Microsoft and Habanero from NCSA. We then illustrate TangoInteractive and the capabilities of such systems in terms of two key applications described in sections A4 and A5; these are distance training (or education) and collaborative computing. We are exploring these in a partnership between DoD, NSF and the NPAC organization at Syracuse University. DoD partners come from the modernization program and include in different ways, the four major resource centers at ARL (Aberdeen), ASC(Dayton), CEWES(Vicksburg) and NAVO(Stennis Center, Miss). NSF activities are part of the NCSA National Computational Science Alliance led by the University of Illinois.

In a computing model based on distributed objects, TangoInteractive is used to share access to and views of these objects. TangoInteractive includes basic collaborative tools, including audio/video conferencing, shared whiteboard, and chat tool which are used in all applications. These are then augmented in any application by custom "shared distributed objects" which can either be client or server based. In general, TangoInteractive can share any applications with a well-defined API that allows one to sense the state of one copy and set the new state in replicated copies that are following the original. As TangoInteractive uses the powerful shared event model, one can choose which properties of an application to share and which to adjust separately for each user.

Originally TangoInteractive was developed for Rome Laboratory as the software integration framework for Command and Control and this illustrates its use for shared decision-making. As an example of the custom support needed here, TangoInteractive provides command and control users a rich environment for shared geographical information systems providing 2D and 3D scenes on which one can overlay events and results of a weather or other simulations which are shared among participants.

In the fall of 1997, Syracuse University and Jackson State University undertook a pilot project for the delivery of "distance training" to DOD Modernization Program users. In sec. A4, we describe how the pilot involved the use of the TangoInteractive collaboratory system to deliver a for-credit, full-semester academic course from instructors located in Syracuse, New York to students located in Jackson, Mississippi using web-based tools. The effort was highly successful, and the course was taught a second time in the spring 1998 term; moreover, the lessons learned from this experience are being used in an increasing number of such distance education and training activities. We have designed a powerful system linking the best of asynchronous and synchronous approaches.

Collaborative computing offers several places where collaborative tools can be used -- these include collaborative visualization and "computational steering" -- real time interactive control of networked systems by a distributed team. As discussed in section A5, another interesting application combines these tools with those of training to provide "interactive help desks" -- collaborative consulting. These are particularly relevant to large supercomputer centers where typically over 80% of the use comes from users who are off site.

In DOD, industry, academia and society, there is a growing interest in geographically distributed linked activities. These are enabled by communication infrastructure such as the Internet and illustrated by the World Wide Web and its growing acceptance as a premier information resource. Distributed collaboration is not a revolutionary new concept but today’s high performance communication and computing technologies are accelerating the creation of what some call knowledge networks. These link people, computers and instruments exchanging data (or if possible wisdom) interactively and asynchronously. The computers are used for at times somewhat mundane storage and dissemination tasks but also for predictive simulations and aids to intelligent planning.

There are several base technologies of importance; for the physical network, ATM and optical fiber illustrate them; however our interest is the higher level distributed system software where the emerging "object web" provides a powerful set of capabilities. The latter include both "web technologies" (from Java to HTML) and distributed object models such as CORBA and COM. Databases are an example of a more complex but critical building block. One builds many high level sophisticated distributed services on top of such base technologies. In this article, we will discuss collaborative capabilities using the system TangoInteractive as an exemplar.

Collaboration has an intuitive meaning – human society is built around this concept with examples at many different emotional and technical levels. Used in distributed (computer-based) systems, collaboration implies technology to share distributed objects. In the simplest such example, two individuals collaborate electronically by reading each other’s web pages or by exchanging e-mail with each other. These are perhaps the simplest asynchronous collaboration mechanisms. At another extreme, we have individuals interacting through avatars in a virtual environment. This could be a detailed representation of a battlefield for command and control or perhaps a peaceful summer meadow for a kindergarten virtual field trip. These are examples of synchronous collaboration in which class we can also put audio-video conferencing, chat rooms and shared whiteboards. We know from general experience in society that not only do essentially all activities involve collaboration but also they involve it in both asynchronous and synchronous forms.

Thus collaboration systems involve the sharing of distributed objects – these can be web-pages, databases, visualizations, sensors, instruments, simulations or indeed general computer programs. The latter could include design packages such as AutoCAD or CATIA, planning aids, word processors, spreadsheets and any authoring package. These shared objects can be client or server side. We need tools to allow two or more individuals to share dynamically any number of these distributed objects and interact with them such that state changes are properly seen by all participants. We can also get shared objects with just one participant as in the case that a doctor gets a replicated copy of the readings of a remote medical instrument with no other individuals involved.

There are many examples of collaboration systems available today – each of which supports just a subset of all the features one might wish to use. Let us briefly discuss three examples: Lotus Notes, Microsoft NetMeeting and Habanero/TangoInteractive. Lotus Notes is a rich system which focuses on support of asynchronous collaboration built around a shared document database. It offers many features including shared calendars (event lists) and threaded discussion lists with a good security model. It is designed to support enterprise administration. Microsoft NetMeeting is an impressive product, which has good support for the basic collaborative tools (audio-video conferencing, chat rooms and whiteboard). In a high performance computing language, we can consider SIMD and MIMD collaboration. In the shared display or SIMD model of collaboration, we have only one copy of a given application and the output (display) of this is shared between each client. NetMeeting supports the shared display model for general PC applications but this means that if one shares, say Microsoft Word, only one individual can interact fully with the application and the others just get a copy of the output frame-buffer and input event queues. NetMeeting uses the more powerful shared event model for its shared whiteboard and chat where each participant runs a separate copy of the application and the collaboration shares the "change-events" – this is for instance the text of messages for chat rooms. NCSA’s Habanero and NPAC’s TangoInteractive use this MIMD (in SPMD Single Program Multiple Data) model for all applications. This provides a much more powerful collaborative model where all participants can interact with each shared object. This interaction is controlled by application-dependent logic that could limit the action of some individuals. For instance when teaching, only the lecturer can choose the web page holding the next set of notes. However these could contain applets or multimedia data. We could set up the sharing rules to allow each student to separately control the invocation of these features from the given curricula page. A more detailed description of this model is given below using our TangoInteractive as an example. We should also discuss interesting commercial systems from Centra, Placeware and ILINC. These have some similarities with the education and training model pioneered by TangoInteractive and described in sec. A4. However none of these supports our full mix of asynchronous and synchronous learning with the power of browser-based synchronous interaction tools together with a Java/XML based curricula and administrative database.

TangoInteractive is a pragmatic collaboration system, which does not always use the most elegant solution but uses a modern architecture and is designed to support a wide range of client and server applications in essentially any language. The overall structure of this system is shown below in fig. 1. TangoInteractive and the associated shared application objects are invoked from a browser which gives them convenient dissemination and user access. This design choice has brought with it a set of difficult software development problems for today’s browsers are a moving target with still erratically changing features and quite serious bugs. TangoInteractive’s object sharing model has the following characteristics:

Each client (users) runs a small control daemon, which is currently structured as a plug-in but this feature is only necessary in Netscape 3 (which is of course of decreasing importance) with its limited security model. The signed applets and JavaScript modules used by TangoInteractive allow the daemon to be moved outside the plug-in in the latest browsers. The daemon is responsible for coordinating messages with the main server and between the linked client-side applications. These use Netscape’s LiveConnect for applets and web pages invoked in the browser but socket communication for standalone applications outside the browser. For Internet Explorer, a functionally equivalent Java/COM communication model replaces the LiveConnect mechanism.

Each client has one special applet called the CA or control applet. The CA provides many useful functions including information on all connected users, applications available for sharing and those that are currently activated.

A group of users can share any one of the applications and this is termed a session. A given application can be launched multiple times with different subsets of the logged-in users involved. Thus many sessions can correspond to the same core application with perhaps multiple chat rooms and whiteboards used with various collaborations.

Each client executes a separate copy of the application in a session. This is usually invoked using the standard web mechanism with the clients fetching the code and data from a web server. The web server can in fact be different for each client and we use such replication of source material to minimize communication on Internet connections in our distance education use of TangoInteractive. As always, one can also invoke client applications outside the browser from client side file systems, in the same way that helper application are used by web browsers. This mechanism is for instance used to share our C++ (Open Inventor) 3-dimensional geographical information system (GIS) and NPAC’s Buena Vista audio video conferencing system, as well as Xemacs (written in LISP) with its powerful set of GNU utilities.

The application developer is responsible for implementing event sharing by supplying routines that send or receive appropriate tagged messages when properties of an application change. These messages are received by TangoInteractive’s Java server and then multicast to the daemon and thence to all other involved client applications. We have an elegant interface – the TangoBean -- under development which assumes the shared objects are built as JavaBeans and then allows one to use visual software development tools to specify which of the Bean’s properties and events are to be shared.

TangoInteractive keeps track of a simple flag indicating the "master" or "slave" status of a client for each session. However the developers can interpret this as they please. For chat rooms and whiteboards all users can send messages or modify the shared view respectively. In most of shared simulations, only the master can set parameters and start the simulation running. In this case one can use the CA to exchange the master token in a flexible fashion among involved clients.

Note that TangoInteractive does not require the developer to share all the change events. In a collaborative visualization, one could choose to share the image but allow the sharing of viewpoint to be an option which can be separately set by each participating client. In fact TangoInteractive does not even require the linked applications to be identical and taking our 3D GIS again, we use this feature to link 2D and 3D client GIS applications. Each client has a separate configuration file specifying the web server and program file associated with each application button in CA. Thus, the system provides support for the independent data views. This is an important concept that enables further work on adaptive session management.

It appears that TangoInteractive’s mechanism only allows one to share client side applications but this is not true. The shared client can invoke a server side object (e.g. as a servlet or CGI script) and in this way we share databases and web-linked simulations. This simple mechanism can share any server side distributed object whose properties can be exported to the web. We intend to link Lotus Notes to TangoInteractive using this approach which is more general than appears at first sight. We expect the new XML standard to be very helpful in specifying object properties in a web compatible fashion. We have used this as the core of our WebWisdomNT curricula database described in sec. A4.2.

What objects can be shared? Essentially, one can share any object where it is known which properties are needed to specify its relevant state and for which there is an API (with get and set methods) to access and change the values of these methods. As this is often true for quite complex applications including those from third parties, one can in fact often share objects without changing the application code. A good example here is JDBC (Java Database Connectivity framework), which allows one to easily share any database with a wrapper that can be developed outside the commercial software.

The flexibility of the event-sharing model is illustrated by TangoInteractive’s support of NetMeeting, which can be invoked from TangoInteractive with the latter supplying session and user control. TangoInteractive allows the slave clients to just have the shared display versions of the invoked application.

Many academic courses at Syracuse University and elsewhere use technology to enhance the learning experience. For example, commercial textbooks often include CD-ROMs complete with databases, online resources, utilities, and hot links to related material on the World Wide Web. Mailing lists and bulletin board systems (such as USENET) are commonly used to make timely course-related announcements and as a medium to facilitate asynchronous group discussions and administer technical forums. Course material is often published on web servers, which students are encouraged to access at their leisure (that is, asynchronously).

There appears to be no shortage of such course material on the web (especially in the technical fields), although the quality of such material varies greatly. What is clearly missing, however, is a vehicle to deliver educational material synchronously, that is, in real time. Although some "distance learning" courses use chat tools to communicate synchronously, there appears to be few (if any) systems that deliver real-time multimedia content in an authentic, two-way interactive format.

The computational science education group at the Northeast Parallel Architectures Center (NPAC) has developed a huge repository of online asynchronous course material, including lectures, tutorials, and programming examples in various languages. We believe, however, that a significant majority of students require regular and sustained interaction (i.e., synchronous learning activities) involving teachers and other learners, in addition to asynchronous learning materials. This motivated us to use TangoInteractive to deliver CSC 499 over the Internet.

The first offering of Jackson State University (JSU) course CSC 499, Programming for the Web, began on August 18, 1997 and ended December 10, 1997. The same course was taught again the following spring semester from January 8, 1998 to May 5, 1998. CSC 499 is equivalent to Syracuse University course CPS 406, which is also taught as a graduate course (CPS 606).

The goal of CSC 499 is to provide the basic programming skills, such as Java, needed to develop World Wide Web applications as developed for the CPS 406/606 course which is taught at Syracuse in traditional lecture-based format. Students are provided access to a wealth of online material and evaluated on a number of projects, which are assigned, submitted, and graded over the web. See http://www.npac.syr.edu/projects/cps606spring98/index.html for a recent offering of CPS 606. Students in CSC 499 also had access to asynchronous learning material on the web. From the course home page http://www.npac.syr.edu/projects/jsuspring98/index.html, students link to resources covering HTML, CGI (including Perl), and Java programming. These pages contain links to lectures, tutorials, documentation, and numerous examples.

CSC 499, offered as a JSU credit course, included regularly scheduled lectures delivered via TangoInteractive as shown in fig. 2. During these lectures, an instructor would show lecture slides on a workstation in Syracuse, while the students attended class in a lab at JSU. Each lecture slide would appear on the students' workstations as the instructor displayed it. The instructor would deliver the lecture via an audio link, and the students would ask questions either through a chat tool or the audio link. This was

implemented by a collaborative tool WebWisdom, which essentially allowed the instructor a sophisticated interface to a curricula database while the students just receive the chosen "educational object". Other tools used extensively was the similar but less structured shared web page, the whiteboard and a custom "raise hands" applet allowing students to alert the instructor.

We are planning several enhancements of the training system but perhaps the most important is the complete integration of a powerful curricula database with the interactive synchronous tools of TangoInteractive. We have observed that even complicated tools such as Lotus Notes are used to provide an asynchronous learning environment, these are really "just a multimedia website" with a few asynchronous tools such as a shared calendar/appointment list. Thus it is important to realize that in the Jackson State classes, the teacher uses TangoInteractive to provide an annotated guided tour of the material on such a web site. Thus we can clearly define a very powerful training and education model which combines the best of both asynchronous and synchronous modes. We can support popular tools such as WLS and WebCT which are viewed as particular authoring tools for material on the backend curricula web site.

There are well known advantages in using databases to generate web site pages. These include consistent propagation of changed common data over multiple pages and the ease to change display modes by "just" changing the template used to generate the page display in HTML. The current WebWisdom tool in TangoInteractive uses an ingenious patented JavaScript approach but new developments involving XML (which separates meaning and display syntax for data) and JDBC can be added to produce a very powerful system which will be deployed in January of 99. The architecture is shown in fig.3 where a custom servlet is used to interpret XML based servlets. This allows one to easily combine features of systems such as Lotus Notes or ColdFusion with those of TangoInteractive. WebWisdomNT will support both curricula and administrative capabilities and we will link the "personal identity modules" used by TangoInteractive to that defined in the PAPI (Personal and Performance(grading) Information) part of the WebWisdomNT database. We will use templates that use the database to generate RDF metadata in the web pages which supports the new standards from IMS and ADL. Templates can be chosen from a suite that includes pure HTML, images in different sizes and display on an interactive whiteboard supporting shared cursor and annotations.

Now although large systems (such as ours where we already can select from over 25000 foils) are advantageously set up using databases, TangoInteractive will still support several shared browser modes that can be used for smaller applications or for installations where databases are not available.

Jackson State is a major historically black University (HBCU) with many computer science graduates where the faculty have heavy teaching loads and so find it harder to incorporate latest research and commercial developments into the curricula. Thus our aim was not to compete with base courses but rather offer add-on courses with "leading edge" material (Web Technology) which give JSU graduates skills that are important in their career but not currently offered through standard JSU courses. The students appreciated the course content, which improved their job prospects. We can generalize this to a scenario where universities "specialize" and only deliver courses in areas of expertise to students from many collaborating institutions. Note that there was a JSU Professor assigned to the course, who in this model acts as a mentor to the students. In our case, these mentors mastered the material and technology so well that JSU hopes to offer such Web Technology courses to other HBCU institutions using TangoInteractive. We also note the need for good technical support at both receiving and teaching sites.

A major requirement was good network quality of service with modest bandwidth; assuming course material mirrored at JSU, one needs a guaranteed 30 (audio) to 100 (video) kilobits per second bandwidth link. Note the shared web pages are implemented by just exchanging a URL and there are no needs for large bandwidth transfer. The TangoInteractive Server only transmits control data and those events specified by modest size messages. The actual digital audio-video information is transmitted over separate channels after the session has been established through the TangoInteractive server and client interactions. For the fall 98 semester we will put the Mirrored Web Server by a CDROM, which would allow us of offer homes, courses using 28.8 kbaud phone links. Note that this course was supported by a large asynchronous resource -- 250 megabytes on the mirrored servers and this illustrates that synchronous and asynchronous collaboration are mutually supportive and in some sense that synchronous delivery is "just a guided tour" of the asynchronous resource.

Our experiments were successful but conservative -- we deliberately tried to reproduce "time honored approach" -- we can expect there to be different optimal approach in distance case than in face to face case -- find by more experiments! Most interesting are new instructional modes such as those with graduate assistant manning ancillary tools such as chat rooms to enhance richness of interactive sessions. Note that professors cannot easily lecture and monitor other tools at the same time. We plan several obvious technology improvements for future such projects. These include the sophisticated database backend described in the previous subsection; use of two-way video on reliable link; interactive annotatable lecture material; archiving of audio-video conferencing, use of curricula material and annotations; chat rooms for asynchronous replay; better link of asynchronous and synchronous tools based on the WebWisdomNT database.

We believe that whereas say K-12 education is mainly interested in synchronous collaboration, DOD and Industry's main thrust is asynchronous electronic training. Our work shows how this can be integrated with synchronous activities to lead to a very powerful new distance learning paradigm. Thus one can out together approaches for any audience that combines the best of both modes of learning.

Fig. 4 shows schematically a typical problem faced by many (super)computer centers. One has many disparate machines which may or may not be on the same site but they accessed by users spread through out the nation. These users may be independent or geographically distributed teams. They are supported by one or more "consultant services" (help-desks), typically located at one or more of the supercomputer sites. This scenario can surely make use of the basic collaborative features of TangoInteractive but also has several interesting special applications. These include:

• Collaborative Visualization where both NPAC and NCSA have demonstrated this in Tango. NCSA took the powerful multi-platform VTK toolkit and used the C++ API to link visualization clients. This could support shared distributed visualizations between high end CAVE VR systems and low end PC's. NPAC linked the Java visualization system SciVis to TangoInteractive.
• Collaborative computational steering links the simulation -- perhaps controlled by a Java servlet on the gateway system in fig. 4 -- to each client. This together with collaborative visualization of intermediate results can allow a team to discuss and modify the behavior of running simulations.
• Collaborative consulting, where we link one or more users with a consultant. This suggests shared debuggers and other tools including the shared links to visualization and simulation discussed above. It also naturally involves the training technology as one component of consulting is pointing to key parts of online resources. This is the use of TangoInteractive as a "tour guide" of a backend electronic document repository. Our first step in this area was the shared program development tool emacs.

• Tango Web Site: http://trurl.npac.syr.edu/tango/
• Habanero Web Site: http://www.ncsa.uiuc.edu/SDG/Software/Habanero/
• Centra Web Site: http://www.centra.com/
• ILINC Web Site: http://www.ilinc.com/
• Placeware Web Site: http://www.placeware.com/
• Detailed Review of Current collaborative Systems and comparison with TangoInteractive: http://trurl.npac.syr.edu/tango/Documentation/Papers/BIS/bis.html
• Discussion of first two semesters using TangoInteractive in distance education: http://www.npac.syr.edu/users/trscavo/jsuspring98/
• EDUCOM's IMS Instructional Management Systems Project: http://www.imsproject.org/
• The Department of Defense ADL Advanced Distributed Learning Project: http://www.adlnet.org/
• A collection of Web Sites associated with Training and Collaboration in the Dod HPC Modernization PET Program: http://www.npac.syr.edu/users/gcf/petcandt/
• The WLS Web Learning System: http://renoir.csc.ncsu.edu/
• The WebCT Web Site: http://homebrew.cs.ubc.ca/webct/