Fox Presentation 1996 Collaborative Interaction and Visualization NPAC- Vanguard Sponsored by Rome Laboratory PR No. C-5-2293/4 Contract F30602-95-C-0273 January 31,1996 NPAC Team: Presenter Geoffrey Fox NPAC Syracuse University 111 College Place Syracuse NY 13244-4100 Abstract of Jan 31 1996 RL CIV Presentation This Presentation summarizes the current status of the Rome Laboratory funded Collaborative Interaction and Visualization Project performed by NPAC and Vanguard This uses 5 component technologies (VR, Network Support, Compression, Video Conferencing, GIS, Multimedia Databases) with Web based Integration These are used with SGI based large screen stereo displays in 4 applications (Electromagnetic and Weather Simulation, Command and Control, Medical Information Systems and Telemedicine) This is second presentation of project which statrted in September 1995. Organization of Presentation General Remarks Technology Integration Plan -- Overview and Background Details Infrastructure -- Hardware, Database, Speech Recognition, Networking, Display 5 Component Technologies -- VR, Network Support, Compression, Video Conferencing, GIS, Multimedia Databases 4 Applications -- Electromagnetic and Weather Simulation, Command and Control, Medical Information Systems and Telemedicine Some Key World and Project Developments In the the "real world": The nature and importance of Web Technology is now much clearer with the early december 95 Java annoucements and continuing evolution of VRML This allows us to put together a clear technology integration plan which builds on past approach but is set to exploit current commercial Web developments We have split technology areas into "component technologies" (the first 5) and integration technologies (The Web and parallel and distributed computing) -- these last are now packaged as technology integration plan At NPAC, the applications have become clearer with (largely funding outside CIV) an increase in effort and understanding in medical application. This has evolved to make clearer synergy between medical systems and command and control Nationally Telemedicine is becoming medical decision support with distributed information systems -- the traditional command and control approach Highlights of Recent Progress in RL CIV Clear Technology Integration Plan Developed with prototypes in JavaScript Network Support presented as a decision support module in CIV framework Excellent 2D adaptive compression technology completed as prototype Geographical Information System developed using key Web Technologies Java based 2D System VRML with object database for 3D -- allows vision of evolution to future VRML 2.0 televirtual environment Video server and Text Indexing of multimedia database operational Electromagnetic, Weather and Medical Applications identified and activities initiated. EMS and Weather designed so can be modules in C2 application. Some Key Developments in Next Quarter Technology Integration Plan including recent Sun and Netscape Java/JavaScript developments and initial WebFlow design delivered First demonstration using full SGI large screen infrastructure Identify expected command and control demonstrations and scenarios for whole project Present initial prototypes at two local conferences -- Telemedicine(May) and Dual-Use(June) -- at OnCenter Medical prototype based on School of Nursing application Continued Progress with other component Technologies and Applications Basic Integration Technology approach using Web Technologies - Prototypes at NPAC and the RL CIV Integration Plan I.O: Integration Technologies Plan -- Topics I.1: Web'95 Technology Revolution I.2: Web'96 Technology Scenario -- Confusion I.3: Web'96 Technology Scenario -- II: Implications for RL CIV I.4:Current prototype of Teacher-Student Interactive Environment - WebFoil I.5:Prototype of Web based Patient Record system I.6: RL CIV Multi-Use Technology Integration I.7: More Technology Background: Complexity of the Expanding Web I.8: Web/Legacy Software Linkages: Plug-ins, Java/CORBA I.9: Possible Language/Protocol Level Integration Technologies I.10: Topologies for Interactive Collaboratory Web Environments -I I.11: Topologies for Interactive Collaboratory Web Environments - II: Typical topologies I.12: Dataflow Based Integration Technology I.13: Integration Concepts: WebFlow, WebTools, WebTop Applications I.14: WebFlow - Web-based Coarse Grain Data (Object) Flow I.15: WebTools - ensemble of reusable WebFlow modules I.16: WebTop Systems - WebFlow based distributed applications I.17:Preliminary Design of WebFlow Production Version of WebFoil I.18:Possible WebFlow Implementation of Patient Record Database I.1: Web'95 Technology Revolution - I: Amazing Progress and Change In early '95, Web was established as a new pervasive information technology infrastructure. Several sites, including NPAC, initiated in-house experiments to adopt and extend base Web technologies for intranet (enterprise) level base computing support. We based RL CIV on Web Technologies At NPAC, we prototyped WebTools (a CGI-based PDA support for content authoring, document management and e-mail handling), Web-Oracle frontends for USENET groups and other (educational) databases, and early Web interfaces to VOD services. Mid'95 brought the first wave of Web technology explosion, including Java language, VMRL protocol and Netscape public offering. At the end of '95, there werea series of software industry coalitions (Netscape plugins, Java licenses) and the '96'Web product announcements. I.1: Web'95 Technology Revolution - II: NPAC's Activities In particular, Netscape'96 products should provide commercial products related to many '95 prototypes developed by NPAC (see table below). NPAC Netscape'96 WebMail Netscape Mail HyperWorld Manager LiveWire Site Manager On-Line HTML Editor Netscape Gold WYSIWYG HTML Editor WebRDBMS LiveWire Pro support for Oracle, Informix, Sybase, MS NPACBoard Netscape Chat Enterprise Int. Sys. Netscape Community System This confirms NPAC's strategy but requires that we carefully evaluate our software development projects such as RL CIV so that they are synergistic rather than competitive with the Web software industry. A major goal of the 'integration technologies' thrust in the RL CIV effort is to provide such a framework (collaborative software development with the world) for all project components. I.2: Web'96 Technology Scenario -- I:Confusion! The initial scene for Web'96 will be set by two major players: Netscape with plugin partners, and Sun/JavaSoft with Java licensees. SGI may also play important role due to the leadership in VRML. Other major vendors such as IBM, Microsoft or Oracle are still trying to play it 'their' rather than the Web way which might result in unpredictable developments and shaping the Netscape/Sun competition/collaboration patterns. Web technologies will quickly acquire significant power and penetrate corporate computing/information systems. No simple unique standards are expected in the near term since Netscape has the largest Internet penetration, Sun has the best Internet software technology, SGI has the best visualization platform etc. I.3: Web'96 Technology Scenario -- II: Implications for RL CIV Complexity of the '96 Web will keep growing together with its power. We will observe systematic shift from client-server to server-server architectures (clients must become reactive, and hence servers), which will help development of collaboration systems. In RL CIV project, NPAC will pursue hybrid strategy Continue focussed component technology and application activities with "today's" technology (this includes beta Netscape 2.0) Develop integration and component understanding and capabilities to exploit which innovations and Companies really succeed/deliver -- The Integration strategy is called WebFlow and will have an interim implementation evolving to future Web I.4:Current prototype of Teacher-Student Interactive Environment - WebFoil Early WebFoil prototype based on Netscape2 multiframe JavaScript is already operational at NPAC and extends initial HotJava Prototype demonstrated at Supercomputing 95. Offers a hierarchical information model where contents in one frame can be scrolled and generate associated information module in main frame 130 foilsets can be scrolled -- Information is abstract foils within foilsets can be scrolled -- information is HTML or Gif version of foil User has control of size, color, font etc and can save configurations Can link notes, further information and audio to information page Can be generalized to any hierarchically organized table with indices at each level I.5:Prototype of Web based Patient Record system This has been developed for both Nursing and general medical applications The current versions have been upgraded to use same WebTool(kit) JavaScript front end first developed for education This illustrates how the Web can develop generic approachs which can be applied across a variety of applications from Education, Medicine to Command and Control! Compared to foilset, new element is Oracle database backend for storage of records Now you can scroll components of database (selecting perhaps Images), patients or later records from particular hospital Video can be part of database if this in multimedia patient record I.6: RL CIV Multi-Use Technology Integration Within the proposed integration framework, the RL CIV project can be formulated in the following way. Component technologies will be wrapped as reusable modules (e.g. VOD) or as reusable WebTool(kit)s (e.g. DB interfaces) The bridge based overall architecture of telemedicine will be used as initial focus and template for the command and control application (for which telemedicine can be viewed as civilian dual-use version) Weather and EMS applications will be viewed as plug-in compute-webs to support proof-of-the concept command and control, based on bridge/telemedicine skeleton. Image processing is compute web module for Telemedicine While sharing the common skeleton and GUI, telemedicine will focus on care portal component (which is most realistic for early deployment), while command and control will explore the commander site in terms of weather and EMS collaborative visualization tools. Both will share database backended information systems Further Details on Web Evolution in '96 and - Relevance of WebFlow in RL CIV Technology Integration Strategy I.7: More Technology Background: Complexity of the Expanding Web Original client-server and HTML/CGI based WWW acquires now several new architectural components. Server side CGI technologies are being challenged by Netscape LiveWire scripted model which is more consistent with the JavaScript at the client-side. The emergent generation of Java based HTTP daemons offers new dimensions for fully programmable and state-aware Web servers. (Critical for Collaboration servers) Documents, previously expressed in terms of easy to learn HTML, are now enriched by Java applets and JavaScript inserts which requires programming expertise for content development. Several middleware/agent technologies are under development, such as Telescript (intelligent agents), helper/plugins, collaboratory servers, whiteboards, chat rooms etc. I.8: Web/Legacy Software Linkages: Plug-ins, Java/CORBA Third party legacy software becomes now rapidly part of the expanding Web using integration/interface/wrapper technologies. Currently the most popular interface technique is via Netscape plug-ins at the client side and the corresponding LiveWire gateways at the server side. Finer grain linkage between distributed applications was previously addressed by CORBA/OLE protocols. With Java taking over the C++ domain, these object broker techniques (focused on compiled OO languages) might be eventually of decreasing relevance. In the near term, companies will require support for object level interaction between their legacy systems and Web software modules. We learn from private communication with Sun that JavaWorks (a HotJava based CASE toolkit for professional Java development) will offer CORBA based interfaces or "C++ gateways". I.9: Possible Language/Protocol Level Integration Technologies At the language/protocol level, we can identify now three potential candidates for integrating/unifying the complex '96 Web systems: 1) HTML/CGI -> JavaScript/LiveWire - this (Netscape) solution promotes all HTML components (forms, frames etc.) to the scripted object level. HTML documents gradually evolve towards dynamic scripts, passed between servers and clients and integrating all multimedia components of HTML with support for user-document interaction. 2) Java OS solution - Sun plans to put Java directly on computer hardware. All backend computers become Java Web servers, all frontends become Java terminals/PDAs. 3) VRML 2.0 -> Televirtuality (TVR) - here the paradigm is shifted from textual/multimedia to 3D visual interactive spaces governed by TVR protocol that integrates VRML and Java. In the near term, none of these solutions will likely dominate and Web'96 will be a mixture of all these components. We conclude that the single language/protocol based integration path in not promising in the near term on the multilingual Web. I.10: Topologies for Interactive Collaboratory Web Environments -I Complex interactive collaboratory '96 Web needs to develop new higher level content authoring techniques to stay intuitive and pervasive. Increasing object orientation will result in better encapsulation and language/protocol independence of the individual functional modules. Analysis of high level topologies or "compute-webs" of such modules provides a useful classification and dataflow based integration paradigm which we adopt for RL CIV project. I.11: Topologies for Interactive Collaboratory Web Environments - II: Typical topologies a) one-to-one - standard client-server based solitary surfing b) one-to-many - webcast, useful e.g. for distance education or training c) many-to-one - e.g. a report form a team of intelligent agents, employed in some websearch task d) many-to-many-direct - a real-time collaboratory such as MOO or chat spaces e) many-to-many-moderated - bridge based shared spaces such as in web based telemedicine with intelligent middleware switch/moderator f) synchronized many to many - a cluster management circuit, running continuously in the background to monitor connectivity and resource utilization of individual servers. (see manufacture of aircraft by distributed company) I.12: Dataflow Based Integration Technology We conclude that the topological decomposition of Web applications in terms of functional modules connected by MIME communication channels offers a promising uniform classification and integration paradigm for distributed/collaboratory Web systems. Such a dataflow framework offers the following attractive features: a) language/protocol independent design b) simple/minimal module API requires (just well defined I/O) c) scalability (compute-webs become composite modules) d) module mobility e) compatibility with existing dataflow models such as AVS/Khoros on compute side and LotusNotes in Information arena. f) maximal parallelism g) intuitive design/authoring based on visual programming I.13: Integration Concepts: WebFlow, WebTools, WebTop Applications We identified three main integration concepts: Two technology components: WebFlow, WebTool(kit)s and applications viewed as WebTop Systems. WebFlow - Web based dataflow infrastructure. Web is viewed as a collection of computation capable servers, with each server managing a collection of modules, connected by dataflow channels. WebTool(kit)s - reusable compute-webs, packaged/maintained as composite modules, and used for quick assembly of distributed applications. WebTop Systems - distributed applications, constructed as compute-webs of webtools modules. RL CIV will protoype WebFlow in the integration technologies sector, package component technologies as WebTools, and use for integrating four selected applications as WebTops. I.14: WebFlow - Web-based Coarse Grain Data (Object) Flow We will develop two trial implementations of the WebFlow infrastructure: a) based on conventional HTTPD + CGI Web servers b) based on Java collaboratory Web servers In model a), modules are CGI processes scheduled by Unix or Windows OS, communicating by HTTP, and maintaining the internal state via MIME files (virtual memory) In model b), modules are Java classes, scheduled by multithreaded Java runtime, communicating via Java sockets, and maintaining internal state in the server memory. A uniform, language/protocol/server technology independent module API will be designed and implemented in this project. All applications and component technologies will be provided with WebFlow module wrappers and selected modules (e.g. for telemedicine) will be developed from scratch in the WebFlow framework. I.15: WebTools - ensemble of reusable WebFlow modules WebFlow will offer a minimal set of base modules such as client, server, MIME document, viewer, frame, and visual compute-web editor. Editor will be written as a Java applet, with the initial design inherited from AVS/Khoros. Using this base set of primitive modules, new composite modules will be constructed and maintained as reusable WebTool(kit)s. WebTools layer will assure interoperability with commercial Web tools such as under development by Netscape. Off-the-shelf tools will be used whenever possible, and only application-specific components will be wrapped or developed from scratch. For example, some base WebRDBMS tools will be provided by Netscape, but the telemedicine application will require advanced application-specific support for patient record databases. The base Oracle API in Netscape will be encapsulated as a core module and then used for developing custom extensions. Also, this Netscape support is not yet available so we will likely develop initial prototoype using WOW or oraperl. I.16: WebTop Systems - WebFlow based distributed applications A WebTop system will typically include some default system level webtools such as cluster management support, a set of reusable webtools (e.g. compute-web editors, monitors, user guide support etc.) and a set of application-specific modules, developed from scratch or constructed as wrappers to existing software packages (e.g. GEMACS or weather simulator). A simple example of a minimal WebTop is Netscape2 based WebFoil package for electronic presentation, recently prototyped at NPAC. A more complex example is a Web based Telemedicine Office, to be prototyped as part of the RL CIV project. More generally, RL CIV will prototype four WebTop systems for Command and Control, Weather Simulation, EMS and Telemedicine. In a hierarchial fashion, Command and Control WebTop will incorporate those of Waether and EMS as submodules I.17:Preliminary Design of WebFlow Production Version of WebFoil WebFlow based production version of WebFoil will be modularized in terms of a set of backend modules (such as foil server, audio server or database server), and a set of frontend modules (such as NavigationBar, Clock, ListPanel, FoilViewer etc.). NavigationBar is a composite module built out of button modules. Buttons are further decomposed - for example the [Next] button is a compute-web including Counter, FoilLoader and ExceptionHandler primitive modules. Buttons are visually authored using the compute-web editor, and the resulting JavaScript functions are constructed automatically be the WebFlow "visual JavaScript compiler" (i.e. graph->source generator) Functional/Dataflow decomposition of WebFoil will make it more robust and less sensitive to rapid changes of the JavaScript specification. I.18:Possible WebFlow Implementation of Patient Record Database The top-level WebFlow topology here is many-to-many-moderated with care portals, care units and the bridge as the major coarse grain modules. Two care portal instances will be developed: home care terminals (based on NeatTools) and the patient record database support for nurses. Frontend GUI authoring and customization will be similar as in the WebFoil example. Backend modules will include DB servers and biosignal processing circuits. The latter can be located either at the client side (.e.g eye movement tracking applet) or at the server side (e.g. voice recognizer or blood pressure monitor). Simple rule-based compute-web in the bridge area will be constructed to help nurses in the assessment and diagnosis process. Infrastructure -- Hardware support SGI Onyx SMP 4 R4400 150 MHz CPUs, 1MB secondary cache/CPU, 256 MB RAM 140 Mbit/s ATM interface directly connected to RL via NYNET 20 GB+ disk space, real-time file system support available Display system: Reality Engine with two texture processors maximal supported resolution: 1600x1200 pixels at 60 Hz (~2Mpixel display) suitable monitor installed A farm of SGI desktop machines for developers, some high-end, some with ATM NIC Acquisition of new MIPS5K/MIPS10K machines with built rendering support in negotiation phase Infrastructure -- SGI Support for stereoscopic viewing Three stereo modes supported on SGI machines Option 1. 1280x492 (512 on Onyx) pixels per right/left field (eye), at 60 field-pairs per second. Supported by all models. Requires shutter glasses. Reduces spatial resolution to one-half of the original display. Only one resolution supported. Option 2. Right/left buffer allocated to the right/left eye. Requires 120 fields/sec, 90 fields/sec is a minimum. Supported by Reality Engine only. Requires shutter glasses. Does not reduce spatial resolution. Supports multiple resolutions. Maximal spatial resolution for stereo viewing depends on pixel bus bandwidth limit. Option 3. Separate video channel dedicated to each eye. Requires Multichannel Video option (MCO) or multiple graphics pipes. Used with VR helmets or with aligned two-head rear projection displays, for which it requires linear polarizers and glasses. Infrastructure -- Proposed Support for stereoscopic viewing Support for option 1 for development systems Support for option 2 for the demo system (Onyx) Demonstration space in the NPAC conference room Stereoscopic viewing support from Stereographics Crystal Eyes VR (shutter glasses and head tracking system) Display unit: stereo ready Barco system Difficulty: maximal spatial resolution of the stereographic system unclear. New Infinite Reality engine from SGI studied as a Reality Engine replacement. Support for option 3 under consideration in the context of new low price helmet systems. Option 1 and option 2 support will be installed and operational within 2 months Infrastructure -- Database Parallel Database Server Facility Currently running of 4 wide nodes of the SP2 Power Server, 1GB RAM, 70 GB of disk space Runs Oracle relational DBMS. Supports both parallel server and parallel data query capability 4 instances of the server run against the same database Network access upgrade to the parallel server pending: switched Ethernet to FDDI backbone. Relational-Object-Oriented Database Illustra DBMS running on the SGI Onyx provides support for multimedia data types very high performance (low overhead) engine directly supports GIS technology Infrastructure -- Networking ATM cluster Two ASX-200 switches, 9 workstations ATM LAN connected to the CASE ATM network: IBM 8220 ATM hubs (25 Mbit/sec), IBM 8260 hub as a gateway integrated with FDDI via LAX-20 (pending release of LAN Emulation software) connection to Rome Laboratory has been successfully tested using Video Server; VoD clients are being installed in RL as a permanent technology addition. Outstanding effort: direct connectivity to the RL Paragon ISDN connectivity Four dial-up lines connected via DigiBoard LAN bridges; operational 8 lines ISDN dial up with CISCO router has been installed and is being tested Infrastructure -- MM Collaborative Environments over ISDN Goal: development of C3I MM collaborative applications using Web/ProShare video conferencing over ISDN networks Status: Evaluated the use of ISDN technology to provide universal Multimedia Collaborative Environments Evaluated existing ISDN video conferencing tools (e.g., Intel ProShare) to deliver multimedia collaborative services over ISDN Ongoing integration of the ProShare system with Web (effort based on ProShare development kit) Infrastructure -- Speech Recognition Support: BBN Hark System Goal: establish speech recognition as a multimedia indexing technology Assigned GRA to project (has Computer Science MS); Installed Hark on an SGI platform Initial Review of BBN Hark System Features -- continuous speech recognition (CSR) supports 2,000 word vocabularies selected from a 100K words microphone, telephone, file input Prototyper Toolkit -- write, compile, test grammar files create application dictionaries contains phonetic dictionary (extendable) grammars used as input for HARK recognizer engine Recognizer -- voice model describes the individual words to be recognized grammar defines legitimate seqeuences of words quality of CSR a function of the quality of defined grammar files and dictionaries Component Technologies T1:Virtual Reality T2:Network Management and Data Transport T2:Compression T3:(Digital Video) Conferencing T4:Geographical Information Systems(GIS) and VRML T5:Distributed and Parallel MultiMedia Databases T1.0: VR Front-Ends: List of Topics T1.1: VR Software: evolving VRML as the ultimate VR standard T1.2: VR Hardware 1: Off-the-shelf peripherals T1.3: VR Hardware 2: Interface Lab at SU/NPAC T1.4: Near term RL CIV project: Neat Tools for Home based Care T1.1: VR Software: evolving VRML as the ultimate VR standard Based on vigorous activities of the Web software community, we assume that the evolving VRML protocol will provide software foundation for the client side VR software environment. Current protocol VRML 1.0 specifies base set of 3D object classes for geometry, rendering and networked composition of navigable and clickable scene graphs (or worlds). VRML 2.0 will (prototypes summer 96) provide support for object behavior (animation) and client-world real-time interaction. object behavior (animation) and client-world real-time interaction. T1.1:Implications of VRML Evolution for RL CIV Multiuser shared virtual environments will naturally emerge as the next step in this VRML evolution process. Good Collaboratory networked VR or 'Televirtuality' (TVR) protocols will not be available during the course of this project. We will therefore focus on 2D based collaboratory and single user 3D VR aspects of CIV systems during this project. Our integration technologies such as WebFlow will be applicable to TVR later on when it emerges as the ultimate Cyberspace protocol. T1.2: Hardware 1: Off-the-shelf peripherals - I We will select, test, customize and use within the RL CIV project a set of standard low cost off-the-self sensory VR peripherals to handle input and output streams. These devices will be used to support flexible user control for and participation in technology demonstrations such as GIS, Weather or EMS. Device selection will be guided by usability, human factors considerations, and interoperability between collaborative wall displays and single user/developer computer monitors. T1.2: Hardware 1: Off-the-shelf peripherals - II Polarized shutter glasses, available at low cost from various vendors, offer a natural front-end candidate for the video output. Alternative low end HMD mode will be also supported to address sensory immersive support for VRML. Here the promising current candidate is from Virtual I/O - a truly lightweight, reasonable resolution and display quality headmount, which also offers an easy switch between fully immersive and augmented reality modes. 3D sound (Crystal River -- Convolvotron) supported by VRML 2.0 will be investigated Cricket-like input devices seem to be most practical for CIV applications. These low cost peripherals typically offer 6 degrees of freedom location capability, tactile feedback and one or a few special purpose control buttons. The device is hand-held, light and easy to operate in various lab settings. T1.3: A related project - Interface Lab at NPAC In parallel with standard/commercial VR hardware/software solutions for the RL CIV project, we are also pursuing at NPAC a research project in advanced human interfaces within the Interface Lab planned by Dave Warner. This project will address innovative techniques (soundchair, biosignal detecting and multiplexing) for the full body sensory immersion and the associated capabilities for real-time control, information integration and holistic decision making support. A more prevasive, general purpose component of this research, based on Warner's "neat thing" input device, and aimed at more near term home based care applications, will be addressed as part of the RL CIV project to illustrate role of VR in C2. T1.4: Near Term Project: NeatTools for Home based Care -- Technology Warner's 'neat thing' system includes low cost sensors (such as bipolar electrodes for biosignal recording), the 'thing' box for signal amplification and multiplexing, and the 'neat' software for real-time signal processing and dataflow based filter authoring. Current version of 'neat' is a PC DOS based program with custom GUI. We will develop for the Web based Command and Control environment, a Java/JavaScript version of 'neat' and package it as a collection of WebFlow modules. Thre resulting NeatTools, written in Java and composable in JavaScript into customized multiframe Netscape2 control panels will facilitate user friendly web based custom interface authoring for specialized applications. T1.4: Near Term Project: NeatTools for Home based Care -- Applications Initial application targets are rehabilitation and chronic disabilities support where the neat thing was used successfully in the past. This will prototype an application independent, web-based VR interface design kit which will allow for the exploration of new, more pervasive application domains for this class of sensory devices. A high-end high-performance version of NeatTools is contemplated for use in Warner's Interface Lab experiments. T2:Web-based Network Management Web-based interface for network monitoring, controling, larger scale computer networks. Implementation of AIX Netview/6000 from IBM. Automatic discovery of the network resources(topology) Event reporting. data collection and display in form of graphs ability to develop sophisticated application management using Netview API. Using UNIX tools for enhancing the Netview report http://afognak.npac.syr.edu:8022 T2:WebComm: Web Communication System for Large HPCC Applications T2: Compression - Motivation Compression technology decreases (by factor 20-200) the time and cost of transmission and storage requirements eliminating spacial, spectral and temporal redundancy Evaluation of several image compression (JPEG, JBIG, Fractal, Wavelets) and video compression (MPEG, motion JPEG, H.261/263, MVC1, etc) technologies show great potential of wavelet-based methods Wavelets : comprehensive, modern approach to advanced signal analysis and processing Public domain software (EPIC, H-compress, Khoros Toolkit) - poor quality Commercial products of high quality (AccuPress) available but without source and API and so cannot be integrated into other components such as VRML texture maps, digital video etc. To proceed with the project we need full access to the wavelet based engine for compression/decompression of images and movies T2: Compression -Goal and Status Goal : development and implementation of a robust, wavelet-based, compression / decompression engine for which will allow us to build our own compression software for still images and video streams Status : 2D Image Finished except for packaging T2: Compression - Detailed Progress Implementation of 2D still image wavelet compression engine done during the last three months Algorithm Orthogonal reversible pyramid algorithm with 9-tap symmetric quadrature mirror filters Zerotree quantization of transform coefficients with embedding Arithmetic coder Major features 24 bit color, 8 bit grey scale, no limit for compression ratio portability: installed on IBM, SGI, Sun, DEC modular structure: easy for testing fully adaptive output stream: one can cut off the tail of the codestream and the rest is the same as the codestream for higher compression ratio object methodology: readability and easy for introducing changes Advantage: source available NPAC developed wavelet compression software (Wv) of quality comparable with the best commercial solutions T2: Compression (2D) Comparison Wv versus AccuPress for 512x512 grayscale Lenna Image T2: Compression - Future Plans Implementation of a player for displaying wavelet compressed images Developing and implementation of wavelet plug-ins for Web browser Investigation of various new video compression technologies including : H263 (Sep 95), MPEG-4 (standard expected in 1998) Development of wavelet-based video-codec. Focus: transfer over low bandwidth ISDN (BRI) network - design of transport and protocol layer required Motion estimation and prediction techniques needed to satisfy real time, low bandwidth and quality requirements T3: (Digital Video) Conferencing Solutions Results of the previous "Collaboratory and Telecommunication Experiments" project (contract No C-4-2803) show that there are lots of proprietary videoconferencing and conferencing solution Multivendor standards exists (T.120, H.320) or are under development. Still huge interoperability problems Currently videoconferencing technologies (InSoft, ProShare, InPerson) are NOT integrated with Web and this is certain to change! Rapid change in technology is observed right now T3: New Commercial Conferencing Developments Collaborative Command and Control environment requires integration of videoconferencing and Web technologies Evaluation of Argonne project - 3D virtual environment based on the MOO paradigm (virtual laboratories) -- Argonne has Java based prototype running Intel works on integration of ProShare with the Web Netscape acquired Collabra Software Inc. (Sep 95) - the leading independent developer of collaborative computing software Cosmo technology announced by SGI in December 95 - development environment for creating media-rich, 3D applications Allows transformation of the Web into fully interactive, multimedia environment - enables creating and viewing Cosmo suite includes: Cosmo Create, Cosmo Code, Cosmo Player, Cosmo Media Base Supports the open standards of the Web including HTML, VRML 2.0 and Java T3: New Commercial Conferencing Developments (Continued) Plug-ins technology in Netscape Navigator 2.0 available for Macintosh and Windows -Inline support for a huge range of Live Objects - delivery of rich multimedia content through Web Unix implementation will be available soon Lightning Strike by Infinet Op - optimized wavelet image codec Real Audio by Progressive Networks - live and on-demand real-time audio over 14.4 Kbps or faster connections VDOLive by VDOnet - TrueSpeech and wavelets; delivery over 28.8 Kbps WebFX by PaperSoftware - 3D VRML platform that enables capability to fly through VRML worlds and run VRML applications written in Java CoolTalk and CoolView by InSoft - allows users to talk to and see each other, share graphics in real time over the Net; includes data collaboration; compatible with IICE - InSoft Internet Collaborative Environment We limited ourselves to observe the new products and technologies developed and compatible with Netscape products as we expect to "just" integrate commercial "plug-ins" and not develop our own "solution" for conferencing T4:Geographical Information Systems Provides geographical interface to spatially referenced information (place names, population, etc) and terrain models for overlaying electromagnetic and weather simulations User can interactively navigate over the terrain, and point and click (or search) to find information (in the form of a web page) for a particular region Terrain model utilizes digital terrain data Elevation data at 100 meter resolution Satellite images at 30 meter resolution Could use other data such as Synthetic Aperture Radar 2D terrain viewer uses only satellite images, has been implemented in Java 3D terrain viewer uses satellite images draped over an elevation grid, has been implemented in VRML Spatially referenced information from a multimedia database is overlaid onto the 2D or 3D image T4:GIS - 2D Terrain Rendering in Java Originally developed using Tcl/Tk for the user interface and customized Unix client-server networking code. Java enabled us to create a simpler, more portable implementation, available via the Web. User interface on the client is a Java applet. Terrain data and other information is stored on the Web server and accessed via standard URL requests. Multi-threading in Java enables navigation of current section of terrain while downloading images of the neighboring sections. Currently creating Java interface to Illustra database used for 3D VRML terrain viewer, so 2D and 3D versions can access same spatially referenced data. T4:GIS - 3D Terrain Rendering in VRML Data is stored in Illustra Database System Terrain shape data - elevation and color data Embedded object data - objects that are on the surface VRML representation is created in real time when requested The same data may be visualized in various ways (terrain, objects) Parameters like resolution, size, altitude magnification, etc. are set by the user T4:Milestones for GIS Achievements in past 3 months Developed Java implementation of 2D terrain viewer Developed VRML implementation of 3D terrain viewer Developed interface to Illustra database for VRML viewer Milestones for next 3 months Add more terrain and GIS data Optimize performance of Java and VRML viewers Milestones for next 6 months Develop Java interface to illustra database so 2D and 3D viewers access the same GIS data Integrate Java and VRML viewers Milestones for next 12 months Overlay electromagnic and weather simulations onto 3D terrain model T5: Multimedia Databases -- Overview Focus on the following two efforts in the reporting period: Integration of the video server technology with database backend and Web frontend Base VOD technology developed under separate contract Goal: provide search and random access capabilities to the video server contents Status: the framework designed and implemented, working prototype in place Database support for VRML Goal: provide efficient means to support object interaction and adaptability in VRML Premise: flat file system storage inadequate in dynamic environment Status: Illustra database selected, installed and tested. Working prototype in place. T5: Multimedia Databases -- Video on Demand Server Overview Video on demand environment has been extended to include both NPAC and Rome Laboratory. Video on Demand is available in RL over the ATM delivery trunk from a number of NPAC servers. Client-server video on demand architecture has been implemented using Windows NT and SGI Irix machines as servers and SGI desktops and PCs as video clients. Streaming, real time video delivery has been implemented and integrated with Web front ends. Support for video indexing via closed-caption/free text search is available. Video database is integrated with the text database for both video indexing/retrieval and video server management. Integrated with commercial Video for Windows technology: off the shelf displays supported. T5: Multimedia Databases -- Architecture of Video on Demand Server T5: Multimedia Databases -- Database Support for VRML Database system is used to store the data fast retrieval of data (indexing) language to query and manipulate data simultaneous access by many users Database keeps information about the components - not the final VRML form easy and powerful update more compact storage VRML representation is created dynamically result is up-to-date multiple views on the same data are possible queries can select parts of the data T5: Database Support for VRML -- System Architecture T5: Database Support for VRML and GIS Application Terrain data - describes the shape and color of the terrain surface An array of elevation and color data indexed for fast retrieval of small parts using various resolution levels Embedded objects data - represents objects on the surface (buildings, trees) A set of objects characterized by position and type Objects may be simple or composed Simple objects are described using VRML Composed objects are represented set of other objects and spatial relations between them Applications A1:Electromagnetic Simulations A2:Weather Simulations A3:Command and Control A4:Medical Information Systems and Telemedicine A1,A3:Electromagnetic Simulation for C2 -- Selection of Application Two Candidates Examined SRC & NPAC code (limited functionality) Rome LaboratoryÕs GEMACS (General Electromagnetic Model for the Analysis of Complex Systems) GEMACS Chosen No proprietary issues for Air Force GEMACS 4 is available (GEMACS 5 is restricted) InfoMall Partner (Ultra) currently under contract to RL/ERST to parallelize GEMACS 5 Ñ will assist NPAC A1,A3:NPAC Planned Use of GEMACS (4) -- Scenarios and Integration into C2 Simulation (This takes into account predeployment, rehearsal, analysis and operational requirements) Here are Two Possible Generic Scenarios Cessna over New York Drug interdiction mission Low flying small aircraft, terrain masking Civil air radar detections (possible cueing from national satellites?) Bosnia theater Ñ JSTARS Air Force mission TBD Dynamic vs Static 3-dimensional radar cross section (RCS) rendering of fly-by No space-time/doppler processing A1:Dynamic EM Modeling Approach Use GEMACS to model complex scattering geometry against surveillance radars Employ high frequency, geometric theory of diffraction, method of moments, etc. methods to compute full 3-dimensional static RCS(radar crosssection) pattern Use radar range equation (mono- or bistatic) to dynamically compute surveillance radar detections Fly target according to flight plan, include terrain masking and surveillance radar geometries (can also include stochastic radar noise/clutter model) Obtain RCS on the fly using above method Forward detection reports to C2 Òsituation assessment and displayÓ functions A2:Real-Time Interactive Distributed Weather Information System -- Overview Choice has been made Oklahoma Advanced Regional Prediction System (ARPS) code. This leverages the several millions of dollars have been invested into this code which is used in the "real world". This is produced by the leading mesoscale group in the world. Current Progress of the code. Test runs have worked properly on NPAC machines. Capabilities of the code. Physics of cloud water, rain water, cloud ice, snow exist. A parallel version of the code also exist which NPAC helped developed. Model Design Philosophy. Future work. A2:Current Progress with ARPS ARPS code has compilied and ran on the cluster of DEC Alpha's. The runs have currently been using data centered around Oklahoma. Current work will make the runs use terrain data around the State of N.Y. Easy to set up new datasets: Use the Terrain data Preprocessor. Input the latitude and longitude for Syracuse. Use the multi-pass analysis since the State of N.Y. is highly variable in terrain elevation, i.e. mountains and lakes are present. Terrain data is first smoothed and then interpolated onto the computation mesh. The North American Terrain data set used with this code is accurate to within 20-30 meters. Higher resolution data is available only for the CONUS. The data is originally obtained by the National Center for Atmospheric Research (NCAR). A2:Current Progress with ARPS(continued) Fidelity of ARPS prediction in RL CIV depend on level of effort needed to customize the code to N.Y. state. This cannot be estimated yet but this uncertainity does not affect ability to deliver good module illustrating capability ARPS can be initialized with a horizontally homogenous inital state or a three-dimensionally varying analysis. Horizontally homogenous data does not vary in the computational domain on the two surface. Three-dimensionally varying data varies in the N,S,E,W and vertical directions. NPAC is investigating sources of local weather data. A2:Capabilities of the ARPS code Non-hydrostatic, compressible dynamics in a terrain-following vertical coordinate. 6 water phases microphysics (water vapor, cloud water, rain water, cloud ice, snow, and hail). Code is designed for incorporating spherical coordinate Doppler radar and other data. Storm-tracking capability. Platforms support. Conventional scalar machines (IBM RS/6000, DEC Alpha, etc.). Vector machines (Cray C-90, etc.). MPP machines (Cray T3D, CM5). Workstation clusters using PVM. Data parallel version for the Cray T3D will be available. A2:ARPS Simulation Capability Other current numerical prediction models lack the spatial resolution required to capture small-scale, short-duration events such as snow bands. General Prediction goals are: Mesoscale Phenomena. 0 to 12 hours. Location of events to within 50 km. Timing of events to within 1 hr. DV ± 5 m/s, DT ± 3 Kelvin, precip rate ± 5 mm/hr. Stormscale Phenomena. 0 to 6 hours. Location of events to within 10 km. Timing of events to within 15 min. DV ± 5 m/s, DT ± 2 Kelvin, precip rate ± 5 mm/hr. Microscale Phenomena. 0 to 1 hours. Location of events to within 1 km. Timing of events to within 5 min. DV ± 2 m/s, DT ± 2 Kelvin, precip rate ± 2 mm/hr. A2:Future RL CIV Work in Weather Application Continued investigation and understanding of the ARPS code. Verify program execution in parallel environment. Define required changes to input parameters/files for local terrain & weather. Design output for VRML engine: Currently data is output to files. Current data is plotted with the NCAR graphics package. Decisions need to be made whether to convert output file to VRML (post processor) or change ARPS output routines. Need to coordinate with other NPAC efforts on terrain rendering and electromagnetic scattering. A4: Telemedicine - List of Topics A4.1: General Framework - Telemedicine Lessons A4.2: General Framework -- The new WebMed Approach A4.3: Project Team A4.4: Planned Projects A4.5: Web Technologies for Telemedicine A4.6: Thrust 1 - Medical Information Gathering A4.7: Thrust 2 - General Purpose Telemedical Services A4.8: Thrust 3 - Specialized Value-Adding Services A4.9: Thrust 4 - WebFlow/Bridge as Integration Framework A4.1: General Framework - Telemedicine Lessons Telemedicine concepts until mid '95 were based on the assumption of rapid onset of the broadband wide area networking infrastructure. Dominant anticipated medium was direct life video linkage between patients and physicians. However this is not considered by many to be very succesful and new approach to Telemedicine is based on decision support for doctors with an environment very similar to that needed by Command and Control Need Image Processing not weather simulation but similar adaptive access to diverse distributed databases Slowdown in the ATM deployment, rapid explosion of Web technologies with variable bandwidth conditions, and new social and economic needs for the managed care based medicine, implies currently the paradign shift in the near term telemedicial environments. A4.2: General Framework -- The new WebMed Approach Warner' team came up with the Bridge Concept which was prototyped by I3 and ECU and succesfully demonstrated with Web Components from NPAC at WWVR'96 in San Diego The Bridge connects patients/care portals with quality healthcare professionals (DOCking stations) via the intelligent middleware station, offering suitable routing and optimizing the message traffic, service quality and expert time utilization. The emerging Web based framework (WebMed) addresses near term implementation in terms of today's networks and matches the social/patients and HMO/economy needs in terms of pervasive low cost infrastructure. New Web based telemedicine initiative at NPAC addresses these issues in a set of planned pilot projects. A4.3: WebMed Project Team Robert Corona - previously family practitioner, now neuropathologist at SUNY HSC, provides both general and specialized medical expertise and connectivity (via CareNet program) with CNY telemedicine activities. Wojtek Furmanski - SU/Physics & NPAC, expertise in interactive Web technologies, distributed software engineering and system integration. Edward Lipson - SU/Physics, expertise in biophysics, medical imaging, connectivity with other SU activities in telemedicine. Roman Markowksi - SU/NPAC, expertise in ATM, networking infrastructure and core technologies (streamlined media, databases) Dave Warner - I3/Loma Linda and SU/NPAC (Nason Fellow), expertise in use of human sensory interfaces for rehabilitation and disabilities, overall vision of and connectivity across domains of the telemedical society. 3 NPAC GRAs, 1 junior NPAC researcher Larger team includes (group of ~30) Warner collaborators in California, Minnesota and North Carolina. A4.4: Possible WebMed Projects - I School Nurse - Web based patient record database with links to medical information (diagnosis, treatment) and 3 hiearchy levels: 1) nurse terminals in schools, connected to 2) nurse practitioner station at the SU Nursing College, connected to 3) pediatrician station at SUNY HSC. This is a Pilot project to prototype an instance of the telemedicine Bridge concept. Medical Imaging Web Server - an advanced image processing toolkit, including publicly available and in-house developed (e.g. wavelet compression or pattern recognition) algorithms, packaged and published as a Web service to aid (possibly collaboratory) diagnosis process in the areas of radiology and pathology. Prototype developed as part of earlier Rome Contract A4.4: Possible WebMed Projects - II Home care terminals - Web (Java/JavaScript) version of Warner's "neat thing" sensory front-end, with rehabilitation and disabilities as initial application target. This builds on Warner's earlier activity in VR for which he is well known Other, very recently identified possible projects include medinfo network for a large ObGyn PPO in PA, Web information network for alternative medicine at the NIH, selected services for chronical disability cases in the local community (for instance parent of quadruplegic child is interested in working with us) A4.5: Web Technologies and Thrusts for Telemedicine We have identified 4 major thrusts aimed at developing reusable web-telemedical technologies: 1) Gathering Medical Information 2) Base/General Purpose Telemedical Services 3) Specialized Value-Adding Services 4) WebFlow/Bridge based Integration RL CIV telemedicine effort is viewed as a "breadth project" aimed at designing and prototyping common infrastructure underlying and shared by Command and Control as well as specialized medical applications listed in A4.4. A4.6: Thrust 1: Gathering Medical Information - I WWW already offers a vast amount of useful information in the healthcare area but its localization and maintenance becomes increasingly complex with the Web expansion. The goal of this thrust is to develop a systematic procedure for scanning the Web, selecting information relevant for the pilot projects, and constructing a set of relational or OO/MM databases. Our current/initial approach uses NPAC WebTool(kit)s to setup a collaboratory (CIV) environment, to be accessed by the team members at SU, NPAC, SUNY HSC, California, Minnesota and North Carolina, and used to collectively accumulate useful URLs. A4.6: Thrust 1: Gathering Medical Information - II Such, initially unstructured, a list is then analyzed, partitioned into a set of major categories (some of them created dynamically during the list inspection), and finally converted into a relational database with categories as keys, and URLs and cross-indices as table entries. Example categories include: Standards (e.g. HL7 for medical record scripting and transfer protocol, ICD-9 coding or ANAD diagnosis patterns), Databases (of images, druges, hospitals, patient record formats, software packages, textbooks etc.), People, Organizations etc. A4.7: Thrust 2 - General Purpose Telemedical Services - I One base service is electronic (Web based) patient record database which we have prototyped with Oracle and JavaScript. Such records will be likely distributed, with components located at home, family physician office and specialist lab. The associated services will offer tools for record design, editing, management/storage, history control, secure transfer, structural/hierarchical presentation, and statistical analysis. Our current application focus in this area is the school nurse project described in next foil. A4.7: Thrust 2 - General Purpose Telemedical Services - II: Application to Nursing Databases Based on Nursing Documentation literature, we selected a sample of diverse record formats used in real care units at various stages of the nursing process (assessment, diagnosis, planning, admission, treatment, discharge, long-term care) and we have started a trial implementation using Java/JavaScript multiframe frontends and Oracle/WOW backends. Deploy Prototype in May 96 for Syracuse TeleMedicine Conference Atomic components of a generic medical record will later be packaged as WebFlow modules, more composite multiframe constructs as as reusable WebTool(kit)s and the full record design and customization will be facilited by the visual WebFlow authoring tools. A4.8: Thrust 3 - Specialized Value-Added Services Medicine is a very broad field and we can initally address only a very narrow, carefully selected subset of specialized services. Our main goal here is to explore the system integration issues related to composing base (text/forms based) and specialized (ofter MM based) services and documentation. Production level implementations will be addressed by the telemedicine software industry. We selected two specialized services: medical imaging at the server side, and sensory frontend for rehabilitation at the client side. Both projects build on software and media developed and accumulated in the previous projects (TelePathology Workstation for imaging, PC version of 'neat' software for rehabilitation). A4.9: WebFlow/Bridge as Integration Framework WebFlow (i.e. Web based dataflow) offers the system integration paradigm, underlying the WebMed Bridge concept. Specific routing patterns, rule based circuits for middleware decision making, and dynamic collaboratory schemes (such as diagnosis whiteboards) are consistently represented in terms of high-level dataflow networks (compute-webs) of information processing modules. WebFlow modules include both the general purpose and specialized plugin services The compute-web based architecture is naturally scalable with successful compute-web designs encapsulated as composite modules (webtools).