
\documentstyle[11pt]{article}

\textwidth=6.0in
\textheight=8.5in
\topmargin=-0.5in
\oddsidemargin=0.25in
 
\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Executive Summary}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The development of powerful, inexpensive microprocessors has created a
revolution in the computing industry. High performance workstations, and
parallel computers that combine the power of many fast processors,  
are offering performance exceeding that of mainframes and traditional
vector supercomputers at a fraction of the cost.
However, as is always the case with new computer technology, 
the development of applications software has lagged behind the development
of new hardware, especially in the area of parallel processing.
This project addresses the development of applications using high performance 
computing, and in particular parallel processing, in the utility industry.

Power utilities have two main constraints -- security and economy. 
The power network must be operated in such a way that it is stable under 
any reasonable conditions, including possible equipment failures. 
This requires the provision of reserve generation and transmission capacity.
The other major constraint is of course economy. Utilities would like 
to provide the most cost-effective generation and transmission 
of the power needed to cover demand and meet security constraints.
A utility will have available many generators with different generation
and transmission costs. Choosing which generators to commit to power
production ({\it unit commitment}),
and how the power required to match the 
forecast demand should be distributed among these generators
({\it economic dispatch}), is a complex optimization problem.
Environmental constraints are also becoming more important. 

Substantial changes are occurring in the utility industry.
There is increasing competition from Non-Utility Generators (NUGs),
more contracts for buying and selling power
from neighboring utilities and NUGs,
and increasing demand for access to transmission networks. 
Environmental and economic concerns mean that generation and 
transmission facilities must be utilized as efficiently as possible.
These changes will make the secure and efficient management of the
power network even more complex and time-critical,
and more advanced computers, computer software, and
man-machine interfaces will be necessary.

Computer simulation currently plays a major role in the utility industry,
in both long-term network planning, 
and the on-line Energy Management System (EMS) used by control room 
operators for handling the day-to-day running of the power grid.
Many of the simulations are compute-bound, and the use of high performance
computing would allow more accurate and more frequent simulations to be
performed, providing increased security and less conservative estimates
of required generation and transmission reserves.
Due to the large scale of the power industry, even a small reduction
in operating overhead translates into large savings.
Estimates have indicated that reducing operating margins by
improving simulations could save NMPC millions of dollars per year.
This would also save energy resources and reduce environmental impact.



The recent TASC report on the application of supercomputing to the utility 
industry 
did a creditable job of summarizing the trends in supercomputing,
and outlining the types of compute-intensive utility problems which
could benefit from high performance computing.
However the report does not emphasize the fact that the hardware used in
high performance computers, such as vector and parallel processing, 
requires special software to be utilized effectively. 
Although vectorizing and parallelizing compilers do exist for many of 
these computers, it is very rare for a standard sequential program to 
obtain anywhere near the peak performance of these machines, and in many
cases the speed-ups obtained over sequential workstations are negligible. 
Obtaining the best performance from parallel computers almost always requires
different methods (or algorithms) to those used on sequential machines.

For some problems, implementing efficient parallel programs can be very 
difficult. Since power networks are sparsely connected (each component
is only connected to a few others), simulations of power systems generally
require the solution of sparse linear equations.
Although dense linear solvers can be parallelized very efficiently,
sparse linear solvers are one of the most difficult applications to 
implement on parallel computers.


\subsection*{Project Overview}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

One of the main goals of this project is to identify the basic algorithms
used in various utility applications, and to determine whether they 
can be parallelized efficiently, or if appropriate parallel algorithms
already exist that could be applied to these problems.
We have therefore undertaken the following activities concerned with
this broad objective:
%
\begin{itemize}
\item A study of the relevant issues involved in using parallel computing
      in the utility industry, and recommendations for the most suitable
      parallel hardware, software, and languages.
\item A survey of utility applications to determine those that are amenable
      to parallel processing.
\end{itemize}
%
We have also concentrated on developing parallel algorithms for the following 
specific applications that could benefit from high performance computing:
\begin{itemize}
\item  unit commitment
\item  contingency evaluation
\item  transient stability analysis
\item  sparse matrix algorithms
\end{itemize}
%
Speeding-up the programs for these applications can open up new areas of use.
A major goal of this project is to speed-up transient stability analysis to 
the point where it can be used on-line in an Energy Management System, 
to set near-real-time stability limits. 
We are therefore also working on the following problems:
%
\begin{itemize}
\item The development of an expert system and user interface for 
      classifying transient stability and finding stability limits.
\item The development of an on-line program for transient stability 
      analysis and determining stability limits.
\end{itemize}

\medskip
\noindent
{\bf Phase 1} of this project involved research to study the feasibility
of using parallel processing and expert systems for utility applications,
and the development of parallel algorithms and expert system rules 
for specific applications.\\
\\
{\bf Phase 2} of the project is to develop and benchmark prototype parallel 
programs and expert systems software. \\
\\
{\bf Phase 3} involves developing, documenting and testing production-grade
software which can be used at NMPC. \\


This report presents the work done in Phase 1 of the project. 
We have studied various applications used at NMPC, 
obtained a detailed knowledge of the programs and the sequential 
algorithms currently used, 
devised strategies for parallelizing these programs, 
developed an expert system knowledge base,
and in some cases have made substantial progress in writing programs
to implement these parallel algorithms and expert systems, which will 
continue in Phase 2 of this project.


\subsection*{Unit Commitment}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The standard method for solving the unit commitment problem is the 
use of dynamic programming optimization techniques.
Previous research at Syracuse University developed parallel 
dynamic programming algorithms that were used to implement a parallel
unit commitment program on a shared memory parallel computer.
We have studied this work and determined
a straightforward procedure for migrating this parallel dynamic 
programming code from a shared memory to a distributed memory architecture. 

We have also investigated the use of simulated annealing, a general-purpose 
optimization technique, for unit commitment.
We devised a strategy for using simulated annealing for unit commitment,
and have begun implementing this program using a
public-domain simulated annealing package.
We have also developed a preliminary implementation of our own improved
simulated annealing program, which uses methods to automatically find good
parameters which otherwise must be tuned by hand for each different
problem in order for simulated annealing to work effectively.
We have also developed a parallel version of this program.

Unit commitment at NMPC is currently done manually using a spreadsheet
that sets up a simple prioritized list of generators, with the least
expensive generators having highest priority. EPRI have developed an
advanced program called DYNAMICS that incorporates unit commitment,
economic dispatch, and environmental and emissions constraints.
NMPC is a test utility for this project. This program runs unit 
commitment for NMPC in only 1 minute on an IBM RS/6000 workstation, 
and the unit commitment group at NMPC are very happy with its performance
and results.  Unit commitment for the whole New York Power Pool
could be run on-line in only a few minutes using this program.
It therefore seems unnecessary to continue the development of 
a parallel unit commitment program. 

The focus of the work in this section of the project has been on parallel 
algorithms for simulated annealing and dynamic programming. 
Simulated annealing is a general technique which can be used for any 
hard optimization problem. 
It is therefore likely that a parallel simulated annealing algorithm
could be of use in other utility applications, 
for example the MAPS and ENGINEOUS programs from General Electric, 
which use optimization techniques to search for the most
cost-effective network configurations for planning studies.
MAPS currently uses linear programming techniques for optimization.
We therefore recommend that research be directed toward developing and
improving parallel algorithms for simulated annealing and linear 
programming.






\subsection*{Contingency Evaluation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

An important aspect of the operation of an electrical power network
is the security model, which simulates the response of the network to 
disturbances (or {\it contingencies}) such as component failure. 
These simulations allow control room operators to identify
and remedy possible sources of cascading failures,
that could cause a disruption of power generation or transmission. 

In the NMPC Energy Management System, contingency evaluation is done on
a list of over 200 independent contingencies. 
A very approximate simulation is first done to screen out the less problematic
contingencies, and a full simulation is done on the 10-20 worst cases.
Parallel processing can be used very simply and very efficiently by 
independently simulating different contingencies on different processors. 
This would allow more frequent contingency evaluation, a more accurate
screening procedure, and the testing of more contingencies, 
with the possibility of evaluating the effects of 
multiple simultaneous contingencies (this is currently not addressed).

Unfortunately the current software used in the NMPC EMS is run on an IBM 
mainframe, and would be extremely difficult to parallelize,
since it uses some non-standard languages and software management techniques
that are far removed from the standardized languages and operating systems
used in modern workstations and parallel computers.
However the EMS software is currently being ported to IBM RS/6000 
workstations, and is expected to be available this year.
Only minimal changes to this new software would be required to implement
an efficient parallel program for contingency evaluation that could run
on a network of IBM RS/6000 workstations or an IBM SP-1 parallel computer.
Implementing such a program would require a collaboration between 
NPAC, NMPC and STAG (the developers of the EMS software).

Parallel processing could also enable the development of an on-line
program to help an operator in devising strategies for preventive or 
corrective action in a situation where the occurrence of a contingency
would violate stability limits.
A program could be used to simulate many possible scenarios in parallel,
in order to identify a workable strategy to deal with a stability problem. 
If the security model could be integrated with a pricing model, then an
even better approach would be to search for not just a stable strategy,
but one that would minimizing incurred costs.



\subsection*{Transient Stability Analysis}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Transient stability analysis involves the study of the electrical 
network and generator dynamics for several seconds following a disturbance. 
The dynamics of each generator in the system are described by multiple 
differential equations, while the coupling of the generators to the 
electrical network is described by non-linear algebraic equations. 
Together these lead to a system of differential-algebraic equations (DAEs).
The standard algorithms used for transient stability analysis decouple
this system into independent differential and algebraic equations.
The differential equations are solved using standard numerical integration
techniques. The algebraic equations are solved using a standard iterative
method, with each iteration requiring the solution of a set of linear
equations using sparse matrix techniques.

We have decided to base our studies on the EPRI transient stability program
ETMSP, rather than the PTI PSS/E program currently used at NMPC.
This has been done for a number of reasons, including availability 
of the source code and documentation, access to the program developers,
and the availability of previous research on parallelism done by EPRI.
We have studied the EPRI ETMSP transient stability program in detail, and
devised an approach for parallelizing this program, drawing on 
recommendations of previous EPRI research.

The generator equations are independent, and thus should be easily 
parallelized. However the ETMSP program is not written in a manner
such that this independence is easily utilized -- there is no clean
loop over generators that can be simply parallelized.
Substantial restructuring of the current program is required before
this can be done. We believe this is straightforward, but may take up
to 6 months to complete, due to the complexity of the program 
(ETMSP has over 200,000 lines of Fortran).
%
The network equations can only be parallelized by using a parallel
sparse matrix solver. We have begun the integration of such a 
solver into ETMSP. We plan to test a number of different
parallel sparse solvers, including the one we are developing.

We believe it is feasible to parallelize the ETMSP transient
stability program and obtain speed-ups of around 10 on 16 processors.
Increasing the number of processors used beyond 16 is unlikely to have 
much effect, due to the lack of scalability of parallel sparse
matrix algorithms to larger numbers of processors.

Another way of speeding-up this program is to use better solution
techniques. We are investigating the use of advanced public-domain
differential-algebraic equation solvers.
These methods solve the coupled differential and algebraic equations
simultaneously, which should provide improved accuracy. They also offer
variable timestep and variable convergence numerical integration techniques
that should allow larger timesteps to be used, and thus faster simulations. 
We have investigated the requirements for integrating DAE solvers 
into the framework of ETMSP.  This will not be as straightforward as 
had been expected, since parts of ETMSP are ``hard-wired'' to the current
solution technique. We are looking into workarounds for this problem.

We have surveyed available DAE solvers and found that DASSL and LIMEX 
appear to be the most suitable for solving the transient stability problem. 
Another advantage in using DAE solvers is that a parallel 
implementation of DASSL, known as Cdassl, already exists. 
We have worked on improving the structure of Cdassl to make it easier 
to interface this program with ETMSP and different parallel sparse matrix
solvers.

During Phase 2 we aim to make substantial progress on developing 
prototype parallel programs based on ETMSP, using both the current 
decoupled methods and new methods incorporating DAE solvers.
We will initially implement these programs using a message passing language
for the IBM SP-1. This will be migrated to standardized portable parallel
languages when good compilers for these new languages become available.
Since we are using the EPRI transient stability program ETMSP, 
we recommend that this part of the project be co-ordinated with EPRI.


\subsection*{Sparse Matrix Algorithms}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Most simulations of power networks require the solution of large sets of 
coupled linear equations, which is done by sparse matrix techniques.
Sparse matrix solvers have complex and irregular data structures,
and are therefore much more difficult to parallelize than dense matrix
solvers, which are very regular, and thus easily distributed among 
multiple processors. The communication of data between processors in a
parallel sparse matrix solver is also much more complicated, 
and the amount of computation required is much less, than for a parallel
dense solver. Obtaining substantial speed-ups for a parallel sparse
matrix solver has thus proven to be an extremely difficult task, and
can usually only be achieved by using problem-specific methods that 
identify and effectively utilize any structure present in the sparse matrices.
General methods, although they exist, are usually less than optimal.

Power networks tend to be hierarchical in structure, and this can be
utilized to order the matrices into a structured 
{\it block-diagonal-bordered} form, where non-zero data elements occur
only in the borders and in blocks along the diagonal of the matrices.
We have performed a theoretical examination of the parallelism available
in block-diagonal-bordered matrices and developed software to
order matrices into this form.

Sparse matrix equations are solved using either {\it direct} or 
{\it iterative} methods. We have concentrated on implementing direct methods,
which are currently used in most power systems programs. However we are
also studying the possible use of iterative methods, which may be more
effective for parallel processing.
The standard direct methods used to solve matrix equations are 
Choleski factorization for symmetric matrices (which occur in many power 
systems problems, such as calculating power flows in a network),
and LU factorization for non-symmetric matrices (which occur in transient
stability analysis).

We have developed a parallel block-diagonal-bordered Choleski solver 
and implemented it on the Thinking Machines CM-5 parallel computer
using a message-passing paradigm.  
Preliminary results show that the speed-up that can be obtained from
this program is very limited (only a factor 2-3), 
due to the limited amount of calculation required. 
LU factorization should give a larger speed-up, since it requires more
calculation.
We are currently working on developing a parallel LU solver for
transient stability analysis.

Using DAE solvers for the transient stability problem produces larger
matrices with additional available calculations, due to the inclusion 
of the generator equations.
These matrices are naturally in block-diagonal-bordered form.
The increased size and computational complexity of these matrices should
allow much greater speed-up for parallel LU factorization.
We believe parallel algorithms can be developed that will give reasonably 
efficient performance for eight or sixteen processors.

Tasks remaining in our work include the development and implementation of two
additional parallel block-diagonal-bordered sparse matrix solvers, a direct
solver based on LU factorization and an iterative Gauss-Seidel solver.
These programs will be ported to the IBM SP-1 parallel computer,
and to a standardized portable parallel language that can be run on any 
distributed memory parallel computer.




\subsection*{Expert System and User Interface}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In order to provide an on-line transient stability program, there must
be an automated procedure for analyzing the output of the program, and
classifying the results as being stable or unstable.
This is currently done off-line by a trained engineer.
Our approach to automating this process is to develop an expert system 
that solves the problem in the same way as a human engineer.

This expert system should be useful not only for on-line applications,
but also in helping the engineer in traditional off-line transient 
stability studies. Programs for transient stability analysis produce
tremendous amounts of graphical output which must be analyzed.
An expert system would be able to prioritize or filter this data,
for example by flagging results which show clear instability,
filtering out results which are clearly stable, and presenting the
engineer those results which are most useful for making a decision,
through an improved user interface.

We have conducted interviews with NMPC and NYPP engineers to determine
criteria for classifying transient stability results, and used this
information to constructed a decision model for this process.
This has been used as input for the knowledge base of an expert system
for classification of transient stability results.
We have implemented a preliminary version of this expert system.
In Phase 2 this will be integrated with the ETMSP transient stability program.
This will involve some post-processing of the transient stability output,
with the results passed to the expert system for stability classification.
We have also been studying other research on using analytic methods and
neural networks to classify stability, which may be a useful complement 
to the expert system approach.

Stability assessment of the simulation results is only one part of the 
process of setting transient stability limits. 
The engineer must also choose appropriate contingencies and 
parameter values in order to find the limiting value for a certain component
or interface, above which the network becomes unstable. 
This is a more difficult problem, but we believe an expert system can 
also be developed for this process.
This will be investigated in Phase 2.

The volume and complexity of data and decisions faced by EMS operators 
is likely to increase in the future. There is a growing movement in the
utility industry toward the use of expert systems and artificial 
intelligence to cope with this problem.
In the face of information overload, issues such as the
human-computer interface will be critically important for the successful 
implementation of these expert system technologies.
We suggest that this is an important area for further 
study, and we plan to investigate this problem starting in Phase 2
of this project. 




\subsection*{On-Line Transient Stability Analysis}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% \epsfbox{flowchart.ps} ??

Off-line transient stability analysis is currently used to set the
component and interface stability limits used by NMPC Operations. 
These are set by infrequent studies (every few months), using a 
worst-case scenario. An on-line program to provide stability limits
on a near-real-time basis using the actual operating state of the 
power network would reduce the amount of excess generation 
required to satisfy security constraints, and increase safe transmission
capability to allow greater transfer of power from areas with lower 
generation costs.
It is estimated that NMPC could save as much as \$3M % ???
per year in this way.
EPRI have given similar arguments for large savings as motivation for
a large project to develop an on-line program for dynamic security
assessment, which aims to incorporate transient stability analysis into
an EMS contingency evaluation.

We have analyzed the requirements for an on-line program for
transient stability analysis, and the determination of stability limits,
and studied the related EPRI project for dynamic security assessment.
We believe it is feasible to develop an on-line system which will 
provide transient stability limits to control room operators. 
There are four main obstacles to be overcome to develop such a program:
%
\begin{enumerate}
\item {\bf Speeding-up transient stability analysis.}\\
      Currently transient stability analysis is too slow to be used for a
      near-real-time on-line program.
      We are using three approaches to tackle this problem:
      \begin{itemize}
      \item Parallelization over contingencies.\\
	    This is simple and efficient
	    but may not be adequate if expert knowledge and an iterative
	    procedure using feedback to affect later steps are used.
      \item Development of a parallel version of ETMSP.\\
            The main difficulty here is that ETMSP is a large and complex
            program, so fully implementing the parallel algorithm will take
            quite some time. 
      \item The use of improved solution techniques such as DAE solvers. \\
	    These offer great promise but integrating them into ETMSP
	    may be difficult.
      \end{itemize}
\item {\bf Development of an expert system for classifying stability.}\\
      An automated on-line program is needed to classify the ETMSP output
      as being stable or unstable.
\item {\bf Development of an expert system for finding stability limits.}\\
      We need some expert system rules that describe how an engineer
      would use the transient stability program to find the stability limit
      for a particular component or interface in the system. 
\item {\bf Integration of the on-line stability program into the control
      room EMS.}\\
      This is quite a large software engineering problem, which would 
      require input from STAG, the developers of the EMS software. 
      The on-line stability program would need to access the network data 
      from the EMS program and to return stability limits in a form that
      is usable by the EMS security model programs and the operators.
\end{enumerate}
%
We have already made substantial progress in the first two areas.
Developing an on-line stability program that can be integrated into the
control room EMS would require an agreement on interfacing software 
developed by STAG, EPRI and NPAC. Assuming that this could be arranged,
we estimate that such a program could be developed in Phase 3 of this
project, and be ready for testing at NMPC by the summer of 1995.



\subsection*{Parallelism and Computational Issues}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


In this section of the project we have studied issues relevant to 
the use of parallel computing in the utility industry. 
Parallel computing has become a much more attractive option for industrial 
applications, due to the following recent developments:
\begin{itemize}
\item Powerful parallel computers developed by mainstream computer 
      companies such as IBM, DEC and Hewlett-Packard, that are
      based on clusters of high performance workstations.
\item Standardized portable parallel languages such as High Performance 
      Fortran (HPF) and Message Passing Interface (MPI).
\item The development of libraries of portable parallel programs, 
      such as the SCALAPACK linear algebra package.
\end{itemize}
 
We believe distributed memory parallel computers based on 
workstation clusters, such as the IBM SP-1, 
are the most suitable high performance computers for the utility industry.
The architecture of these parallel machines is well-suited to any of
the applications discussed here, from the simple independent parallelism
of contingency evaluation, to problems where fast inter-processor
communication is critical, such as parallel sparse matrix solvers.
These machines also allow a much simpler integration and parallelization of
programs developed for single workstation platforms, which are becoming a
standard for computationally intensive utility applications.
Workstation clusters offer an extremely good ratio of price to performance.

Standardized parallel languages such as HPF and MPI should be used
for developing commercial parallel software. However good compilers for
these languages are not expected to be available for at least another
year. In the meantime, parallel programs should be written in a manner
that will allow easy migration of the software to these standard languages,
and will preferably be portable among different parallel computers. 
Where possible, standard libraries of portable parallel programs should be
utilized.
If specialized routines such as parallel block-diagonal-bordered sparse matrix
algorithms need to be developed, they should be written in such a way that
they have similar interfaces to programs in standard parallel libraries.






\subsection*{Survey of High Performance Computing and Parallel Processing 
         in the Utility Industry}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


We have surveyed a variety of applications used in the utility industry,
identified the basic algorithms used in these applications, 
and indicated whether parallel versions of these
algorithms already exist or could be developed. 
The survey has so far concentrated on applications related to the security
and economics of power network management and planning.
We plan to extend the scope of this survey in Phase 2 of the project.

We have so far identified the following areas as being the most promising
applications of parallel processing to the utility industry:
\begin{itemize}
\item Parallelism over independent contingencies for contingency evaluation
      and dynamic security assessment.
\item Use of parallel transient stability methods, including parallel 
      sparse matrix and DAE solvers, for mid-term and long-term
      stability analysis programs such as EPRI's ETMSP and LTSP.
\item Use of existing parallel eigenvalue solvers for voltage and
      small signals stability programs.
\item Development of parallel optimization techniques such as 
      simulated annealing and linear programming for 
      production costing programs such as MAPS.
\item Use of parallel processing to integrate multiple simulation programs
      into an advanced on-line Energy Management System.
\item Use of job-level parallelism, i.e. running different programs on 
      different processors rather than time-sharing on a single processor,
      to increase turn-around time for the many programs that are run
      concurrently in a central corporate computing facility.
\end{itemize}
 


\subsection*{Recommendations for Future Work}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In Phase 2, we are planning to focus on the follow action items:
\begin{enumerate}
\item Restructuring the ETMSP code for solving the generator equations,
      to allow independent parallelism over generators.
\item Developing parallel block-diagonal-bordered LU and iterative sparse
      matrix solvers for use within ETMSP.
\item Porting our parallel sparse matrix programs to the IBM SP-1.
\item Integrating of a number of parallel sparse matrix solvers into ETMSP.
\item Benchmarking the performance of various parallel sparse matrix solvers
      within ETMSP.
\item Integrating a parallel DAE solver into ETMSP for solving the transient
      stability equations.
\item Implementing a prototype expert system for the classification of 
      transient stability output data.
\item Developing a decision model for the process of determining stability
      limits.
\item Formulating specifications for an on-line program for determining 
      stability limits.
\item Investigating the structure of the STAG EMS software for the IBM
      mainframe and RS/6000 platforms, with regard to parallelizing the
      contingency analysis program and integrating an on-line program 
      for setting stability limits.
\item Studying the feasibility of parallelizing the MAPS program.
\item Investigating the possibly use of improved linear programming and
      simulated annealing optimization techniques within MAPS.
\item Studying the feasibility of using parallel versions of MAPS and ETMSP
      within ENGINEOUS. 
\item Extending our survey of application programs used at NMPC and the
      possible use of parallel processing for these programs.
\end{enumerate}

\end{document}


