%\documentclass[psfig]{article}
\documentclass{article}

\usepackage[dvips]{graphicx}

\begin{document}

%\title{Draft Working Proposal for Petasim V0.3 and UMD's Emulator}
\title{ Petasim V0.3.3 Report }

\author{%Yuhong Wen, Geoffrey C. Fox
\vspace{5mm}\\
Northeast Parallel Architectures Center\\
Syracuse University\\
111 College Place\\
Syracuse, NY 13244--4100\\
%\em \{wen, gcf\}@npac.syr.edu
}

\maketitle

\begin{abstract}

In the design of parallel computer systems and the developing of the 
applications,
it is very important to have some good performance prediction tools to get
an estimation result before the real architecture
construction and application implementation. This paper describes
a new approach -- {\em \tt Petasim}
, which will get the
abstractions of the applications and the underlying machine architecture
to give performance estimation. In this approach of performance estimate,
one can easily modify all the architecture parameters of a computer system, and the
behavior of the application to get the performance and execution cost
very quickly, which will greatly help the new computer architecture  and
application deign.

\end{abstract}

%% \paragraph{Keywords} 
%% Performance Estimation, Architecture Description, \\
%%  Application Execution Script 

%\section{Description of PetaSim and Emulator}
\section{Introduction}

Performance prediction for a large scale application, 
such as weather forecasting,
oil exploring, etc, is a very important and rather complex problem
for scientific computing, especially when involing the massively parallel computers. 
Provided with a good performance estimator, one can easily construct the application
model, adjust the system parameters to find out a suitable computing model.
We introduce a new approach in our design of {\em \tt Petasim}, which
will benefit both the computer system design and the application design.

%\begin{figure}[btp]
%\centerline{\psfig{figure=Image54.eps,height=4.6in}}
\begin{figure}[btp]
\begin{center}
\includegraphics[height = 3in]{Image54.eps}
\caption{\label{fig:performance}The Performance Prediction Process}
\end{center}
\end{figure}

Performance estimator {\em Petasim} is built around the performance prediction process sketched in
Fig.\ref{fig:performance} \cite{PAWS report}. The distinctive feather of our approach is the use 
of machine and problem abstractions which although less accurate than
detailed complete representations, can be expected to be more robust and
further quite appropriate for the rapid prototype needed in the design
of new machines, software and algorithms. The heart of this performance 
prediction process  are two technologies - HLAM (Hierarchical Application
Modeling Framework) and Petasim \cite{PAWS report}, 
\cite{FoxFurm97}. In this report, we will address the design
of {\em Petasim}, which will take three key inputs from HLAM, which 
describe respectively the target machines, application, script specifying
execution of the application on the machine, to get an estimation of the 
performance of the application running on the machine.


\section{Description of Petasim}



Performance estimator PETASIM is aimed at supporting the
(conceptual and detailed) design phases of parallel algorithms, systems software and hardware
architecture. Originally this was designed as a result of the two week long workshops - PAWS and
PetaSoft - aimed at understanding hardware and software architectures ten years from now when
Petaflop (\(10^{15}\)) scale computing can be expected. It was clear from these meetings that the
community needed better aids for performance estimation as the 8 groups (and 8 different machine
designs) present found it difficult to compare designs and in particular differed by a factor of
million (PAWS report) in estimating performance of even a set of extremely simple algorithms -
the PetaKernels -- on their new designs \cite{PAWS report},
\cite{PCW}. The most sophisticated PetaKernel was a regular finite
difference problem solved by simple Jacobi iteration which is of course well understood. These
workshops emphasized the need for tools to allow one to describe complex memory hierarchies
(present in all future and most current machine designs) and the mapping of problems onto them in
a way that allows reliable (if high level) performance estimates in the initial design and rapid
prototyping stages.

PETASIM is aimed at a middle ground - half way between detailed instruction level machine
simulation and simple "back of the envelope" performance estimates. It takes care of the
complexity - memory hierarchy, latencies, adaptivity and multiple program components which
make even high level performance estimates hard. It uses a crucial simplification - dealing with
data in the natural blocks (called aggregates in HLAM) suggested by memory systems - which
both speeds up the performance simulation and in many cases will lead to greater insight as to the
essential issues governing performance. We motivate and illustrate the design of PETASIM by the
well known formulae for parallel performance of simple regular applications on nodes without
significant memory hierarchy. Then we will have:
%(chapter 3 of PCW) one finds,
\\

Speed Up = Number of Nodes/(1 + Overhead)
\\

where the Overhead is proportional to \((Grain Size)^{-g} (t_{comm} / t_{calc} )\)
\\

where in this case natural data block size is the Grain Size or number of data points in each node.
The power g measures edge over area effects and is 1/d for a system of geometric dimension d.
\((t_{comm} / t_{calc} )\) represents a ratio of communication to compute performance of the hardware. Such
a formula shows the importance of identifying natural data block and how such high level analysis
allows one to understand the relation of performance to memory sizing, I/O and compute
capabilities of the hardware. PETASIM generalizes this "back of the envelope" estimate to more
complex problems and machines. It also includes not just primitive messaging performance (node
to node as used in above estimate) but also collective (such as multicast) mechanisms which are
present in most applications but ignored in many simple analyses. Note that the simple
performance estimate above is valid (with straightforward generalizations) on machines with the
simple two level distributed memory hierarchy - namely memory is either on or off processor -
which is essentially model built into the current generation of parallel programming systems as
typified by HPF or MPI. As we described, it is essential to generalize this machine
model whether we want to provide input to either parallel programming tools or to performance
estimation systems. Thus we believe our experience with HLAM and PETASIM will be very
valuable in helping designing the new generation of parallel programming environments needed
for the increasing complex high performance systems coming online.

Fortunately we have good evidence that we can generalize this naive analysis to more complex
problems and machines for indeed the Rutgers/UCSB group has studied granularity issues
\{cluster,dsc\} to identify the natural data block sizes and computation clusters based on
computation/communication ratios in more general hierarchical memories. They have developed
preliminary performance prediction and optimization tools (PYRROS \{PYRROS\}, D-PYRROS
\{jjiao\},RAPID \{RAPID\}) based on task graphs in which the impact of single-processor memory
hierarchy is addressed in the intra-task level and the impact of inter-processor communication
delay is identified in the inter-task level. These techniques have been shown effective for a
number of adaptive and static applications \{Iter-2,ship,Fernandez,SparseLU\} and it will be
naturally integrated into PETASIM as this Rutgers/UCSB technology is the basis of HLAM.

As well as a machine description, PETASIM requires a problem description where we will use the
HLAM as described above. Note also that application emulators also use the same
"aggregate" description of applications needed by PETASIM and these will be used as input. It is
not clear yet if we will convert these directly to the HLAM or allow a separate API for this style
of program description. We will provide even more work here later.

\section{The Design of Petasim}

\subsection{The Specifications of Petasim}

There are two parts of descriptions in the petasim performance estimation
system, architecture description and application description. The most 
general computer architectures can be specified using the PetaSIM {\em nodeset,
linkset} while the applications can be specified using {\em dataset, 
distribution} objects.

A {\em nodeset} is a collection of entities with current types allowed as:

\begin{description}
\item  $\circ$      memory with cache (with flushing rules) as special case 
\item  $\circ$      disk which is essentially same as a memory. 
\item  $\circ$      CPU where results can be calculated 
\item  $\circ$      pathway such as a bus, switch or network cable which concentrates data
\end{description}

A {\em linkset} connects {\em nodesets} together in various ways. {\em distributions} specify the horizontal
(geometrical) connectivity of {\em nodesets} and {\em linksets}. Typically these are arranged in a natural
default for the classic homogeneous architectures. The default mapping is inferred from sizes of
{\em nodesets} and done in a simple one-dimensional block fashion. The vertical (flow of information)
connectivity in the architecture is specified in the execution script with defaults implied in
architecture specification.

The application is specified by a {\em dataset} object whose implementation is controlled by a
{\em distribution} object that specifies classic HPF style geometric decomposition across memories and
CPU objects. The computation is specified by the execution script, which also specifies data
movement.

{\em nodeset, linkset, dataset and distribution} are Java classes that are subclassed as necessary to give
particular special cases with particular capabilities. They have methods that are defaulted for
simple cases but can be overridden for complicated cases.

\begin{enumerate}
\item \textbf {{\em nodeset} Object Structure in PetaSim Estimator}

{\em nodeset} has the following properties:

\begin{description}
\item $\circ$ \textbf {name:} one per {\em nodeset} object
\item $\circ$ \textbf {type:} choose from \textbf {memory, cache, disk, CPU, pathway}
\item $\circ$ \textbf {number:} number of members of this {\em nodeset} in the computer architecture 
\item $\circ$ \textbf {grainsize:} size in bytes of each member of this {\em 
nodeset} (only relevant for memory cache or disk
\item $\circ$ \textbf {bandwidth:} this is maximum bandwidth allowed in any one member of this {\em nodeset}
\item $\circ$ \textbf {floatspeed:} performance on floating point arithmetic specified as a time to do a single
         operation for entities in cache. Only used by a CPU 
\item $\circ$ \textbf {calculate():} method for CPU nodesets that performs computation implied by
         \textbf {floatspeed} and other architectural features. 
\item $\circ$ \textbf {cacherule:} controls persistence of data in a memory or cache
\item $\circ$ \textbf {portcount:} number of ports on each member of nodeset
\item $\circ$ \textbf {portname[]:} ports connect to linksets and a member of a nodeset has one or more
         ports -- each of which has a name. A port corresponds to a class of connections and
         depending on number of members involved, a given port can correspond to multiple
         connections 
\item $\circ$ \textbf {portlink[]:} name of linkset connecting to this port

\end{description}

\item \textbf {{\em linkset} Object Structure in PetaSim Estimator}

A basic {\em linkset} has the following properties. A derived linkset is gotten by concatenating several
basic {\em linksets} together. Derived linksets could be specified by special scripts or just written
directly in Java.

\begin{description}

\item $\circ$ \textbf {name:} one per {\em linkset} object
\item $\circ$ \textbf {type:} choose from \textbf {updown, across}. If \textbf 
{across}, this is a network of given \textbf {topology},
         linking members of a single {\em nodeset}. If \textbf {updown}, this is a link between two different
         {\em nodesets} 
\item $\circ$ \textbf {nodesetbegin:} name of initial {\em nodeset} joined by this {\em linkset }
\item $\circ$ \textbf {nodesetend:} name of final {\em nodeset} joined by this {\em
 linkset}. \textbf {Nodesetend} and
         \textbf {nodesetbegin} are identical if type is \textbf {across} 
\item $\circ$ \textbf {topology:} used for \textbf {across} networks to specify linkage between members of a single
         {\em nodeset }
\item $\circ$ \textbf {duplex:} choose from \textbf {full} or \textbf {half}. If \textbf {half} only allow transmission from \textbf {nodesetbegin}
         to \textbf {nodesetend}. If \textbf {full} allow either direction with bandwidth limiting sum of both
         directions.
\item $\circ$ \textbf {number:} number of members of this {\em linkset} in the computer architecture 
\item $\circ$ \textbf {latency:} time to send zero length message across any member of this {\em linkset }
\item $\circ$ \textbf {bandwidth:} this is maximum bandwidth in bytes per second allowed across any link
         in this {\em linkset}. Time \textbf {\(T_{lk}\)} to transfer information from nodeset \textbf {{\em l}} to nodeset \textbf {{\em k}} is
         expressed as \textbf {{\em latency + bandwidth }}. number of bytes. Here \textbf {{\em l}} refers to \textbf {nodesetbegin} and
         \textbf {{\em k}} to \textbf {nodesetend} 
\item $\circ$ \textbf {send():} method that calculates implications of sending information through given
         {\em linkset}. For a derived {\em linkset}, this method can include multiple references to properties
         and methods of basic {\em linksets} and {\em nodesets}. 
\item $\circ$ \textbf {distribution:} name of geometric distribution controlling this {\em 
linkset}

\end{description}

\item \textbf {{\em distribution} Object Structure in PetaSim Estimator}

{\em distribution} has the following properties:

\begin{description}

\item $\circ$ \textbf {name:} one per {\em distribution} object
\item $\circ$ \textbf {type:} choose from \textbf {block1dim, block2dim, block3dim} (can obviously add more to
         this in analogy with HPF) to specify geometrical structure of entity being distributed.
         Note most computer architectures are implicitly done as one dimensional block
         {\em distribution}

\end{description}

\item  \textbf {{\em dataset} Object Structure in PetaSim Estimator}

{\em dataset} has the following properties:

\begin{description}

\item $\circ$ \textbf {name:} one per {\em dataset} object
\item $\circ$ \textbf {type:}  choose from \textbf {grid1dim, grid2dim, grid3dim}, specifies type of {\em dataset }
\item $\circ$ \textbf {bytesperunit:}  number of bytes in each unit. If 5 field values at each grid point and
         double precision used, then \textbf {bytesperunit} is 40 
\item $\circ$ \textbf {floatsperunit:} update cost as a floating point arithmetic count. Differences between
         double or single precision , should be reflected in values of {\em CPUnodeset.floatspeed}
         and {\em dataset.bytesperunit}
\item $\circ$ \textbf {operationsperunit:} operations in each unit. A dataset contains {\em totalsize} of units,
         {\em operationsperunit} then reflects the operations in each unit for the {\em CPUnodeset} to
         calculate the computing time.
\item $\circ$ \textbf {update():} ethod that updates given {\em dataset} which is contained in a CPU {\em nodeset} and
         with a grainsize controlled by last memory {\em nodeset} visited.
\item $\circ$ \textbf {transmit():} method that calculates cost of transmission of {\em dataset} between memory
         levels including either communication (between distributed nodes) or movement up
         and down hierarchy. Note classic grid problems are assumed to be done using ghost
         cells so that edges of regions transmitted.

\end{description}

\item \textbf {Computation/Communication Instructions (Execution Script)}

Much of execution is controlled by methods in {\em nodeset, linkset} and {\em dataset} objects. Some typical
additional commands that implicitly invoke these methods are:

\begin{description}

\item $\circ$ \textbf {send} DATAFAMILY \textbf {from} MEM-LEVEL-L \textbf {to} MEM-LEVEL-K \\
Here DATAFAMILY is a {\em dataset} specified by name \\
MEM-LEVEL-K, MEM-LEVEL-L are {\em nodesets} labeled by name which must be linked by
     a {\em linkset}.
\item $\circ$ \textbf {move}  DATAFAMILY \textbf {from} MEM-LEVEL-L \textbf {to} MEM-LEVEL-K
\item $\circ$ \textbf {Use distribution} DISTRIBUTION \textbf {on} NODESET1,…,LINKSET1,…,DATASET1 
\item $\circ$ \textbf {compute}  DATAFAMILY-A,DATAFAMILY-B .. {\textbf {on} MEM-LEVEL-L}
\item $\circ$ \textbf {synchronize} synchronizes all processors (loosely synchronous barrier). Pipelining is stopped
     by this. Otherwise it is assumed

\end{description}

The difference between the send and move command is that send command will send the data from
source node to the destination node, and then after the computing, send the data back to the original
source. While move command will just move the data from source to destination, which does not
have that "loop" operation.

\end{enumerate}

%\subsubsection{Possible Changes to PetaSim}
\subsection{Current Status of PetaSim}

A prototype of PetaSim performance estimation system has been implemented in
NPAC. It's based on the architecture description ({\em nodeset} class and {\em linkset}
class) and application description ({\em dataset} class and {\em distribution} class, plus
{\em execution script} to describe the operations on the {\em dataset}) as we
discussed above. These two kinds of descriptions correspond to the Target
Machine Specification, High Level Representation of Application and Execution
Script in the design of HLAM (Hierarchical Application Modeling Framework), 
which was taken as input in PetaSim performance estimator. University of Maryland
has some kinds of emulators right now, which take applications and generate
the abstractions of the applications. After the initial co-operation with
UMD's emulators, we found that in order to 
make petasim available for the design of HLAM, we
need to provide the {\em Petasim} with the following features:

\begin{enumerate}

\item \textbf {{\em nodeset} member identical}

In the current version of PetaSim right now, PetaSim has the ability to 
describe the architecture's specifications into {\em nodeset} and {\em
linkset}, according the Target Machine's memory hierarchy. In order to
describe detail architecture information of the machine, we also need to
describe each nodeset member's feature and its linkage features in the
architecture description files. Meanwhile, in the execution script, 
performance estimation also wants to estimate the cost of each nodeset
member. So we provide the nodeset member identical in the architecture
and application description files. In this case, we can estimate the 
performance cost on each nodeset member.

\item \textbf {{dataset} hierarchy}

In this version of PetaSim, an application may contain a set of dataset, 
each contains {\em totalsize} units of data. In the assumption of PetaSim,
all computation is supposed to be in block-block form with both:

\begin{description}

\item $\circ$ "real" blocks of nonzero size and
\item $\circ$ the "virtual zero size blocks formed by for instance ghost cells
around the edge of domain

\end{description}

The blocks are formed from {\em dataset} objects with a particular distribution
on a particular memory {\em nodeset}.

Each statement in the execution script will operate on a specific dataset,
for examples: \textbf {move} one dataset from one {\em nodeset} to another
{\em nodeset}, or \textbf {compute} one dataset on a CPU {\em nodeset}. 

In some applications, like the applications from UMD, the applications may
have the feature of hierarchy. An application contains a set of {\em dataset}, 
and a dataset may contain a bunch of {\em sub-dataset}. A statement in execution
script may operate on a specific {\em sub-dataset} or directly operate on
a parent dataset. For example:

An application has {\em dataset1, dataset2,} and {\em dataset3}, and {\em 
dataset1} contains {\em sub\_data1, sub\_data2, sub\_data3} and {\em sub\_data4}.

And the execution script may be like :

\begin{verbatim}
move sub_data1 from memory_node to cpu_node
compute sub_data1 on cpu_node		; which works on sub_dataset
move dataset1 from disk_node to cache_node
mode dataset1 from cache_node to memory_node	
		; which automaticly operates on sub_data1, sub_data2, 
		  subdata3 and sub_data4
\end{verbatim}

We also provide this feature of dataset hierarchy in the current version 
of Petasim. This will benefit the application's HLAM to generate easy 
execution script, and make it more readable.

\item \textbf {execution script library}

Right now, in the {\em execution script} of PetaSim,  we support four different
kinds of statement, {\em send, move, Use distribution, compute} and {\em
synchronize} commands, which is enough to express the abstractions of operations
in the applications, data movement and computation. But this simplicity of
abstraction somehow will make the execution script of applications very long,
especially to the real large applications. So we should build 
an execution script library, which contains aggregates of data movement and 
computation in a function call. For example:

Calculate dataset on CPU\_nodeset

will involve the following operations :

\begin{verbatim}
move dataset from disk_nodeset to disk_controller
move dataset from disk_controller to cache_nodeset
move dataset from cache_nodeset to memory_nodeset
compute dataset on CPU_nodeset
\end{verbatim}

In this case, we will make the execution script of applications shorter and
easier to read and understand.

\item \textbf {data dependency}

In the current version of PetaSim, the performance estimation of an application
running on a specific machine will based on the features of target machines,
dataset of the applications, and the execution script of the application. And
the estimation operation is just in the order the statements of the execution
script. The petasim will not try to reorder the statements, like the real
application execution.

But in the real case, some operations may not have the order of statements in 
the execution script. They will depend on the finish of previous statements.
For examples:

\begin{verbatim}
s1: move data1 from node1 to node2
s2: mode data2 from node3 to node2
s3: move data3 from disk2 to node2	; according to the arrival of data1
s4: mode data4 from disk2 to node2	; according to the arrival of data2
\end{verbatim}

In this example, s3, s4 may not the execution order in real execution. They
just depend on the finish of s1 and s2. If s1 finish first, s3 may execute
first on node2, and if s2 finish first, s4 may execute before s3. They just
have the data dependency with the previous statement.

Right now, Petasim doesn't have the mechanism to deal with this kind of data
dependency problem. Any way, we will try to support that later.

\end{enumerate}

%\subsection{Description of Emulator}

%\subsubsection{Description of Applications and Current Status}

%\subsubsection{Possible Changes to Emulators}

%\section{Interface between PetaSim and Emulator}

%\subsection{Proposed Interface}
\subsection{Interface Petasim with HLAM}

Right now, PetaSim takes architecture description and application description
input from files. Here, we have 5 files, corresponding to nodeset, linkset,
dataset, distribution description, and execution script, respectively. In order
to provide {\em Petasim} use for HLAM design, and try to avoid the large
amount of execution script file, we may use socket communication between
the Petasim and HLAM.

%\subsection{Description of Machines, and Applications}

In the architecture
 Description of Machines, we have two parts, nodeset and linkset, which
aimed to describe the detailed memory hierarchy and the linkage relationship
between each hierarchy level. In the nodeset description file, it contains
all the information about each nodeset, and all of its {\em nodeset members},
and the linkage neighbors of this {\em nodeset member}. While in the linkset
file, it contains the features of all the linkset, and each linkset member,
which link two of the nodeset members.

%\subsection{Generation of Execution Scripts}

In the aspect of execution script of an application, 
PetaSim will read the execution script from a file. And it supposes
that the total statements of execution script have already existed there. In 
order to use the socket format to work with HLAM, we need to take execution script input
dynamicly, and give an estimation. Which means, from the HLAM, it will dynamicly
generate the execution script statements according to the applications and then
send to petasim to give an estimation. This is also a way to try to avoid the
long execution script of an application.

Here, we plan to send and receive execution script statements in a block way, 
which means, it supposes that each bunch of execution script statements petasim
receive from emulator will have a rather independent execution behavior. Petasim
doesn't need to care about the data dependency of the statements in this set with
the execution script received from emulator previous or later. Petasim will
estimate the cost according to the hardware available now and the execution
script received this time.

%%  \section{Examples and Experiments}
%%  
%%  The following table show some of the real application's performance
%%  estimate results on {\em Petasim}: (The architecture and application
%%  description files are generated by University of Maryland's emulators.)
%%  
%%  \small
%%  \begin{center}
%%  \begin{tabular*}{4.255in}{|l| l| l| l| l|} \hline
%%  & Petasim & IBM SP2 & \multicolumn {2}{l|}{Petasim Execution Time} \\
%%  %& Petasim & IBM SP2 & Petasim Execution &  \\
%%  \cline{4-5} 
%%  & Estimate Time & Running Time & SGI Challenge & SUN Ultra-2 \\
%%  \hline
%%  test1 & 105.234s & 114 secs & 7.12s & 18.24s \\
%%  \hline
%%  test2 & 318.509s & 347 secs & 21.61s & 55.32s \\
%%  \hline
%%  test3 & 316.071s & N/A & 41.69s & 99.46s \\
%%  \hline
%%  test4 & 159.45 s & 166 secs & 8.49s & 21.92s \\
%%  \hline
%%  test5 & 484.51 s & 497 secs & 25.7s & 66.11s \\
%%  \hline
%%  test6 & 710.477 s & N/A & 78.99s & 193.96s \\
%%  \hline
%%  \end{tabular*}
%%  \end{center}
%%  
%%  \begin{verbatim}
%%  test1: 15 processors, titan application, 9k  data blocks
%%  test2: 15 processors, titan application, 27K data blocks
%%  test3: 32 processors, titan application, 27K data blocks
%%  test4: 15 processors, pathfinder application, 9K data blocks
%%  test5: 15 processors, pathfinder application, 27K data blocks
%%  test6: 32 processors, pathfinder application, 55K data blocks
%%  \end{verbatim}
%%  \normalsize
%%  
%%  From the benchmarks we can see that the {\em Petasim} estimate results are
%%  very close to the real application's running time on SP2 machines. But it
%%  takes much less time to get the performance estimate cost. This will be a
%%  great help to the system and application design.
%%  
%%  The reason that {\em Petasim} a little bit under-estimate is that in the real
%%  computing, there are some memory swap cost due to the multi-applications
%%  running on the SP2 system, while in the Petasim, we suppose there is only
%%  the application running on the machines. We calculate the cost based on
%%  the full capacity of the system. When we meet the resource
%%  conflict in the computing, e.g. communication, we estimate for each operation
%%  the half speed of bandwidth, while in the real computing, it's normally a
%%  little bit lower than that.

\section{Further Work}

As discussed above, the early experience of {\em Petasim} prototype shows
that if we can get the correct abstraction of the machine architecture
and the execution behavior of the applications, {\em Petasim} can produce
a performance prediction result in a very short time. This will make great
benefit to the new architecture design and the application development.

The early experience also shows that in order to make {\em Petasim} more
usable to the design of HLAM, we may also consider the binary expression
in the architecture and application description. Right now, we take the
ASCII expression in all the description files, which may make them very 
large files, especially in the massively parallel computing. The binary
code expression will make them in a short size. However, in order not to loose
the flexibility of {\em Petasim} we may also provide some kind of translator
between the ASCII user interface and the binary code expression. We may provide
a much attractive programming environment based on our early experience
\cite{Dincer97}.

Typically {\em Petasim} is executed on a sequential machine as it is aimed 
at relatively crude estimation. However if for instance an accurate 
modeling of cache, requires a small aggregate size (and corresponding
large number of aggregates), the performance estimate may become very 
expensive computationally.  Very adaptive problems could lead to a similar 
situation. In this case, we may need to use parallel estimation and the
operation of {\em Petasim} needs to be parallelized.

We are also considering to improve {\em Petasim} to estimate the program's
execution even more detail to make it an application simulator. If it works
in a parallel way, we can dynamically simulate the execution behavior of
an application on a sequential / parallel machine.

\bibliography{petasim_v0.3}

\end{document}
