\documentstyle[art10,npac_proto,palatino]{article}

\begin{document}
%\makeNPACtitle\newpage
\begin{project}
\begin{titlepage}
	\projtitle{NPAC Sailfish}
	\doctitle{User Guide for the Monte Carlo Data Measurement Package}
		\author{Miloje S. Makivic}
		\date{November 17, 1995}
		\version{0.2}
		\summary{Delivered November 30, 1995}
	\id{Measurements}
	\docauthor{M~S~Makivic}
%	\doccontributor{KAH}
	\security{Commercial in Confidence}
\maketitle
	\approval{NPAC}
\vfill
	\begin{history}
		\release{
			\author{Miloje S. Makivic}
			\date{November 17, 1995}
			\version{0.1}
			\summary{Initial draft}
			}
	\end{history}
\end{titlepage}

\pagenumbering{roman}\setcounter{page}{1}
\begin{abstract}
Monte Carlo Data Measurement Package contains library routines for
accumulation, management, storage and retrieval of data and functions
evaluated in the course of a Monte Carlo simulation. The library is designed to
be used in conjunction with the NPAC Monte Carlo Simulation modules, although 
many routines provide generic functionality and can be used with any
Monte Carlo simulator.
\end{abstract}

{\parskip=0pt   % hack to patch format of \tableofcontents
\tableofcontents
%\newpage
\medskip\hspace{2.0cm}\hrulefill\hspace{2.0cm}\bigskip
}

\pagenumbering{arabic}\setcounter{page}{1}

%=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
\section{Introduction}
Typically, in the course of a Monte Carlo simulation, one generates 
configurations drawn from a given probability distribution. For the purposes
of Sailfish Risk Management applications, configurations generated by the
NPAC Monte Carlo Simulator consist of
histories of risk factors, which are stored in the data structure of type {\bf PATH}. 
Each generated
configuration constitutes a Simulation Step. For a subset of Simulation Steps,
one evaluates functions which correspond to various quantities
dependant on risk factors (or Monte Carlo configurations in general). 

For example, to compute a fair price of an option, one would
evaluate the payoff function for each history of the underlying
generated by Monte Carlo. 
The evaluations are stored on disk files
for later statistical
analysis. For example, the option price can be computed as the average value
of the payoff function.  The analysis of statistical quality of
Monte Carlo data may require computation of higher order moments,
correlations and other quantities. 

The combined evaluation and storage steps constitute a
Measurement Step. Routines in the Data Measurement Package
enable the implemention of a Measurement Step for arbitrary function
evaluations. Depending on available hardware and optimization considerations, 
Measurement Step can be implemented as either a subroutine or a coroutine.

The computational model described above assumes that function
evaluations are performed on Monte Carlo configurations as they are generated,
and only {\it function evaluations} are stored for analysis.
In some applications, it may be advantageous to store Monte Carlo generated
{\it configurations} instead
on a disk file for later evaluation and analysis. This will happen in 
situations when user requires higher level of interactivity. For example,
user might want to experiment with different methodologies for
extracting information from statistically generated data sets. 
In such a case, simulation has to be repeated for each new method introduced,
unless all Monte Carlo configurations are recorded.
One has to evaluate trade-offs between CPU time cost and cost of secondary
storage, since storage requirements of the latter model
may be quite high for long simulations
and large numbers of factors. Data Measurement Package provides routines to
support both computational models. 
%=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
\section{Library Contents}
Simulation Module generates PATH variables, which contain histories of 
market factors. Typically, to perform a useful computation one must
extract a subset of data from the PATH structure. For example, one may
retrieve a history of a single factor, or values of all market factors
on a time slice. Data Measurement Package provides
routines to gather predefined subsets of data from a PATH: 
\begin{itemize}
\item
  \begin{verbatim}
static REAL *GatherMeasurementArgs
                                 (
                                 PATH *CurrentPath,
                                 int MapDimension,
                                 int *Map
                                 )
  \end{verbatim}
\end{itemize} 
  This is the top level gather routine. It uses the index array, {\it Map},
  with {\it MapDimension} array elements,
  to retrieve requested data from {\it CurrentPath}. The type of retrieval
  is determined by  $Map[0]$. Depending on this value, {\it GatherMeasurementArgs()}
  will call one of the following  routines, with self-explanatory names:
\begin{itemize}
\item
  \begin{verbatim}
static void Gather_SLICES
                                 (
                                 REAL *ArgArray,
                                 PATH *CurrentPath,
                                 int MapDimension,
                                 int *Map 
                                 )
  \end{verbatim}
\item
  \begin{verbatim}
static void Gather_SINGLE_GRID_POINT
                                   (
                                   REAL *ArgArray,
                                   PATH *CurrentPath,
                                   int MapDimension,
                                   int *Map 
                                   )
  \end{verbatim}
\item
  \begin{verbatim}
static void Gather_HISTORIES
                                    (
                                    REAL *ArgArray,
                                    PATH *CurrentPath,
                                    int MapDimension,
                                    int *Map 
                                    )
  \end{verbatim}

\end{itemize}
The interpretation of {\it Map} array elements and its size are
as follows:
\begin{itemize}
\item For {\it Gather\_SLICES()}
  \begin{enumerate}
    \item {\it Map}$[0] = SLICES$ (defined in MEAS.h)
    \item {\it Map}$[1] = N$, number of slices to be gathered
    \item {\it Map}$[2\cdots N+1] =$ indices of slices to be gathered
    \item {\it MapDimension} $= N+1$
  \end{enumerate}
\item For {\it Gather\_HISTORIES()}
  \begin{enumerate}
    \item {\it Map}$[0] = HISTORIES$ (defined in MEAS.h)
    \item {\it Map}$[1] = N$, number of histories to be gathered
    \item {\it Map}$[2\cdots N+1] =$ indices of histories to be gathered
    \item {\it MapDimension} $= N+1$
  \end{enumerate}
\item For {\it Gather\_SINGLE\_GRID\_POINT()}
  \begin{enumerate}
    \item {\it Map}$[0] = SINGLE\_GRID\_POINT$ (defined in MEAS.h)
    \item {\it Map}$[1] =$ factor index
    \item {\it Map}$[2] =$ time slice index
    \item {\it MapDimension} $= 2$
  \end{enumerate}
\end{itemize}
{\it Gather\_SLICES()} and {\it Gather\_HISTORIES()} return pointers
to an array which conatins concatinated slices or histories, in
increasing factor or time slice index order. {\it Gather\_SINGLE\_GRID\_POINT()}
returns a pointer to the floating point value of the specified 
factor on the
specified time step.

Gather routines return arrays which will be passed to user specified
functions along with other required parameters, which do not depend
on factor histories (for example, strike price of an option).
There are two types of user-supplied functions:
S\_MEAS type are scalar functions and they return a single
floating point number; functions of type V\_MEAS are vector-valued 
and they return a pointer to a floating point array. NPAC provided
an example of a
V\_MEAS type function:
\begin{itemize}
\item
  \begin{verbatim}
void ComputeCovarianceMatrix
                           (
                           REAL *Args,
                           int NumIntParams,
                           int *IntParams,
                           int NumRealParams,
                           REAL *RealParams,
                           int NumResults,
                           REAL *Results
                           )
  \end{verbatim}
\end{itemize}
The interface to a generic S\_MEAS function is:
\begin{itemize}
\item
  \begin{verbatim}
REAL S_MEAS_TYPE_Func
                     (
                     REAL *Args,
                     int NumIntParams,
                     int *IntParams,
                     int NumRealParams,
                     REAL *RealParams
                     )
  \end{verbatim}
\end{itemize}
In either case, the {\it Args} input array is typically constructed
by one of the gather routines.
For the {\it ComputeCovarianceMatrix()} routine, {\it Gather\_SLICES()} is used
to obtain values of market factors on the beginning and ending date of the
simulation. They are concatinated into the {\it Args} array and passed to
{\it ComputeCovarianceMatrix()}.
{\it IntParams} and {\it RealParams} are lists of additional function parameters
and {\it NumIntParams} and {\it NumRealParams} are sizes of additional integer
and real parameter list respectively. For the {\it ComputeCovarianceMatrix()} routine,
$RealParams=NULL$ and $NumRealParams=NULL$, while the integer parameter 
list has a single
element ($NumIntParams=1$), which gives the number of market factors in the 
{\it Args} array. This number is used in the {\it ComputeCovarianceMatrix()} to
distinguish between factor values at the 
beginning and ending dates in the {\it Args} array. 
{\it ComputeCovarianceMatrix()} then calculates the covariance matrix of
factor changes between the two dates and fills the {\it Results} array with
concatinated rows of the covariance matrix. The size of
this array is equal to $NumResults = IntParams[0]*IntParams[0]$.
For an {\it S\_MEAS\_TYPE\_Func} last two arguments do not exist, since it returns
a scalar value.

All of these functions are private, as indicated by the {\it static} keyword.
Application developer is not supposed to use these functions directly. Their
services are requested by the only publicly available functions,
\begin{itemize}
\item
  \begin{verbatim}
REAL S_Measurement
                 (
                 PATH *CurrentPath,
                 S_MEAS *Func,
                 int MapDimension,
                 int *Map,
                 int NumIntParams,
                 int *IntParams,
                 int NumRealParams,
                 REAL *RealParams
                 )

  \end{verbatim}
\item
  \begin{verbatim}
REAL *V_Measurement
                  (
                  PATH *CurrentPath,
                  V_MEAS *Func,
                  int NumResults,
                  int MapDimension,
                  int *Map,
                  int NumIntParams,
                  int *IntParams,
                  int NumRealParams,
                  REAL *RealParams
                  )
  \end{verbatim}
\end{itemize}
which should be used to perform scalar or 
vector measurements on the {\it CurrentPath}.
{\it Func} is a pointer to the evaluation function of S\_MEAS or V\_MEAS type
respectively. The other arguments are already explained above: 
they are just passed to
the appropriate gather routine and the requested evaluation function {\it Func}.
 
Once a function evaluation (measurement) is performed, it has to be recorded.
There are a number fo factors to be considered:
\begin{itemize}
\item It is likely that number of measurements to be performed on a single PATH
      will be large, to amortize the costs of generating a PATH 
\item Different measurements will be most likely recorded in different
      files for easier management of later statistical data analysis
\item Data sets obtained by measurements will be much smaller than PATH
      structure itself 
\end{itemize}
Therefore, it is important for performance considerations to cache measurements
in RAM and flush measurement caches in large blocks infrequently during
simulation. Measurement caches are implemented as dynamically allocated
stacks. Before any measurement can be attempted, the measurement routine
must call:
\begin{itemize}
\item
  \begin{verbatim}
static int CreateRecordStack
                           (
                           int RecordSize,
                           int NumRecords
                           )
  \end{verbatim}
\end{itemize}
Each type of measurement will have its own stack.
Up to {\it NumRecords} of size {\it RecordSize} can reside on the stack.
The bigger the {\it NumRecords} is requested, fewer file writes will be
neede during the simulation, but it will also require more RAM for caching
the measurements.
The function returns a handle to the record stack, which must be used by
other routines to access the stack.
The first call to this function will transparently generate a call to:
\begin{itemize}
\item
  \begin{verbatim}
static void StackPoolInit()
  \end{verbatim}
\end{itemize}
which will initialize the stack pool. 

Total number of stacks
that can be maintained ({\it MAX\_NUM\_OF\_RECORD\_STACKS}, defined
in MEAS.h),is equal to $100$,
which means that $100$ different measurements can be performed concurrently.
If this number is exceeded, measurements will not be cached. Instead they will
be written to the file immediately, which will degrade performance, but will
not generate an error. This will also happen if memory cannot be allocated
for a stack. The {\it CreateRecordStack()} routine will try to reduce the
stack size by halving the requested {\it NumRecords} iteratively until
sufficient memory can be allocated. If {\it NumRecords} becomes less than 1
in this process, it implies that memory is exhausted and an error condition
is returned.

Measurements are pushed onto the
record stack using:
\begin{itemize}
  \begin{verbatim}
static int Push
              (
              REAL *Meas,
              int MeasSize,
              int StackID
              )
  \end{verbatim}
\end{itemize}
where {\it Meas} is the pointer to the data buffer and {\it MeasSize} specifies
how many items at the buffer location are to be pushed  onto the stack identified
by {\it StackID}. Once the stack overflows, function:
\begin{itemize}
  \begin{verbatim}
static int DumpStack
                   (
                   int StackID,
                   char *RecordFile
                   )
  \end{verbatim}
\end{itemize}
must be called to flush the stack into the {\it RecordFile}. Stack pointers
are reset and measurements can be cached again. {\it DumpStack()} will
transparently issue a call:
\begin{itemize}
\item
  \begin{verbatim}
static int DumpRecordsToFile
                           (
                           int NumRecords,
                           int RecordSize,
                           REAL *RecordBuffer,
                           char *RecordFile
                           )
  \end{verbatim}
\end{itemize}
Once the stack is not needed, it can be released by a call to:
\begin{itemize}
\item
  \begin{verbatim}
static int CloseRecordStack
                          (
                          int StackID
                          )
  \end{verbatim}
\end{itemize}
which will return its handle to a list of available stack identifiers and
free memory. If {\it StackID} is bigger than $MAX\_NUM\_OF\_RECORD\_STACKS$,
it will do nothing since no memory allocation was performed anyway.

Once a stack is created,
stack management routines are hidden under the interface provided by:
\begin{itemize}
\item
  \begin{verbatim}
static int RecordMeasurements
                            (
                            int Final,
                            int StackID,
                            char *RecordFile,
                            int RecordSize,
                            REAL *Meas
                            )
  \end{verbatim}
\end{itemize}
The only argument that needs explanation is {\it Final}. It is
a flag supplied by the caller routine, which tells the stack manager that
simulation is about to end and all stacks should be flushed, regardless of
whether they are full ot not.

Note that all these routines are private. Applications are supposed to
access their services from a public higher level routine, which performs
function evaluations and measurement recording. As an example, NPAC supplied
the function:
\begin{itemize}
\item
  \begin{verbatim}
int MeasCovariance
                 (
                 PATH *CurrentPath,
                 int nmBlocks,
                 char *OutFile
                 )
  \end{verbatim}
\end{itemize}
which computes the covariance matrix of factor changes between the
start and the end date of the simulation. It should be viewed as a template for writing
measurement routines. Its inputs are the PATH generated in the current
simulation step, {\it CurrentPath}, total number of measurement steps {\it nmBlocks}
(needed to set the {\it Final} flag properly) and the name of the storage
file, {\it OutFile}. It returns number of measurement records stored in the
file.

As discussed earlier, instead of doing measurement immediately,
one may want to record
whole paths. This may be accomplished
using:
\begin{itemize}
\item
  \begin{verbatim}
int DumpPathToFile
                 (
                 PATH *Rates,
                 char *PathDumpFile
                 )
  \end{verbatim}
\end{itemize}
Before file dump, PATH structure is disassembled into arrays containing
histories of factors. These arrays are concatinated and downloaded into the
file. Therefore, the first {\it NumTimeSteps} +1 numbers are values of the
first factor for all time slices, followed by {\it NumTimeSteps} +1 values
for the second factor and so on. Once written to a file in this fashion, PATH
can be treated as a record which has $NumRates*(NumTimeSteps+1)$ items.

Records written to a file can be uploaded and processed using
the following set of routines, which support the second computational model
discussed earlier. These routins can be used to upload measurement records,
gather subsets of data from measurement records, evaluate scalar and
vector functions of these subsets if necessary
and store results in dynamically allocated
arrays. These arrays are later processed by Statistical Analysis Tools to
produce final results such as averages, moments, correlations, histograms etc.
The public routines are:
\begin{itemize}
\item
  \begin{verbatim}
REAL *LoadScalarMeas
                   (
                   int *NItems,
                   int Discard,
                   int ItemSize,
                   int BlockSize,
                   char *DataFile,
                   S_MEAS *Func,
                   int MapDimension,
                   int *Map,
                   int NumIntParams,
                   int *IntParams,
                   int NumRealParams,
                   REAL *RealParams
                   )
  \end{verbatim}
\item
  \begin{verbatim}
REAL **LoadVectorMeas
                    (
                    int *NItems,
                    int Discard,
                    int ItemSize,
                    int BlockSize,
                    char *DataFile,
                    V_MEAS *Func,
                    int NumResults,
                    int MapDimension,
                    int *Map,
                    int NumIntParams,
                    int *IntParams,
                    int NumRealParams,
                    REAL *RealParams
                    )
  \end{verbatim}
\end{itemize}
These routines provide functionality similar to {\it S\_Measurement()} and
{\it V\_Measurement()}. The only difference is that input Monte Carlo
data are records stored in the {\it DataFile} file. If {\it NItems} is
a positive integer, the routines assume that it is the number of records
of size {\it ItemSize} stored in the file. The first {\it Discard} will
be thrown away and the rest will be uploaded. Data are buffered,
{\it BlockSize} items at
a time to improve I/O performance. 
If {\it NItems} is supplied as negative, the routines will first figure out
the number of items in the file before uploading them. This will result
in a somewhat poorer performance as opposed to the known record number
case. In either case,
{\it NItems} will be overwritten with
the actual number of uploaded records not counting {\it Discard} items.
The remaining arguments are exactly the same as for {\it S\_Measurement()}
and {\it V\_Measurement()}. The important difference is that gather function
does not operate on a PATH but on an array representing a record. 
To gather data from a record into an input array for the {\it V\_MEAS} or 
{\it S\_MEAS} function, user must
supply the correct {\it Map} index array to the {\it GatherFromBin()} 
function:
\begin{itemize}
\item
  \begin{verbatim}
static REAL *GatherFromBin
                         (
                         REAL *Arg,
                         int MapDimension,
                         int *Map
                         )
  \end{verbatim}
\end{itemize}
{\it MapDimension} specifies how many numbers from the record are gathered
into the {\it Arg} array, and {\it Map} specifies their indices. Just like in
{\it S\_Measurement()} or {\it V\_Measurement()}, gathered data are processed
via user-supplied {\it V\_MEAS} or {\it S\_MEAS} functions. If the final result 
is a scalar, {\it LoadScalarMeas()} will return an array. Each array element
corresponds to a single Monte Carlo measurement. 
Similarly, if the
measurement result is an array, {\it LoadVectorMeas()} will return a matrix.
Every row in a matrix corresponds to a single Monte Carlo measurement.
Both scalar and vector measurements are
meant to be processed by Statistical Analysis Tools.  

%=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
\section{Files}
Monte Carlo Data Measurement module is stored in "meas.c". Measurement functions
of {\it V\_MEAS} or {\it S\_MEAS} type are supplied in a separate file. Presently,
the file "func.c" contains only the example function {\it ComputeCovarianceMatrix()}.
There are four header files: 
\begin{itemize}
\item DEFS.h: genral purpose definitions used in the data measurement package
\item  FUNC.h: declarations related to {\it V\_MEAS} or {\it S\_MEAS} functions
\item  MEAS.h, \_meas.h: declarations related to data measurement routines
\end{itemize}
Public header files use all capital letters. 
Private header files have an underscore prefix.

%=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
\section{Examples}
The "Examples" subdirectory contains example code which 
uses Path Integral Monte Carlo simulator and routines from
the Data Measurement package. {\it MeasCovariance()} routine is called
from within the Metropolis driver, {\it MetropolisMonteCarlo()} (note that
"MEAS.h" must be include in "modmc.c" file). {\it MeasCovariance()} stores
the covariance matrix data in a file called "cov.out". That file is used
as input to the record upload from file test routine in "test.c". Test routine
prints to standard output elements of the covariance matrix.

To comile the Monte Carlo simulator with data measurement package, type
\begin{verbatim}
make sail
\end{verbatim}
in the "Examples" subdirectory.
To run the test code, follow the instructions given for the Monte Carlo simulator.
Example input files are already supplied. The output file name for the example
covariance measurement is hardwired: "cov.out".

To compile the upload-from-file example, type
\begin{verbatim}
make tload
\end{verbatim}
To run the upload example, you could type:
\begin{verbatim}
tload cov.out -1 100 10
\end{verbatim}
This command will require the upload routines to find out how many records
are stored in the "cov.out" file, to discard the first $100$ records and to
read them in blocks of $10$ records. 

\end{project}
\end{document}


