\section{Komodo: a distributed resource monitor}
\label{sec:komodo}

For different applications, different resource constraints are likely
to govern the decision to migrate - for example network latency,
network bandwidth, server load (as in number of server connections
available), CPU cycles etc. We have propose that a single monitor be
used for all resources.  Using a single monitor facilitates
applications that might need information about multiple resources. It
also reduces communication requirements for distributed monitoring as
information about multiple resources can be sent in the same message.

In our design, each host runs a monitor daemon which communicates with
peers on other hosts. The monitoring daemons are loosely-coupled and
use UDP for communication as well as for monitoring the network.  A
simple timeout-based scheme is used to handle lost packets and
re-transmissions. 

Applications register monitoring requests with the local daemon. If
the resource mentioned in the request can be monitored from the current
host then the local daemon handles the request.  Requests that cannot
be handled locally, for example, network latency between two remote
sites, are forwarded by the local daemon to the daemon on the
appropriate host.

Applications can request the current availability of a resource
(on-demand monitoring) or they can request periodic checks on resource
availability (continuous monitoring). On-demand monitoring returns a
single snapshot. Continuous monitoring applies a resource-specific
filter to eliminate jitter in resource levels. Eliminating jitter
helps reduce the reporting requirements (and therefore the
communication needed) without impacting application performance.  Data
corresponding to remote requests for continuous monitoring is
forwarded to the requesting sites as and when the filtered value of
the resource changes.  Requests for continuous monitoring may also
specify a sampling frequency, subject to an upper bound. Komodo
enforces an upper bound on this frequency to keep the monitoring cost
at an acceptable level.

Each daemon supports a limited number of monitoring requests. This
limit applies to both local and remote requests. Together with the
limit on sampling frequency, this ensures the monitoring load on
individual hosts is within acceptable limits.

Each request has an application-specified {\em time-to-live}. There is
an upper bound on the {\em time-to-live} which allows the daemons to
clean-up requests made by applications or hosts that have since
crashed.  Applications need to refresh requests within the {\em
time-to-live}. Requests that are not refreshed are dropped.  If a
daemon runs out of entries in its monitoring table, the least recently
requested entry is ejected.

Monitoring requests are passed from Sumatra to the local Komodo daemon
using a well-known Unix domain socket. The resource information is
made available by the daemon in a read-only shared memory
segment. This allows applications to rapidly access the latest
available monitoring information.

\subsection{Implementation}
\label{sec:komodo-impl}

The current implementation of {\em Komodo} monitors network bandwidth
and latency.  Each Komodo daemon pings a network link for which it has
received monitoring requests, by sending a probe packet to the daemon
on the other end of the link of interest.  If an echo is not received
within an expected interval, (the maximum of the ping period or five
times the current round trip time estimate) the packet is
retransmitted.

Bandwidth values are reported without filtering whereas the latency
values are filtered to eliminate short-term variations.  This filter
was based on the results of an extensive study of Internet
latency~\cite{AcharyaS96Study}. This study revealed that: (1) there is
a lot of short-term jitter in the latency measures but in most cases,
the jitter is small; (2) there are occasional jumps in latency that
appear only for a single observation; and (3) In most cases, a short
window of values around the mode contains a large fraction of the
observations (this indicates that the mode would be a good
characteristic value for RTT distributions).  Based on this, we have
developed a filter for latency measures that returns a moving window
mode if there is a well-defined mode, else it returns a moving window
mean.

\subsection{Discussion of the design of Komodo}
\label{sec:komodo-discussion}

In this section, we discuss the design of Komodo and how it has been
influenced by the {\em awareness} and the {\em authority}
requirements.

Komodo provides both on-demand and continuous monitoring. It allows
applications to select the monitoring mode on a per-resource and a
per-host basis (or host-pair basis for network latency and bandwidth).
For resources that are monitored in a continuous mode, Komodo allows
applications to control the frequency with which the resource is
monitored. This allows the applications to control how much effort is
spent in monitoring on their behalf and is in keeping with the {\em
authority} requirement. To safeguard hosts from malicious or runaway
applications, Komodo enforces an upper bound on the monitoring
frequency.

Komodo provides both a synchronous interface and an asynchronous
interface. In the current implementation, asynchronous notification is
implemented using UNIX signals. For the asynchronous interface, Komodo
allows resource-specific filters which pre-process the information
about individual resources; individual applications do not have to
replicate this functionality. They can, however, control its operation
by providing their own values for the filters' parameters.  This
allows the applications to track only the stable changes.

For network-related resources, there is the choice of active versus
passive monitoring. Komodo provides only active monitoring. It assumes
that passive monitoring is the responsibility of the applications. For
applications that do not coordinate their operations, this can lead to
some loss of information. For example, if two applications
independently access the same server, they could share information
about the network connection to server but if they have not been
designed to cooperate in this manner, they will not be able to utilize
the information acquired by the other. Since Komodo is a user-level
monitoring system, it can keep track of network performance for other
programs in only two ways: (1) applications are required to measure
performance for every network access and supply the information to the
local Komodo daemon or (2) all messages are routed through
Komodo. Neither of these are attractive options.  We believe that
cooperating applications should share such information instead of
requiring all applications to acquire/provide it. Such an approach was
been used by Mummert~et~al~\cite{MummertES95} in the {\em Coda} file
system which is able to adapt to changes in network
connectivity. Individual components of {\em Coda} cooperate in
monitoring the bandwidth and maintain the information in a shared
location.


