Examples of Message Passing Architecture
Examples of Message Passing Architecture
Example 1: IBM SP-2
A summary of the machine's basic characteristics:
- Constructed essentially out of almost complete RS6000
workstations.
- Some modifications are made to package the RS6000s into standing
racks.
- The rack-level topology varies from machine to machine but it is
typically a variant of a butterfly network..
- The physical topology is not fixed in hardware so the network
interface inserts the route for each outgoing message via a table
lookup.
- Since a complete system is used for the node, the communication
assist and network interface (NI) are part of a card that plugs into
the system. This is a MicroChannel adapter card connecting
to a switch in the base of the rack.
- The above diagram illustrates in some details the organization of
an SP-2 node. As shown, a network interface card (NIC) connected to
the MicroChannel I/O bus. The NIC contains the drivers for the actual
link into the network, a large amount of memory to buffer message
data, a direct memory access (DMA) engine, and a complete i860
microprocessor to move data between host memory and the network.
- Not shown in the figure is the network itself which is a
butterfly-like structure, constructed by cascading 8 x 8 crossbar
switches. The individual links operate at 40 MB/sec in each direction,
which is the full capability of the I/O bus.
- Between 8 to 16 nodes fill a rack, so large configurations are
built by cabling together a number of racks, including additional
switch racks.
- With a complete system at every node, there is the option of
distributed disks and other I/O devices over the entire machine. Disks
on compute nodes are typically used only for swap space and temporary
storage. Most of the I/O devices are concentrated on dedicated I/O
nodes.
- This style of design allows for some degree of heterogeneity among
the nodes since all that is required is that the network interface card
can be inserted. Therefore, SP systems support several models of
workstations as nodes, including SMP nodes.
Example 2: Intel Paragon
- This diagram shows in some details a single node of the
Intel Paragon. Also, it shows the overall 2D grid network with
processing nodes, each attached to every switch.
- The Paragon illustrates a much tighter packaging of nodes than the
SP-2.
- Each card is an SMP with two or more 50-MHz i860XP processors, a
network interface (NI) chip, and 16 or 32 MB of memory, connected by a
64-bit, 400-MB/sec, cache-coherent memory bus.
- In addition to that, two DMA engines (one for sending and the
other for receiving) are provided to burst data between memory and the
network.
- One of the processors is designated as a communication processor
(CP) to handle network transactions and message-passing protocols
while the other is used as a compute processor for general computing.
- The NI chip connects the 64-bit synchronous memory bus to 16-bit
asynchronous (self-timed) network links.
- The transfer rate between main memory and the NI chip is at
400MB/sec. The memory region is specified as a physical address,
aligned on a 32-byte boundary, with a length between 64 bytes and 16
KB (one DRAM page) in multiple of 32 bytes (a cache block).
- During DMA transfer, the DMA engine snoops on the processor caches
to ensure consistency.
- The send- and receive-DMA engines periodically kick in and move
data to or from network buffers in 400-MB/sec bursts.
- The CP must take action on a number of events including the
user's program writing a message into the shared queue, the network
delivering a message into the NI input buffer, the NI output buffer
going empty as a result of the network accepting a message, the
send-DMA engine completing, and the receive-DMA engine completing.
The bandwidth of the CP is determined by the time it takes to detect
and dispatch on these various events. Also, while the CP handling any
of these events all others are effectively locked out.
- The net is a 3D grid, much like the Cray T3E, with links at 175
MB/s in each direction.
Other examples: