Neural Network Simulations
A neural network simulation is an attempt to solve problems using a structure modeled after the neurons in a nervous system. Nodes perform calculations, while being interconnected via a weighted network. Different types of neural networks specify different node functions and various interconnect strategies.
Neural nets usually work be training and reinforcement. They are presented with a problem expressed as a pattern (e.g. a bit block). They are then told what the answer to that problem should be (e.g. as another bit block). The network then adjusts itself based on these responses so that if it should see the input problem again, it will more likely output the answer pattern it previously saw.
A popular type of neural net is the back propagation network. Shown above, it consists of an input, middle, and output layers with weighted interconnects. Each node in a layer is connected to every other node in the successive layer. Also, the number of nodes in each layer can vary as the problem changes. The output of a node is simply the inner product of the input vector and the vector of weights in the interconnect from where the inputs came. Learning is accomplished by calculating errors at the output nodes and "back propagating" a share of the blame for the error back down the network. This blame sharing is used to alter the weights of the interconnects and is governed by the "delta rule".
In order to speed learning the help to avoid traps, some tricks are used to attempt to contain the learning process. A sigmoid curve (above), limits the intensity of output from a node. Additionally, simulated annealing or biasing the node towards an output can significantly improve the process.
Neural network simulations are one of those embarrassingly parallel applications. For each round of computation, a node calculates its output value based on the input it received the last round, it then communicates the results through its output connections. When the network (i.e. a back propagation network) is being trained, the processes is just reversed. Error values are passed down the output connections into the previous layer.
In this architecture, each node can be thought of as a separate and independent entity. It doesn't have to share any state information with other nodes. In addition, only a trivial level of synchronization is necessary as messages are passed among the nodes. Both allow a lot of parallelism in the simulation.
Within a node, parallelism may also be exploited, depending on the type of network. In a back propagation network, most of the work deals with taking inner products of vectors. This could take advantage of any support for very fine grained parallelism in an architecture.
A wide variety of systems have been built which run on systems from single processor boxes to fast custom machines with exotic interconnects.
In order to get good performance, control over communication speed and latency become important, especially since the messages being exchanged are often small and frequent. Also, the computations of each node are usually limited, for which general purpose CPU's would likely be overkill. As a result, custom hardware and interconnects seem to be the norm. In addition, better performance is probably gained by a more efficient layout on VLSI processes.