4: Relevance of Parallelism

Parallel computing for web-based training.
When discussing this topic there are two issues that need to be addressed:

When dealing with a parallel/distributed computing infrastructure, for example, a large cluster or network of workstations, any web-based training (distance learning) can be conducted on the network. With this kind of hardware setup, we can have concurrent training sessions over the network. Perhaps, we could start by partitioning the overall cluster into sub-clusters and have a different web-based collaborative training session run on each partition. The relevance of this is that it will allow us to tackle various parts of the overall problem in an independent fashion, at the same time, any necessary communication between the partitions can be conducted in another level of collaboration. Basically, collaboration will cover the entire cluster while the computational part will be distributed between various sets of individual nodes, which are, of course , members of the cluster.
In a nutshell, what we end up having is a two-level parallelism, more or less. The first, a highly concurrent level covering the entire domain or available nodes, and the second, is also distributed/parallel level of computation running in each locality to deal with an independent part of the overall problem.
As far as I know this has not be done yet.

The second issue in this question to deal with the teaching of parallel computing in a web-based collaborative environment. This is already has been done successfully. One thing that can be added is to have actual parallel programs run during training on either the cluster used by the collaborators or on another cluster just to demonstrate in real time various aspects, etc. of parallelism to those involved in the class. Perhaps, the node used by the instructor can act as master node, partitioning the task and distributing the work to others in a message-passing fashion. If the audience involved in training are widely situated then this may be impractical to do other than dissemination of information and other collaborative things that are currently done.

Parallel Computing for Recognition and Learning
Paradigms such as neural networks are proven to be good tools for tackling learning and recognition problems. Certainly, this can be done in parallel, how?
Well, there is an inherent parallelism in those networks, making their mapping into either a cluster of processors or a multiprocessor system quite natural. For example, for feedforward type nets, backprop can be used to train it to recognize a class of objects or images. Object recognition and function approximation are perhaps the two most important tasks that these class of networks are good at. Training can easily be done in parallel and there have been great deal of this type of work in the past, especially on "special-purpose" SIMD type architectures. Later, the same process was carried out on MIMD hardware, such as the late CM-5, and on networks of workstations.
Not only feedforward nets were used but also others, such as feedback or the Hopfield-style nets. Parallelism can play quite an important role here. Well, for one thing, we can vary the mapping of networks on the hardware. So, instead of using one large network and map it into one large machine, we can have multiple nets distributed over a cluster of SMPs and then train it on whatever training set we have. Throughout the training process, the cluster of nets can communicate between them regarding training patterns or perhaps on how to go about optimizing their training parameters to speed up the learning cycle. Another option is that, one need not to train all of them using the same set of patterns but cluster these patterns into various groups based on some common characteristics, etc. and then concurrently train each member of the herd with one or more cluster of the samples. Hints and other useful aspects can be communicated between the networks during training. I believe this is the most useful and sensible way of training a colony of nets to handle difficult jobs such as function approximation or pattern recognition.

Saleh Elmohamed