\chapter{Conclusions}
\label{ch:Conclusion}

In this dissertation we proposed
various extensions of Principal Component
Analysis in relation to neural network
learning theory
and we established
the full relationship between
the Hebbian rule and standard PCA.
We also discussed the usefulness of these
extensions for different signal/image processing applications.
In the last chapter we focused our attention to the
very important motion compensation problem for the coding
of motion sequences and we proposed a new scheme for
more efficient coding without compromising performance
or computational cost at the decoder.

In Chapter \ref{ch:PCA} we studied the relationship between
PCA and the Hebbian learning rule.
We proposed a new model for multiple component extraction
(APEX) which uses only the normalized Hebbian rule for all the
synaptic weights without any additional non-neuronal
or off-line processing in contrast to the old methods.
The new model featured a novel structure
of a lateral connection network
for the orthogonalization
of the synaptic weight vectors.
We formally proved that APEX
exponentially extracts the desired components and furthermore, we
were able to theoretically predict the convergence rate of the components
and successfully compare it with simulation results which
verified the validity of our predictions.
Moreover, the model is faster than
the previous neural PCA schemes
and is extensible: one can increase
the order of the network by adding one more neuron
or can decrease it by removing one or more neurons
without need for retraining the old or remaining units.
Via a connection with the Recursive Least
Squares algorithm (RLS) we derived the optimal
value for the step-size parameter $\beta$
which significantly improves the model performance.
We then extended our results by showing that
the APEX network is applicable to
Constrained Principal Component Analysis.
The optimality criterion for CPCA is variance maximization
as with standard PCA but with an external
subspace orthogonality constraint.
The constraint subspace may represent for example,
the subspace of an undesirable signal.

In Chapter \ref{ch:OPCA} we introduced Oriented
Principal Component Analysis (OPCA). OPCA involves
two signals the goal being to maximize the
variance ratio between the two (after projection
onto a small dimensional subspace).
We proposed two extensions of the Hebbian rule
for extracting principal oriented components.
We mathematically justified both models and
verified them via simulation experiments.
We finally discussed the relationship of OPCA with matched
filtering and Fisher linear discriminant analysis
as well as with other techniques used for various applications in signal
processing and control.

In Chapter \ref{ch:APCA} we
investigated another extension of PCA which we called
Asymmetric PCA. We identified two subclasses of APCA,
namely linear-approximation and cross-correlation.
The first subclass relates linear regression and Generalized Singular Value
Decomposition (GSVD) since
it is shown that under reduced number of hidden units
the BP network solves a GSVD problem special cases of
which are linear regression (Wiener Filtering) and SVD
or Principal component analysis.
Although BP extracts the APCA subspace it does not produce
the exact asymmetric principal components (i.e. the generalized
eigenvectors of the GSVD problem) so we proposed
an extension of the model
introducing a deflation transformation
similar to the one used for the standard eigenvalue problem.
This deflation was also implemented by a lateral
connection network as with the APEX model.
We showed the universality of the lateral network
structure not only for the PCA problem but also for APCA.
Again we demonstrated via simulated experiments the
agreement between theoretical and numerical results.

The other subclass of APCA problems called the
Cross-correlation APCA is equivalent to the singular value decomposition
of the cross-correlation of the two signals involved. We proposed
and proved a learning model that extracts these components
again applying the lateral network structure
for extracting multiple components

In Chapter \ref{ch:Applications} we tried to justify
all these new models by showing some selected applications.
The problems discussed were related to signal detection,
system identification and image compression. We showed detailed
simulation comparisons with other techniques in the corresponding
problems and we established the superiority or limitations of the
learning methods.

In Chapter \ref{ch:Motion} we attacked the problem
of motion estimation for compression and coding of
image sequences, an application which has become increasingly
important due to the prospects
of High Definition Television (HDTV)
and Integrated Services Digital Networks (ISDN) that will 
handle large amounts of images.
A major problem of the traditional block-matching
techniques for motion estimation is the artifacts created
by the artificial square borders of the matching
process. We proposed to amend this problem
segmenting the image locally, according to gray level, and matching each 
segment with a similar segment in the next frame.
We compared the new technique with the standard MPEG
algorithm for similar MSE performance
and showed that the proposed scheme
significantly reduces the bit-rate required.
In addition the block artifacts are practically
absent and the picture quality is acceptable to the human eye
even at the rate of (approximately) 0.1 bits per pixel.
A further advantage of the technique is the simple
decoding algorithm which can even be simpler than the MPEG
decoder if the DCT coefficients are not transmitted
in which case there is no need for inverse DCT transform.

\section{Future Work}

The research in this dissertation has shed new light in the properties
of linear learning networks especially in relation to principal
component analysis.
However some questions still remain open
while various extensions of the current work
yield promising directions for future effort.
An interesting research topic for example,
is the extension of the current results to non-linear networks
such as the standard neural networks containing sigmoidal units.
PCA techniques are essentially linear however, a non-linear network
can be locally approximated by a linear mapping. In fact, linearization
is a widely used technique in studying non-linear systems since our
mathematical tools are much more efficient in the linear case.
It would also be interesting to incorporate the new structures
such as the lateral connection network,
in standard sigmoidal networks like multi-layer perceptrons.
We conjecture that even with sigmoidal units, LON
will play an important role in helping the net
find more efficient representations of the data, as it did with
the linear units via the property of orthogonality. In the linear
case orthogonality is a synonym to statistical uncorrelation.
It is really the absence of correlation between
the activation of the units that maximizes
the representation power of the system since it implies that
no unit repeats any information included in other units.
In the non-linear case some extended concept of orthogonality
will be introduced by the lateral network, but the underlying principle
should be similar. This, in addition to the
network growing idea, will hopefully result in increased
performance of the network, both in representation power
(i.e. MSE error) and time to train, since only one unit
is trained each time. We propose to investigate the merits
of such an approach (at least via simulation experiments)
in the future.

Furthermore, all the possible applications of the proposed PCA
networks are certainly not exhausted in this thesis. For example,
in speech processing, and in particular in phoneme recognition,
one is interested in extracting the basic frequencies
({\em formants}) of a speech segment that are usually sufficient
to identify the phoneme uttered in this segment.
The traditional technique used for
this task is Linear Predictive Coding (LPC) where essentially
an AR model is constantly fitted to the incoming data. The
coefficients of the AR model correspond to the poles of the
model which in turn correspond to the characteristic frequencies of the phoneme.
Typically the 3 smallest frequencies serve the purpose of phoneme identification.
The adaptivity of the scheme is {\em absolutely
necessary} because phonemes are uttered in succession
(in natural speech) so the formants change
with time depending on the current phoneme and the system
has to track the changing statistics.

From signal-processing theory
however, it is also known that the $2p$-dimensional
principal component subspace of a signal containing $p$ sinusoids
in white noise has one-to-one correspondence with the
frequencies contained in the signal.
In addition, the smallest eigenvalues
of the autocorrelation matrix
are equal to each other and correspond to the
additive white noise, so removal of these eigenvalues
implies removal of the noise as well.
If the frequencies change then the
$2p$-dimensional PCA
subspace also changes, and conversely, knowing the subspace one
can uniquely derive the $p$ frequencies. This is in essence
the heart of most eigenvalue-based
harmonic-retrieval techniques in signal processing.
So PCA is an alternative way, relative
to the LPC coefficients, for representing the frequencies
contained in the speech signal.
Naturally, the question arises of comparison
between PCA and LPC in frequency identification applications.
More specifically, in applications like speech
where we already observed the necessity of adaptive signal processing,
what would be the performance merits of techniques like APEX 
in relation to LPC?
LPC is a closed-form problem (it basically
requires the inversion of a matrix at each step
which is finite step algorithm)
as opposed to eigenvalue decomposition which has no closed-form
solution. Nevertheless, as we saw in Chapter \ref{ch:Applications},
approximate solutions
of the eigenvalue problem are good enough in many applications,
while the exponential convergence of networks like APEX
show that such approximate solution can be achieved rather quickly.
So it is not clear at this point whether LPC is superior to PCA
(or vice-versa) either in terms of complexity or in terms of performance.
However we believe that this is an important problem that
deserves further investigation.

The noise-removal capability of PCA discussed above
may also be useful for the generalization
problem in learning theory. Generalization is the
prediction of future patterns (data) given a finite set
of representative training data.
If no assumption
is made regarding the data distribution
the problem is ill-posed
and admits infinitely many solutions.
Although it is 
difficult to find a universal assumption that is good for
all problems in the real world, for most cases
a smoothness assumption is both good and sufficient
to make the problem well-posed (i.e. make it
admit a unique solution). PCA's noise removal property
may serve as an alternative way to imposing smoothness on the
data and may also be a useful concept/tool for generalization.
However, to our knowledge, such a connection has not been
made in the past and we propose to look, in the future,
at the usefulness of proposed PCA networks, primarily
as a preprocessing smoothing stage to prediction
(generalization) models.

In image coding, the proposed technique should be extended to cover
color images and should also be incorporated in the MPEG
scheme for bi-directional prediction (B-frames). So far
only uni-directional prediction -- i.e.
the prediction of P-frames from I-frames -- has been studied.
Apart from that, the parallel implementation of the new
method is a very important issue, since a single processor
cannot deliver real-time performance required for example, in television.
Fortunately, both the segmentation and the segment-matching tasks
are highly parallelizable since each block
can be processed independent of the others.
However, the optimal (or simple a cost-efficient) parallel
VLSI implementation of the scheme remains an open question
that will be addressed in a future work.
