%%INTRODUCTION
\chapter{Introduction}
\label{ch:introduction}

Understanding the underlying principles of biological
perceptual systems is very important not only for
the neuroscientist who wishes to know how the brain works
but also for the engineer and the computer scientist
who wish to use that knowledge to develop artificial
perceptual systems.
In recent years significant advances in neurobiology,
neural network theory and neuro-computation modeling
shed new light into our efforts of understanding
biological perceptual processing. Although the problem is far from
solved, some partial answers have already been formulated
in particular regarding certain subsystems,
such as the mammalian visual cortex for example.
It was found that at least in the beginning stages
of the mammalian visual processor the neurons self-organize
(i.e. learn) to recognize specific features of the environment.
It was conjectured that the self-organizing principle for
these feature-analyzer cells is related to information-theoretic
criteria or statistical criteria which maximize the information
compression into few representation parameters, in this case
the synaptic weights of the neural network \cite{Linsker88}.

As early as 1949, Donald Hebb \cite{Hebb49} proposed a
simple, yet biologically plausible rule for adjusting the
synaptic weights during a neural network learning process.
The Hebbian rule is a correlation-type rule, i.e. the strength
of the connection between two cells is proportional to the
correlation between their output activations. Surprisingly,
this simple rule turns out to be closely related
with the mathematical principle of maximum statistical
compression known for many years in the literature
as Principal Component Analysis \cite{Hotelling33}.
PCA identifies the
most important features (subspaces) of a high-dimensional statistical
distribution (such as for example,
the distribution of the data processed by the retina)
in the sense that the projection error onto those
feature subspaces is minimal.
Alternatively, PCA can be interpreted as
maximizing the projection variance of the stochastic signal
since both the variance criterion
and the projection-error criterion lead to the same PCA solution.
The optimal solution under both criteria is the subspace spanned by the
eigenvectors of the signal autocorrelation matrix associated
with the largest eigenvalues (henceforth called principal eigenvectors).
The relation between the Hebbian rule and PCA was first shown
by Oja in 1982 \cite{Oja82} for a simplified linear
neuron: a normalized version of the Hebbian rule
applied on a linear unit extracts the principal component
of the input sequence, i.e. it converges to the principal eigenvector
of the input autocorrelation matrix.
Although actual neurons are not simple linear units
the relationship of the Hebbian rule
with PCA can not be just a happy coincidence and it gives us
insight for the nonlinear case as well. After all, nonlinear
mappings can be locally approximated by linear ones.

Furthermore, PCA is a significant mathematical
tool in itself, beyond its relationship with neural networks. 
It has been known for example, that Principal Component
Analysis (PCA), Singular Value
Decomposition (SVD), Eigenvalue Decomposition (ED) and the
Karhunen-Lo\`{e}ve Transform (KLT) are all related techniques
with established applications in signal/image processing, control
theory, pattern recognition, statistical analysis, etc.
In this thesis we study various extensions of Principal Component
Analysis after we first establish the full relationship between
the Hebbian rule and standard PCA. All new methods are based
on extensions of the PCA optimization criterion
which are suitable for different applications. For each
extension we also provide a suitable neural implementation using
the Hebbian learning rule on simplified linear networks.
We also look closely at some of the applications of the new models
which require adaptive solution methods.
We devote the last chapter to one application (motion compensation
for image coding) where more traditional (non-neural net)
techniques are used. More precisely, this thesis is divided
in the following chapters.

In Chapter \ref{ch:PCA} we investigate the relation of
PCA with the Hebbian learning rule.
Although it was shown by Oja that one linear unit,
after being trained by
the normalized Hebbian rule
extracts the principal component,
the extension to multiple components
had been so far rather artificial and biologically implausible.
We show how to train a multi-component network
using only the normalized Hebbian rule for all the
synaptic weights without any additional non-neuronal
or off-line processing. The new model is called APEX, an
acronym for Adaptive Principal component EXtraction.
The network features a novel structure
compared to previous neural PCA models
by incorporating a lateral
connection network which is instrumental for the orthogonalization
of the synaptic weight vectors.
We formally prove that our model
extracts the required components and in fact, the convergence
is exponential with respect to the iterations of the algorithm.
In addition, we
are able to theoretically predict the decay rate of the components
as the network is trained given the value of the step-size parameter $\beta$.
The algorithm is faster than
the previous neural PCA schemes
and has the advantage of extensibility: one can increase
the order of the network by adding one more neuron
or can decrease it by removing one or more neurons
without need for retraining the old or remaining units.
We also show the connection with the Recursive Least
Squares algorithm (RLS) via which we derive the optimal
value for the step-size parameter. Finally,
we back our theoretical claims by simulation results.

Next we show that
the APEX network is applicable to a new kind of Principal
component analysis called Constrained PCA (CPCA). This is 
useful in applications where one wants to find the directions
containing the most information regarding a signal
under an orthogonality constraint with respect to some
prespecified subspace. This subspace could contain for example
an interference signal which must be avoided entirely,
thus we project the signal to an orthogonal direction
that maximizes the variance of the projection.
We explicitly show the convergence rates of APEX for CPC
(as we did for standard PCA) and we show that exponentially
the network converges to the required eigenvectors.

In Chapter \ref{ch:OPCA} we propose another extension of
standard PCA by introducing what we call Oriented
Principal Component Analysis (OPCA). OPCA involves
two signals (usually one is called signal and the other noise)
the goal being to maximize the signal to noise power ratio (SNR)
after projecting the signals onto a small dimensional subspace.
In the special case where the noise is white, OPCA
and PCA become the same.
In analogy to PCA, which is related to the
Eigenvalue problem of the autocorrelation of a sequence,
the OPCA problem is related to the Generalized Eigenvalue
problem of the autocorrelations of two signals.
We propose two models based on extensions of the
Hebbian rule which are capable of extracting principal oriented
components of a pair of sequences.
For both models we show the mathematical justification
of the algorithm along with simulation experiments
which display the actual convergence of the methods
to the desired components.
We finally discuss the relationship of OPCA with matched
filtering and Fisher linear discriminant analysis
as well as with other techniques used for various applications in signal
processing and control.

In Chapter \ref{ch:APCA} we make the connection between
PCA and linear regression and Generalized Singular Value
Decomposition (GSVD).
The standard back-propagation learning rule
for neural networks which is suitable
for primarily approximation problems is shown in the
linear case, to be related to a specific Generalized
Eigenvalue Problem involving the input and the teacher sequence.
It is shown that under reduced number of hidden units
the BP network is solving a GSVD problem special cases of
which are linear regression (Wiener Filtering) and SVD
or Principal component analysis. When the teacher
and the input to the BP network are equal then the
problem is symmetric and is related to PCA. In the general
case when the input and teacher signals are different,
then we call the problem Asymmetric Principal Component
Analysis (APCA). Although BP tackles APCA it does not produce
the exact asymmetric principal components (i.e. the generalized
eigenvectors of the GSVD problem) but rather some unpredictable
combination of them. This could be a disadvantage when we may want
to grow the network by one unit later: in that case the net
must be trained from the beginning. However, the exact components
can be extracted by introducing a deflation transformation
similar to the one used for the standard eigenvalue problem.
This deflation can also be implemented by a lateral
connection network as with APEX. We identify two
learning procedures for this network in general
(covering the APEX case as well): the dynamic and the
immediate orthogonalization rules. We show via theoretical analysis
and via simulation that the proposed network converges
asymptotically to the desired solution.

Another version of the APCA problem called the {\em
Cross-correlation} APCA is the problem of finding two
projection directions instead of one that will maximize
the cross-correlation of the projections. This problem
is equivalent to the singular value decomposition of the
cross-correlation of the two signals involved. We propose
and prove a neural model that extracts these components.
We again apply the lateral network structure to the
net for extracting multiple components and we show analytically
and via simulation that the network indeed extracts
the desired components.

%**********Beef up *********************
In Chapter \ref{ch:Applications} we show some selected
applications of the PCA, OPC, and APCA ideas and methods.
The problems are related to signal detection,
system identification and image compression. We show detailed
simulation comparisons with other techniques in the corresponding
problems and we establish the superiority or limitations of the
neural techniques.

In Chapter \ref{ch:Motion} we shift our attention to a different
application: motion estimation for compression and coding of
image sequences. This application has become increasingly
important due to the prospects
of High Definition Television (HDTV)
and Integrated Services Digital Networks (ISDN) that will 
handle and transmit images in digital form
requiring a very large bit rate. To reduce the bandwidth
and make these applications viable image compression
becomes essential. In the case of still photographs
the Joint Photographics Experts Group (JPEG) has
submitted a proposal for standardizing image compression.
A similar effort is under way for motion pictures as well
by the Moving Pictures Experts Group (MPEG) and other
organizations (notably the CCITT) for various target
bit rates and image qualities. Most applications
considered include video-telephony and video-conferencing,
HDTV, CD-ROM picture storage, etc. A major problem
in motion picture compression is how to take advantage of the
temporal correlation of consecutive images
to help predict the next frame and reduce the number of bits
required to be sent by the encoder.
Block matching is a commonly used technique for motion
estimation between consecutive frames. One drawback
of this technique is the high frequency artifacts created
by the artificial square borders of the matching
process. Given the sensitivity of the human visual system (HVS)
to broken lines and artifacts of similar type the problem
can be potentially serious especially for very low bit rate
communications. We propose to amend this problem by relaxing
the shape of the blocks to be matched.
The new technique divides the image into segments
locally, according to gray level, and matches each 
one with a similar segment in the next frame. The technique
significantly reduces the bit-rate required for similar
Mean Square Error (MSE) performance compared to the standard
MPEG algorithm even without sending to the decoder the (coded)
prediction error. In addition the block artifacts are practically
absent and the picture quality is acceptable to the human eye
even at the rate of (approximately) 0.1 bits per pixel.
For television applications it is almost imperative that
the decoder process remains simple so that it can be cost-effectively
implemented on a home television set. Our scheme obeys this
requirement since the decoder complexity is not more than
the standard MPEG algorithm.

\section{Thesis Contributions}

The major contributions and original ideas included in this thesis
are summarized below:

\begin{itemize}
\item We propose a new learning network model
for Principal Component extraction (called APEX)
based completely on the normalized Hebbian rule.
The network features a novel connection structure where in
addition to the feed-forward connections between inputs and outputs
there are lateral connections between output units themselves.
The lateral connection network enforces orthogonality
between the feed-forward weights which is essential for the
extraction of the orthogonal principal components.
The novel structure also facilitates the growing or shrinking
of the network whenever an order update is required.

\begin{itemize}
\item Mathematically rigorous proofs of the asymptotic properties
of the network are provided, as well as of the orthogonality
property between the feed-forward weights.

\item A connection is made
between APEX and the Recursive Least Squares (RLS) algorithm
which provides us with an optimal value for the step-size parameter
and drastically improves the performance of the algorithm.

\item We show that the convergence rate of the network to be
exponential and we are in fact able to analytically approximate
it for each component while
we verify our prediction via simulation.

\item We show that APEX is superior to previous neural PCA models
in terms of computational complexity, flexibility of order increase/decrease,
and lack of numerical analysis in the previous methods regarding
the optimal size of the learning step-size parameter.

\item We put forward a simple argument justifying the feasibility of a
parallel APEX model, which we then simulate and study its convergence
properties. Theoretical proof of the parallel model is given elsewhere
\cite{Chen91}.
\end{itemize}

\item Extension to the constraint PCA problem: we mathematically prove
that the APEX network is capable of extracting the constraint components
of the input sequence and verify our theory via simulation experiments.

\item Introduction and mathematical definition of the Oriented PCA problem.
Furthermore, two learning network models based on the Hebbian rule
are proposed for extracting the Principal Oriented Component of
a pair of signals. Mathematical analysis based on approximation assumptions
proves the asymptotic convergence of the first model to the desired
component. For the second model we only provide a less rigorous
analysis based on an appropriate Lyapunov energy function for the
problem. Simulation experiments accompany both methods showing the
asymptotic convergence of these techniques.

\item We prove that in a two-layer supervised linear feed-forward network
with fewer hidden units than input and output units the
optimal solution to the least-squares criterion is related
to the Generalized Singular Value Decomposition
between the input data matrix and the input-output outer product
matrix. This holds true even if the input data are rank-deficient,
i.e. the input autocorrelation matrix is non-singular.
This is an extension of the work reported in \cite{Baldi89}.
We call the problem Linear Approximation Asymmetric PCA since
PCA is a special case when the teacher of the network is equal
to the input of the network. Wiener filtering is also shown to
be a special case of linear approximation APCA.
We prove that the least-squares
cost function contains no local minima under reasonable assumptions
and therefore, the gradient descent type algorithms like
Back-Propagation will be able to achieve the global minimum.
However the minimum is not unique and BP extracts some linear
combination of the Asymmetric PCA components rather than the components
themselves.

In order to extract the exact generalized singular components
related to linear approximation APCA we propose to use
a lateral connection network among the hidden units
in addition to the feed-forward connections that exist in the 
standard BP network. We propose two types of learning rules
for the lateral net: the dynamic and the immediate
orthogonalization rules. We show that the APEX lateral
learning rule falls into the immediate orthogonalization
category and imply that the lateral network has broader
applicability as well. We provide simulation experiments
that depict the performance of our proposed model
in extracting the desired generalized singular vectors.

\item Introduction of another extension of standard PCA:
the cross-correlation APCA problem. This problem is related
to the SVD of the cross-correlation matrix of two signals
and can be tackled by another proposed linear learning network.
The network again features the lateral connections we found
useful in both the PCA and the linear approximation APCA problems.
It is formally shown that our model indeed extracts the
desired singular vectors of the cross-correlation matrix
and the accompanying simulations verify this claim and also
imply that the convergence is exponential.

\item We demonstrate the applicability of the above introduced
techniques in three different types of applications:
\begin{itemize}
\item drifting-sinusoids detection in white gaussian noise,
where we show that our proposed APEX-based scheme has superior performance compared
to previously proposed methods at the cost however of increased
computational complexity. Our scheme has the additional
advantage that it can self-tune to the statistics of
the drifting phase signal so that perform well in a (slowly)
changing environment.
\item system identification, where a linear approximation APCA
network can be used for identifying a reduced-rank
linear system, even if it is driven by reduced-rank input data.
\item image coding, where the optimality of the Karhunen-Lo\`{e}ve
transform can be utilized for coding sequences of images.
Our technique is based on successive applications of the APEX
model by both the encoder and decoder on a sequence of images so that 
the decoder can estimate the optimal PCA subspace for the
next incoming frame. We show that this technique performs better
than traditional DCT coding for the same number of components.
\end{itemize}

\item In coding moving pictures, we introduce a new technique
for motion compensation
that practically eliminates the block artifacts produced by the
standard block-matching method and at the same time achieves
significant reduction in the bit-rate for same MSE performance compared
to MPEG coded standard DCT-based technique. The computational
complexity of the encoder is increased but not the decoder
who is even simplified in the case where the DCT can be avoided.
\end{itemize}
