Read the
details here. Or continue on for a
general overview.
The first stage of biological mental development is development of the perceptual system. The purpose of this early learning stage is to create effective internal representation of the environment using a limited storage capacity. The constant stream of high-dimensional sensory inputs cannot be stored exactly, but there is a high degree of repeated, similar structure that the sensory system can take advantage of. The single-layer Lobe Components Analysis (LCA) algorithm is designed with a similar principle in mind: learning an optimal representation of a set of input samples using a much smaller set of vectors, each called a lobe component. Each lobe component is the most efficient estimator for the samples in its region. LCA is an "in-place" algorithm, in the sense that it operates incrementally (no samples are stored), does not require any memory space to save the higher order statistics, such as the covariance matrix or kurtosis, and the network does not use a separate developer, such as gradient-based error backpropogation, to learn - the network develops and learns as a side effect of competitive interactions.
The first stage of biological mental development is development of the perceptual system. The purpose of this early learning stage is to create effective internal representation of the environment using a limited storage capacity. The constant stream of high-dimensional sensory inputs cannot be stored exactly, but there is a high degree of repeated, similar structure that the sensory system can take advantage of. The single-layer Lobe Components Analysis (LCA) algorithm is designed with a similar principle in mind: learning an optimal representation of a set of input samples using a much smaller set of vectors, each called a lobe component. Each lobe component is the most efficient estimator for the samples in its region. LCA is an "in-place" algorithm, in the sense that it operates incrementally (no samples are stored), does not require any memory space to save the higher order statistics, such as the covariance matrix or kurtosis, and the network does not use a separate developer, such as gradient-based error backpropogation, to learn - the network develops and learns as a side effect of competitive interactions.
Above shows a set of samples,
designated
by crosses, around the origin. The samples are represented by
the
three vectors v1, v2, and v3. Each vector is a lobe component,
and is the best single-vector representation (in both direction and
length) for the set of samples within that region. The regions
are designated by R1, R2, and R3.
The lobe components converge to
represent well-separated regions using the neural learning
mechanisms of
Hebbian learning and lateral inhibition: for each sample, only the
closest lobe component(s) in terms of direction is the "winner", and
will
update the
vector direction to be closer to the sample vector direction. The
variance of the samples that each lobe component updated for is
preserved as the length, or energy, of the vector.
Above is a result of the unsigned
version of the algorithm. Each vector represents samples by the
asbolute value of their direction. The version of LCA available
for download on this page is unsigned, although that can be changed
within the program.
When the input to LCA is a set of
samples from images
of nature, such as the one above, the lobe components converge to a set
of mostly localized edge detectors, as seen below. This is
the same result as the Independent Components Analysis algorithm, which
seeks to extract signals based on some measure of independence, such as
maximum kurtosis or minimum mutual information. But LCA is not
designed to minimize or maximize any criteria in particular. The
reason the results are similar is that natural images are
super-Guassian. The LCA algorithm will converge to the most
independant signals because of the nature of super-Gaussian input
-
correlated areas of high density. So, LCA is ICA for
super-Gaussians.
The above is a grid of 256 lobe
components developed from natural
images. They are sorted from the top left to the bottom right by
number of times updated. LCA naturally leads to sparse responding
features for computer vision. Observe that most of the area of
the lower order components is gray (zero). The remaining area,
the edge area, is the receptive field of this neuron. It will not
respond very well unless there is similar activity
in the input. Good feature seperation with sparse response is a
desirable criteria for a real world perceptual
system, in terms of efficiency. These low level features are
similar to those observed in
the first area of the visual cortex. They can
be combined in different ways over a larger receptive field to create
higher-level features for contour detection. This recursive use
of shared features is another useful property of a perceptual system
with a limited resource.
Considering the
many examples of
super-Gaussian data, such as real
images or real audio, LCA can be a very efficient way to extract the
independent components. As seen above, the LCA algorithm (using
the amnesic averaging fast learning technique)
converges to
a known set of independent super-Gaussian source signals (comparison
with ground truth - the y axis is % error) in many fewer iterations
than two ICA algorithms. The FastICA algorithm is a batch
algorithm,
and Extended Infomax (which does not converge at all for this case) is
a
block-incremental algorithm. LCA is more constrained than either
of
these, yet it outperforms both here.
An extension of
LCA enforces lobe
component position within a
two-dimensional grid. An updating unit will also update its
neighbors
in a limited size neighborhood window. Above is an example of
topographic LCA using "high-level" digit data. Below is an
example of the result with low-level features. Use of topographic
updating leads to sparsely responding areas, instead of single
units. Within this area are variations of the feature.
Representing each feature in several different ways is
desirable because of the many possible variations of each.
There is a description of the
code and a brief tutorial here.