Topographic Class Grouping via Discriminative Sparse Coding

How the higher-level representation might alter lower-level representation is still largely unknown in learning deep networks. To investigate this problem, we developed a biologically inspired Multilayer In-place Learning Network (MILN). In such a network, a discriminative sparse coding model is used to allow concurrent information flows from both bottom-up and top-down directions.  The sparse coding model is l0-norm constrained and supervised via top-down propagation from labels. Its winner-take-all solution leads to a highly efficient online learning, which does not require iterative steps to infer the internal states.



The introduction of top-down information flows helps to group reconstructive features belonging to the same class, which is called topographic class grouping (TCG).   The topographic class grouping enables development of  feature discriminance regarding classification tasks. Compared to unsupervised sparse coding via only bottom-up directions, the bidirectional discriminative approach improves the visual recognition performance a lot.  Further studies also showed that recurrent top-down connections can provide temporal context and such context assists perception in a continuously but gradually changing physical world.

Experiments are conducted on multiple data sets including MSU-25, GM car vs. non-car, handwritten digits and 3-D objects. As shown in the example of MSU-25 data set, features belonging to the same object class are grouped together for better recognition performance.  And in the next figure, temporal context via top-down connections even improves performance to nearly 100% after the transition periods from one rotating object to the next.

Reference

J. Weng, H. Lu, T. Luwang and X. Xue, "Multilayer In-place Learning Networks for Modeling Functional Layers in the Laminar Cortex",  Neural Networks, vol. 21, no.2-3, pp. 150-159, 2008. PDF file. (An introduction of learning concepts and algorithms for Multilayer In-place Learning Networks.)

M. Luciw and J. Weng, "Topographic Class Grouping with Applications to 3D Object Recognition", in Proc. IEEE World Congress on Computational Intelligence: International Joint Conference on Neural Networks, Hong Kong, June 1-6, 2008. PDF file. (First paper described topographic class grouping (TCG) via bottom-up and top-down directions in MILN, with application to 3D object recognition.)

M. Luciw, J. Weng, S. Zeng, "Motor Initiated Expectation through Top-Down Connections as Abstract Context in a Physical World", in Proc. 7th IEEE International Conference on Development and Learning, Monterey, CA, pp. 115-120, Aug. 9-12, 2008. PDF file. (Top-down connections are provided as temporal contexts, enabling vehicle recognition for disjoint tests to become almost perfect.)

M. Solgi and J. Weng, "Developmental Stereo: Emergence of Disparity Preference in Models of Visual Cortex", IEEE Transactions on Autonomous Mental Development, vol. 1, no. 4, pp. 238-252, 2009. PDF file. (TCG was utilized to learn temporal stereo without explicit stereo matching, with analysis.)

M. Solgi, "Cortex-Inspired Developmental Learning Networks for Stereo Vision", PhD Dissertation, Computer Science Department, Michigan State University, 2013.  PDF file. (Dr. Solgi's dissertation describing TCG for temporal stereo in more details.)

M. Luciw and J. Weng, "Top-Down Connections in Self-Organizing Hebbian Networks: Topographic Class Grouping",IEEE Transactions on Autonomous Mental Development, vol. 2, no. 3, pp. 248-261, 2010. PDF file. (Extended journal version of TCG; More analysis is conducted on why using top-down connections result in discriminative features.)

Z. Ji, M. Luciw, J. Weng, and S. Zeng, "Incremental Online Object Learning in a Vehicular Radar-Vision Fusion Framework," IEEE Transactions on Intelligent Transportation Systems, vol. 12, no. 2, pp. 402-411, 2011. PDF file. (Online object learning system based on radar-vision fusion framework. Topographic class grouping is utilized to develop discriminative features on V1-coded representation, rather than original pixels, leading to the better performance in vehicle identification.)

Z. Ji, W. Huang, G. Kenyon, L. M. A. Bettencourt, "ierarchical Discriminative Sparse Coding via Bidirectional Connections", in Proc. IEEE International Joint Conference on Neural Networks, San Jose, Aug. 2011. PDF file. (Bidirectional l0-norm sparse coding. Its gradient descent search solution leads to an efficient online format similar to MILN, which does not require iterative steps, and shows the property of TCG.  Divide-and-conquer strategy is further provided to solve a bidirectional network with deeper layers.)
Note: This PDF link provides the latest updated version. The citation of Luciw and Weng's 2010 IEEE-TAMD paper "Top-Down Connections in Self-Organizing Hebbian Networks: Topographic Class Grouping" was missing in the original submission under IEEE Xplore.