General-Purpose Vision Architecture, Invariance, Attention and Reasoning

Tutorial for CVPR 2010

Afternoon, Monday, June 14, 2010


Juyang (John) Weng, Professor
Embodied Intelligence Laboratory
Department of Computer Science and Engineering,
Cognitive Science Program, and Neuroscience Program
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388, 617-253-8024


Much progress has been made in understanding and modeling the brain architecture.  This tutorial reviews major published agent architectures, with an emphasis on their relationships.   There is a large gap between connectionist modeling and symbolic modeling.    Connectionist modeling works on low-level sensory data, while symbolic modeling deals with high-level abstract symbols, but there is relatively little study for intermediate representations.   This tutorial will focus on intermediate representation suited for invariant object recognition, scene classification, spatiotemporal event detection, attention in the presence of complex natural backgrounds, and reasoning under abstract contexts (e.g., goal directed) with pixels.

A fundamental challenge for connectionist representations is abstraction which symbolic representations enjoy through human hand-design.  The term “abstract” refers to properties disassociated with any particular form or instance.  The capability of attention is essential for internal abstraction from iconic stimuli.   After the review, I introduce an architecture of the primate visual system, suited for computer vision. It deals with two major challenging open questions (1) how the system deals with space and time in generating intermediate internal representations at various levels, (2) how the system abstracts from concrete sensory experiences through attentions, (3) how the abstract context can be used to reason with pixels.

For a related IJCNN 2010 Panel Session and email discussion, visit

Tutorial topics

1.     Agents

2.     History of agent research

3.     Symbolic monolithic representations and agent architectures, such as 2.5-D maps, optical flows, SLAM and SMPA framework.

4.     Symbolic contextual representations and argent architectures, such as Bayesian Net, HMM, MDP, POMDP, Kismet, Cog.

5.     Connectionist emergent representations and architectures, such as Neocognitron, Cresceptron, SOM, LVQ, LISSOM, ICA, LCA, LDA, SVM, HDR, AGREL, MILN.

6.     Connectionist temporal representations and architectures, such as Elman net, Jordan net, Hopfield net, Boltzman machines, LSTM, WSA, SASE.

7.     Biological development: genes, mitosis, genomic equivalence, cell migration, cell signaling, morphogen and epigenesis.

8.     Early processing. receptors, retina, and LGN.

9.     Laminar architecture of all cerebral cortex, and its implication.

10.  Development of feature detectors for different levels.

11.  Architecture of the visual cortex, dorsal and ventral representation.

12.  Attention: bottom-up, top-down based on location, top-down based on type.

13.  Deliberative reasoning with the abstract and the concrete.

14.  Space and time complexities of the general-purpose vision systems.

Biographic sketch of the lecturer

Juyang (John) Weng is a professor at the Department of Computer Science and Engineering,  the Cognitive Science Program, and the Neuroscience Program, Michigan State University, East Lansing, Michigan, USA. He received his BS degree from Fudan  University in 1982, his MS and PhD degrees from University of Illinois at Urbana-Champaign, 1985 and 1989, respectively, all in Computer Science.  From August 2006 to May 2007, he is also a visiting professor at the Department of Brain and Cognitive Science of MIT.   His research interests include biologically inspired neural systems, computer vision, audition, touch, human-machine multimodal interface, and intelligent robots.  He is the author or coauthor of over two hundred research articles.  He is an editor-in-chief of International Journal of Humanoid Robotics and an associate editor of the new IEEE Transactions on Autonomous Mental Development. He was a program chairman of the NSF/DARPA funded Workshop on Development and Learning 2000 (1st ICDL), the Chairman of the Governing Board of the International Conferences on Development and Learning (ICDLs) (2005-2007,, a general chairman of 7th ICDL (2008) and general chairman of 8th ICDL (2009), chairman of the Autonomous Mental Development Technical Committee of the IEEE Computational Intelligence Society (2004-2005), an associate editor of IEEE Trans. on Pattern Recognition and Machine Intelligence, an associate editor of IEEE Trans. on Image Processing. He is a Fellow of IEEE.

Some related publications

J. Weng, N. Ahuja and T. S. Huang, “Learning recognition and segmentation using the Cresceptron,” International Journal of Computer Vision, vol. 25, no. 2, pp. 105-139, Nov. 1997.

D. L. Swets and J. Weng, “Hierarchical discriminant analysis for image retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 386 - 401, May 1999.

W. S. Hwang and J. Weng, “Hierarchical Discriminant Regression,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1277 - 1293, Nov. 2000.

J. Weng, J. McClelland, A. Pentland, O. Sporns, I. Stockman, M. Sur and E. Thelen, “Autonomous Mental Development by Robots and Animals,” Science, vol. 291, no. 5504, pp. 599-600, Jan. 26, 2001.

J. Weng and I. Stockman, “Autonomous Mental Development: Workshop on Development and Learning,” AI Magazine, vol. 23, no. 2, pp. 95-98, 2002. 

J. Weng, Y. Zhang and W. Hwang, “Candid Covariance-free Incremental Principal Component Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 8, pp. 1034-1040, 2003. 

J. Weng, “Developmental Robotics: Theory and Experiments,” International Journal of Humanoid Robotics, vol. 1, no. 2, pp. 199-235, 2004.

J. Weng and W. Hwang, “From Neural Networks to the Brain: Autonomous Mental Development,” IEEE Computational Intelligence Magazine, vol. 1, no. 3, pp. 15-31, 2006.

J. Weng, “On Developmental Mental Architectures,” Neurocomputing, vol. 70, no. 13-15, pp. 2303-2323, 2007.

Y. Zhang and J. Weng, “Task Transfer by a Developmental Robot,” IEEE Transactions on Evolutionary Computation, vol. 11, no. 2, pp. 226-248, 2007.

X. Huang and J.Weng, “Inherent Value Systems for Autonomous Mental Development,” International Journal of Humanoid Robotics, vol. 4, no. 2, pp. 407-433, 2007.

J.Weng, T. Luwang, H. Lu, X. Xue, “Multilayer In-place Learning Networks for Modeling Functional Layers in the Laminar Cortex,” Neural Networks, vol. 21, no.2-3, pp. 150-159, 2008.

Z. Ji, J. Weng and D. Prokhorov, “Where-What Network 1: ‘Where’ and ‘What’ Assist Each Other Through Top-down Connections,” IEEE International Conference on Development and Learning, Monterey, CA, pp. 61-66, Aug. 9-12, 2008.

M. Luciw, J. Weng, S. Zeng, “Motor Initiated Expectation through Top-Down Connections as Abstract Context in a Physical World,” IEEE International Conference on Development and Learning, Monterey, CA, 115-120, Aug. 9-12, 2008.

J. Weng, “Task Muddiness, Performance Metrics and the Necessity of Autonomous Mental Development,” Minds and Machines, vol. 19, pp. 93-115, 2009.

J. Weng and M. Luciw, “Dually Optimal Neuronal Layers: Lobe Component Analysis,” IEEE Transactions on Autonomous Mental Development, vol. 1, no. 1, pp. 68-85, 2009.

 Back To Weng's Home Page: