Behavior Learning with Multimodal Sensors

Graduate

We are interested in a system that has the capability of sensing the real world through multimodalities and behaves accordingly as the human user wishes it to be. In other words, such a system should be able to learn behaviors through the interaction with the environment, including the trainer. This kind of system has prosperous applications in autonomous robot, wearable computing, HCI, multimedia index and etc. Behavior learning with multimodal sensors, however, is a very challenging task because of the unpredictable world and the tremendous amount of information involved. Some of the major difficulties include,

• Online learning mode: The designer of the system would never be able to anticipate all the situations that the user of the system would be facing. The designer can only embed as powerful learning capability as possible to the system and let the system learn behaviors on-the-fly.
• Attention: An image or a sequence of utterance may include a lot of different patterns. Some are relevant to the task and of interest while others are not. How to focus on the important ones and ignore others?
• Association: association among sensory inputs from different modalities and imposed or intended actions. How to handle high dimensionalities and heterogeneous nature of sensors?

Return to Workshop Listing