Language Acquisition and Robotics Group

University of Illinois at Urbana and Champaign

Department of Electrical and Computer Engineering  

and Beckman Institute for Advanced Science and Technology

Contents
Illy, Alan, and Norbert

Introduction to Our Laboratory

 Here at the Language Acquistion and Robotics Laboratory, we are aiming to develop intelligent robots with the ability to learn natural language. While there are many aspects of this research, the most important are the following ideas: Our research is based around these ideas. We work with three heavily modified robots from Arrick Robotics (pictured above and below), and have developed and integrated modules for various cognitive processes, including modules for sensory input processing, speech recognition, speech generation, navigation, and associative learning. Our ultimate goal in this endeavor is nothing less than the construction and explanation of a mechanical ``mind''.

Back to Contents...

Our Robots
Alan
Alan 		Illy
Illy 		Norbert
Norbert

The three robots we work with are all from Arrick Robotics. Alan is a first-generation Trilobot, with a monocular camera, microphones, and a microwave transmitter and wireless modem for control from a remote workstation. Illy and Norbert are both second-generation, heavily modified Trilobots, with capabilities for binaural hearing, stereo vision, tactile sense, and basic proprioceptive control.

Back to Contents...

Demonstration Videos

Semantic Based Learning of Syntax

In these videos, Illy shows its knowledge of two-word sentences. The words that Illy knows are "kitten", "puppy", "can", "stay", "move", and "gone", which it learned using its associative memory - see the "Associative Learning" video and/or Kevin Squire's Ph.D. thesis below. The meanings of these words should be obvious, except for "gone", which to Illy means that the object has moved far away.

Illy has learned about syntax by hearing example two-word sentences (from an experimenter) that describe events in its environment. Illy then used its knowledge of the words to deduce the syntactic information. It is important to note that in these videos, Illy produces two-word sentences that it has never heard before - while it was trained with the single word "puppy", it did not hear the word "puppy" during the syntax training.

In the "Syntax Demo" video, Illy searches for objects to play with. Illy also demonstrates its sound source localization ability by turning toward the experimenter when called. Once the experimenter gets Illy's attention in this manner, Illy is directed to look for a specific object when it hears the object's name. In the "Syntax Test" video, the experimenter places objects in front of Illy for it to play with.

Videos:

Syntax Demo

Syntax Test

Vision-Based Localization and Map Learning

In this experiment, our robot Illy demonstrates its ability to acquire the mental map of its environment and use the mental map for localization. During the learning phase, Illy is put in an unknown environment and memorizes its navigational experience while exploring. This experience, which is recorded as sequences of images collected from Illy’s camera, is then consolidated to form a map of the environment using our proposed Learning Nonlinear Manifolds from Time Series (Poster Presentation, 314) algorithm. Once Illy has acquired the mental map, it can accurately locate itself in the environment.

The video shows an episode of the localization experiment. The image on the left is Illy’s visual perception, and figure on the right shows Illy’s mental map. There are nine probabilistic maps shown in a 3x3 grid. The eight maps which form the border of the grid show the conditional probability distribution of Illy’s x-y position given a particular (discretized) direction that it is facing. The map on the center is the final probability distribution of Illy's x-y position with the direction parameter marginalized out. As can been from the video, Illy can accurately infer it position and orientation from the visual input.

Video and Figures:

Associative Learning

In this demonstration, our robot Illy is wandering around in her pen. For some time now, we've been teaching her the names of some objects, and in this video, I call to her, and tell her to play with the cat. After she approaches the cat, I tell her the word for cat a few times to help strengthen the association between that word and the object she sees in front of her. After this she plays with the cat briefly, then looks around for her other toys.

Video:

Food Hunting by Illy (2001 Beckman Open House)

Illy has to save enough food before another snow storm comes to Urbana-Champaign. Illy’s favorite food is in some specially designed cans. She needs to find those cans and move them to her nest. The demonstrator eagerly helps Illy with her food hunting. He calls Illy every time he finds a can. When Illy hears the call, she instinctively turns toward the demonstrator and looks for her food. This demonstration shows the following capabilities:

Videos:

First Attempts (2000 Beckman Open House)

Videos:



Back to Contents...

People

 Adviser:
Stephen E. Levinson

Ph.D Students:
Matthew McClain
Shankar Rao

Masters Students:
Ryan Rivera

Former Students:
Christopher Dodsworth (MS '01)
Danfeng Li (PhD '03)
Ruei-sung Lin (PhD '05)
Qiong Liu (PhD '00)
Kevin Squire (PhD '04)
Weiyu Zhu (PhD '03)


Back to Contents...

Publications

Books

Theses

Journal Publications

Technical Reports

Conference Publications

Workshop Publications



Back to Contents...

Copyright Notice

The following notice is required by IEEE:

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and conditions invoked by each authors copyright. In most cases, these works may not be reposted without explicit permission from the copyright holder