Perceiving and Acting
Some AI systems can directly interact with the outside world (whether physical or virtual). Such a sensorimotor mechanism is also a necessary front-end for a language interface.
1. Perception
Roughly speaking, sensors convert external stimuli (physical, chemical, biological, etc.) into sensations represented within AI systems, which are usually not identical to human sensations, though they may be similar to various degrees.
Perception is the process of interpreting sensory input and constructing an internal representation of the environment. Unlike sensation, which merely records stimuli, perception organizes, filters, abstracts, and relates sensory data to the system's existing knowledge and goals.
AI research on perception has focused primarily on vision and sound. The major challenge is to choose proper features for each level of abstraction. Initially, the levels and the features at each level are chosen by the designers. For instance, in computer vision, an influential approach was proposed by David Marr, who viewed vision as transforming a 2D projective image on the retina into a 3D model of objects and events in the world.
Traditional computer vision relied heavily on handcrafted features and explicit models of visual processing. Recent deep learning approaches shift much of this design effort to learning algorithms that acquire useful representations directly from data. In this "feature learning" process, features are generated and selected by a learning algorithm according to their contribution to the overall task.
Research in cognitive science suggests alternative approaches for AI perception:
Related fields:
- In spoken language processing, deep learning has greatly improved the quality of speech-text mapping in both directions (Speech recognition and Speech synthesis), providing an additional stage in NLP.
- AI has also been applied to Music and Art, spanning both perception and composition/creation. In recent years, AI-Generated Content (AIGC) has received considerable attention. Whereas perception maps sensory signals to abstract representations, these studies perform the reverse transformation by generating sensory content from internal representations, and can thus be viewed as an early computational form of imagination.
2. Robots and Agents
Robots interact directly with their environment through sensors and actuators.
Some categorical clarifications:
- Intelligent robots differ from industrial robots by emphasizing flexibility.
- A humanoid robot is a special type of robot, though even such a robot will not have inputs and outputs identical to a human's.
- In a broad sense, an agent or "bot" can be considered a robot interacting with a virtual environment, such as a computer game or a "metaverse".
- Cyber-Physical Systems (CPS) represent a broader category that includes robots as a specialized subset.
Robots are often equipped with specialized hardware, a subject primarily studied in electrical and mechanical engineering. Robot software is closer to AI and addresses unique requirements:
- Managing diverse sensors and actuators,
- Handling novel situations with real-time responses,
- Maintaining various degrees of autonomy.
An especially important idea is that perception and action should not be viewed as independent processes. Intelligent agents often act in order to improve perception, while perceptual information continuously guides action. Intelligent behavior emerges from a continuous feedback loop in which perception guides action and action creates new perceptual experiences.
This feedback loop is evident in robotics, autonomous vehicles, and reinforcement learning systems, where successful behavior depends on constant adaptation to environmental changes.
Robots provide a common platform for many AI techniques, with current special emphasis on action and body control. Several control paradigms exist for robots:
| Architecture |
Main Idea |
Limitation |
Examples |
| Deliberative |
Plan before acting |
Slow in dynamic environments |
Shakey, BDI agents |
| Reactive |
Respond directly to stimuli |
Limited long-term planning |
Cog, RL robots/agents |
| Hybrid |
Combine planning and reaction |
Integration complexity |
AuRA, Robust-TAMP |
Examples of perception-action systems include autonomous vehicles, mobile robots, drone navigation, reinforcement-learning agents, and embodied AI systems. These systems continuously interact with their environments rather than solving isolated problems. Modern AI approaches emphasize learning ability—whether symbolic or subsymbolic—and have expanded into multi-agent and swarm systems.
From AI's perspective, the key issue in robots and agents is not their programmed or controlled behavior, but their learned behavior, especially in changing environments. Various approaches include:
The public image of "intelligent robots" often differs from what AI researchers actually pursue in this field. Examples include:
3. Theoretical Issues
Readings
- Russell and Norvig: Chapters 25, 26