An Oz-Centric Review of Interactive Drama and Believable Agents

Industry

1. Cyberlife (makers of Creatures): http://www.cyberlife.co.uk/ Creatures are Alife pets that you raise from eggs. Their technology is distinctive in its level of biological modeling. A Creature has a neural net for action selection, artificial biochemistry (including hormonal effects on the neural net), an immune system, and a reproductive system (a genome encodes for creature traits).

2. Extempo Systems: http://www.extempo.com/ Extempo Systems was founded by Barbara Hayes-Roth, leader of the Virtual Theater Project at Stanford. Extempo is creating architectures and authoring tools for the creation of improvisational characters. Their first demo is Erin the bartender, a character who serves drinks and chats with customers in a virtual bar.

3. Fujitsu Interactive (makers of Fin Fin): http://www.fujitsu-interactive.com/ Fin Fin is a half-bird half-dolphin creature who lives in a world called Teo. Users interact with Fin Fin via a microphone, proximity sensor and mouse. Fin Fin is shy; a user has to slowly build up a relationship with Fin Fin over time. Fin Fin utilizes technology developed by the Oz group at Carnegie Mellon.

4. Motion Factory: http://www.motion-factory.com/ Motion Factory is developing “Intelligent Digital Actor technology.” Digital actors generate their own animation (motion) based on interactions with the environment. Motion Factory is an example of work converging on believable characters from the graphics community rather than the artificial intelligence community.

5. Persona Project at Microsoft Research: http://www.research.microsoft.com/research/ui/persona/home.htm The Persona project at Microsoft Research is developing the technologies required to produce conversational assistants—lifelike animated characters that interact with a user in a natural spoken dialog. Their first prototype is Peedy, a character that responds to requests to play music. Gene Ball, a researcher in the Persona Project, organizes the conference Lifelike Computer Characters.

6. P.F. Magic (makers of Petz): http://www.pfmagic.com/ Petz are autonomous pets that live on your screen.

7. Tamagocchi (from Bandai): http://www.virtualpet.com/vp/farm/lleg/lleg.htm Tamagocchi is a small, egg shaped plastic toy with an LCD screen and 3 buttons. Users must nurture a creature that lives on the screen by feeding it, giving it medicine, disciplining it, and cleaning up excrement. If the user is negligent in these tasks, the creature dies. This product is a craze in Japan. While Tamagocchi possesses neither sophisticated personality nor sophisticated behaviors, it is an example of the powerful effect (in terms of effect on users) of even a small amount of lifelike behavior.

8. Zoesis Zoesis was recently founded by Joseph Bates (head of the Oz project) and Oz project alumni. Its goal is to build interactive story experiences utilizing believable agents.

   Google Scholar 

Groups Collecting and Disseminating Research

1. Contact Consortium: http://www.ccon.org/ A group that promotes avatar spaces.

2. Loebner Prize: http://acm.org/~loebner/loebner-prize.htmlx The Loebner Prize contest, held each year, awards $2000.00 to the author of the program which does the best job passing a limited form of the Turing test.

3. Virtual Pet Home Page: http://www.virtualpet.com/vp/vpindex2.htm A page discussing research and commercial products related to virtual pets.

Academic Research Projects

1. Affective Reasoning Project (Depaul University): http://condor.depaul.edu/~elliott/ar.html Led by Clark Elliott. The goal of this project is to build agents that can reason about emotion. Currently they have systems that can detect emotion in human voice, express emotion through facial expressions and speech inflection, and “have” emotions (in the sense that emotions detected in the user trigger emotions in the agent).

2. Center for Human Modeling and Simulation (University of Pennsylvania) http://www.cis.upenn.edu/~hms/index.html Home of Jack, a graphical human simulation package. The research at the Center is focused around building behavior and physics-based simulations of human figures.

3. The Cog Shop (MIT AI Lab): http://www.ai.mit.edu/projects/cog/ Led by Rodney Brooks, the father of subsumption architecture. Rodney has been arguing for over a decade that the road to intelligence consists of building situated, embodied, broad agents (in his case, robots) which employ no semantic representations. Cog is a humanoid robot. As Cog interacts with the world using a body similar to a human body, it is hoped that Cog will learn to think the way humans do.

4. The Cognition and Affect Project (University of Birmingham) http://www.cs.bham.ac.uk/~axs/cog_affect/COGAFF-PROJECT.html A project led by Aaron Sloman and Glyn Humphries. The goal of this project is to explore the design space of AI architectures in order to understand the relationship between what kinds of architectures are capable of what kinds of mental phenomena. They are interested in the whole range of human mental states; in particular they wish to discover whether emotions are an accident of evolution or fundamental to the design of any resource-limited intelligent agent.

5. Entertainment Technology Center (Carnegie Mellon University) Founded by Don Marinelli and Randy Pausch. They are charged with developing an entertainment technology program at CMU. Their current focus is Synthetic Interviews, an interactive video technology with which a user can have a conversation with some character.

   Google Scholar 

6. Gesture and Narrative Language (MIT Media Lab) http://gn.www.media.mit.edu/groups/gn/ Led by Justine Cassell. Using ideas from discourse theory and social cognition, this group designs agents which have discourse competence (e.g. knowing how to integrate gestures and speech to communicate, knowing how to take turns in a conversation, etc.).

7. IMPROV Project (NYU Media Research Lab): http://www.mrl.nyu.edu/improv/index.html This project is led by Ken Perlin and Athomas Goldberg. “The IMPROV Project at NYU’s Media Research Lab is building the technologies to produce distributed 3D virtual environments in which human-directed avatars and computer-controlled agents interact with each other in real-time, through a combination of Procedural Animation and Behavioral Scripting techniques developed in-house.” An example of convergence towards believable characters from the graphics side (vs. AI).

8. Interactive Cinema Group (MIT Media Lab): http://ic.www.media.mit.edu/ A project at the Media Lab led by Glorianna Davenport. They study techniques for bringing interactivity to the traditional cinematic medium (with notable exceptions such as Tinsley Galyean’s Dogmatic, which is set in a virtual world). In general, this involves breaking down a linear medium (such as video) into a database of clips, somehow annotating those clips, and then intelligently choosing the right clips at the right time as a user interacts with the system. The video may be accompanied by other media such as email (e.g. Lee Morgenroth’s Lurker).

9. IntelliMedia (North Carolina State University) http://www.csc.ncsu.edu/eos/users/l/lester/www/imedia/ Led by James Lester. This group focuses on intelligent multimedia. Currently they are focusing on animated pedagogical agents.

10. Jouhou System Kougaku Laboratory (University of Tokyo) http://www.jsk.t.u-tokyo.ac.jp/index.html A robotics research lab, including remote-brained and humanoid robotics.

11. Julia (Carnegie Mellon University): http://www.fuzine.com/mlm/julia.html The home page for Julia, a chatterbot that lives in TinyMUDS.

12. Karl Wurst (Robotics and Puppetry, University of Connecticut) http://www-rfcc.cse.uconn.edu/www/KarlHome.html Karl Wurst, in collaboration with the University of Connecticut’s world-renowned Puppet Arts Program, is building robotic versions of the Woggles.

13. MIRALab (University of Geneva): http://miralabwww.unige.ch/ Led by Nadia Thalmann. This group works on virtual humanoids. Focus is on realistic modeling of human faces, movement, clothing, etc. Now starting to do work on autonomous systems.

14. Neo (University of Massachusetts) http://eksl-www.cs.umass.edu/research/conceptual-systems/index.html Led by Paul Cohen. This group is building a baby that interacts in a simulated world. The goal is for the baby to learn the conceptual structure of the world through physical interaction.

15. Oz Project (Carnegie Mellon University) http://www.cs.cmu.edu/afs/cs.cmu.edu/afs/cs.cmu.edu/project/oz/web/oz.html Led by Joseph Bates, founder of Zooesis. The goal of the Oz project is to build interactive story worlds containing personality rich, believable characters. A drama manager ensures that the user experiences a high-quality story.

16. Phil Agre: http://dlis.gseis.ucla.edu/people/pagre/ An alumnus of the MIT AI Lab, Phil Agre developed Pengi, a system which played the video game Pengo. Pengi is an instance of “alternative AI”: it employed reactive behaviors and deictic (context dependent) representations. He has written elegantly on why classical AI is inappropriate for building agents which engage in situated, embodied, routine activity.

17. Selmer Bringsjord: http://www.rpi.edu/~brings/ Primarily a philosopher of AI, Selmer also does research in story generation. His forthcoming book, AI, Story Generation and Literary Creativity: The State of the Art will describe BRUTUS, his latest story generation system.

18. Social Responses to Communication Technology (Stanford University) http://www.stanford.edu/group/commdept/ A project led by Clifford Nass and Byron Reeves. They are studying the way people apply social rules and schemas to their interactions with technology.

19. Software Agents Group (MIT Media Lab) http://agents.www.media.mit.edu/groups/agents/ Led by Patti Maes. The software agent group explores the use of autonomous agents in a wide variety of contexts. Much of their work tends to have an artificial life flavor (by which I mean that the work focuses on useful behavior emerging out of the interactions of many software agents). Agents as synthetic characters was explored by Bruce Blumberg in the ALIVE and Hamsterdam projects. The synthetic character work has how shifted to a new group being started by Bruce. He developed an ethologically motivated action selection mechanism to drive his synthetic characters.

20. Virtual Environments for Training (USC Information Sciences Institute) http://www.isi.edu/isd/VET/vet.html Led by W. Lewis Johnson. This group has built a pedagogic agent named Steve that trains humans in virtual worlds. Steve teaches people how to perform tasks, gives advice as it watches users perform tasks, and answers student’s questions.

21. Virtual Theater Project (Stanford): http://www-ksl.stanford.edu/projects/cait/index.html Led by Barbara Hayes-Roth, founder of Extempo. The metaphor informing their work is that of an improvisational actor. That is, they build actors who try to improvise behavior in different situations. An actor’s improvisational choices may be influenced by an explicitly specified personality (a set of values along some dimensions of personality). They are also exploring how a human might exert high level control over one of these actors.

22. Waseda Humanoid Project (Waseda University) http://www.shirai.info.waseda.ac.jp/humanoid/index.html They are building a humanoid robot including sensing, recognition, expression and motion subsystems.

Articles and Books

1. Articles written by the OZ Project (CMU) http://www.cs.cmu.edu/afs/cs.cmu.edu/project/oz/web/papers.html On-line articles available about the OZ project. Articles include overall descriptions of the goals of the project, the action architecture, the emotion architecture, and natural language generation (for the text based worlds).

2. Articles written by the Software Agents Group (MIT Media Lab) http://agents.www.media.mit.edu/groups/agents/publications/ On-line articles from the Software Agents Group. Articles relevant to believable agents are listed under “Modeling Synthetic Characters: Applications and Techniques.” Articles include descriptions of ALIVE, action-selection architectures, and the role of artificial life in entertainment.

3. Articles written by the Virtual Theater Project (Stanford) http://www-ksl.stanford.edu/projects/cait/publicity.html On-line articles available about the Virtual Theater Project. Articles include descriptions of their approach to emotion, personality, and user control of improvisational puppets.

4. Special Issue on Situated Cognition: Cognitive Science 17 (1993) The articles in this issue discuss the relationship between “alternative AI” (sometimes called behavioral AI, or situated action) and “classical AI.” Simon and Vera wrote an article in which they argue that all of the specific work that falls under the rubric of situated action can not be construed as refutations of the physical symbol system hypothesis. Situated action is just a subset of symbolic AI which focuses on perception and motor control. The rest of the issue consists of articles written by various situated action proponents responding to Simon and Vera’s article.

   Google Scholar 

5. Agre, P.: The Dynamic Structure of Everyday Life. A.I. Memo 1085. Artificial Intelligence Lab, MIT (1988) ftp://publications.ai.mit.edu/ai-publications/1000-1499/AITR-1085/AITR-1085.ps Agre’s Ph.D. thesis. Describes Pengi, a program that can play a video game called Pengo. Pengi is able to play the game without employing any traditional planning.

6. Agre, P., Chapman, D.: What are plans for? A.I. Memo 1050a. Artificial Intelligence Lab, MIT (1988) ftp://publications.ai.mit.edu/ai-publications/1000-1499/AIM-1050A.ps Argues for a view of plans as plans-for-communication (as opposed to the classic view of plans-as-programs).

7. Aristotle: The Poetics. Dover, Mineola, New York (1997) (first written 330 BC)

   MATH  Google Scholar 

8. Bledsoe, W.: I Had a Dream: AAAI Presidential Address. AI Magazine (Spring 1986) 57–61 Bledsoe describes the dream that brought him (and many AI researchers) into AI research in the first place: the dream of building computer companions.

   Google Scholar 

9. Brooks, R.: Intelligence Without Reason. A.I. Memo 1293. Artificial Intelligence Lab, MIT (1991) ftp://publications.ai.mit.edu/ai-publications/1000-1499/AIM-1293.ps.Z Argues for a situated, embodied, semantic-symbol-free approach to achieving intelligence in artificial systems.

10. Brooks, R.: Elephants Don’t Play Chess. Robotics and Autonomous Systems 6 (1990) 3–15 Argues for a situated, embodied, semantic-symbol-free approach to achieving intelligence in artificial systems.

    Article  Google Scholar 

11. Cohen, P., Atkin, M., Oates, T., Beal C: Neo: Learning Conceptual Knowledge by Sensorimotor Interaction with an Environment. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 170–177 Describes a simulated baby who learns concepts by “physically” interacting with a simulated world. This work comes out of the Neo project.

    Google Scholar 

12. Damasio, A.: Descartes’ Error: Emotion, Reason and the Human Brain. Avon Books (1994) Describes recent research findings in neuropsychology which seem to indicate that emotion plays a fundamental role in human intelligence. Much of traditional cognitive psychology and artificial intelligence has assumed that emotion is not critical to understanding intelligence

    Google Scholar 

13. Egri, L.: The Art of Dramatic Writing: Its Basis in the Creative Interpretation of Human Motives. Simon and Schuster (1946) Describes how plays work via a theory which relates character, motive and story.

    Google Scholar 

14. Elliott, C: I Picked Up Catapia and Other Stories: A Multimodal Approach to Expressivity for “Emotionally Intelligent” Agents. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 451–457 Describes an agent which communicates emotionally with people using speech recognition, text-to-speech conversion, real-time morphed schematic faces and music. This work comes out of the Affective Reasoning Project.

    Google Scholar 

15. Fujita, M., Kageyama, K.: An Open Architecture for Robot Entertainment. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997)435–442 Describes a standard defined by Sony Corporation for household entertainment robots.

    Google Scholar 

16. Galyean, T.: Narrative Guidance of Interactivity. Ph.D. thesis, MIT Media Lab, MIT (1995)

    Google Scholar 

17. Grand, S., Cliff, D., Malhotra, A.: Creatures: Artificial Life Autonomous Software Agents for Home Entertainment. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 22–29 Describes the architecture behind virtual pets which employ Alife technology (see Cyberlife).

    Google Scholar 

18. Hara, F., Kobayashi, H.: A Face Robot Able to Recognize and Produce Facial Expression. Proceedings of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems. Senri Life Science Center, Osaka, Japan (1996) 1600–1607. Describes a robot with a human-like face that can recognize and produce human facial expressions.

    Google Scholar 

19. Hayes-Roth, B., van Gent, R., Huber, D.: Acting in Character. Proceedings of the AAAI Workshop on AI and Entertainment (1996) ftp://www-ksl.stanford.edu/pub/KSL_Reports/KSL-96-13.ps Describes a system that portrays a role change between a master and a servant. The master and servant improvise within the constraints of a script.

20. Inaba, M., Nagasaka, K., Kanehiro, F., Kagami, S., Inoue, H.: Real-Time Vision-Based Control of Swing Motion by a Human-form Robot Using the Remote-Brained Approach. Proceedings of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, Senri Life Science Center, Osaka, Japan (1996) 15–22 Describes a humanoid robot that can swing on a swing using visual tracking for control.

    Google Scholar 

21. Jones, C: Chuck Amuck: The Life and Times of an Animated Cartoonist. Farrar, Straus and Giroux. (1989) The autobiography of Chuck Jones, an animator at Warner Bros. Describes the Warner Bros. approach to creating characters and story.

    Google Scholar 

22. Kelso, M., Weyhrauch, P., Bates, J.: Dramatic Presence. Presence: The Journal of Teleoperators and Virtual Environments Vol. 2 No. 1, MIT Press (1993) http://www.cs.cmu.edu/afs/cs.cmu.edu/project/oz/web/papers/CMU-CS-92-195.ps Describes a series of live experiments to test the effect of interactive freedom on the dramatic experience. Also includes a description of plot graphs.

23. Laurel B.: Computers as Theater. Addison-Wesley (1991) Draws on Aristotle’s theory of drama to define a new approach to designing dramatic human-computer interfaces.

    Google Scholar 

24. Laurel, B.: Toward the Design of a Computer-Based Interactive Fantasy System. Ph.D. thesis, Drama department, Ohio State University (1986) Describes a hypothetical drama manager that guides an interactive story experience.

    Google Scholar 

25. Lebowitz, M.: Story Telling as Planning and Learning. Poetics 14 (1985) 483–502 Describes the use of plan-like plot-fragments in UNIVERSE, a system that writes soap opera-like stories.

    Article  Google Scholar 

26. Lebowitz, M.: Creating Characters in a Story-Telling Universe. Poetics 13 (1984) 171–194 Describes the representations of characters in UNIVERSE, a system that writes soap opera-like stories.

    Article  Google Scholar 

27. Lester, J., Stone, B.: Increasing Believability in Animated Pedagogical Agents. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 16–21 http://www.csc.ncsu.edU/eos/users/l/lester/Public/dap-aa-97.ps Describes a competition-based behavior sequencing engine which produces life-like behavior while maintaining pedagogical appropriateness (e.g. don’t distract a learner with some fancy behavior when they are problem solving). This work is part of the IntelliMedia project.

28. Loyall, A. B.: Believable Agents. Ph.D. thesis, Tech report CMU-CS-97-123, Carnegie Mellon University (1997) Describes requirements for believability derived from the character arts. These requirements motivate the description of Hap, an agent language designed to facilitate writing believable agents. The thesis then describes several examples of agents written in Hap. Finally, a method for doing believable, embodied natural language generation in Hap is described. This work is part of the Oz Project.

    Google Scholar 

29. Loyall, A. B., Bates, J.: Personality-Rich Believable Agents That Use Language. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 106–113 Describes the integration of embodied natural language generation into a behavioral agent architecture. This work is part of the Oz Project.

    Google Scholar 

30. McCloud, S.: Understanding Comics: The Invisible Art. HarperCollins (1993) Written in comic book form, this book describes the semiotics of comics.

    Google Scholar 

31. Meehan, J.: The Metanovel. Ph.D. Dissertation, Yale University (1976) Describes a system that generates Aesop fable-like stories. It generates stories by using planning to achieve the goals of characters.

    Google Scholar 

32. Neal Reilly, W. S.: A Methodology for Building Believable Social Agents. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 114–121 Describes a methodology for building social behaviors on a character-by-character basis. The philosophy behind this approach is that generic taxonomies of social behavior and personality are inappropriate for building believable characters. This work is part of the OZ Project.

    Google Scholar 

33. Neal Reilly, W. S.: Believable Social and Emotional Agents. Ph.D. thesis. Tech report CMU-CS-96-138, Carnegie Mellon University (1996) http://www.cs.cmu.edu/afs/cs.cmu.edu/project/oz/web/papers/CMU-CS-96-138-lsided.ps Describes a system that maintains emotional state and a methodology for incorporating emotion into the behaviors of believable agents. The thesis then describes a methodology for building believable social behaviors. This work is part of the Oz Project.

34. Perlin, K., Goldberg, A.: Improv: A system for Scripting Interactive Actors in Virtual Worlds. Proceedings of SIGRAPH 96. New Orleans, LA, USA (1996) 205–216 http://www.mrl.nyu.edu/improv/sig96-paper/ Describes the interactive character architecture of the Improv project. An animation engine manipulates the control points of a graphical model. A behavior engine allows the user to specify higher level scripts which control the characters motions. The scripts are written in an English-like scripting language.

35. Pinhanez, C: Interval Scripts: a Design Paradigm for Story-Based Interactive Systems. Proceedings of CHI97. Atlanta, GA, USA (1997) 287–294 http://pinhanez.www.media.mit.edu/cgi-bin/tr_pagemaker Describes a method whereby interaction can be scripted with a temporal calculus that represents the relationships between intervals. A constraint propagation mechanism is used to determine the temporal value of each interval. Intervals can be associated with sensors and effectors.

36. Rich, C, Sidner, C: COLLAGEN: When Agents Collaborate with People. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 284–291 Describes a toolkit that supports the construction of agents who follow the rules of collaborative discourse. This work comes out of MERL.

    Google Scholar 

37. Thomas, F., Johnston, O.: The Illusion of Life: Disney Animation. Hyperion (1981) Written by two Disney animators, this book describes the history of animation at Disney and what techniques the animators developed to make their characters seem believable. This book has been highly influential in the OZ Project at CMU.

    Google Scholar 

38. Thorison, K.: Communicative Humanoids: A Computational Model of Psychosocial Dialogue Skills. PhD Thesis. MIT Media Laboratory (1996) http://kris.www.media.mit.edu/people/kris/abstr.html Describes a system called Gandalf that models human dialog competence in order to communicate with a human using speech and gesture.

39. Wavish, P., Connah, D.: Virtual Actors that Can Perform Scripts and Improvise Roles. Proceedings of the First International Conference on Autonomous Agents. Marina del Rey, CA, USA (1997) 317–322 Describes a script based architecture developed at Phillips Research Labs for controlling virtual characters.

    Google Scholar 

40. Weizenbaum, J.: ELIZA — A computer program for the study of natural language communication between man and machine. Communications of the ACM 9(1) 1966 36–45 Original paper describing ELIZA, a template-based pattern-matching program that simulates the conversational patterns of a non-directive therapist.

    Article  Google Scholar 

41. Weyhrauch, P.: Guiding Interactive Drama. Ph.D. thesis, Tech report CMU-CS-97-109, Carnegie Mellon University (1997) Describes the Oz drama manager, a search-based system for guiding an interactive story experience. This work is part of the Oz project.

    Google Scholar 

Download references