Talking Robots seminar, Winter 2011/2012


This course is about talking robots. It’s about robots that interact with people, to do something together. And that makes dialogue to be about a lot more than just words.

The course has three goals: Teaching you about the issues that arise in creating talking robots, how to build talking robots, and then — doing it. We reflect this in the organization of the course.

First, we start off with several lectures in which we explain what it means for dialogue to be set in collaborative activity (the “intentional perspective”), and what it means for dialogue to be about the world, and acting therein (“situatedness”). This is based in research performed in our group. We illustrate the issues that arise in creating talking robots on several domains, ranging from child-robot interaction in medical settings, to human-robot teaming in urban search & rescue missions. By the end of these lectures, you will have a broad insight in the field of human-robot interaction, and the challenges we face in making a robot link language to acting in a dynamic world.

Second, after we have discussed the key issues in creating talking robots, we introduce you to the Talking Robots Toolkit. The toolkit provides a collection of APIs for creating a basic platform for processing spoken collaborative dialogue, and linking that to robot hardware. Functionality for making a robot move, map the environment, and recognize objects is all provided. Our focus here is on dialogue processing, and on seeing how some of the issues discussed before can be addressed in a real system. We use a comprehensive set of tutorials to introduce the different aspects of the toolkit, ranging from speech recognition to dialogue management to situated belief modeling to content planning. And speech synthesis. By the end of these lectures, you will have experience in building small modules with the toolkit, coupled to an insight in how to deal with key issues like referring to aspects of the world, and the uncertainty involved in that.

Third, you form teams of 3-4 students (ideally, already before or at the beginning of the course), and you get to work with real robots! This block takes up about half of the semester. Each team gets to define their own “mission” (indoor or outdoor), present that, and then implement it. Each team has access to a robot at our lab. During this time, each team is supported by one or two members of the Talking Robots group. We supervise each team with the explicit goal to teach you skills for working in teams, ranging from purely technical ones (svn, trac, etc.) to social ones (coordination, communication, etc.). We continue to have weekly meetings, to discuss development issues with the entire group.

At the end of the semester, we organize a Grand Presentation at which each team gets to present and demonstrate their robot to a larger audience.


The course is organized in three blocks: A block of 4-5 lectures “theory,” then a block of 2 lectures with exercises in using the toolkit, followed by weekly meetings during team-based development.

Lectures take place on Tuesday, 0830-1000h, in C7 2, U15.

  • Tuesday October 25 2011: First meeting
  • Tuesday November 1 – Tuesday 15: Reading assignments (no sessions)
  • Tuesday November 22 2011: Introduction; Robots that talk
  • Tuesday November 29 2011: Talking about objects you see
  • Tuesday December 6 2011: Talking about scenes you see
  • Tuesday December 13 2011: Talking about the world you are in
  • Tuesday December 20 2011: Talking about things to do in the world
  • Tuesday January 3 2012: Introduction to TaRoT – the Talking Robots Toolkit
  • Tuesday January 10 2012: Introduction to TaRoT – the Talking Robots Toolkit
  • Tuesday January 17 2012: Introduction to TaRoT – the Talking Robots Toolkit
  • Tuesday January 24 2012: Presentations of team project proposals
  • January – March 2012: Lab sessions
  • mid-April 2012: Team project presentations at DFKI
After the teams have presented their proposals, we will adjust the schedule to facilitate working in the Talking Robots lab.

Each team is supervised by a member of the Talking Robots group, among others GJ Kruijff, Hendrik Zender, Heriberto Cuayahuitl, and Sergio Roa.

Course materials

  • Slides for the lectures
  • Chapters from G.J.M. Kruijff, “Talking Robots” (in preparation).
  • Tutorials for the Talking Robots Toolkit
  • Several types of indoor- and outdoor robots
  • Infrastructure for collaborative software development
Reading assignments

The following articles and book chapters form the reading assignments. The articles by Fong et al and Goodrich & Schultz give a good overview of the field of human-robot interaction, and current issues. The article by Murphy shows how these current issues find themselves instantiated in the field of rescue robotics. The book chapter by Kruijff et al outlines an early approach to situated dialogue processing for talking robots.

  • Terry Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. “A survey of socially interactive robots.”
    Robotics and Autonomous Systems, 2003 [ PDF ]
  • Michael Goodrich and Alan Schultz. “Human-Robot Interaction: A Survey.” Foundations and Trends in Human-Computer Interaction, 1(3), 2007, pp 203-275. [ PDF ]
  • Robin Murphy. “Human-Robot Interaction in Rescue Robotics.” IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews. May 2004, pages 138-153. [ PDF ]
  • Geert-Jan M. Kruijff, P. Lison, T. Benjamin, H. Jacobsson, H. Zender, and I. Kruijff-Korbayová. “Situated Dialogue Processing for Human-Robot Interaction.” In: H.I. Christensen, G.J.M. Kruijff and J.L. Wyatt (eds), Cognitive Systems. Cognitive Systems Monographs (COSMOS), Springer Verlag. 2010. pages 311-364. [ PDF ]
  • Introduction [ PDF ]
  • Talking & seeing [ PDF ]
  • Talking about where you are [ PDF ]
  • Talking & acting [ PDF ]
  • Introducing Tarot, Part 1: Logical forms, the MOLOKO grammar, and content planning [ PDF ]
Additional slides:
  • How to make talking robots (Tutorial at RO-MAN 2007) [ PDF ]

During the course, each student team member will give a presentation on the team project s/he is involved in. At the end of the course, each student team member will write an (individually identifiable) contribution to the team project report. The overall grade, individual for each student, is determined as 25% presentation, 55% report, and 20% involvement in team efforts.


The course is intended as a Software Project for advanced BSc students (12 LP), or Vorlesung mit Übung for MSc (8 CP).

The toolkit we use is in Java. A basic knowledge of Java and some programming experience is needed.