CS 577 Computational Models of Human Behavior

Paper [Discussion Leader]

The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users. Eric Horvitz, Jack Breese, David Heckerman, David Hovel, Koos Rommelse. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, Madison, WI, July 1998, pages 256-265. Morgan Kaufmann: San Francisco. [Sangho]

N. Oliver, E. Horvitz, and A. Garg. Layered Representations for Recognizing Office Activity, Proceedings of the Fourth IEEE International Conference on Multimodal Interaction (ICMI 2002) [Sangho]

Inferring ADLs from Interactions with Objects Matthai Philipose, Kenneth P. Fishkin, Mike Perkowitz, Donald J. Patterson1, Dirk Hähnel, Dieter Fox, and Henry Kautz.  IEEE Pervasive Computing, volume 3, number 4, pages 50-56, 2004. [Hemayet]

 

Fine-Grained Activity Recognition by Aggregating Abstract Object Usage. Donald Patterson, Dieter Fox, Henry Kautz, Matthai Philipose. Proceedings of the IEEE International Symposium on Wearable Computers, Osaka, Japan, Oct. 2005. [Hemayet]

Learning and Inferring Transportation Routines Lin Liao, Dieter Fox, and Henry Kautz.  Best Paper Award, Nineteenth National Conference on Artificial Intelligence, San Jose, CA, 2004. [Arrvindh]

Suplementary Reading:

 

Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services. Donald J. Patterson, Lin Liao, Krzysztof Gajos, Michael Collier, Nik Livic, Katherine Olson, Shiaokai Wang, Dieter Fox, and Henry Kautz. Sixth International Conference on Ubiquitous Computing, Nottingham, England, 2004. [Arrvindh]

Compare/Contrast Readings for Learning and Inferring Transportation Routines:

Predestination: Inferring Destinations from Partial Trajectories (2006). John Krumm Eric Horvitz. Ubiquitous Computing/Handheld and Ubiquitous Computing. [Craig]

Learning and Detecting Activities from Movement Trajectories Using the Hierarchical Hidden Markov Models (2005). Nam T. Nguyen, Dinh Q. Phung, Svetha Venkatesh, Hung Hai Bui. Computer Vision and Pattern Recognition. [Sangho]

Inferring Complex Agent Motions from Partial Trajectory Observations (2007).
Finnegan Southey, Wesley Loh, Dana F. Wilkinson. International Joint Conference on Artificial Intelligence. [Joseph]

Location-Based Activity Recognition using Relational Markov Networks  Lin Liao, Dieter Fox and Henry Kautz.  Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, 2005. [Tongxin Bai]

Compare/Contrast Reading:

Training Conditional Random Fields Using Virtual Evidence Boosting (2007) Lin Liao, Tanzeem Choudhury, Dieter Fox, Henry A. Kautz. International Joint Conference on Artificial Intelligence. [Hemayet]

Sangho Park will present a lecture on computer vision. For background, everyone should read:

Shinko Y. Cheng, Sangho Park, Mohan M. Trivedi, "Multi-spectral and
multi-perspective video arrays for driver body tracking and activity
analysis," Computer Vision and Image Understanding: Special Issue on Advances in Vision Algorithms and Systems Beyond the Visible Spectrum, Vol. 106, Issues 2-3, May-June 2007.

J. Wu, A. Osuntogun, T. Choudhury, M. Philipose, J. Rehg. A Scalable Approach to Activity Recognition Based on Object Use. In Proceedings of ICCV 2007. Rio de Janeiro, Brazil, October 2007. [Arrvindh]

Compare/Contrast Readings:

What, where and who? Classifying events by scene and object recognition (PDF) Li-Jia Li, Li Fei-Fei. In Proceedings of ICCV 2007. Rio de Janeiro, Brazil, October 2007. [Tongxin Bai]

Objects in Context (PDF) Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, Serge Belongie. . In Proceedings of ICCV 2007. Rio de Janeiro, Brazil, October 2007. [Hemayet]

Designing for seniors: The use of an intelligent prompting system for people with dementia. Alex Mihailidis, Jennifer Boger, Marcelle Canido, Jesse Hoey. July 2007 Interactions, Volume 14 Issue 4
Publisher: ACM

A Decision-Theoretic Approach to Task Assistance for Persons with Dementia(2005). Jennifer Boger Pascal Poupart Jesse Hoey Craig Boutilier Geoff Fernie Alex Mihailidis Conf: International Joint Conference on Artificial Intelligence.

Topic: Plan recognition in natural language understanding.
Discussion leader: Henry Kautz. Slides. Video clip.

Philip R. Cohen & C. Raymond Perrault, Elements of a Plan-Based Theory of Speech Acts, Cognitive Science, Vol. 3, No. 3, Pages 177-212, 1979.

C. Raymond Perrault & James F. Allen, A plan-based analysis of indirect speech acts, Computational Linguistics, Volume 6 , Issue 3-4, Pages 167 - 182, 1980.

1. Build a system that recognizes human behavior from sensor data using a hidden Markov model (HMM) and/or other probabilistic model. Data could come from one or more of the following sources:

(1) The data used in the paper Fine-Grained Activity Recognition by Aggregating Abstract Object UsageYou can download a zip file of all data here: a5data.zip.

Important data: Included are 10 days of data (e.g., day01.data). Also included are concatenations of all but one of those days (e.g., allbut01.data). The latter files allow you to train from multiple days. By training on all but the first day, then using the first day as test data, you have independent training and test sets. You can repeat using all but the second day and testing on the second day, etc.

Other files: If you're curious about the meanings of the numbers, state_mapping.txt and tag_mapping.txt give the actual state and object names. Also included are extended data files that give a time stamp for each reading, along with all secondary activities being performed concurrently at each time step (e.g., full.002.01.data.) Sensor readings for single activities are provided as well (e.g., single.door.01.data).

Training and test data are in the following format:

<objectID> <activityID>
<objectID> <activityID>
<objectID> <activityID>

etc. Different lines correspond to distinct sensor readings, ordered by time.

(2) The MIT PlaceLab public data sets:

http://architecture.mit.edu/house_n/data/PlaceLab/PlaceLab.htm

(3) Data you help collect in our own Laboratory for Assisted Cognition Environments.

(4) Data you collect in your own office by instrumenting your own desktop or laptop computer.

For an alternative to an HMM, see the following paper that uses a conditional random field model to explicitly recognize concurrent activities:

Hsu-yu Wu, Chia-chun Lian, and Jane Yung-jen Hsu. Joint Recognition of Multiple Concurrent Activities using Factorial Conditional Random Fields. In Christopher Geib and David Pynadath, editors, 2007 AAAI Workshop on Plan, Activity, and Intent Recognition, Technical Report WS-07-09. The AAAI Press, Menlo Park, California, July 2007.

To make this a project with enough substance to qualify for graduate credit, you should pursue one of the following paths:

1. Implement a basic HMM model from scratch (both for training and for inference), using a description of the algorithm from a textbook such as Russell & Norvig.
2. Or: use an off-the-shelf implementation that you grab from web, but experiment with different ways of modeling the domain. For example, you can try modeling:
   • Concurrent activities
   • Multiple steps within an activity
   • Multiple users etc.

2. Design and implement a route-finding algorithm that automatically learns user preferences, both about specific places (e.g., never drive down a particular street) and general rules (e.g., avoid highways unless using one saves a large amount of time).

Create a simulated world (with a simulated driver) to test your algorithm.

For a very ambitious project, go on to real data, by obtaining a data logger to carry in your car, and a street map of Rochester in computer-readable form.

One way to think about this problem: Both the system and the user find routes using A*. The cost of a route is a linear function of factors such as its length, the number of turns, the kind of road, etc. The user's cost function is unknown. However, by watching the user, the system adjusts the parameters of its cost function so that it more accurately matches that of the user.