Research Direction
Objective
Using multimedia signals captured from multiple sensors, we realize human-machine interaction systems that
- Understand the meaning of human behaviors such as gesture and utterance;
- Generate and recognize natural information presentation (response) timing for humans.
How to share a sense of time?
Understanding the meaning of user commands and presenting appropriate information to a user is one of the primary objectives of human-machine interaction systems. Most of the existing approaches, therefore, set the goal to realize interaction systems that understand semantic information specified by a user and generate attractive presentation to a user using multimedia data such as text, graphs, pictures, video, sound, and so on.
While such multimedia interaction systems are important, users sometimes feel frustration when the systems get out of human interaction protocols. That is, the systems often ignore dynamic features such as acceleration patterns, pause lengths, tempo speed, and rhythms, which convey rich nonverbal and non-semantic information in human communication. So, how to realize the systems that understand and control these features, and share a sense of time with humans?
Interval-Based Hybrid Dynamical System
In this research, we attempt to model such dynamic features or temporal structures in verbal and nonverbal communication based on a novel computational model, named Interval-Based Hybrid Dynamical System. A hybrid dynamical system is the integration of two types of dynamical systems: one described by differential equations, which is suitable for describing physical phenomena (consider time as physical metric entity), and a discrete-event system, which is suitable for describing human subjective or intellectual activities (consider time as ordinal state transition). In other words, these systems are designed based on "objective time (physical time)" and "subjective time (cognitive time)", respectively. We believe that the integration of these two concepts of time is crucial to design the systems that share a sense of time with humans.
Applications of Interval-Based Hybrid Dynamical System
- Segmentation of lip image sequences
- Facial Score: Representation of Timing Structure in Faces
- Timing Structure Model of Multimedia Signals