Research

Main Page >> Research

Research Direction

Objective

Using multimedia signals captured from multiple sensors, we realize human-machine interaction systems that

How to share a sense of time?

Understanding the meaning of user commands and presenting appropriate information to a user is one of the primary objectives of human-machine interaction systems. Most of the existing approaches, therefore, set the goal to realize interaction systems that understand semantic information specified by a user and generate attractive presentation to a user using multimedia data such as text, graphs, pictures, video, sound, and so on.

While such multimedia interaction systems are important, users sometimes feel frustration when the systems get out of human interaction protocols. That is, the systems often ignore dynamic features such as acceleration patterns, pause lengths, tempo speed, and rhythms, which convey rich nonverbal and non-semantic information in human communication. So, how to realize the systems that understand and control these features, and share a sense of time with humans?

Interval-Based Hybrid Dynamical System

In this research, we attempt to model such dynamic features or temporal structures in verbal and nonverbal communication based on a novel computational model, named Interval-Based Hybrid Dynamical System. A hybrid dynamical system is the integration of two types of dynamical systems: one described by differential equations, which is suitable for describing physical phenomena (consider time as physical metric entity), and a discrete-event system, which is suitable for describing human subjective or intellectual activities (consider time as ordinal state transition). In other words, these systems are designed based on "objective time (physical time)" and "subjective time (cognitive time)", respectively. We believe that the integration of these two concepts of time is crucial to design the systems that share a sense of time with humans.

Applications of Interval-Based Hybrid Dynamical System

Dialog Analysis

Pamphlet for Open Laboratory (PDF)