Interval-Based Hybrid Dynamical System for Event Segmentation and Learning
Integration of discrete-event systems and dynamical systems
Dynamic features, such as acceleration patterns, pauses, tempo speed, and rhythms, convey rich nonverbal and non-semantic information in human communication. However, these features are often ignored in multimedia interaction systems. This research attempt to model such dynamic features or temporal structures in verbal and nonverbal communication based on a novel computational model, named Interval-based Hybrid Dynamical System (IHDS, in short). A hybrid dynamical system is the integration of two types of dynamical systems: one described by differential equations, which is suitable for describing physical phenomena (consider time as physical metric entity), and a discrete-event system, which is suitable for describing human subjective or intellectual activities (consider time as ordinal state transition) (Figure 1).
Interval-Based Hybrid Dynamical System
We developed IHDS based on the following rationale. Firstly, we assume that a complex human behavior consists of dynamic primitives, which are often referred to as motion elements, movemes, visemes, and so on. For example, a cyclic lip motion can be described by a cyclic sequence of simple lip motions such as "open", "close", and "remain closed". Once the set of dynamic primitives is determined, a complex behavior can be partitioned into "temporal intervals", each of which is characterized by a dynamic primitive and its temporal duration.
Secondly, we assume that not only temporal orders of motion elements but also their duration lengths or temporal differences among beginning and ending timing of the temporal intervals convey rich information in human communication. For example, some psychological experiments suggest that duration lengths of facial actions play an important role for human judgments of basic facial expression categories.
Based on the assumptions above, we proposed IHDS for modeling dynamic events in terms of temporal intervals. The system has a two-layer architecture consisting of a finite state automaton and a set of linear dynamical systems. In this architecture, each linear dynamical system represents the dynamics of a motion primitive and corresponds one to one to a discrete state of the automaton. In other words, the automaton controls the activation order and timing of the linear dynamical systems. Thus, IHDS can model and generate multimedia signals that represent complex human behaviors.
Hybrid system identification
In spite of the flexibility of the systems, the learning process has a difficulty due to its paradoxical nature; that is, it requires us to solve temporal segmentation and system identification problems simultaneously. We therefore propose a two-step learning method. The first step of the method estimates the number of linear dynamical systems and its parameters based on the hierarchical clustering of dynamical systems, and the second step refines overall system parameters. Experiments on simulated and real image data show that the proposed method successfully solves segmentation and hybrid system identification problems from input time-varying signals.
Applications
Applying IHDS to various human verbal and nonverbal communication behaviors, we can successfully extract dynamic features of the behaviors based on relations of temporal intervals. Figure 2 shows the result of the proposed learning process applied to an input lip sequence. We see that the lip sequence is divided by four extracted dynamical systems, which represent "remain closed", "open", "remain open", and "close".
Using multiple IHDS, we can classify fine-grained facial expressions (see also "Facial Score") and analyze synchronization/delay mechanisms between mouth motion and speech utterance (see also "multimedia timing structure model").
Figure 1. Hybrid Dynamical System.
Figure 2. Lip motion segmentation based on hierarchical clustering of linear dynamical systems.
References
- [PDF] Chapter 1: Introduction, Doctoral Dissertation, 2007.
- [PDF] Chapter 2: Interval-Based Hybrid Dynamical System, Doctoral Dissertation, 2007.
- [PDF] Chapter 3: Learning Method for the Interval-Based Hybrid Dynamical System, Doctoral Dissertation, 2007.
- [PDF] Hiroaki Kawashima, Takashi Matsuyama, "Multiphase Learning for an Interval-Based Hybrid Dynamical System", IEICE Trans. Fundamentals, Vol.E88-A, No.11, pp.3022-3035, 2005 (in English).
- [PDF] Hiroaki Kawashima, Takashi Matsuyama, "Hierarchical Clustering of Dynamical Systems based on Eigenvalue Constraints", 3rd International Conference on Advances in Pattern Recognition (S. Singh et al. (Eds.): ICAPR 2005, LNCS 3686), pp. 229--238, 2005.
- [PDF] Hiroaki Kawashima, Kimitaka Tsutsumi, Takashi Matsuyama, "Hybrid Dynamical System for Dynamical Event Segmentation, Learning, and Recognition", Information Technology Letters (FIT2004) pp.175-178, 2004 (in Japanese) (FIT Paper Award).