DiGest - Dictionary of Gestures

Project

Action 2102: Cross-Modal Analysis of Verbal and Non-verbal Communication

The main objective of the Action is to develop an advanced acoustical, perceptual and psychological analysis of verbal and non-verbal communication signals originating in spontaneous face-to-face interaction, in order to identify algorithms and automatic procedures capable of identifying human emotional states. Several key aspects will be considered, such as the integration of the developed algorithms and procedures for application in telecommunication, and for the recognition of emotional states, gestures, speech and facial expressions, in anticipation of the implementation of intelligent avatars and interactive dialogue systems that could be exploited to improve user access to future telecommunication services. This Action profits from two former COST Actions (COST 277 and COST 278) that identified new appropriate mathematical models and algorithms to drive the implementation of the next generation of telecommunication services such as remote health monitoring systems, interactive dialogue systems, and intelligent avatars.

The funding of our participation on the COST 2102 project is provided by a grant of the Slovak Academy of Sciences (so called MVTS grant) "Nonverbal speech gestures in Slovak in multimodal and multicultural context"

One of the outputs of this project is:

DiGest, web-based multicultural and multimodal Dictionary of Gestures

The first ideas leading to this project were presented at Patras COST 2102 conference [ 1 ].

First ideas towards a definition of GestBase - an open database of gestures

In our attempt towards building an open database of gestures GestBase we found out that we have too little knowledge on conversational grunts, face gestures, body postures and their classification we decided to split the process into two steps. In the first step we want to get more information and skills on grunts, gestures, postures, its classification schemes, annotation conventions, coding of their sound, photo and video representations, coding of their features etc. Therefore we decided to take the advantage of the fact, that Eva Ruzickova has created and published her "Picture Dictionary of Gestures, American, Slovak, Japanese, and Chinese" [2]. At the initial phase of the project we have (with a kind permission of the author) adopted the basic structure of the dictionary, the classification scheme and even the descriptions of gestures. We have prepared a web-page that will serve as a DiGest user-interface. We have broadened the choice of languages, added video and sound (namely grunts that are of great interest in speech communication!) and started to load the data to the database. We hope that this first draft version of DiGest will serve as a good starting point for collaboration with researchers from other countries and will open a wide discussion on different aspects of gestures in different modalities, different cultures, languages, their representation, classification, coding, similarity measures, etc.

[1] RUSKO, Milan - JUHAR, Jozef. Towards annotation of nonverbal vocal gestures in Slovak. In Verbal and nonverbal features of human-human and human-machine interaction. Editor Anna Esposito, Nikolaos G. Bourbakis, Nikolaos Avouris, Ionnis Hatzilygeroudis. - Berlin : Springer, 2008. ISBN 978-3-540-70871-1. ISSN 0302-9743, p. 255-265.
[2] RUZICKOVA, Eva. Picture Dictionary of Gestures, American, Slovak, Japanese and Chinese.Comenius University Publishing House, Bratislava 2001. ISBN 80-223-1675-X.