ИСТИНА

Emotion expression encompasses various types of information, including face and eye movement, voice and body motion. Emotions collected from real conversations are difficult to classify using one channel. That is why multimodal techniques have recently become more popular in automatic emotion recognition. We collected The Russian Acted Multimodal Affective Set (RAMAS) the first multimodal corpus in Russian language. This database contains approximately 7 hours of high-quality close-up video recordings of subjects faces, speech, 3D motion-capture data and such physiological signals as electro-dermal activity and photoplethysmogram. Ten actors played out interactive dyadic scenarios. Each scenario involved one of basic emotions: Anger, Sadness, Disgust, Happiness, Fear, Surprise, and social behavior – Domination and Submission. Emotions that subjects really felt during the scenarios were collected with short questionnaires (self-reports). The records were marked by 21 annotators (at least five annotators marked each scenario). The average Krippendorff’s Alpha statistics for RAMAS dataset is 0.44. The proposed dataset is suitable for solving multimodal emotion recognition problem. We achieved 52,5% weighted accuracy with stacked bidirectional long short-term memory recurrent neural network and decision-level feature fusion. Analysis of self-reports revealed that actors experienced the same emotions they had played out in scenarios, and there were no significant differences between dominative and submissive scenarios for each experienced emotion. RAMAS is an open corpus that provides research community with synchronous multimodal recordings of faces, speech, gestures and physiology data. Such material is useful for various studies and automatic affective systems development.