SFB/Transregio 62
SFB/Transregio 62: 'A Companion Technology for Cognitive Technical Systems' Subproject C4 : 'Psychobiological Emotion Recognition'
The individual and continuous emotion recognition based on emotion patterns constitutes the aim of psychobiological emotion recognition. The project attempts to reliably classify and model the emotions of cognitive technical system users. This emotion information is fused with other emotion sources and made available to other cognitive technical system modules to adapt their functions to the user's emotional state. Because only very few psychobiological patterns universally correlate with emotions, a new calibration process using standardized emotional stimuli will be used to determine the individual user's psychobiological emotion patterns.
Emotion Patterns are individual parameters for psychobiological emotional behaviors. A computer model for emotional simulation is to determine if it is possible to manage the interaction between humans and the computer (human computer interaction - HCI) with emotion patterns. The current studies on 'emotion computing' focus on the analysis and utilization of this information. The objective of this project is to determine individual psychobiological parameters that modify them in accordance with stable personal variables, model them in accordance with their procedural characteristics, and to thus make them available to develop the capacity of cognitive technical systems to display emotional empathy. The project is also studying and testing technical implementation possibilities.
Due to the partially contradictory findings on the use of individual psychobiological parameters and their combination for emotion identification purposes, some unsolved issues must be worked on and solved. Researchers must look for algorithms for the analysis of emotional reactions, which are able to reliably identify emotional states, even with weak emotional activation and significant individual variance. One important approach is the development of personality-moderated patters from variance sources and a not yet tested individual calibration.
This question is of medical-psychological significance for assisting persons with unpracticed or restrictive cognitive abilities (e.g. older persons) because a confrontation with complex situations often leads to avoidance behaviors, withdrawal, or helplessness. The continuous gathering of psychobiological emotional parameters is especially important for this group, because it is going to show expressive mimic, intentional gastric and prosodic behavior only during special moments of interaction with a cognitive technical system. These persons are not very familiar with using complex everyday technology. Therefore, the short-term communicated and learned behavior must be proactively practiced and strengthened. In this regard, a cognitive technical system is best, which maintains or improves the independence and participation of such persons in an increasingly technical everyday life. Companion systems promote self-management when establishing decision-making competencies and self-dependent handling of technical innovations.
SFB/Transregio 62: Project Link
FEEL - Facially Expressed Emotion Labeling
The FEEL test objectively measures the ability to recognize the basic emotions anger, disgust, fear, joy, surprise, and sadness in facial expressions.
The images that were used were taken from the JACFEE image set by Matsumoto und Ekman (1998). Each of the full-face color photographs of people shows one defined basic emotion in the facial expression. Each emotion is displayed by 4 people of Caucasian and Asiatic origin, 2 women and 2 men. The displayed emotions were coded with the FACS (Facial Action Coding System; Ekman & Friesen, 1976). The actors were given precise instructions on which emotion-specific facial muscles they were to use.
The FEEL test begins with the presentation of a pre-series of 6 images, in which each emotion is shown one time. This helps test subjects familiarize and orient themselves. During the pre-test, the test subjects are told if their answer was correct or wrong after having been shown the emotion and provided with time to respond. This pre-test is followed by the actual test with 42 images. Each emotion is seen on 7 images. Each correct answer during the actual test is worth one point.
The stimulus presentation and response entry for all 42 images follows the same procedure. Prior to the emotional facial expression, a person’s face with a neutral facial expression is displayed on the screen for 1.5 seconds. After a 1 second interval, that person displays the respective emotion. The stimulus presentation lasts 300 ms. The image sequence for each test run is determined by a random generator to prevent a sequence effect.
In total, test subjects can achieve 42 points in the FEEL test (all emotions were correctly recognized). Test subjects can achieve up to 7 points per emotion. At the end of the test, the most important results can be obtained immediately. Aside from the overall score and the score per emotion, researchers can also see the test subject's response times, typical mix-up tendencies, and compare the achieved results with a control group. It is thus relatively easy to generate a profile for each test subject, which provides information on his or her emotion recognition capability.
For research purposes, the data of several test subjects can be exported in a format, which current statistics programs can read without any problems.
Fig.: Screenshot of the FEEL test: On the right side, the test subject can see his/her score (red) compared to the results of the control group (gray).
Reliability and Application
The test is objective, since it is run and analyzed exclusively on the computer. FEEL was used on more than 400 healthy test subjects and its maximum reliability index (Cronbach alpha of r = .077) fulfills the requirements necessary to measure group and individual differences. In addition, the FEEL test has been used so far with the following patient groups: panic disorder, stroke, alcoholism, eating disorder, depression, somatoform disorders, focal dystonia, etc.
Details on the dest design and accuracy criteria can be found here:
Facially Expressed Emotion Labeling (FEEL) (113 KByte)
FEMT - Facial Expression Morphing Tool
We developed the FEMT (Facial Expression Morphing Tool), which can be used to generate a film sequence from a neutral and an emotional facial expression, which shows the development of emotion on the face. This makes it possible to to generate high-quality sequences and therefore dynamic stimuli.
Morphing is a computer-based process, in which one object gently morphs into another. This technology was developed by the special effects industry in the 1980s for entertainment. In the simplest case scenario, the process starts with a start and a target image. The software calculates the possible interim images through geometric distortions and color interpolations of the start images. The individual images can be put together in a continuous film of variable speed. In the case of the FEMT, this creates the impression that the person’s face changes from a neutral expression to an expressive emotional expression.
The FEMT can use two different state-of-the-art morph techniques. We also optimized the software for emotional faces by restricting the morphing to the respective facial areas and to achieve a natural-looking result. We additionally developed a particular technology, which makes the teeth look natural, when the person laughs.
LEAS-C
The alexithymia concept is important both for psychosomatics and emotion research. The term was originally used to describe psychosomatic patients and refers to difficulties in recognizing and describing feelings, understanding that physical sensations are part of an emotion, a stunted phantasy life with a consecutive, functional way of thinking, and an externally oriented mentality. Later, alexithymia was defined as a deficit, which originates in the insufficient development of emotional capabilities and constitutes a risk factor for various mental and somatic disorders.
The LEAS (Level of Emotional Awareness Scale, English R. Lane, German Cl. Subic-Wrana) is a valid tool to diagnose alexithymia. It measures the ability to determine the emotional content of short behavioral scenes. It is especially suitable to measure performance compared to self-assessment, since the test subjects are asked to read a scene with two persons, and must then describe in their own words how they and the other person would feel (performance test). The quality of the response allows for conclusions on the level of emotional perception the test subject is capable of. This way, alexithymia is measured from a performance perspective.
One problem of this test in the clinical-psychological field is that it requires a lot of time and personnel to analyze the test and trained raters. A psychometric analysis with templates is not possible. As a result, this tool is only used by a relatively small group of scientifically interested persons to diagnose alexithymia.
The LEAS-C, which was developed by the Emotion Lab of the University of Ulm now makes it possible to conduct the LEAS on the computer and to analyze it in a fully automated manner. Thus, the advantages of the competency test can be combined with a user-friendly analysis. With the LEAS-C, the LEAS can be entered and analyzed entirely on the computer. During the test, the subject views the situation description of the LEAS on the screen and types in his/her response. The software uses the response text to calculate the LEAS score in three steps (preparation, assignment, analysis). A specifically adapted text analysis algorithm assesses the level of emotional perception on the basis of a valid glossary of emotionally relevant words. After the test, the screen displays the SEAS score of each test subject, and how many points were achieved for the self and other category, and how the computer arrived at its score (see illustration). The latter helps the user understand how the score was calculated and allows for a correction of software errors. The user can easily change the assessment of individual words, their assignment to the self or other and the awarding of more points.
Fig.: Screenshot of the LEAS-C. The test subjects are evaluated in this part.
To date, the LEAS-C has been used on N=187 healthy subjects. In the total LEAS score, the subjects achieved an average of 34.3 of 50 possible points (SD = 6.0). The scores calculated by the software mostly correlated with the assessment of trained human raters (Rater 1: 34.9; Rater 2: 34.8; LEAS-C: 35.4 of 50 possible points in a smaller sample). The interrater reliability of values between r = 0.86 and r = 0.90 is very high and fulfills the requirements of a reliable evaluation process.
The software itself is still in its last development stage. The LEAS-C is expected to be finalized by the end of 2008 and can then be used in research and the clinical field. Any new information on the development stage is going to be posted on this page.
SIMPLEX
One of the most important innovations in the study of man/machine interaction (MMI) are developments in the area of “artifical emotions”. These can improve problem solutions of cognitive systems in complex scenarios. In practical application, artificial emotions are predominantly used to design the MMI in a more human-like and realistic manner. Human users are more likely to accept technical systems, if they show emotions.
The computer emotion model SIMPLEX (Simulation of Personal Emotion Experience), which was developed in the Emotion Lab in Ulm, is such a method to generate artificial emotions. In the SIMPLEX, events are appraised, which then, depending on the person's personality and mood, leads to an emotion or changes the mood accordingly. The design uses the so-called OCC model by Ortony, Clore und Collins, which generates emotions by appraising events, which are based on the desirability of the results of the event and the laudability of the actions of others.
This appraisal is modified by personality variables, captured by the FFM (Five Factor Model). The five variables are: Extroversion, diligence, compatibility, neuroticism, and directness. During the appraisal of the events and the development of the mood, the emotion model is influenced by the personality on several levels. The mood itself is displayed in the three-dimensional PAD space (pleasure, arousal, dominance). Since also slight emotion (as a result of the assessment according to OCC) can be displayed in this space, the mood is attracted from the emotional perspective and thus constitutes a medium-term storage of the emotions. The proximity of the current emotion to the mood also decides, whether the emotion is truly active or still below the threshold. Personality (long-term), mood (medium-term), and emotions (short-term) therefore represent three semantic and temporal levels of the emotion model which interact with each other in a realistic manner. Fig. 1 shows a simplified diagram of the SIMPLEX model.
Fig.: Simplified diagram of the SIMPLEX emotion model
In an experimental evaluation of the model, we had virtual characters, which function in accordance with the SIMPLEX system, interact with each other in thousands of scenarios, and assessed the resulting emotions. Emotions such as pity, joy, fear, etc. known from human interactions, were replicated in a realistic manner. In addition, the personalities, which were defined according to the principles of the FFM, influenced the emotions that a virtual character displayed during an interaction.
Detailed information on the development, design, and first experimental data of the model can be found here:
SIMPLEX - Simulation of Personal Emotion
Experience (487 KByte)
Emotional Avatar
The Emotion Lab developed an Avatar, which can display many different facial expressions in real time, to visualize expressive facial emotions. The Avatar can be accessed from almost any application via TCP/IP.
Fig.: Avatar including access interface
Psychometric Operationalization
Scales on emotion regulation, the ERQ and the AEQ-G18 and AEQ-10 were translated, revised, and the theory of the tests studied to determine individual differences in emotion regulation. A German scale for the assessment of daily stress was developed on the basis of the Daily Stress Inventory (Brantley, Waggoner, Jones und Rappaport, 1987), and used for studies on the connection between daily stress and the symptoms of psychosomatic and chronic illnesses.
Emotion Regulation Questionnaire
The English Emotion Regulation Questionnaire (ERQ) by Gross & John is one of the first validated tools to scientifically research emotion regulation processes. It makes it possible to determine preferences for two frequently used emotion regulation strategies, i.e. suppression and reappraisal. The German version was tested in three translation steps on a group of students (n=113/167/174) each. The main objective was to closely copy the English original and to optimize the factor lading on the two components. A factor analysis with iterative communality analysis and Varimax rotation was used. In accordance with our objective, the alpha values (inner consistency) as a reliability parameter for the suppression and reappraisal reached the average values of the original American questionnaire. We developed a tool which is easy to answer within a short period of time (5-10 minutes) and which includes both emotion regulation styles reappraisal and suppression. Some of the possible areas of application are the research of depression and anxiety disorders and the assessment of psychotherapeutic processes.
Ambivalence over Emotions Questionnaire
A 10 item and 18-item German version of the Ambivalence over Emotions Questionnaire (AEQ) (King & Emmons, 1990) was developed, the AEQ-10 and AEQ-G18. The connections between emotional ambivalence, illness behavior, depression, and social support were examined in three separate studies. In the first series of studies, a factor analysis of the translated AEQ-items revealed a factor structure involving two factors, one concerning ambivalence over the ability to show mainly positive emotions (competence ambivalence), and one concerning the ambivalence over the consequences of mainly negative emotions (effect ambivalence). In all studies (whose samples were: the general population, students, and general practitioner patients), reported physical symptoms, depression symptoms and the lack of social support were related to emotional ambivalence. However, this relation was less strong after controlling for neuroticism (Deighton & Traue, 2006). The second series of studies aimed at the validation of the AEQ -G18. The questionnaire AEQ-G18, a two item screener for depression, the Profile of Mood States, the revised Beck depression inventory and the questionnaire Fragebogen zum Gesundheitszustand, SF-36 were administered on a representative German population. The two-factor-structure (effect ambivalence and competence ambivalence) could be proofed partly. Woman scored higher at the scale effect ambivalence. Persons with higher levels of education showed lower levels of ambivalence over emotional expression. Ambivalence over emotional expression was positive correlated with depression and negative mood (depression/anxiety, fatigue, anger) and negative correlated with quality of life and positive mood (vigor). A short version, AEQ-G10, and detailed reference values for the AEQ-G18 and the AEQ-G10 are published. Norm data of the two versions of the clinically relevant, valid and economical German version of the ambivalence over emotional expressiveness questionnaire (AEQ-G18 and the short form AEQ-G10) are presented (Albani, 2007).
Daily Stress Questionnaire
A German scale to determine daily stress was developed from the Daily Stress Inventory (Brantley, Waggoner, Jones und Rappaport, 1987), and used for studies of the connection between daily stress and psychosomatic and chronic illnesses. The scale was tested for its correlation with the statistical values of the original scale and German usability in two studies with a sample of 115 healthy test subjects and 451 patients. Data from the 16-PF questionnaire by Cattell and the stress management questionnaire by Janke were used to determine convergent validity. The connection between the symptoms for Morbus Crohn, Colitis Ulcerosa, and headaches and daily stress were examined in time series analyses.