Emotion recognition from physiological signals

Emotion recognition is one of the great challenges in human–human and human–computer interaction. Accurate emotion recognition would allow computers to recognize human emotions and therefore react accordingly. In this paper, an approach for emotion recognition based on physiological signals is proposed. Six basic emotions: joy, sadness, fear, disgust, neutrality and amusement are analysed using physiological signals. These emotions are induced through the presentation of International Affecting Picture System (IAPS) pictures to the subjects. The physiological signals of interest in this analysis are: electromyogram signal (EMG), respiratory volume (RV), skin temperature (SKT), skin conductance (SKC), blood volume pulse (BVP) and heart rate (HR). These are selected to extract characteristic parameters, which will be used for classifying the emotions. The SVM (support vector machine) technique is used for classifying these parameters. The experimental results show that the proposed methodology provides in general a recognition rate of 85% for different emotional states.     

Subject-independent emotion recognition based on physiological signals: a three-stage decision method

Background

Collaboration between humans and computers has become pervasive and ubiquitous, however current computer systems are limited in that they fail to address the emotional component. An accurate understanding of human emotions is necessary for these computers to trigger proper feedback. Among multiple emotional channels, physiological signals are synchronous with emotional responses; therefore, analyzing physiological changes is a recognized way to estimate human emotions. In this paper, a three-stage decision method is proposed to recognize four emotions based on physiological signals in the multi-subject context. Emotion detection is achieved by using a stage-divided strategy in which each stage deals with a fine-grained goal.

Methods

The decision method consists of three stages. During the training process, the initial stage transforms mixed training subjects to separate groups, thus eliminating the effect of individual differences. The second stage categorizes four emotions into two emotion pools in order to reduce recognition complexity. The third stage trains a classifier based on emotions in each emotion pool. During the testing process, a test case or test trial will be initially classified to a group followed by classification into an emotion pool in the second stage. An emotion will be assigned to the test trial in the final stage. In this paper we consider two different ways of allocating four emotions into two emotion pools. A comparative analysis is also carried out between the proposal and other methods.

Results

An average recognition accuracy of 77.57% was achieved on the recognition of four emotions with the best accuracy of 86.67% to recognize the positive and excited emotion. Using differing ways of allocating four emotions into two emotion pools, we found there is a difference in the effectiveness of a classifier on learning each emotion. When compared to other methods, the proposed method demonstrates a significant improvement in recognizing four emotions in the multi-subject context.

Conclusions

The proposed three-stage decision method solves a crucial issue which is ’individual differences’ in multi-subject emotion recognition and overcomes the suboptimal performance with respect to direct classification of multiple emotions. Our study supports the observation that the proposed method represents a promising methodology for recognizing multiple emotions in the multi-subject context.

     

The effect of cognitive reappraisal on physiological reactivity and emotional memory

We investigated the effect of cognitive reappraisal on emotional arousal, facial expressivity and subsequent memory. Men and women viewed emotionally negative pictures while they attempted to either increase or decrease negative emotions elicited by the pictures, or to simply view the pictures. Neutral pictures were also presented with instructions to simply view the pictures. Concurrent changes in emotional arousal and valence were assessed with skin conductance responses (SCRs) and facial corrugator electromyographic responses (EMG), respectively. Picture memory was assessed with an immediate recall test and a delayed recognition test. Relative to simply viewing pictures, voluntary reappraisal to increase negative emotion generated greater facial corrugator EMG and SCR responses, and reappraisal to decrease negative emotion generated decreased corrugator EMG responses. Men showed enhanced recognition for pictures presented during the increase and decrease conditions, whereas women showed comparable recognition performance across all regulation conditions. The modulation of subsequent recognition memory associated with decreasing emotion was inversely associated with changes in physiological responses. Our results suggest that sex is an important factor to consider in determining how reappraisal-induced physiological changes are associated with subsequent changes in memory. These findings contribute to our understanding of how reappraising emotion exerts both immediate and enduring influences on physiological responses and subsequent memory.     

Mapping emotions into acoustic space: the role of voice production

Research on the vocal expression of emotion has long since used a “fishing expedition” approach to find acoustic markers for emotion categories and dimensions. Although partially successful, the underlying mechanisms have not yet been elucidated. To illustrate that this research can profit from considering the underlying voice production mechanism, we specifically analyzed short affect bursts (sustained/a/vowels produced by 10 professional actors for five emotions) according to physiological variations in phonation (using acoustic parameters derived from the acoustic signal and the inverse filter estimated voice source waveform). Results show significant emotion main effects for 11 of 12 parameters. Subsequent principal components analysis revealed three components that explain acoustic variations due to emotion, including “tension,” “perturbation,” and “voicing frequency.” These results suggest that future work may benefit from theory-guided development of parameters to assess differences in physiological voice production mechanisms in the vocal expression of different emotions.     

Emotion recognition in schizophrenic and depressed inpatients

The performance of schizophrenic inpatients (N = 14), depressed inpatients (N = 15), and normal hospital employees (N = 15), all females, was compared on Izard's (1971) test of emotion recognition. Subjects were required to match photographs of facial expressions of emotion with the correct label for the emotion. The average numbers of errors of four types were examined: Positive or neutral emotions mislabelled as other positive or neutral emotions; positive or neutral emotions mislabelled as negative; negative emotions mislabelled as positive or neutral; and negative emotions mislabelled as other negative emotions. The principal findings were (1) the schizophrenics were less accurate than normals, but they were not less accurate than the depressives, nor were they less accurate on negative than positive or neutral emotions; (2) the depressives did not display any form of negative bias in emotion recognition; and (3) both groups of inpatients were less accurate than normals when they labelled an emotion as positive or neutral, but not when they labelled it as negative. Implications of the results for the etiology and maintenance of schizophrenia and depression were noted.     

Effects of joy and sorrow on pulse-graph parameters in healthy female college students based on emotion-evoked experiments

ObjectiveTo explore the effects of joy and sorrow on pulse-graph parameters in healthy subjects and analyze their potential effect to provide preliminary evidence that pulse diagnosis can identify human emotional changes.Design and interventionForty healthy female college students from Beijing University of Chinese Medicine were recruited. The emotion-evoking experiment was conducted to observe the effects of joy and sorrow on pulse-graph parameters, taking neutral emotion as a control. The experiment included the following criteria. (1) Emotions evoked and evaluated: the same subjects watched three emotional videos which were, respectively, neutral, joyful and sorrowful to evoke the corresponding emotions. They completed the “subjective emotion evaluation form” (SEEF) before watching each video (T₀), immediately after watching the video (T₁) and 15 min after watching the video (T₂) to conduct subjective emotion evaluation. Simultaneously, their heart rate, galvanic skin response and heart rate variability were collected with a 16-lead physiological recorder to conduct an objective evaluation of emotional arousal. (2) Collection of pulse-graph parameters: pulse-graph parameters of the subjects at T₀, T₁ and T₂ were collected with a pulse condition analyzer.ResultsWhen watching the joy video, the rapid ejection phase extended significantly, both the left ventricular diastole and the cardiac cycle shortened significantly. When watching the sorrow video, the left ventricular diastole extended significantly, the cardiac cycle showed an upward trend, and heart rate showed a downward trend.ConclusionJoy and sorrow have a certain effect on pulse-graph parameters of the subjects. The mechanisms may be that joy contributes to rapid ejection phase extension and a shortening of the left ventricular diastole and the cardiac cycle, while sorrow extends the left ventricular diastole. According to the theory of Traditional Chinese Medicine (TCM), the visceral functions can be mediated through human qi flow. Moderate joy and sorrow can promote, respectively, the smooth qi flow of the heart and the lung. However, excessive or long-term joy and sorrow can lead to disharmonious qi flow of the two viscera. Therefore, maintaining moderate emotions is beneficial to the physiological functions of the viscera. Great emotional fluctuations may damage the visceral functions. Future studies with effects of other emotions on pulse-graph parameters are warranted to determine the reliability of the association of emotions and pulse diagnosis.     

Emotions in everyday life: an ambulatory monitoring study with female students

Additional heart rate as an indicator of emotional arousal was monitored throughout the day with a special ambulatory device. Fifty female students received acoustic feedback every 10-20 min. The feedback was based either on events (additional heart rate present) or was random without additional heart rate. Following the feedback the subjects were asked to disclose their emotions. The following emotions were listed on the display of the monitoring device: no emotion, happiness, anger, anxiety/fear, sadness, surprise, and disgust. The frequency and quality of the emotions were not different between event-related and random feedbacks, indicating that the subjects were not able to discriminate between events with and without additional heart rate correctly. Accordingly, the physiological profiles of the differing emotions compared to conditions with "no emotion" were equivocal. The psychological ratings of excitement and enjoyment, however, came up to expectations. The results show that cognitive schemata and personality dimensions are more important in emotion perception than physiological activation.     

The influence of attention and arousal on emotion perception in adults with severe traumatic brain injury

Many people with traumatic brain injury (TBI) have poor emotion recognition, with negative emotions more frequently impaired. They can also display abnormal affective responses to emotionally charged material, however, the mechanisms underpinning such deficits are unclear. This study examined whether affective responsivity can be improved by focusing attention and whether responsivity is associated with perception accuracy. Eighteen adults with moderate-to-severe TBI and 18 control participants viewed facial expressions while skin conductance (SCR) and evoked cardiac deceleration (ECD) (used as indices of orientation) and skin conductance levels (SCL) (used as an index of phasic arousal) were monitored. They viewed two blocks of faces (8 angry and 8 happy per block), passively in the first block and with the instruction to identify the emotional expression in the second. No differences between conditions, emotions or groups were found using SCR. Both groups demonstrated increasing ECD for the attend condition relative to the passive condition. For the passive task the control group showed increasing SCL (sensitisation) over trials when viewing angry faces and decreasing SCL (habituation) to happy faces. No differences between emotions were shown for the TBI group who rapidly habituated to both expressions. For the attend task, there was no evidence of habituation for either expression for either the control or TBI participants. Physiological measures did not correlate to accuracy in recognising emotions. The results suggest that increasing attentional demands improves orientation and emotional engagement (arousal) to emotional faces following TBI. However, the relationship to this and emotion perception accuracy remains unclear.     

基于脑电的情绪识别研究综述

情绪是综合了人的感觉、思想和行为的一种状态,在人与人的交流中发挥着重要作用。情绪识别是通过获取人的生理或非生理信号对人的情绪状态进行自动辨别,以实现更加友好和自然的人-机交互。基于脑电的识别方法是一种常用且有效的情绪识别方式。文中从脑电获取及预处理、特征提取和特征选择、情绪模式的学习和分类等几个方面,介绍了基于脑电的情绪识别的研究进展、应用前景以及目前存在的主要问题。     

Emotion recognition system using short-term monitoring of physiological signals

A physiological signal-based emotion recognition system is reported. The system was developed to operate as a user-independent system, based on physiological signal databases obtained from multiple subjects. The input signals were electrocardiogram, skin temperature variation and electrodermal activity, all of which were acquired without much discomfort from the body surface, and can reflect the influence of emotion on the autonomic nervous system. The system consisted of preprocessing, feature extraction and pattern classification stages. Preprocessing and feature extraction methods were devised so that emotion-specific characteristics could be extracted from short-segment signals. Although the features were carefully extracted, their distribution formed a classification problem, with large overlap among clusters and large variance within clusters. A support vector machine was adopted as a pattern classifier to resolve this difficulty. Correct-classification ratios for 50 subjects were 78.4% and 61.8%, for the recognition of three and four categories, respectively.     

Smart Vest: wearable multi-parameter remote physiological monitoring system

The wearable physiological monitoring system is a washable shirt, which uses an array of sensors connected to a central processing unit with firmware for continuously monitoring physiological signals. The data collected can be correlated to produce an overall picture of the wearer's health. In this paper, we discuss the wearable physiological monitoring system called ‘Smart Vest’. The Smart Vest consists of a comfortable to wear vest with sensors integrated for monitoring physiological parameters, wearable data acquisition and processing hardware and remote monitoring station. The wearable data acquisition system is designed using microcontroller and interfaced with wireless communication and global positioning system (GPS) modules. The physiological signals monitored are electrocardiogram (ECG), photoplethysmogram (PPG), body temperature, blood pressure, galvanic skin response (GSR) and heart rate. The acquired physiological signals are sampled at 250 samples/s, digitized at 12-bit resolution and transmitted wireless to a remote physiological monitoring station along with the geo-location of the wearer. The paper describes a prototype Smart Vest system used for remote monitoring of physiological parameters and the clinical validation of the data are also presented.     

What our eyes tell us about feelings: Tracking pupillary responses during emotion regulation processes

Emotion regulation is essential for adaptive behavior and mental health. Strategies applied to alter emotions are known to differ in their impact on psychological and physiological aspects of the emotional response. However, emotion regulation outcome has primarily been assessed via self‐report, and studies comparing regulation strategies with regard to their peripheral physiological mechanisms are limited in number. In the present study, we therefore aimed to investigate the effects of different emotion regulation strategies on pupil dilation, skin conductance responses, and subjective emotional responses. Thirty healthy females were presented with negative and neutral pictures and asked to maintain or up‐ and downregulate their upcoming emotional responses through reappraisal or distraction. Pupil dilation and skin conductance responses were significantly enhanced when viewing negative relative to neutral pictures. For the pupil, this emotional arousal effect manifested specifically late during the pupillary response. In accordance with subjective ratings, increasing negative emotions through reappraisal led to the most prominent pupil size enlargements, whereas no consistent effect for downregulation was found. In contrast, early peak dilations were enhanced in all emotion regulation conditions independent of strategy. Skin conductance responses were not further modulated by emotion regulation. These results indicate that pupil diameter is modulated by emotional arousal, but is initially related to the extent of mental effort required to regulate automatic emotional responses. Our data thus provide first evidence that the pupillary response might comprise two distinct temporal components reflecting cognitive emotion regulation effort on the one hand and emotion regulation success on the other hand.     

Effects of unrecognized physiological residual arousal on emotional experience

This study investigated the effect of residual arousal on perceived positive and negative emotion, and the relationship between the valence of emotion and cardiovascular and cardiorespiratory reactivity. A total of 74 participants were randomly assigned to either a Residual Arousal (exercise‐induced) group or a No Residual Arousal (no exercise) group. Following the arousal manipulation, participants watched a video that elicited positive emotion and a video that elicited negative emotion. Within‐person differences revealed greater discrepancies between participants’ reports of positive and negative emotions in response to the videos, indicating that residual arousal caused stronger positive and negative emotions. With regard to physiological reactivity, participants in the No Residual Arousal group exhibited lower heart rate, respiration amplitude, and heart rate variability (LF /HF ratio) during the negative video than during the positive video, suggesting that the positive and negative videos had different influences on physiology. These results support the hypothesis that autonomic activation may be nonspecific with regard to the genesis of emotion, but once a person becomes emotional, physiological reactivity may differ between emotions.     

The facilitative effect of facial expression on the self-generation of emotion.

Twenty-seven female undergraduates completed three tasks: (1) feel four emotions (happiness, sadness, anger, peacefulness); (2) express these emotions, without trying to feel them; and (3) feel and express clearly these four emotions. During each trial subjects pressed a button to indicate when they had reached the required state, and the latency from emotion cue to button press was measured. Heart rate, skin conductance and EMG from four facial sites (brow, cheek, jaw and mouth) were recorded for 15 s before and after the button press and during a baseline period prior to each trial. Self-reports were obtained after each trial. Facial EMG and patterns of autonomic arousal differentiated among the four emotions within each task. Shorter self-generation latency in the Feel-and-Show versus the Feel condition indicated the facilitative effect of facial expression on the self-generation of emotion. Furthermore, the presence of autonomic changes and self-reported affect in the Show condition supports the sufficiency version of the facial feedback hypothesis. The self-generation method employed as an emotion elicitor was shown to reliably induce emotional reactions and is proposed as a useful technique for the elicitation of various emotional states in the laboratory.     

From emotion perception to emotion experience: emotions evoked by pictures and classical music.

Most previous neurophysiological studies evoked emotions by presenting visual stimuli. Models of the emotion circuits in the brain have for the most part ignored emotions arising from musical stimuli. To our knowledge, this is the first emotion brain study which examined the influence of visual and musical stimuli on brain processing. Highly arousing pictures of the International Affective Picture System and classical musical excerpts were chosen to evoke the three basic emotions of happiness, sadness and fear. The emotional stimuli modalities were presented for 70 s either alone or combined (congruent) in a counterbalanced and random order. Electroencephalogram (EEG) Alpha-Power-Density, which is inversely related to neural electrical activity, in 30 scalp electrodes from 24 right-handed healthy female subjects, was recorded. In addition, heart rate (HR), skin conductance responses (SCR), respiration, temperature and psychometrical ratings were collected. Results showed that the experienced quality of the presented emotions was most accurate in the combined conditions, intermediate in the picture conditions and lowest in the sound conditions. Furthermore, both the psychometrical ratings and the physiological involvement measurements (SCR, HR, Respiration) were significantly increased in the combined and sound conditions compared to the picture conditions. Finally, repeated measures ANOVA revealed the largest Alpha-Power-Density for the sound conditions, intermediate for the picture conditions, and lowest for the combined conditions, indicating the strongest activation in the combined conditions in a distributed emotion and arousal network comprising frontal, temporal, parietal and occipital neural structures. Summing up, these findings demonstrate that music can markedly enhance the emotional experience evoked by affective pictures.     

EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm

This paper addresses the emotion recognition problem from electroencephalogram signals, in which emotions are represented on the valence and arousal dimensions. Fast Fourier transform analysis is used to extract features and the feature selection based on Pearson correlation coefficient is applied. This paper proposes a probabilistic classifier based on Bayes' theorem and a supervised learning using a perceptron convergence algorithm. To verify the proposed methodology, we use an open database. An emotion is defined as two-level class and three-level class in both valence and arousal dimensions. For the two-level class case, the average accuracy of the valence and arousal estimation is 70.9% and 70.1%, respectively. For the three-level class case, the average accuracy is 55.4% and 55.2%, respectively.     

Recognition of Facial Expressions: Selective Impairment of Specific Emotions in Huntington's Disease

Recognition of facial expressions of basic emotions was investigated in HL and UJ, two people with Huntington's disease who showed little evidence of general cognitive deterioration. No impairments were found in tests of the perception of age, sex, familiar face identity, unfamiliar face identity, and gaze direction, indicating adequate processing of the face as a physical stimulus. Computer-interpolated ("morphed") images of facial expressions of basic emotions were used to demonstrate deficits in the recognition of disgust and fear for UJ. HL also showed a deficit in the recognition of disgust, and was not very adept (but not significantly impaired) at recognising fear. Other basic emotions (happiness, surprise, sadness, anger) were recognised atnormal levels of performance by HL and UJ. These results show that impairments of emotion recognition can be circumscribed; affecting some emotions more than others, and occurring in people who do not show pronounced perceptual or intellectual deterioration. Questionnaires examining self-assessed emotion indicated normal experience of anger by HL and UJ, but possible abnormalities for disgust and fear. The processes involved in recognising other people's emotions may therefore be linked to those involved in experiencing emotion, and the basic emotions of fear and disgust may have separable neural substrates.     

Pervasive emotion recognition deficit common to alexithymia and the repressive coping style

OBJECTIVE: Previous research has demonstrated a deficit in the ability to recognize emotions in alexithymic individuals. The repressive coping style is thought to preferentially impair the detection of unpleasant compared with pleasant emotions, and the degree of deficit is typically thought to be less severe than in alexithymia. We compared emotion recognition ability in both individuals with alexithymia and those with the repressive coping style. METHODS: Three hundred seventy-nine subjects completed the 20-item Toronto Alexithymia Scale, the Levels of Emotional Awareness Scale, the Marlowe-Crowne Scale (a measure of repressive defensiveness), the Bendig Short Form of the Taylor Manifest Anxiety Scale, and the Perception of Affect Task. The Perception of Affect Task consists of four 35-item emotion recognition subtasks: matching sentences and words, faces and words, sentences and faces, and faces and photographs of scenes. The stimuli in each subtask consist of seven emotions (happiness, sadness, anger, fear, disgust, surprise, and neutral) depicted five times each. Recognition accuracy results were collapsed across subtasks within each emotion category. RESULTS: Highly alexithymic subjects (for all, p<.01) and those with low emotional awareness (for all, p<.001) were consistently less accurate in emotion recognition in all seven categories. Highly defensive subjects (including repressors) were less accurate in the detection of anger, sadness, fear, and happiness (for all, p<.05). Furthermore, scores on the Levels of Emotional Awareness Scale accounted for significantly more variance in performance on the Perception of Affect Task than scores on the Marlowe-Crowne Scale (p<.01). CONCLUSIONS: The results indicate that alexithymia and the repressive coping style are each associated with impairments in the recognition of both pleasant and unpleasant emotions and that the two styles of emotional self-regulation differ more in the magnitude than in the quality of these impairments.     

The Autonomic Differentiation of Emotions Revisited: Convergent and Discriminant Validation

The convergent and discriminant validity of three models of physiological emotion specificity were compared. Forty-two female students served as subjects in a 2 (Context of emotional inductions: real-life, imagery) X 3 (Emotion: fear, anger, control) +1 (Happiness induced in real-life context) repeated measures design. The dependent measures included self-reports of emotion, Gottschalk-Gleser affect scores, back and forearm extensor EMG activity, body movements, heart period, respiration period, skin conductance, skin temperatures, pulse transit time, pulse volume amplitude, and blood volumes. Self-report data confirmed the generation of affective states in both contexts, as intended. Planned multivariate comparisons between physiological profiles established discriminant validity for fear and anger in the real-life context, whereas under imagery, emotion profiles were essentially equal. Convergent validity could not be substantiated. Implications for models of physiological specificity of emotion were discussed.     

Tears of joy,aesthetic chills and heartwarming feelings: Physiological correlates of Kama Muta

Situations involving increased closeness or exceptional kindness are often labeled as moving or touching and individuals often report bodily symptoms, including tears, goosebumps, and warmth in the body. Recently, the kama muta framework has been proposed as a cross-cultural conceptualization of these experiences. Prior research on kama muta has mostly relied on subjective reports. Thus, our main goal of the present project was to examine the pattern of physiological responses to kama muta inducing videos and compare it to the patterns for the similar, though distinct emotions of sadness and awe. One hundred forty-four Portuguese and Norwegian participants were individually exposed to all three emotion conditions. Several psychophysiological indexes of the autonomic nervous system were collected continuously during exposure, including cardiovascular, respiratory, and electrodermal activity, facial EMG, skin temperature, as well as piloerection and lachrymation using cameras. Overall, the results partly replicated previous findings on being moved experiences and self-report studies. Strong self-reported experiences of kama muta were associated with increased phasic skin conductance, skin temperature, piloerection, and zygomaticus activity, while they were associated with reduced heart rate, respiration rate, and tonic skin conductance. The physiological profile of kama muta was successfully distinguished from sadness and awe, partly corroborating self-report evidence. We obtained no clear evidence of a kama muta association with the occurrence of lachrymation or heart rate variability. Our findings provide a systematic overview of psychophysiological response to experiences of kama muta, and help to inform future research on this emotion and positive emotions in general.