Emotion recognition from physiological signals

Emotion recognition is one of the great challenges in human-human and human-computer interaction. Accurate emotion recognition would allow computers to recognize human emotions and therefore react accordingly. In this paper, an approach for emotion recognition based on physiological signals is proposed. Six basic emotions: joy, sadness, fear, disgust, neutrality and amusement are analysed using physiological signals. These emotions are induced through the presentation of International Affecting Picture System (IAPS) pictures to the subjects. The physiological signals of interest in this analysis are: electromyogram signal (EMG), respiratory volume (RV), skin temperature (SKT), skin conductance (SKC), blood volume pulse (BVP) and heart rate (HR). These are selected to extract characteristic parameters, which will be used for classifying the emotions. The SVM (support vector machine) technique is used for classifying these parameters. The experimental results show that the proposed methodology provides in general a recognition rate of 85% for different emotional states.     

Subject-independent emotion recognition based on physiological signals: a three-stage decision method

Background

Collaboration between humans and computers has become pervasive and ubiquitous, however current computer systems are limited in that they fail to address the emotional component. An accurate understanding of human emotions is necessary for these computers to trigger proper feedback. Among multiple emotional channels, physiological signals are synchronous with emotional responses; therefore, analyzing physiological changes is a recognized way to estimate human emotions. In this paper, a three-stage decision method is proposed to recognize four emotions based on physiological signals in the multi-subject context. Emotion detection is achieved by using a stage-divided strategy in which each stage deals with a fine-grained goal.

Methods

The decision method consists of three stages. During the training process, the initial stage transforms mixed training subjects to separate groups, thus eliminating the effect of individual differences. The second stage categorizes four emotions into two emotion pools in order to reduce recognition complexity. The third stage trains a classifier based on emotions in each emotion pool. During the testing process, a test case or test trial will be initially classified to a group followed by classification into an emotion pool in the second stage. An emotion will be assigned to the test trial in the final stage. In this paper we consider two different ways of allocating four emotions into two emotion pools. A comparative analysis is also carried out between the proposal and other methods.

Results

An average recognition accuracy of 77.57% was achieved on the recognition of four emotions with the best accuracy of 86.67% to recognize the positive and excited emotion. Using differing ways of allocating four emotions into two emotion pools, we found there is a difference in the effectiveness of a classifier on learning each emotion. When compared to other methods, the proposed method demonstrates a significant improvement in recognizing four emotions in the multi-subject context.

Conclusions

The proposed three-stage decision method solves a crucial issue which is ’individual differences’ in multi-subject emotion recognition and overcomes the suboptimal performance with respect to direct classification of multiple emotions. Our study supports the observation that the proposed method represents a promising methodology for recognizing multiple emotions in the multi-subject context.

     

The effect of cognitive reappraisal on physiological reactivity and emotional memory

We investigated the effect of cognitive reappraisal on emotional arousal, facial expressivity and subsequent memory. Men and women viewed emotionally negative pictures while they attempted to either increase or decrease negative emotions elicited by the pictures, or to simply view the pictures. Neutral pictures were also presented with instructions to simply view the pictures. Concurrent changes in emotional arousal and valence were assessed with skin conductance responses (SCRs) and facial corrugator electromyographic responses (EMG), respectively. Picture memory was assessed with an immediate recall test and a delayed recognition test. Relative to simply viewing pictures, voluntary reappraisal to increase negative emotion generated greater facial corrugator EMG and SCR responses, and reappraisal to decrease negative emotion generated decreased corrugator EMG responses. Men showed enhanced recognition for pictures presented during the increase and decrease conditions, whereas women showed comparable recognition performance across all regulation conditions. The modulation of subsequent recognition memory associated with decreasing emotion was inversely associated with changes in physiological responses. Our results suggest that sex is an important factor to consider in determining how reappraisal-induced physiological changes are associated with subsequent changes in memory. These findings contribute to our understanding of how reappraising emotion exerts both immediate and enduring influences on physiological responses and subsequent memory.     

Effects of joy and sorrow on pulse-graph parameters in healthy female college students based on emotion-evoked experiments

ObjectiveTo explore the effects of joy and sorrow on pulse-graph parameters in healthy subjects and analyze their potential effect to provide preliminary evidence that pulse diagnosis can identify human emotional changes.Design and interventionForty healthy female college students from Beijing University of Chinese Medicine were recruited. The emotion-evoking experiment was conducted to observe the effects of joy and sorrow on pulse-graph parameters, taking neutral emotion as a control. The experiment included the following criteria. (1) Emotions evoked and evaluated: the same subjects watched three emotional videos which were, respectively, neutral, joyful and sorrowful to evoke the corresponding emotions. They completed the “subjective emotion evaluation form” (SEEF) before watching each video (T₀), immediately after watching the video (T₁) and 15 min after watching the video (T₂) to conduct subjective emotion evaluation. Simultaneously, their heart rate, galvanic skin response and heart rate variability were collected with a 16-lead physiological recorder to conduct an objective evaluation of emotional arousal. (2) Collection of pulse-graph parameters: pulse-graph parameters of the subjects at T₀, T₁ and T₂ were collected with a pulse condition analyzer.ResultsWhen watching the joy video, the rapid ejection phase extended significantly, both the left ventricular diastole and the cardiac cycle shortened significantly. When watching the sorrow video, the left ventricular diastole extended significantly, the cardiac cycle showed an upward trend, and heart rate showed a downward trend.ConclusionJoy and sorrow have a certain effect on pulse-graph parameters of the subjects. The mechanisms may be that joy contributes to rapid ejection phase extension and a shortening of the left ventricular diastole and the cardiac cycle, while sorrow extends the left ventricular diastole. According to the theory of Traditional Chinese Medicine (TCM), the visceral functions can be mediated through human qi flow. Moderate joy and sorrow can promote, respectively, the smooth qi flow of the heart and the lung. However, excessive or long-term joy and sorrow can lead to disharmonious qi flow of the two viscera. Therefore, maintaining moderate emotions is beneficial to the physiological functions of the viscera. Great emotional fluctuations may damage the visceral functions. Future studies with effects of other emotions on pulse-graph parameters are warranted to determine the reliability of the association of emotions and pulse diagnosis.     

基于脑电的情绪识别研究综述

情绪是综合了人的感觉、思想和行为的一种状态,在人与人的交流中发挥着重要作用。情绪识别是通过获取人的生理或非生理信号对人的情绪状态进行自动辨别,以实现更加友好和自然的人-机交互。基于脑电的识别方法是一种常用且有效的情绪识别方式。文中从脑电获取及预处理、特征提取和特征选择、情绪模式的学习和分类等几个方面,介绍了基于脑电的情绪识别的研究进展、应用前景以及目前存在的主要问题。     

Mapping emotions into acoustic space: the role of voice production

Research on the vocal expression of emotion has long since used a “fishing expedition” approach to find acoustic markers for emotion categories and dimensions. Although partially successful, the underlying mechanisms have not yet been elucidated. To illustrate that this research can profit from considering the underlying voice production mechanism, we specifically analyzed short affect bursts (sustained/a/vowels produced by 10 professional actors for five emotions) according to physiological variations in phonation (using acoustic parameters derived from the acoustic signal and the inverse filter estimated voice source waveform). Results show significant emotion main effects for 11 of 12 parameters. Subsequent principal components analysis revealed three components that explain acoustic variations due to emotion, including “tension,” “perturbation,” and “voicing frequency.” These results suggest that future work may benefit from theory-guided development of parameters to assess differences in physiological voice production mechanisms in the vocal expression of different emotions.     

Emotion recognition in schizophrenic and depressed inpatients

The performance of schizophrenic inpatients (N = 14), depressed inpatients (N = 15), and normal hospital employees (N = 15), all females, was compared on Izard's (1971) test of emotion recognition. Subjects were required to match photographs of facial expressions of emotion with the correct label for the emotion. The average numbers of errors of four types were examined: Positive or neutral emotions mislabelled as other positive or neutral emotions; positive or neutral emotions mislabelled as negative; negative emotions mislabelled as positive or neutral; and negative emotions mislabelled as other negative emotions. The principal findings were (1) the schizophrenics were less accurate than normals, but they were not less accurate than the depressives, nor were they less accurate on negative than positive or neutral emotions; (2) the depressives did not display any form of negative bias in emotion recognition; and (3) both groups of inpatients were less accurate than normals when they labelled an emotion as positive or neutral, but not when they labelled it as negative. Implications of the results for the etiology and maintenance of schizophrenia and depression were noted.     

Effects of unrecognized physiological residual arousal on emotional experience

This study investigated the effect of residual arousal on perceived positive and negative emotion, and the relationship between the valence of emotion and cardiovascular and cardiorespiratory reactivity. A total of 74 participants were randomly assigned to either a Residual Arousal (exercise‐induced) group or a No Residual Arousal (no exercise) group. Following the arousal manipulation, participants watched a video that elicited positive emotion and a video that elicited negative emotion. Within‐person differences revealed greater discrepancies between participants’ reports of positive and negative emotions in response to the videos, indicating that residual arousal caused stronger positive and negative emotions. With regard to physiological reactivity, participants in the No Residual Arousal group exhibited lower heart rate, respiration amplitude, and heart rate variability (LF /HF ratio) during the negative video than during the positive video, suggesting that the positive and negative videos had different influences on physiology. These results support the hypothesis that autonomic activation may be nonspecific with regard to the genesis of emotion, but once a person becomes emotional, physiological reactivity may differ between emotions.     

Emotion recognition system using short-term monitoring of physiological signals

A physiological signal-based emotion recognition system is reported. The system was developed to operate as a user-independent system, based on physiological signal databases obtained from multiple subjects. The input signals were electrocardiogram, skin temperature variation and electrodermal activity, all of which were acquired without much discomfort from the body surface, and can reflect the influence of emotion on the autonomic nervous system. The system consisted of preprocessing, feature extraction and pattern classification stages. Preprocessing and feature extraction methods were devised so that emotion-specific characteristics could be extracted from short-segment signals. Although the features were carefully extracted, their distribution formed a classification problem, with large overlap among clusters and large variance within clusters. A support vector machine was adopted as a pattern classifier to resolve this difficulty. Correct-classification ratios for 50 subjects were 78.4% and 61.8%, for the recognition of three and four categories, respectively.     

What our eyes tell us about feelings: Tracking pupillary responses during emotion regulation processes

Emotion regulation is essential for adaptive behavior and mental health. Strategies applied to alter emotions are known to differ in their impact on psychological and physiological aspects of the emotional response. However, emotion regulation outcome has primarily been assessed via self‐report, and studies comparing regulation strategies with regard to their peripheral physiological mechanisms are limited in number. In the present study, we therefore aimed to investigate the effects of different emotion regulation strategies on pupil dilation, skin conductance responses, and subjective emotional responses. Thirty healthy females were presented with negative and neutral pictures and asked to maintain or up‐ and downregulate their upcoming emotional responses through reappraisal or distraction. Pupil dilation and skin conductance responses were significantly enhanced when viewing negative relative to neutral pictures. For the pupil, this emotional arousal effect manifested specifically late during the pupillary response. In accordance with subjective ratings, increasing negative emotions through reappraisal led to the most prominent pupil size enlargements, whereas no consistent effect for downregulation was found. In contrast, early peak dilations were enhanced in all emotion regulation conditions independent of strategy. Skin conductance responses were not further modulated by emotion regulation. These results indicate that pupil diameter is modulated by emotional arousal, but is initially related to the extent of mental effort required to regulate automatic emotional responses. Our data thus provide first evidence that the pupillary response might comprise two distinct temporal components reflecting cognitive emotion regulation effort on the one hand and emotion regulation success on the other hand.     

Family resemblance—category structure of joy and shame

Examined the category structure of “joy” and “shame” emotions. In this study the organization of emotions into abstract categories with a prototype structure based on family resemblances was proposed. Forty male introductory psychology students judged 56 slides of human faces (posed by other such students) as expressing either joy or shame and rated the degree of typicality of each face. Faces rated as more typical were recognized signicantly more quickly and accurately. Correlations between typicality and latency were also significant. More intense expressions of both emotion categories were rated as more typical of the category. A reliable instrument for future study of recognition of the two emotions was developed. The family resemblance-prototype structure for emotion categories was supported.     

From emotion perception to emotion experience: emotions evoked by pictures and classical music.

Most previous neurophysiological studies evoked emotions by presenting visual stimuli. Models of the emotion circuits in the brain have for the most part ignored emotions arising from musical stimuli. To our knowledge, this is the first emotion brain study which examined the influence of visual and musical stimuli on brain processing. Highly arousing pictures of the International Affective Picture System and classical musical excerpts were chosen to evoke the three basic emotions of happiness, sadness and fear. The emotional stimuli modalities were presented for 70 s either alone or combined (congruent) in a counterbalanced and random order. Electroencephalogram (EEG) Alpha-Power-Density, which is inversely related to neural electrical activity, in 30 scalp electrodes from 24 right-handed healthy female subjects, was recorded. In addition, heart rate (HR), skin conductance responses (SCR), respiration, temperature and psychometrical ratings were collected. Results showed that the experienced quality of the presented emotions was most accurate in the combined conditions, intermediate in the picture conditions and lowest in the sound conditions. Furthermore, both the psychometrical ratings and the physiological involvement measurements (SCR, HR, Respiration) were significantly increased in the combined and sound conditions compared to the picture conditions. Finally, repeated measures ANOVA revealed the largest Alpha-Power-Density for the sound conditions, intermediate for the picture conditions, and lowest for the combined conditions, indicating the strongest activation in the combined conditions in a distributed emotion and arousal network comprising frontal, temporal, parietal and occipital neural structures. Summing up, these findings demonstrate that music can markedly enhance the emotional experience evoked by affective pictures.     

Tears of joy,aesthetic chills and heartwarming feelings: Physiological correlates of Kama Muta

Situations involving increased closeness or exceptional kindness are often labeled as moving or touching and individuals often report bodily symptoms, including tears, goosebumps, and warmth in the body. Recently, the kama muta framework has been proposed as a cross-cultural conceptualization of these experiences. Prior research on kama muta has mostly relied on subjective reports. Thus, our main goal of the present project was to examine the pattern of physiological responses to kama muta inducing videos and compare it to the patterns for the similar, though distinct emotions of sadness and awe. One hundred forty-four Portuguese and Norwegian participants were individually exposed to all three emotion conditions. Several psychophysiological indexes of the autonomic nervous system were collected continuously during exposure, including cardiovascular, respiratory, and electrodermal activity, facial EMG, skin temperature, as well as piloerection and lachrymation using cameras. Overall, the results partly replicated previous findings on being moved experiences and self-report studies. Strong self-reported experiences of kama muta were associated with increased phasic skin conductance, skin temperature, piloerection, and zygomaticus activity, while they were associated with reduced heart rate, respiration rate, and tonic skin conductance. The physiological profile of kama muta was successfully distinguished from sadness and awe, partly corroborating self-report evidence. We obtained no clear evidence of a kama muta association with the occurrence of lachrymation or heart rate variability. Our findings provide a systematic overview of psychophysiological response to experiences of kama muta, and help to inform future research on this emotion and positive emotions in general.     

Emotion-on-a-chip (EOC): Evolution of biochip technology to measure human emotion using body fluids

Recent developments in nano/micro technology have made it possible to construct small-scale sensing chips for the analysis of biological markers such as nucleic acids, proteins, small molecules, and cells. Although biochip technology for the diagnosis of severe physiological diseases (e.g., cancer, diabetes, and cardiovascular disease) has been extensively studied, biochips for the monitoring of human emotions such as stress, fear, depression, and sorrow have not yet been introduced, and the development of such a biochip is in its infancy. Emotion science (or affective engineering) is a rapidly expanding engineering/scientific discipline that has a major impact on human society. The growing interest in the integration of emotion science and engineering is a result of the recent trend of merging various academic fields. In this paper we discuss the potential importance of biochip technology in which human emotion can be precisely measured in real time using body fluids such as blood, saliva, urine, or sweat. We call these biochips emotion-on-a-chip (EOC). The EOC system consists of four parts: (1) collection of body fluids, (2) separation of emotional markers, (3) detection of optical or electrical signals, and (4) display of results. These techniques provide new opportunities to precisely investigate human emotion. Future developments in EOC techniques will combine social and natural sciences to expand their scope of study.     

Emotion-on-a-chip (EOC): Evolution of biochip technology to measure human emotion using body fluids

Recent developments in nano/micro technology have made it possible to construct small-scale sensing chips for the analysis of biological markers such as nucleic acids, proteins, small molecules, and cells. Although biochip technology for the diagnosis of severe physiological diseases (e.g., cancer, diabetes, and cardiovascular disease) has been extensively studied, biochips for the monitoring of human emotions such as stress, fear, depression, and sorrow have not yet been introduced, and the development of such a biochip is in its infancy. Emotion science (or affective engineering) is a rapidly expanding engineering/scientific discipline that has a major impact on human society. The growing interest in the integration of emotion science and engineering is a result of the recent trend of merging various academic fields. In this paper we discuss the potential importance of biochip technology in which human emotion can be precisely measured in real time using body fluids such as blood, saliva, urine, or sweat. We call these biochips emotion-on-a-chip (EOC). The EOC system consists of four parts: (1) collection of body fluids, (2) separation of emotional markers, (3) detection of optical or electrical signals, and (4) display of results. These techniques provide new opportunities to precisely investigate human emotion. Future developments in EOC techniques will combine social and natural sciences to expand their scope of study.     

Psychophysiological effects of emotional responding to goal attainment

Effects of positive performance feedback on self-reported emotion and associated physiological responding and their relation to motivational engagement were investigated in an achievement context. To create a situation of self-relevant goal striving and goal attainment, appraisals of goal relevance and goal conduciveness were manipulated by presenting 65 female undergraduate students with a psychological test, followed by positive performance feedback. Emotional responding during the 1-min feedback showed elicitation of various positive achievement-related emotions associated with broad sympathetic activation (decreased pre-ejection period, increased cardiac output, and increased skin conductance and response rate). Individual-level emotion reports indicated distinct subgroups of participants experiencing primarily either interest, joy, pride, or surprise. Between-participants physiological reactivity was found to differ based on primary self-reported feelings. We discuss motivational antecedents and consequences in achievement-related emotions.     

Emotions in everyday life: an ambulatory monitoring study with female students

Additional heart rate as an indicator of emotional arousal was monitored throughout the day with a special ambulatory device. Fifty female students received acoustic feedback every 10-20 min. The feedback was based either on events (additional heart rate present) or was random without additional heart rate. Following the feedback the subjects were asked to disclose their emotions. The following emotions were listed on the display of the monitoring device: no emotion, happiness, anger, anxiety/fear, sadness, surprise, and disgust. The frequency and quality of the emotions were not different between event-related and random feedbacks, indicating that the subjects were not able to discriminate between events with and without additional heart rate correctly. Accordingly, the physiological profiles of the differing emotions compared to conditions with "no emotion" were equivocal. The psychological ratings of excitement and enjoyment, however, came up to expectations. The results show that cognitive schemata and personality dimensions are more important in emotion perception than physiological activation.     

A longitudinal study of emotion regulation and anxiety in middle childhood: Associations with frontal EEG asymmetry in early childhood

We investigated whether brain electrical activity during early childhood was associated with anxiety symptoms and emotion regulation during a stressful situation during middle childhood. Frontal electroencephalogram (EEG) asymmetries were measured during baseline and during a cognitive control task at 4 ½ years. Anxiety and emotion regulation were assessed during a stressful situation at age 9 (speech task), along with measures of heart rate (HR) and heart rate variability (HRV). Questionnaires were also used to assess anxiety and emotion regulation at age 9. Results from this longitudinal study indicated that children who exhibited right frontal asymmetry in early childhood experienced more physiological arousal (increased HR, decreased HRV) during the speech task at age 9 and less ability to regulate their emotions as reported by their parents. Findings are discussed in light of the associations between temperament and development of anxiety disorders. © 2010 Wiley Periodicals, Inc. Dev Psychobiol 52:197–204, 2010     

大学生学业情绪成套问卷编制的理论构想

学业情绪研究源于教育情境中的情绪研究。目前学业情绪测量问卷的编制理论主要是教育情境中的情绪研究理论之一—德国教育心理学家Pekrun的学业情绪理论。本文主要阐述了以这一理论为基础、以大学生为研究主体所构建的大学生学业情绪成套问卷(CSAEQ-B)的结构,预计本成套问卷由一般学业情绪、课程相关学业情绪、课堂相关学业情绪、课外相关学业情绪和考试相关学业情绪等5个问卷组成,并分析了问卷项目编制的依据。     

Smart Vest: wearable multi-parameter remote physiological monitoring system

The wearable physiological monitoring system is a washable shirt, which uses an array of sensors connected to a central processing unit with firmware for continuously monitoring physiological signals. The data collected can be correlated to produce an overall picture of the wearer's health. In this paper, we discuss the wearable physiological monitoring system called ‘Smart Vest’. The Smart Vest consists of a comfortable to wear vest with sensors integrated for monitoring physiological parameters, wearable data acquisition and processing hardware and remote monitoring station. The wearable data acquisition system is designed using microcontroller and interfaced with wireless communication and global positioning system (GPS) modules. The physiological signals monitored are electrocardiogram (ECG), photoplethysmogram (PPG), body temperature, blood pressure, galvanic skin response (GSR) and heart rate. The acquired physiological signals are sampled at 250 samples/s, digitized at 12-bit resolution and transmitted wireless to a remote physiological monitoring station along with the geo-location of the wearer. The paper describes a prototype Smart Vest system used for remote monitoring of physiological parameters and the clinical validation of the data are also presented.