Interaction between musical emotion and facial expression as measured by event-related potentials

We examined the integrative process between emotional facial expressions and musical excerpts by using an affective priming paradigm. Happy or sad musical stimuli were presented after happy or sad facial images during electroencephalography (EEG) recordings. We asked participants to judge the affective congruency of the presented face–music pairs. The congruency of emotionally congruent pairs was judged more rapidly than that of incongruent pairs. In addition, the EEG data showed that incongruent musical targets elicited a larger N400 component than congruent pairs. Furthermore, these effects occurred in nonmusicians as well as musicians. In sum, emotional integrative processing of face–music pairs was facilitated in congruent music targets and inhibited in incongruent music targets; this process was not significantly modulated by individual musical experience. This is the first study on musical stimuli primed by facial expressions to demonstrate that the N400 component reflects the affective priming effect.     

Laterality effects in cross-modal affective priming

ABSTRACT

The present study pursued M. P. Bryden's legacy by investigating how contextual factors can affect laterality effects. Specifically, a cross-modal affective priming paradigm was used in two experiments to determine whether priming with facial expressions would affect responses to emotional sounds. Experiment 1 established that cross-modal priming could be obtained when presenting the emotional sounds binaurally by showing more accurate responses when prime and target were congruent than when they were incongruent, although this extended to response time only for the happy emotion. This priming effect justified Experiment 2, in which the priming paradigm was integrated into a dichotic listening task. The central finding of Experiment 2 was a congruency by ear interaction on number of correct reports, showing that presentation of a facial emotion congruent with a left target produced a large left ear advantage that was reduced when a right ear congruent prime or an incongruent pairing was used. Implications of these findings for emotion processing in the context of Bryden's legacy are discussed.     

Effect of mild head injury on event-related potential correlates of Stroop task performance.

The effect of mild head injury on event-related potential (ERP) correlates of Stroop task performance was explored with the aim of further elucidating the basis of processing impairments after mild head injury. Computer- and card-based Stroop tasks were employed to assess attention function. A sequence of incongruent color words were presented followed by a sequence of congruent color words (printed in congruent colors). Control performance was equivalent on computer- and card-based versions of the incongruent task and faster on the congruent card task than the preceding congruent computer task. The mild head injury group were as fast as controls on the computer-based task but made more errors. However, they were relatively slower on both the congruent and incongruent parts of the card-based task and made more errors in the incongruent task. ERP correlates of computer-based Stroop task performance suggested a greater allocation of attention resources in the incongruent condition in both groups in the form of relatively greater negativity in the latency range 350 to 450 ms with a distribution consistent with the activation of the anterior cingulate gyrus. In addition the mild head injured group showed relatively greater enhancement than the control group in this latency range in both congruent and incongruent conditions. There was, however no evidence of reduced amplitude P1, N1, N2, or P3b deflections. Trails, digit symbol, digit span and auditory verbal learning tests (AVLT) were also administered. Mild head injured participants were impaired only on the AVLT. The finding of greater ERP negativity in the mild head injured group is consistent with greater allocation of attention resources to achieve equivalent performance in the computer-based Stroop task.     

Perceiving emotions from bodily expressions and multisensory integration of emotion cues in schizophrenia

Most studies investigating emotion recognition in schizophrenia have focused on facial expressions and neglected bodily and vocal expressions. Furthermore, little is known about affective multisensory integration in schizophrenia. In the first experiment, the authors investigated recognition of static, face-blurred, whole-body expressions (instrumental, angry, fearful, and sad) with a two-alternative, forced-choice, simultaneous matching task in a sample of schizophrenia patients, nonschizophrenic psychotic patients, and matched controls. In the second experiment, dynamic, face-blurred, whole-body expressions (fearful and happy) were presented simultaneously with either congruent or incongruent human or animal vocalizations to schizophrenia patients and controls. Participants were instructed to categorize the emotion expressed by the body and to ignore the auditory information. The results of Experiment 1 show an emotion recognition impairment in the schizophrenia group and to a lesser extent in the nonschizophrenic psychosis group, and this for all four expressions. The findings of Experiment 2 show that schizophrenia patients are more influenced by the auditory information than controls, but only when the auditory information consists of human vocalizations. This shows that schizophrenia patients are impaired in recognizing whole-body expressions, and they show abnormal affective multisensory integration of bimodal stimuli originating from the same source.     

High contextual sensitivity of metaphorical expressions and gesture blending: A video event-related potential design

Human communication in a natural context implies the dynamic coordination of contextual clues, paralinguistic information and literal as well as figurative language use. In the present study we constructed a paradigm with four types of video clips: literal and metaphorical expressions accompanied by congruent and incongruent gesture actions. Participants were instructed to classify the gesture accompanying the expression as congruent or incongruent by pressing two different keys while electrophysiological activity was being recorded. We compared behavioral measures and event related potential (ERP) differences triggered by the gesture stroke onset. Accuracy data showed that incongruent metaphorical expressions were more difficult to classify. Reaction times were modulated by incongruent gestures, by metaphorical expressions and by a gesture-expression interaction. No behavioral differences were found between the literal and metaphorical expressions when the gesture was congruent. N400-like and LPC-like (late positive complex) components from metaphorical expressions produced greater negativity. The N400-like modulation of metaphorical expressions showed a greater difference between congruent and incongruent categories over the left anterior region, compared with the literal expressions. More importantly, the literal congruent as well as the metaphorical congruent categories did not show any difference. Accuracy, reaction times and ERPs provide convergent support for a greater contextual sensitivity of the metaphorical expressions.     

Electrophysiological correlates of block-wise strategic adaptations to consciously and unconsciously triggered conflict

The role of consciousness in conflict adaptation has been a topic of much debate. The purpose of the current study was to investigate the neural correlates of block-wise conflict adaptations elicited by conscious and unconscious conflicting stimuli in a meta-contrast masked priming task. Event-related potentials (ERPs) were recorded while individuals responded to prime-target pairs in mostly congruent (80% congruent trials, 20% incongruent trials) and mostly incongruent blocks of trials (20% congruent trials, 80% incongruent trials). Mean response times and error rates revealed that the conflict effect (incongruent trials–congruent trials) was reduced in mostly incongruent blocks relative to mostly congruent blocks. Furthermore, conflict related ERP signals (the amplitude difference between congruent and incongruent trials) for three ERP components (early occipito-parietal negativity, the fronto-central N2 and the centro-parietal P3) were attenuated in mostly incongruent blocks compared to mostly congruent blocks, reflecting block-wise adaptation to the frequency of conflict. The conflict-related frontal N2 component differentiated most strongly between visibility conditions. These results further specify the electrophysiological correlates of block-wise strategic adaptations to consciously and unconsciously elicited conflict.     

Electrophysiological correlates of language processing in schizotypal personality disorder.

OBJECTIVE: This study examined whether the electrophysiological correlates of language processing found previously to be abnormal in schizophrenia are also abnormal in schizotypal individuals. The authors used the N400 component to evaluate language dysfunction in schizotypal individuals. METHOD: Event-related potentials were recorded in 16 comparison subjects and 17 schizotypal individuals (who met full DSM-III-R criteria) to sentences presented both visually and aurally; half of the sentences ended with an expected word completion (congruent condition), and the other half ended with an unexpected word completion (incongruent condition). RESULTS: In the congruent condition, the N400 amplitude was more negative in individuals with schizotypal personality disorder than in comparison subjects in both the visual and auditory modalities. In addition, in the visual modality, the N400 latency was prolonged in the individuals with schizotypal personality disorder. CONCLUSIONS: The N400 was found to be abnormal in the individuals with schizotypal personality disorder relative to comparison subjects. The abnormality was similar to the abnormality the authors' laboratory reported earlier in schizophrenic subjects, in which the N400 amplitude was found to be more negative in both congruent and incongruent sentence completions. The N400 abnormality is consistent with the inefficient use of context.     

The modulatory effect of semantic familiarity on the audiovisual integration of face‐name pairs

To recognize individuals, the brain often integrates audiovisual information from familiar or unfamiliar faces, voices, and auditory names. To date, the effects of the semantic familiarity of stimuli on audiovisual integration remain unknown. In this functional magnetic resonance imaging (fMRI) study, we used familiar/unfamiliar facial images, auditory names, and audiovisual face‐name pairs as stimuli to determine the influence of semantic familiarity on audiovisual integration. First, we performed a general linear model analysis using fMRI data and found that audiovisual integration occurred for familiar congruent and unfamiliar face‐name pairs but not for familiar incongruent pairs. Second, we decoded the familiarity categories of the stimuli (familiar vs. unfamiliar) from the fMRI data and calculated the reproducibility indices of the brain patterns that corresponded to familiar and unfamiliar stimuli. The decoding accuracy rate was significantly higher for familiar congruent versus unfamiliar face‐name pairs (83.2%) than for familiar versus unfamiliar faces (63.9%) and for familiar versus unfamiliar names (60.4%). This increase in decoding accuracy was not observed for familiar incongruent versus unfamiliar pairs. Furthermore, compared with the brain patterns associated with facial images or auditory names, the reproducibility index was significantly improved for the brain patterns of familiar congruent face‐name pairs but not those of familiar incongruent or unfamiliar pairs. Our results indicate the modulatory effect that semantic familiarity has on audiovisual integration. Specifically, neural representations were enhanced for familiar congruent face‐name pairs compared with visual‐only faces and auditory‐only names, whereas this enhancement effect was not observed for familiar incongruent or unfamiliar pairs. Hum Brain Mapp 37:4333–4348, 2016. © 2016 Wiley Periodicals, Inc.     

Visual recalibration and selective adaptation in auditory-visual speech perception: Contrasting build-up courses

Exposure to incongruent auditory and visual speech produces both visual recalibration and selective adaptation of auditory speech identification. In an earlier study, exposure to an ambiguous auditory utterance (intermediate between /aba/ and /ada/) dubbed onto the video of a face articulating either /aba/ or /ada/, recalibrated the perceived identity of auditory targets in the direction of the visual component, while exposure to congruent non-ambiguous /aba/ or /ada/ pairs created selective adaptation, i.e. a shift of perceived identity in the opposite direction [Bertelson, P., Vroomen, J., & de Gelder, B. (2003). Visual recalibration of auditory speech identification: a McGurk aftereffect. Psychological Science, 14, 592-597]. Here, we examined the build-up course of the after-effects produced by the same two types of bimodal adapters, over a 1-256 range of presentations. The (negative) after-effects of non-ambiguous congruent adapters increased monotonically across that range, while those of ambiguous incongruent adapters followed a curvilinear course, going up and then down with increasing exposure. This pattern is discussed in terms of an asynchronous interaction between recalibration and selective adaptation processes.     

The influence of semantic and syntactic context constraints on lexical selection and integration in spoken-word comprehension as revealed by ERPs

An event-related brain potential experiment was carried out to investigate the influence of semantic and syntactic context constraints on lexical selection and integration in spoken-word comprehension. Subjects were presented with constraining spoken sentences that contained a critical word that was either (a) congruent, (b) semantically and syntactically incongruent, but beginning with the same initial phonemes as the congruent critical word, or (c) semantically and syntactically incongruent, beginning with phonemes that differed from the congruent critical word. Relative to the congruent condition, an N200 effect reflecting difficulty in the lexical selection process was obtained in the semantically and syntactically incongruent condition where word onset differed from that of the congruent critical word. Both incongruent conditions elicited a large N400 followed by a left anterior negativity (LAN) time-locked to the moment of word category violation and a P600 effect. These results would best fit within a cascaded model of spoken-word processing, proclaiming an optimal use of contextual information during spoken-word identification by allowing for semantic and syntactic processing to take place in parallel after bottom-up activation of a set of candidates, and lexical integration to proceed with a limited number of candidates that still match the acoustic input.     

Conflict adaptation and sequential trial effects: support for the conflict monitoring theory

The purpose of this study was to investigate the cognitive control process of conflict adaptation and the recruitment of cognitive control across sequential trials-termed higher-order trial effects-using the N2 and P3 components of the scalp-recorded event-related potential (ERP). High-density ERPs were obtained from 181 healthy individuals (93 female, 88 male) during a modified Eriksen flanker task. Behavioral measures (i.e., error rates, reaction times [RTs]) and N2 and P3 amplitudes showed reliable conflict adaptation (i.e., previous-trial congruencies influenced current-trial measures). Higher-order trial effects were quantified across multiple sequential presentations of congruent or incongruent trials (e.g., four consecutive incongruent trials). For higher-order trial effects, P3 amplitudes and RTs reliably decreased across both congruent and incongruent trials. Consistent with the conflict monitoring theory, N2 amplitudes decreased across incongruent trials and increased across congruent trials. N2 amplitudes were positively correlated with incongruent-trial RTs; no significant correlations were found for P3 amplitudes and RTs. Effects remained when stimulus-response repetitions were removed. Results indicate that RTs and ERP measures are sensitive to modulations of cognitive control associated with conflict across multiple congruent and incongruent trials. Implications for the conflict monitoring theory of cognitive control are discussed.     

Attenuation of auditory N1 results from identity‐specific action‐effect prediction

The auditory N1 event‐related potential has previously been observed to be attenuated for tones that are triggered by human actions. This attenuation is thought to be generated by motor prediction mechanisms and is considered to be important for agency attribution. The present study was designed to rigorously test the notion of action prediction‐based sensory attenuation. Participants performed one of four voluntary actions on each trial, with each button associated with either predictable or unpredictable action effects. In addition, actions with each hand could result in action effects that were either congruent or incongruent with hand‐specific prediction. We observed no significant differences in N1 amplitude between predictable and unpredictable tones. When contrasting action effects that were congruent or incongruent with hand‐specific prediction, we observed significant attenuation for prediction‐congruent compared to prediction‐incongruent action‐effects. These novel findings suggest that accurate action‐effect prediction drives sensory attenuation of auditory stimuli. These findings have important implications for understanding the mechanisms of action‐effect prediction and sensory attenuation, and may have clinical implications for studies investigating action awareness and agency in schizophrenia.     

Conflict control processing in adults with developmental dyslexia: An event related potentials study

ObjectiveThe present study investigated the time course for processing conflict in dyslexic adults using a flanker task.MethodsSixteen dyslexic and 15 control adults performed a flanker task comprising congruent and incongruent trials in which participants had to indicate the direction of targets surrounded by flankers. Early negative potentials associated with orienting of attention (i.e., N1) and conflict monitoring (i.e., N2) and two positive potentials associated with conflict resolution (i.e., P3b and Nogo P3) were recorded.ResultsThe behavioral data showed differences between incongruent and congruent trials for reaction times in both groups but for error rate only in dyslexics. As in previous studies, controls displayed greater N1, N2 and NoGo P3 as well as a smaller P3b in incongruent trials. Dyslexics lacked N1, N2 and P3b modulation whereas NoGo P3 effect was preserved.ConclusionDyslexics showed impairments in conflict monitoring and in some aspects of conflict resolution (i.e., the allocation of attentional resources) whereas other aspects of conflict resolution (i.e., the inhibition) were preserved.SignificanceThis is the first study to investigate conflict control processing in dyslexic adults using ERPs. Results provide evidence for deficits in orienting of attention, conflict monitoring and allocation of attentional resources in dyslexics.     

Electrophysiological and behavioral indices of cognitive conflict processing across adolescence

Cognitive control enables goal-oriented adaptation to a fast-changing environment and has a protracted development spanning into young adulthood. The neurocognitive processes underlying this development are poorly understood. In a cross-sectional sample of participants 8–19 years old (n = 108), we used blind source separation of EEG data recorded in a Flanker task to derive electrophysiological measures of attention and conflict processing, including a N2-like frontal negative component and a P3-like parietal positive component. Outside the recording session, we examined multiple behavioral measures of interference control derived from the Flanker, Stroop, and Anti-saccade tasks. We found a positive association between age and P3 amplitude, but no relationship between age and N2 amplitude. A stronger N2 was age-independently related to better performance on Stroop and Anti-saccade measures of interference control. A Gratton effect was found on the Flanker task, with slower reaction times on current congruent and better accuracy on current incongruent trials when preceded by incongruent as opposed to congruent trials. The Gratton effect on accuracy was positively associated with age. Together, the findings suggest a multifaceted developmental pattern of the neurocognitive processes involved in conflict processing across adolescence, with a more protracted development of the P3 compared to the N2.     

Effects of facial attractiveness on personality stimuli in an implicit priming task: an ERP study

Objective: Using event-related potentials (ERPs) in a priming paradigm, this study examines implicit priming in the association of personality words with facial attractiveness.

Method: A total of 16 participants (8 males and 8 females; age range, 19–24 years; mean age, 21.30 years) were asked to judge the color (red and green) of positive or negative personality words after exposure to priming stimuli (attractive and unattractive facial images). The positive personality words primed by attractive faces or the negative personality words primed by unattractive faces were defined as congruent trials, whereas the positive personality words primed by unattractive faces or the negative personality words primed by attractive faces were defined as incongruent trials.

Results: Behavioral results showed that compared with the unattractive faces trials, the trials that attractive faces being the priming stimuli had longer reaction times and higher accuracy rates. Moreover, a more negative ERP deflection (N2) component was observed in the ERPs of the incongruent condition than in the ERPs of the congruent condition. In addition, the personality words presented after the attractive faces elicited larger amplitudes from the frontal region to the central region (P2 and P350–550 ms) compared with the personality words after unattractive faces as priming stimuli.

Conclusion: The study provides evidence for the facial attractiveness stereotype (‘What is beautiful is good’) through an implicit priming task.     

Perceptual and motor contributions to performance and ERP components after incorrect motor activation in a flanker reaction task.

OBJECTIVES: The aims of this study are to evaluate contributions of response and perceptual processes to reaction performance in a flanker reaction task, and to investigate whether event-related potential (ERP) component N2 and error negativity N(e) represent similar or functionally distinct cortical mechanisms. METHODS: ERPs, lateralized readiness potentials (LRPs) and reaction performance were measured in a flanker task with arrows as targets and congruent or incongruent flankers. Squares were used as neutral flankers. Target color signaled a response of the hand indicated by (PRO) or against (ANTI) the target arrow's pointing direction. RESULTS: On both PRO and ANTI conditions, performance was facilitated by congruent and impaired by incongruent flankers. In the ERPs on trials with late response errors an N2 was evident before an N(e). In addition, ERPs on correct trials showed an N2 particularly after incongruent flankers on PRO but for each flanker type on ANTI conditions. On incongruent ANTI trials, two successive response conflicts occurred but only a single N2 appeared. CONCLUSIONS: The results indicate that differences in perceptual processing contribute significantly to the flanker effects on task performance and provide further evidence that N2 and N(e) represent different cortical mechanisms. The data also suggest that N2 is not a real-time correlate of incorrect response suppression.     

Unseen stimuli modulate conscious visual experience: evidence from inter-hemispheric summation

Emotional facial expression can be discriminated despite extensive lesions of striate cortex. Here we report differential performance with recognition of facial stimuli in the intact visual field depending on simultaneous presentation of congruent or incongruent stimuli in the blind field. Three experiments were based on inter-hemispheric summation. Redundant stimulation in the blind field led to shorter latencies for stimulus detection in the intact field. Recognition of the expression of a half-face expression in the intact field was faster when the other half of the face presented to the blind field had a congruent expression. Finally, responses to the expression of whole faces to the intact field were delayed for incongruent facial expressions presented in the blind field. These results indicate that the neuro-anatomical pathways (extra-striate cortical and sub-cortical) sustaining inter-hemispheric summation can operate in the absence of striate cortex.     

Mismatch and conflict: neurophysiological and behavioral evidence for conflict priming

Conflict-related cognitive processes are critical for adapting to sudden environmental changes that confront the individual with inconsistent or ambiguous information. Thus, these processes play a crucial role to cope with daily life. Generally, conflicts tend to accumulate especially in complex and threatening situations. Therefore, the question arises how conflict-related cognitive processes are modulated by the close succession of conflicts. In the present study, we investigated the effect of interactions between different types of conflict on performance as well as on electrophysiological parameters. A task-irrelevant auditory stimulus and a task-relevant visual stimulus were presented successively. The auditory stimulus consisted of a standard or deviant tone, followed by a congruent or incongruent Stroop stimulus. After standard prestimuli, performance deteriorated for incongruent compared to congruent Stroop stimuli, which were accompanied by a widespread negativity for incongruent versus congruent stimuli in the event-related potentials (ERPs). However, after deviant prestimuli, performance was better for incongruent than for congruent Stroop stimuli and an additional early negativity in the ERP emerged with a fronto-central maximum. Our data show that deviant auditory prestimuli facilitate specifically the processing of stimulus-related conflict, providing evidence for a conflict-priming effect.     

正常成人汉语句子识别的事件相关脑电位N400实验研究

目的探讨正常成人汉语句子结尾词匹配和非匹配事件相关脑电位（ERP）N400的特点。方法应用ERP仪,对58名19～63岁正常成人右利手受试者,进行汉语正常句子结尾词（匹配）与句子结尾歧义词（非匹配）N400的实验研究。结果正常成人中央区（Cz）、前额区（Fz）及顶区（Pz）脑区汉语句子结尾词匹配的N400平均潜伏期为364ms,平均波幅为8.9μV;非匹配的N400的平均潜伏期为409ms,平均波幅为12.9μV。正常成人各脑区匹配与非匹配的汉语句子结尾词的N400潜伏期、波幅比较有统计学差异。结论正常成人汉语句子识别的N400稳定、可靠;非匹配的汉语句子结尾词的N400潜伏期长,波幅高。     

Neural correlates of multisensory integration of ecologically valid audiovisual events

A question that has emerged over recent years is whether audiovisual (AV) speech perception is a special case of multi-sensory perception. Electrophysiological (ERP) studies have found that auditory neural activity (N1 component of the ERP) induced by speech is suppressed and speeded up when a speech sound is accompanied by concordant lip movements. In Experiment 1, we show that this AV interaction is not speech-specific. Ecologically valid nonspeech AV events (actions performed by an actor such as handclapping) were associated with a similar speeding-up and suppression of auditory N1 amplitude as AV speech (syllables). Experiment 2 demonstrated that these AV interactions were not influenced by whether A and V were congruent or incongruent. In Experiment 3 we show that the AV interaction on N1 was absent when there was no anticipatory visual motion, indicating that the AV interaction only occurred when visual anticipatory motion preceded the sound. These results demonstrate that the visually induced speeding-up and suppression of auditory N1 amplitude reflect multisensory integrative mechanisms of AV events that crucially depend on whether vision predicts when the sound occurs.