“We respond to gestures... in accordance with an elaborate and secret code that is written nowhere, known by none and understood by all.” Edward Saphir, 19211

Introduction

Interest in both emotions and their regulation has grown substantially in recent years, particularly in the neurosciences and specifically in psychiatry. In experimental studies measuring facial expressions, researchers have sought to identify certain patterns of expression.

A vast quantity of data is available; some of these data have been confirmed and some contradicted, depending on the emotion examined and the method used to measure the emotion. A major problem stems from interpreting data which have not always been fully confirmed, and are based on Paul Ekman's theory of six basic patterns of expression (happiness, anger, disgust, fear, sadness, and surprise).2-8

Only the first of Ekman's “basic emotions,” the feeling of happiness with its expressive feature of the “smile,” is observably related to the underlying physiological and facial pattern of expression. Considerable scientific controversy exists regarding Ekman's other basic patterns of expression.

Universality of facial expression

Investigations into the expression of emotion have raised the question of whether basic emotions are shared across cultures. If so, it is important to examine which types of feelings can be described as “basic emotions.” Over the last 50 years, many apparently comprehensive and plausible emotion theories have been proposed. These theories provide different terms to denote “basic emotions.”

Among these theories, two approaches to the universality of facial expressions have evolved. One is the leading neurocultural theory, and the other the behavioral ecology theory, complemented by a compromise between these two theories proposed by Frijda.9,10 The neurocultural theory of facial expression is based on the investigations by Charles Darwin,11 the first scientist to systematically examine the universality of facial expression by carrying out intercultural comparisons. His methods included observing the emotional expressions of young children, comparing emotional expressions in dual interactions, observing the expression of emotions of people born blind, and analyzing the emotional expressions of persons suffering from a mental illness.

From his observations, Darwin found strong similarities in facial expression between people of different cultures (as well as between animals and men), from which he drew conclusions about the universality of expression of emotion. Based on Darwin's scientific works, Ekman completed a great number of studies demonstrating the universality of facial expression, and subsequently identified the types of emotions that he considered universal, ie, observable basic emotional expressions in unconnected cultures. Ekman assumed that similarity between emotional forms of expression indicated their genetic basis. In 1982, he postulated six basic emotions: anger, disgust, fear, happiness, sadness, and surprise, and supplemented these in the 1990s with 11 additional emotions (amusement, contempt, contentment, embarrassment, excitement, guilt, pride in achievement, relief, satisfaction, sensory pleasure, and shame).2-8

Ekman's findings were followed by other scientific contributions on the psychological development of the newborn and people who were born blind. In the newborn, Sroufe12 found reactive crying and native smiling. In an experimental study by Steiner,13 newborn babies who were exposed to a sugar solution with vanilla flavoring reacted with a smile, compared with an expression of disgust in response to a strong smell of rotten eggs.

Izard14 found the facial expression of irritation to be present from the 6th month of life, while the facial expression of fear presents later. Charlesworth15 found a similarity between children who could see and those who were blind with respect to the expression of emotions such as happiness, fear, sadness, anger and trouble, or surprise.

In contrast to the neurocultural theory, Fridlund,16 in his biological behavior-based mimic theory assumed that facial behavior is not automatically related to basic emotions, but rather originates from social contexts and is used more or less indiscriminately. Although the behavioural ecology theory has not been strongly supported, and related evidence is still limited, Frijda9 has suggested a synthesis between these conflicting theories, in that phylogenetic expressions related to types of emotions do exist, and represent basic emotions which, however, are influenced by lifetime experience. The “Facial Feedback” theory assumes that emotions originating from facial behavior are identical to those derived from emotional experience.17

Measuring facial expression of emotion

Three methods are used to measure facial expression of emotion. Each has specific advantages and disadvantages.

Facial Action Coding System; Emotion Facial Action Coding System

These methods involve an analysis of changes in expression in the face of a person. This can range from real-life observation of a person interacting in a group to videotaped interactions in which facial expressions are documented under laboratory conditions upon experimental triggering of a specific emotion.

The pioneer of this approach to measuring facial expression of emotion was Ekman, who, based on his theory of six basic emotions, developed the Facial Action Coding System (FACS).8 This method allows the identification of basic emotions over time, using the image of a videotaped face for image analysis and documenting specific expression changes (called “Action Units”). The advantage of this method is that facial expression of emotion can be studied without biases by the investigator or the technique. Its disadvantages are the dependence on relatively strong emotions and the time needed to analyze the fixed images one by one using the Facial Action Coding System (EMFACS) method. The time required to use this method is one of the reasons that so few studies have been conducted on expression, compared with those on recognition of emotion.

One contribution of EMFACS research is the identification of different forms of smile: expressions of happiness have been extensively investigated as an expression of emotion. Firstly, different studies have shown that M. zygomaticus is the reference muscle for every type of expression of happiness. Secondly, various types of happiness and different types of laughing as indirect indicators of activation of facial muscles have been described and investigated.3-5

The “felt smile” (also “real laughing”) involves the M. zygomaticus and M. orbicularis oculi muscles, which results in spontaneous involuntary smiling/laughing as an expression of perceived happiness. Another type of smiling is the well-known “phoney smile” or “social smile” which involves the isolated use of M. zygomaticus major and is important in social contexts and personal relationships.8

Electromyography method

An electromyography (EMG) method has been developed to recognize activation of facial muscles as accurately and distinctly as possible by using surface electrodes. Due to technical difficulties, for a long period, only two different facial muscles, the M. zygomaticus and M. corrugator supercilii, could be investigated. The methodological validity remained limited because it was unclear whether the recorded muscle activities actually reflected the muscle over which the electrodes had been fixed or another neighboring muscle. From 2005 onwards, the technical performance of systems has reached an acceptable level due to highly selected amplifiable sensitivity. This technical progress allowed identification and independent recording of the activities of subtle visible face muscles. Using a new EMG system, Wolf and colleagues identified the specific facial muscle patterns used to display disgust, appetite, relaxed joy, and aroused joy.18 In validating the system correlations were also found between hungry versus super-satiated states and between the intensity of activation of facial muscles and subjective emotional state. The validity of another EMG system was demonstrated by Schumann,19 who identified emotion-related activations of facial muscles.

The advantages of an EMG system to detect subtle visible facial muscle activity are undisputed for basic research. EMG systems may be useful to develop a consensus-based, stable, empirically founded theory of emotion. The disadvantages of an EMG system are its technical complexity and the restriction of its use in an experimental context. It is therefore not possible to use this system to examine the facial expression of a person in his/her natural social situation.

Automatic face recognition

Commercially available methods for automatic face recognition have improved substantially in quality. These methods have the advantage of allowing an analysis of facial expression of emotion in different settings, where people are studied in their natural social environment without the influence of technical recordings (eg, the Sophisticated High-speed Object Recognition Engine [SHORE™] system).20

The SHORE™ system, as the world's leading system in face detection, is the culmination of years of developments in the field of intelligent systems. SHORE™ resulted in the creation of a highly-efficient real-time C++ software library. The software enables the detection of objects and faces as well as detailed facial analyses based on Ekman's EMFACS-System. Because of the high degree of optimization, it can be adapted to nearly any platform and operating system, especially mobile devices, which makes it suitable for research in real-life settings. An outstanding advantage of SHORE™ is its ability to function in real time with Google Glass.

This opens up an entire spectrum of new smart eyewear applications, including communication aids and even biofeedback aids for patients with disorders like autism or schizophrenia, many of whom have different kinds of difficulty interpreting emotions through facial expression. The missing information could be superimposed in the person's field of vision with data glasses.

To conclude, SHORE™ might supersede EMG systems extensively, and FACS/EMFACS completely.

Facial expression of emotion in metal illnes

Most information on facial expression of emotion is obtained from schizophrenic patients, who have been found to exhibit a paucity of facial expression, particularly in the muscles involved in laughter. This deficit correlates with the amount of negative symptoms.21-26 Hyperactivity of the levator labii muscle is seen in patients with schizophrenia and correlates with positive symptoms (on the Positive and Negative Symptom Scale).23 To date, the importance of M. levator hyperactivity has not yet been conclusively established. Ekman2-8 found M. levator to be the characteristic muscle involved in disgust, as well as in basic aggressive emotional aspects.18 M. levator activity in patients suffering from schizophrenia, however, may not reveal any specific significant emotion, but rather an emotion-specific cophenomenon of “mimic disintegration”.21-32

Mimic disintegration is defined as the inability to organize specific facial muscle movements as an integrated whole, thus making it difficult for observers to decode the emotional state and establish contact or develop a relationship. Using an EMG system, two signs of mimic disintegration were identified: undefined mimic reactions and lack of mimic consistency.

Facial expression of emotion was also examined in depressed patients by Trémeau,33 who studied their capacity to express emotions via facial expressions, compared with both schizophrenic patients and healthy subjects. The participants were asked to create faces showing different expressions of emotions (anger, disgust, fear, happiness, sorrow, and surprise) by using their own verbal expressions. In a second stage, the participants were asked to talk for 2 minutes about each of the six emotions, specifically about their sensations or experience, and to describe how these emotions were present in themselves. During both these tasks, participants* facial expressions were recorded by video and then analyzed using the Ekman FACS.6,8 Both patient groups (schizophrenic and depressed patients) exhibited less spontaneous facial expression of emotion than healthy people and, compared with the schizophrenic patients, depressed patients showed a greater deficit. A discrepancy between minimal facial expression of sorrow and an intense subjective feeling of sorrow was found in depressed patients.33,39

The future of measuring the facial expression of emotion

As a result of the complex measurement methods used to analyse facial emotional expression (FACS/EMFACS and EMG), only a few studies involving small numbers of cases have been carried out, with many basic and applied research questions left unanswered.

This situation might change, as some methods providing a fully automatic face recognition system are already commercially available and provide sufficiently valid data, the quality of which is improving rapidly (eg, the aforementioned SHORE™ system).20 In particular, dynamic aspects of emotion regulation are expected to be more amenable to investigation. A fundamental problem is a limitation in interpreting experimental data because of a lack of a scientifically consensual emotion theory, which has great repercussions on the analysis of facial expression.

The emotion theory forwarded by Hans Lungwitz40,41 offers a persuasive argument for five basic feelings and for both mixed and subsidiary feelings, providing a richly detailed analysis of the dynamic of emotions. Lungwitz defined five “basic emotions” (hunger, anxiety, pain, sorrow, joy) as biologically encoded and occurring in a predetermined sequence. Based on these five sequential emotions, the subjective dynamic reality with unique mixed emotions and feelings is constructed by the individual brain. The term “sequence” means that every individual's action needs an initiating emotion, called “hunger,” which is followed by the emotion “anxiety,” psychologically known as activation preparing the individual for action. The cycle is then completed with the emotion of joy after fulfilling the concrete action. This theory can be considered as a step towards modern theories of action, combining motion and motivation.

Conclusions

A detailed clarification of how emotions are regulated and how the dynamics of facial expression of emotion can be explained could contribute to improved knowledge about basic research in a social setting. Even more importantly, it could improve understanding of, and therapeutical intervention on, the interactive and social consequences of emotional expression deficits in persons with mental illness. Innovative studies in the field of facial expression of emotion may provide detailed answers to unresolved questions in emotion research.