Earticle

Emotion recognition is rapidly advancing within the realm of artificial intelligence (AI), spurred by progress in deep learning, multimodal data handling, and broader availability of large datasets. This paper offers an in-depth overview of emotion recognition strategies, focusing on bio-metric signals (e.g., audio, visual, physiological, and brain signals), conventional machine learning methods, and the latest deep learning architectures, including Transformers, two-dimensional Con-volution Neural Networks (CNNs) 2D CNNs, and three-dimensional CNNs (3D CNNs). We further examine multimodal systems and the role of Large Language Models (LLMs) in merging textual, audio, and video information for more precise emotion assessments. Key challenges such as real-time implementation, data biases, and cultural diversity are also highlighted. Ethical issues related to privacy and the potential misuse of emotion recognition technologies receive attention, as does a discussion of emerging applications in healthcare, human-computer interaction (HCI), mental health monitoring, and education. In sum, this review aims to contribute to the scholarly conversa-tion around evolving emotion recognition methodologies and their applications in practical systems.

Our research aims to enhance the modeling of speech signals for more effective extraction of node features and analysis of relationships between nodes. To achieve this, we model speech signals as cyclic or linear graphs. Our model combines layers of Graph Convolutional Networks (GCN) and Graph Attention Networks (GAT) to leverage their respective strengths in processing graph data. Specifically, we utilize GCN to aggregate information from neighboring nodes, which helps capture local relationships among nodes. Additionally, we employ GAT mechanisms to assign varying attention weights to different neighboring nodes, facilitating a better capture of complex global relationships between nodes. In our experiments, we validate our approach using the IEMOCAP dataset and demonstrate comparable performance to state-ofthe- art models in emotion recognition tasks. This research outcome provides new insights and methodologies for further exploration in the field of speech signal processing.

Automated speech emotion recognition (SER) by efficient long-term temporal context modeling is a challenging task of the digital audio signal processing domain. However, by default, the recurrent neural network (RNN) is employed to incorporate the temporal dependencies in sequence to investigate the relationships among sequences and features. In this study, we design a parallel convolutional neural network (PCNN) for SER by using a squeeze and excitation network (SEnet) with the self-attention module. Additionally, we adopt the residual learning strategy in both module, SEnet and self-attention, which is further improve the performance of the network. Our proposed SER system utilizes speech spectrogram as input and extracts utterancelevel discrete features by using the PCNN model. We experimentally evaluated our proposed system by standard speech corpus, interactive emotional dyadic motion capture (IEMOCAP). The prediction result reveals the significance and robustness of the proposed PCNN system, which obtained a high recognition rate of 72.01% over state-of-the-art (SOTA) methods.

우리 춤은 우리 고유의 정서를 담고 있는 종합예술이므로 우리 춤을 분석하고 이해하는 것은 큰 의미가 있다. 본 논문에서는 기존의 춤 동작의 정량적인 분석을 통한 감정인식 기술을 이용하여 우리 춤에 내포된 감정 패턴의 변화를 살펴본다. 먼저 한국 전통춤으로부터 무용전문가들의 정성적 분석에 기반하여 추출된 우리 춤사위를 정해진 각 감정별로 재구성하여 창작하고 창작된 우리 춤을 무용전문가가 시연한다. 이를 카메라를 이용하여 획득하고， 영상처리를 통해서 시연자의 실루엣을 뽑아낸 후， 정량적 특정량들을 추출한다. 이러한 특정량을 분석하기 위한 기존의 PCA (Principle Component Analysis) 방법 보다는 조금 더 유용한 ICA(Independent Component Analysis) 방법을 적용한다. 이어 신경회로망을 이용하여 각 감정별 춤사위를 학습 시킨 후， 임의의 춤사위에 내포된 감정을 인식 한다. 또한， 시스템에 의해 인식된 감정 패턴과 변화의 정성적 의미를 이해하기 위해 무용전문가들에 의해 정립된 정성적 분석 결과와 비교， 분석한다. 이는 정성적인 분석에만 국한되던 우리 춤의 특성에 대한 이해를 객관적이고 정량화된 분석을 통한 이해의 차원으로 확장시키는 것으로， 우리 춤의 특성을 새롭게 정의하는 계기를 마련할 수 있다. 다양한 장르의 한국 전통춤 가운데 우리 춤을 대표할 수 있는 춤사위를 선정하고， 정성적 / 정량적으로 분석함으로써 우리 춤의 특성을 이해하기 위한 체계적인 틀을 제공하고자 한다.

Recently, endeavors for recognizing human emotion from full body movements. Especially, dance which is close to common human motions that are the most typical of expressing human emotion, have been made. Korean dance includes Korean peculiar emotion, which discriminates Korean dance from Western dance. Therefore, the unique method for analyzing and recognizing emotion from Korean dance has been strongly required. To the best of our knowledge, there is no method for recognizing emotion from korean dance in the literature. In this thesis, we propose a method for recognizing emotion from Korean dance and analyzing Korean dance using the recognition results. The method consists of four steps. First, dance performances associated with pre-defined emotions are extracted from several representative Korean dances based on professional dancers' qualitative analysis and danced by professional dancers. Second, the dance performances are captured using multiple cameras and quantitative features associated with dance performances are extracted from the camera images using a series of image processing algorithms. Furthermore, the features are analysed by ICA(independent Coponent Analysis) more useful than PCA(Principle Component Analysis).Third, the mapping between the dance performances and the pre-defined emotions is estimated using neural network. Finally, given arbitrary dance performances, their corresponding emotions are recognized using the neural network. Thus, the purpose of this thesis is extracting sequential emotional patterns of Korean dance using the emotion recognition method. and proposing a criterion for specifying the characteristics of Korean dance quantitatively. The quantitative analysis can concrete the results of the existing researches on understanding the characteristics of Korean dance depend only on the qualitative analysis. We expect that this thesis provides a systematic framework for understanding the characteristics of the whole Korean dance through the qualitative and quantitative analysis of some representative Korean dances.

경두개 자기자극(Transcranial magnetic stimulation, TMS)은 대뇌피질을 자극하는 비침습형 측정방법으로 뇌의 특정위치에 코일을 통한 강력한 전류의 순간적 강한 자기장을 가하므로써, 이를 통하여, 뇌의 특정영역의 신경 세포를 활성화 혹은 억제시키는 뇌 자극술이다. 특히, 배외측 전전두엽 피질(Dorsolateral prefrontal cortex, DLPFC)에 적용된 저주파의 반복적으로 가해지는 TMS 자극법은 감정처리와 관련되어 유효성이 검증되면서 의학, 심리학 등 다양한 정신질환 및 심리치료에서 활용되고 있다. 이러한 TMS의 유효성을 바탕으로, 본 연구에서는 사 용자가 감정문장을 읽는 동안, TMS를 사용하여 뇌의 특정영역에 자극을 가한 후, TMS 전후 감정변화에 대하여 알아보고자 한다. 즉, TMS 자극 전/후의 감정에 대한 반응을 사건관련전위(Event-related potential, ERP)방 법으로 뇌파를 기록하고, 감정과 관련된 ERP 성분(Component)을 검출 후, 이를 기계학습 활용하여 그 성능을 평 가하고자 한다. 먼저, 6개의 감정(분노, 흥분, 두려움, 지루함, 슬픔, 행복)과 중립의 감정문장을 제시하고, 이를 2번으로 나누어, 총 280개를 사용하여 데이터를 수집하였다. 측정된 뇌파는 방해파 제거 등의 전처리 과정을 거친 후, 감정과 관련된 ERP 성분인 후기 정적 복합체(Late positive complex, LPC)를 검출하였다. 마지막으로, 100ms당 1개의 스펙트로그램(Spectrogram)으로 이미지를 생성하고, 이에 대한 성능 평가를 진행하였다. 그 결 과, TMS 자극 전/후 비교해보면, TMS 자극 후가 자극 전보다 감정인식 성능이 높게 나타났으며, 합성신경망 (Convolutional Neural network, CNN)을 사용할 때, 4개의 감정에 대해서 86.39%의 가장 높은 성능을 얻었 다. 본 연구는 TMS 처치를 통한 뇌 인지기능조절을 통한 감정반응의 효과를 확인하였고, 감정조절과 뇌 반응에 효 과적이라는 것을 확인할 수 있었다.

Transcranial magnetic stimulation(TMS) is a non-invasive measurement method that stimulates the cerebral cortex to examine human brain function. For this reason, many fields, ranging from medicine and psychology to computer science, focus on identifying and recognizing human emotions to better understand individual states and comprehend specific situations. This study aims to determine emotional responses by controlling brain cognitive functions through TMS. In particular, low-frequency repetitive TMS applied to the dorsolateral prefrontal cortex(DLPFC) has been verified as effective in emotional processing. Hence, it is used in various mental illnesses, psychotherapy, medicine, and psychology. In this paper, we investigate the effects of low-frequency repetitive TMS applied over the DLPFC while reading emotional sentences. In addition, we record EEG(electroencephalogram) using Event-related potential(ERP) to examine brain responses before and after TMS stimulation, detect ERP components related to emotion, and evaluate with machine learning. First, we selected 280 emotional sentences as standardized elicitation samples for self-induced emotion from EEG signals using ERP that described six discrete emotions, i.e., anger, excited, fear, bored, sadness, happiness, and neutral in the baseline. We focus on the emotionrelated ERP component, Late Positive Complex(LPC), extracted by raw EEG signals. Then, we generate the EEG signals to spectrogram as images per 100 ms. As a result, we confirm that emotion recognition after TMS stimulation is higher than before. Also, we achieved the best recognition accuracy of 86.39% after TMS treatment compared to before TMS treatment using CNN. In conclusion, our study means that TMS treatment is effective for emotion control and brain response. In the future, it will be possible to consider individual variance differences or conduct research using various deep-learning methods.

최근 인공지능 기술의 발달과 더불어 감정인식은 중요한 연구 분야로 대두되고 있다. 기존의 음성, 얼굴을 기반으로 한 감정연구에서 최근에는 보다 객관적인 뇌파 등과 같은 생체신호를 활용하는 연구로 확대되고 있다. 본 연구에서 는 뇌파 데이터를 활용하여 긍/부정에 대한 뇌파에 특징요소를 추출하고, 대표적인 기계학습 분류기인 SVM(Support Vector Machine) 분류기를 활용하여 감정분류를 진행하였다. 뇌파 신호에 대한 전처리를 통하여 몸과 눈의 움직임(eye-blick)과 같은 방해파를 제거하였다. 특징요소 추출을 위하여 전처리된 뇌파 데이터를 10ms 당 데이터의 평균값을 생성하고, 이를 50%로 중첩하는 방법으로 1ms 단위의 이미지를 형성하였다. 추출된 특징요 소는 대표적인 기계학습 분류기인 SVM을 사용하여 감정분류를 진행하였다. 그 결과, 3가지 감정에 대하여 평균 94.6%의 결과를 도출하였다. 본 연구의 결과는 감정분류에 있어서 시계열 데이터를 이미지화하는 방법으로 특징을 추출하였다. 향후 시계열 데이터 처리에 있어서 새로운 특징요소 추출방법으로 활용 가능하며, 또한, 감정을 세분화 하여 다양한 기계학습 알고리즘에 적용할 수 있는 새로운 접근법을 제안한 것에 의의를 둔다.

According to the development of artificial intelligence, emotion recognition has emerged as an important element from psychology to engineering. In the previous research, there are various of materials such as voice, motion, and behaviors. However, these responses have a weakness to identify real emotional states because of making the faking voice or hiding facial expression unlike their emotion as intended by oneself. Some studies related to emotion recognition have extended to study emotion recognition using physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), skin conductance (SKT) and respiration. In this study, we performed the emotion recognition using SVM (Support Vector Machine) with public open database. The EEG signals conducted the preprocessing for removal to artifacts such as muscle and eye blicks. In the feature extraction, the window size is 10 seconds and 50% overlap and the output image data is produced per 1ms. As a result, we achieved the classification average accuracy of 94.6% among three emotion, positive, negative and calm. In conclusion, this proposed method is a novel approach for deal with time-series data such as voice and physiological signals. Also, we proposed that feature extraction relates to emotion recognition for machine learning.

최근 들어 뇌파(electroencephalogram, EEG) 신호와 같은 생리적 신호를 활용한 감정 인식 연구가 활발하게 이루어지고 있다. 특히, 다양한 딥러닝 기반의 하이브리드 모델이 도입되면서 EEG 신호의 시간적․공간적 특징을 효과적으로 추출하는 방법이 주목받고 있다. 이러한 접근 방식은 감정 분류 성능 향상에 중요한 역할을 하며, 최신 연구에서는 다양한 딥러닝 기술의 결합을 통해 EEG 감정 인식의 성능을 높이고 있다. 본 연구에서는 EEG 기반 감정 인식의 정확도를 향상시키기 위해 CNN과 CBAM(Convolutional Block Attention Module)을 결합하여 EEG 신호의 채널 및 공간 정보를 강조하여 초기 특징 추출을 강화하고, Bi-LSTM을 통해 시간적 맥락을 학습하며, Transformer의 Self-Attention을 적용하여 전역적 시계열 정보를 통합한 모델을 제안하였다. 또한, 각 모듈의 결합 순서가 감정 분류 성능에 미치는 영향을 실험적으로 비교․분석하여 최적의 구조를 도출하였다. DEAP 데이터셋의 EEG 신호를 활용하여 실험한 결과, CNN을 적용한 모델의 정확도는 정서가(Valence) 56.39%, 각성도(Arousal) 56.16%였으며, 초기 특징 추출 강화를 위해 CNN에 CBAM을 결합한 모델의 정확도는 정서가는 62.75%, 각성도는 65.27%로 향상되었다. 이후 추가적으로 Bi-LSTM과 Self-Attention을 순차적으로 적용한 결과 정서가의 정확도는 62.75%에서 74.15%로 개선되었고, 각성도의 정확도는 65.27%에서 71.54%로 개선되었으며, 이는 기존 선행연구 대비 유의미한 성능 향상임을 확인하였다. 본 연구는 시공간적 정보를 효율적으로 통합하여 EEG 기반 감정 인식의 정밀도를 유의미하게 개선하였으며, 개인별 편차를 고려한 레이블링으로 모델의 일반화 가능성을 확인하였다.

This study proposes a hybrid deep learning model to improve the accuracy of EEG-based emotion recognition by effectively extracting spatial and temporal features. Initially, Convolutional Neural Networks (CNN) were combined with the Convolutional Block Attention Module (CBAM) to enhance initial feature extraction by selectively emphasizing channel and spatial information in EEG signals. Subsequently, Bidirectional Long Short-Term Memory (Bi-LSTM) networks were employed to learn temporal contextual information, and the Self-Attention mechanism from Transformers was incorporated to integrate global temporal dependencies. The study also conducted experimental comparisons of different module combination orders to identify the optimal model architecture. Using the DEAP dataset, the CNN-only model achieved classification accuracies of 56.39% for valence and 56.16% for arousal. The model augmented with CBAM improved these accuracies to 62.75% and 65.27%, respectively. Further sequential application of Bi-LSTM and Self-Attention resulted in significantly enhanced accuracies of 74.15% for valence and 71.54% for arousal. These results demonstrate that the hybrid architecture, which efficiently integrates spatial and temporal information, substantially improves emotion recognition performance based on EEG signals. Additionally, by incorporating individual differences through personalized labeling, the study confirmed the model’s generalizability across diverse participants.

최근 들어 뇌파(electroencephalogram, EEG) 신호와 같은 생리적 신호를 활용한 감정 인식 연구가 활발히 진행되고 있으며, EEG 신호의 시간적·공간적 특징을 효과적으로 추출하는 것이 감정 분류의 성능 향상에 핵심으로 작용하고 있다. 본 연구에서는 EEG 기반 감정 인식 성능을 높이기 위해 EEG 신호의 채널과 공간에서 중요한 특징에 가중치를 부여하는 CBAM (Convolutional Block Attention Module )과 양방향으로 시간적 맥락의 특징을 추출하는 Bi- LSTM(Bidirectional Long Short-Term Memory)과 전체적인 시계열 정보를 통합하는 Self-Attention 모듈을 순차적으로 적용한 방법을 제안한다. CNN 모델을 기준으로 CBAM을 적용한 결과 긍정과 부정을 나타내는 정서가(Valence)에서는 정확도가 56.39%에서 62.75%로 개선되었고 감정의 정도를 나타내는 각성도(Arousal)의 정확도는 56.16%에서 65.27%로 개선되었으며, Bi-LSTM과 Self-Attention을 순차적으로 적용한 결과 정서가(Valence)에서는 정확도가 62.75%에서 74.15%로 개선되었고, 각성도(Arousal)의 정확도는 65.27%에서 71.54%로 개선되었다. 본 모델은 시공간적 정보를 효율적으로 통합하여 EEG 기반 감정 인식의 정밀도를 유의미하게 개선하였다.

In this paper, we have implemented speeches that utilized the emotion information of the user’s speech and image matching and recommendation system. To classify the user's emotional information of speech, the emotional information of speech about the user’s speech is extracted and classified using the PLP algorithm. After classification, an emotional DB of speech is constructed. Moreover, emotional color and emotional vocabulary through factor analysis are matched to one space in order to classify emotional information of image. And a standardized image recommendation system based on the matching of each keyword with the BM-GA algorithm for the data of the emotional information of speech and emotional information of image according to the more appropriate emotional information of speech of the user. As a result of the performance evaluation, recognition rate of standardized vocabulary in four stages according to speech was 80.48% on average and system user satisfaction was 82.4%. Therefore, it is expected that the classification of images according to the user’s speech information will be helpful for the study of emotional exchange between the user and the computer.

다른 장르의 게임에 비해 포커는 게이머의 심리적 요소가 많은 영향을 끼친다. 본 논문에서는 CNN과 SVM을 기반으로 온라인 포커 게임에 게이머와 아바타 간의 감성연결을 실현하기 위한 새로운 감성 인식방법을 제안한다. CNN모델을 이용하여 원래 얼굴 이미지의 특징을 추출하고, 다중 클래스 SVM분류기를 사용하여 목표 이미지를 인식하고 분류한다. FER-2013데이터베이스에서 이 방법은 감성인식률 68.79 %를 달성하였다. 기존의 다른 감성 인 식 모델과 비교하면, 이 모델은 뚜렷한 장점을 보일 수 있다. 본 게임은 Socket 통신방식을 통해 감성인식결과를 Seven Poker로 전송하여 아바타가 게이머와 같은 감성을 표현하도록 설계하였다. 온라인 포커 게임에 감성연결 기술을 이용하면 게임과 인간의 상호작용이 향상될 뿐 아니라 게 이머가 상대방의 심리적인 활동을 효과적으로 분석할 수 있다. 감성연결 기술은 게임에서 게이머 들에게 새로운 게임 경험을 제공할 수 있는 기술이라고 생각된다.

Compared to other types of games, poker game is a psychological game based on gamer's psychological activity. This paper proposes a method based on convolutional neural network (CNN) and support vector machine (SVM) to realize the emotion recognition to link the gamer and his avatar in online poker game. The CNN model is used to extract feature of the original face images, and the multi-class SVM classifier is used to classify the emotions. On the FER-2013 database, the proposed method achieves 68.79% emotion recognition rate, and has obvious advantages compared with most other emotion recognition methods. Next, through the socket communication, the result of the emotion recognition is transferred to the designed seven poker game to realize the emotion linkage between the gamer and his avatar. More importantly, the emotion linkage technology not only helps the gamer to analyze the opponent’s psychological state, but also enhances the interaction of the game. It is undoubtedly a new breakthrough in game play that will give gamers a whole new gaming experience.

본 논문에서는 열화상 스테레오 카메라를 사용하여 춤 동작 분석을 위한 인체 검출 방법을 제안한다. 이 방법은 무용수와 스튜디오 배경의 온도차이로 무용수 영역을 검출하고 배경 차연산과 노이즈필터를 통해 무용수의 실루엣을 추출함으로써, 실내에서만 국한된 춤의 분석 장소를 실외 환경으로 확장이 가능하다. 또한 스테레오 카메라 정합에 기반한 무용수의 3차원 공간 위치를 추정하여 춤 분석을 위한 무용수 움직임의 공간정보를 알 수 있다. 이는 무용수들이 춤사위에 내포된 감정을 잘 표현 할 수 있도록 의상 및 공간의 제약 없이 춤 분석이 가능한 방법을 마련할 수 있다. 향후 무용 콘텐츠의 하나인 모션 캡쳐 등의 모델링에 도움이 되며, 다양한 첨단 분야에서 휴먼 인식의 신뢰성을 높일 수 있을 것으로 기대된다.

In this paper, we propose a human detection for dance performance analysis using infrared thermal stereo camera. The propose method detects a dancer’s silhouette difference between temperature of dancer and temperature of background. The method can be extended to the analysis of dance performance in both out-door and in-door environments by detecting silhouette features of a dance performance. Then 3D spatial coordinate estimation of a dancer based on stereo matching determines spatial data of a dancer’s region movement. The method makes it possible to analyze emotions expressed by dance performance without any limitation of clothes and space. We are looking forward to expect further improvement of human recognition in the up-to-date Science

우리 춤은 우리 고유의 정서를 담고 있는 종합예술이므로 우리 춤을 분석하고 이해하는 것은 큰 의미가 있다. 본 논문에서는 기존의 춤 동작의 정량적인 분석을 통한 감정인식 기술을 이용하여 우리 춤에 내포된 감정 패턴의 변화를 살펴본다. 먼저 한국 전통춤으로부터 무용전문가들의 정성적 분석에 기반하여 추출된 우리 춤사위를 정해진 각 감정별로 재구성하여 창작하고 창작된 우리춤을 무용전문가가 시연한다. 이를 카메라를 이용하여 획득하고, 영상처리를 통해서 시연자의 실루엣을 뽑아낸 후, 정량적 특징량들을 추출한다. 이어 신경회로망을 이용하여 각 감정별 춤사위를 학습시킨 후, 임의의 춤사위에 내포된 감정을 인식한다. 본 논문에서는 정면, 좌, 우 세 시점에서 획득된 다시점 영상을 이용하여 학습시킴으로써 보다 안정적으로 동작하는 인식시스템을 제안한다. 그리고, 시스템에 의해 인식된 감정패턴과 변화의 정성적 의미를 이해하기 위해 무용전문가들에 의해 정립된 정성적 분석결과와 비교, 분석한다. 이는 정성적인 분석에만 국한되던 우리 춤의 특성에 대한 이해를 객관적이고 정량화된 분석을 통한 이해의 차원으로 확장시키는 것으로, 우리 춤의 특성을 새롭게 정의하는 계기를 마련할 수 있다. 다양한 장르의 한국 전통춤 가운데 우리 춤을 대표할 수 있는 춤사위를 선정하고, 정성적/정량적으로 분석함으로써 우리 춤의 특성을 이해하기 위한 체계적인 틀을 제공하고자 한다.

Recently, endeavors for recognizing human emotion from full body movements. Especially, dance which is close to common human motions that are the most typical of expressing human emotion, have been made. Korean dance includes Korean peculiar emotion, which discriminates Korean dance from Western dance. Therefore, the unique method for analyzing and recognizing emotion from Korean dance has been strongly required. To the best of our knowledge, there is no method for recognizing emotion from korean dance in the literature. In this thesis, we propose a method for recognizing emotion from Korean dance and analyzing Korean dance using the recognition results. The method consists of four steps. First, dance performances associated with pre-defined emotions are extracted from several representative Korean dances based on professional dancers' qualitative analysis and danced by professional dancers. Second, the dance performances are captured using multiple cameras and quantitative features associated with dance performances are extracted from the camera images using a series of image processing algorithms. Third, the mapping between the dance performances and the pre-defined emotions is estimated using neural network. Finally, given arbitrary dance performances, their corresponding emotions are recognized using the neural network. The proposed method has more reliable performance by using multiple cameras unlike the privious methods using a single camera. The other purpose of this thesis is extracting sequential emotional patterns of Korean dance using the emotion recognition method. and proposing a criterion for specifying the characteristics of Korean dance quantitatively. The quantitative analysis can concrete the results of the existing researches on understanding the characteristics of Korean dance depend only on the qualitative analysis. We expect that this thesis provides a systematic framework for understanding the characteristics of the whole Korean dance through the qualitative and quantitative analysis of some representative Korean dances.

This study examined social anxiety and facial emotion recognition with and without face mask-wearing in a social threat sit- uation. A total of 309 adult males and females were screened using the Korean version of the Social Avoidance and Distress Scale (K-SADS) and the Korean version of the Center for Epidemiologic Studies Depression Scale-Revised (K-CES-DR). Of the participants, 56 individuals were divided into the social anxiety and non-social anxiety groups. Both groups were then ex- posed to threatening social situations. Using the signal detection theory, the sensitivity and cognitive bias in emotion recogni- tion with and without face mask-wearing was examined. Results showed that the social anxiety group had significantly lower perceptual sensitivity and more negative interpretation bias to faces, with or without face masks. Specifically, the perceptual sensitivity was significantly lower to faces with masks; however, there was no significant difference in interpretation bias. Al- though both groups showed no significant difference in reaction time with the emotional expression on masked faces, the so- cial anxiety group responded faster to faces wearing masks. As this is the first prospective study to investigate facial emotion recognition with face mask-wearing related to social anxiety, it provides valuable insight and data for future research.

There is a growing demand for emotion recognition systems (ERS) to be adopted in everyday life from various fields, particularly automotive, education, and social security. Recently, the use of cardio-based physiological signals, electrocardiogram (ECG), and photoplethysmogram (PPG) in ERS has yielded promising results. Furthermore, the development of wearable devices equipped with cardio-based physiological sensors has significantly aided towards the adoption of ERS in daily life. This paper systematically reviews emotion recognition using cardio-based physiological signals, encompassing emotion models, emotion elicitation methods, and ERS development methods, emphasizing feature extraction, feature selection methods, feature dimension reduction methods, and classifiers. A summary and comparison of recent studies are presented to highlight existing studies’ gaps and suggest future research for better ERS especially using cardio-based signals.

Speech emotion recognition (SER) is a critical field within affective computing, aiming to detect and classify emotional states from speech signals, which vary dynamically over time. These signals encode complex relationships between features at multiple time scales, effectively reflecting a speaker’s emotional state. Despite significant progress, SER faces the persistent challenge of labeled data scarcity, a major obstacle given the data-intensive requirements of deep learning models. This scarcity often results in small, imbalanced datasets that hinder model generalization. Various strategies, including feature selection, data augmentation, domain adaptation, and fusion techniques, have been employed to mitigate these issues. However, comprehensive reviews that critically analyze these methods remain limited. In this paper, we provide an extensive review of these data scarcity strategies in SER, assessing their merits and limitations in terms of efficiency and robustness. Special attention is given to how these strategies enhance the performance of both acoustic and multimodal SER systems when operating on limited datasets. Additionally, we highlight the potential of fusion strategies combined with attention mechanisms as promising solutions to improve convergence and reduce model complexity.

In the evolving field of emotion recognition, which intersects psychology, human?computer interaction, and social robotics, there is a growing demand for more advanced and accurate frameworks. The traditional reliance on single-modal approaches has given way to a focus on multimodal emotion recognition, which offers enhanced performance by integrating multiple data sources. This paper introduces TMNet, an innovative multimodal emotion recognition framework that leverages both speech and Electroencephalography (EEG) signals to deliver superior accuracy. This framework utilizes cutting-edge technology, employing a Transformer model to effectively fuse the CNN-BiLSTM and BiGRU architectures, creating a unified multimodal representation for enhanced emotion recognition performance. By utilizing a diverse set of datasets RAVDESS, SAVEE, TESS, and CREMA-D for speech, along with EEG signals captured via the Muse headband. The multimodal model achieves impressive accuracies of 98.89% for speech and EEG signal processing.

Recent multimodal emotion recognition (MER) approaches often assume that fusing audio and video modalities through cross-attention inherently yields superior feature representations for MER. However, our analysis reveals that different emotions manifest more prominently in specific modalities, leading to frequent misclassifications when an unsuitable modality dominates. Building on this, we propose a framework that dynamically adjusts each modality’s contribution using weighted integration of self & cross-attention for emotion, called WISE-Net. In addition, as certain emotions benefit from merging complementary audio?video information, we introduce a cross-label emotion matching task to explicitly capture fine-grained emotion-related distinctions across modalities. Experiments confirm that WISE-Net surpasses state-of-the-art methods. Ablation studies highlight the significance of balancing self-attention for modality expressiveness and cross-attention for emotion-aware feature fusion.

In this paper, we present facial expression recognition using feature-adaptive motion analysis and ID3 decision tree. Our method optimized the information gain heuristics of ID3 and consists of three parts: (1) simple geometric face model representation, (2) effective facial feature extraction using feature selection and feature-adaptive motion analysis, and (3) computationally inexpensive rule-based classification using ID3 tree. Our methods avoids complicated face model representation such as 3D modeling of human face, rather it uses FAC5-alike but much simpler face model by using fewer features and their actions. Feature selection and feature-adaptive motion analysis estimate motion patterns in fast and effective manner by assigning computational complexity on necessary parts only. Moreover, information gain heuristics of ID3 tree forces the classification to be done with minimal Boolean comparison. The performance was acceptable with overall recognition accuracy of 77% for JAFFE database (95 expressed images).

This paper proposed a Feature-level fusion technique that combines facial expression and audio modalities for multimodal emotion recognition. The learning model utilizes a hybrid approach combining CNN and LSTM to learn the spatiotemporal characteristics of video and audio modalities effectively. Compared to a unimodal approach, speech emotion recognition achieved 74% accuracy, and facial emotion recognition achieved 83% accuracy, while the proposed multimodal approach achieved 93% accuracy, demonstrated that multimodal emotion recognition is more accurate than unimodal emotion recognition. Furthermore, in tests using the RAVDESS dataset, the proposed model achieved higher emotion recognition rates compared to related studies. This study demonstrated the possibility of multimodal emotion recognition and designed a model capable of recognizing emotions in various environments and situations. Through this, we aim to contribute to the advancement of emotion recognition technology.

Facial emotion recognition (FER) has gained increasing attention in human–computer interaction and affective computing; however, existing methods often suffer from high computational cost and limited generalization, especially in real-world scenarios with subtle expressions and noisy inputs. To cope with these issues, this study proposes a knowledge distillation-based framework for FER. The teacher network utilizes a parallel architecture of NasNetMobile and MobileNet as dual backbones for comprehensive feature extraction, further, these features are enhanced by a Deformable Attention (DA) module, which primarily focuses spatial feature representations. To transfer this rich knowledge effectively, we introduce a lightweight student model, TinyNasNet, inspired by the internal architecture of NasNetMobile. In this framework, the student model is trained to mimic the behavior of the teacher network, aiming to achieve higher performance while maintaining computational complexity. Moreover, extensive experiments were conducted over two benchmarks, such as FER and KDEF. In contrast, the proposed network offers higher performance compared to various competitive networks, demonstrating a highly efficient yet robust solution for real-time FER.