Earticle

In everyday communication, sign language is vital for those unable to speak. However, sign language education faces challenges such as mobility risks as well as limitations in solo practice. Recent attention has turned to Virtual Reality (VR) technology, which has shown positive effects in healthcare and rehabilitation. VR environments enhance learner immersion and support various physical activities, similar to traditional methods. Despite this, research on VR-based sign language education content is limited. This study proposes an immersive educational system that enables users to practice sign language in a virtual environment using a Head Mounted Display (HMD) and Leap Motion Controller(LMC) 2, with the aim of establishing a foundation for future VR sign language education content.

The problem of discrimination due to information alienation among Korean sign language users is continuously mentioned, and sign language translation research is actively being conducted to solve this problem. However, due to technological limits in translating text into Korean Sign Language, the need for specialist equipment causes annoyance and spatial constraints. Furthermore, it isn't easy to replicate the vocabulary and grammatical structure of the Korean language in Korean Sign Language. Furthermore, the service's commercialization is complicated by the need for more nonmanual signal (NMS) identification technology.

본 논문에서는 농인들의 일상 소통을 개선하고 비장애인들의 수화 교육을 위해 AI를 활용하여 수화 통역 과 교육을 할 수 있는 수화 번역 프로그램을 개발한다. 기존의 수화 언어는 한국어와 동등한 언어로 인정되었지 만, 수화 통역 및 교육은 여전히 부족한 상황이다. 농인들은 일상 소통에서 언어적인 격차와 문화적인 어려움을 경험하며, 특히 일반 학교에서 수업을 받는 농인들은 어려움을 겪고 있다. 수화 번역 프로그램은 미디어파이프, OpenCV와 같은 AI 기술을 활용하여 손 동작을 인식하여 수화로 번역한다. 손 동작은 KNN을 사용하여 분석되 어 수화를 인식하고 수화 교육 플랫폼 및 의사소통 도구로 활용된다. 이 연구는 농인들의 일상생활의 불편함을 최소화하고 의사소통과 사회적 참여를 촉진하며, 수화 교육의 접근성을 높이는 데 기여할 것으로 기대된다.

In this paper, we develop a sign language translation program that can use AI for sign language interpretation and education to improve daily communication among farmers and to educate non-disabled people about sign language. Although the existing sign language has been recognized as an equivalent language to Korean, sign language interpretation and education is still lacking. Farmers experience linguistic gaps and cultural challenges in their daily communication, especially those who are educated in mainstream schools. The sign language translation program utilizes AI technologies such as MediaPipe and OpenCV to recognize hand gestures and translate them into sign language. The hand gestures are analyzed using KNN to recognize sign language and used as a sign language education platform and communication tool. This research is expected to contribute to minimizing the inconvenience of farmers' daily life, facilitating communication and social participation, and increasing the accessibility of sign language education. Translated with DeepL.com (free version)

청각장애인들은 수어통역 없이 상담서비스를 이용하기에는 어려움이 있다. 수어 통역사 인력이 많이 부족 하여 수어 통역사까지 상담이 연결되는데 많은 시간이 걸리거나 연결이 되지 않는 경우가 많이 발생하고 있다. 이에 본 논문에서는 OpenCV와 CNN(Convolutional Neural Network)을 이용하여 수어를 영상으로 촬영하고 수어 동작을 인식하여 수어가 뜻하는 의미를 텍스트 형태의 데이터로 변환하여 사용자에게 제공하는 시스템을 제안한다. 상담사는 저장된 수어번역 상담내용을 열람하여 상담을 진행할 수 있다. 전문 수어 통역사가 없어도 상담이 가능하여 수어 통역사를 기다려야 하는 부담을 줄일 수 있다. 제안 시스템을 청각장애인 상담서비스에 적 용할 경우 상담 효과의 향상과 향후 청각장애인 상담에 관한 학문적 연구를 촉진하는 계기가 될 것으로 기대된다.

It is difficult for the hearing impaired to use the counseling service without sign language interpretation. Due to the shortage of sign language interpreters, it takes a lot of time to connect to sign language interpreters, or there are many cases where the connection is not available. Therefore, in this paper, we propose a system that captures sign language as an image using OpenCV and CNN (Convolutional Neural Network), recognizes sign language motion, and converts the meaning of sign language into textual data and provides it to users. The counselor can conduct counseling by reading the stored sign language translation counseling contents. Consultation is possible without a professional sign language interpreter, reducing the burden of waiting for a sign language interpreter. If the proposed system is applied to counseling services for the hearing impaired, it is expected to improve the effectiveness of counseling and promote academic research on counseling for the hearing impaired in the future.

수어는 음성이 아닌 시각적 제스처(Gesture)를 매체로 의미를 전달한다. 시각적인 제스처를 이용하여 의미를 주고 받기 어려운 단어나 고유명사를 전달하기 위해서는 손과 손가락의 모양으로 한글의 자음과 모음을 표현한 지문자가 주로 사용된다. 본 연구에서는 MediaPipe Hands API를 이용하여 손 이미지로부터 Landmark를 추출하고 scikit-learn SVM(Support Vector Machine)을 이용하여 31개의 한글 지문자에 대한 학습 모델을 생성한다. ‘ㄲ’과 같은 지문자는 ‘ㄱ’에 대한 지문자가 오른쪽으로 이동하는 제스처로 표현되는데, 제안한 방법에서는 ‘ㄱ’ 무게 중심이 이동하는 것을 검출하여 ‘ㄲ’을 인식할 수 있다. 기존의 연구에서는 특정 수어 문장 또는 특정 지문자를 제한 적으로 인식하는 연구가 주를 이루고 있지만, 제안한 방법에서는 한글 지문자를 구성하는 36개의 지문자를 인식할 수 있기 때문에 청각장애인들의 지문자 학습, 수어 번역 등을 위해 유용하게 이용될 수 있을 것으로 기대된다.

Sign Language are languages that use a visual means of communicating using gestures and facial expression. Finger spelling uses hand movements and shapes to spell out the proper nouns or words, which are difficult to be represented. In this study, Korean finger spelling recognition system is proposed. Landmarks are extracted from hand images by utilizing MediaPipe Hands API. Also SVM(Support Vector Machine) of scikit-learn is used to train 31 Korean finger spellings. Finger spelling such as a double consonant ‘ㄲ’ is represented by moving ‘ㄱ’ in the right direction. The proposed method recognized ‘ㄲ’ by detecting the movement of the bounding box centroid for finger spelling. Most of the previous methods are not able to recognize all the finger spellings for Korean sign language, but the proposed technique can recognize all 36 Korean finger spellings. Also, it is expected that our methods are utilized in finger spelling learning or sign language translation for hearing-impaired persons.

The communication gap between the deaf and public is the concern for both parents and the government of Bhutan. The deaf school urges people to learn Bhutanese Sign Language (BSL) but learning Sign Language (SL) is difficult. This paper presents the BSL digits recognition system using the Convolutional Neural Network (CNN) and a first-ever BSL dataset which has 20,000 sign images of 10 static digits collected from different volunteers. Different SL models were evaluated and compared with the proposed CNN model. The proposed system has achieved 97.62% training accuracy. The system was also evaluated with precision, recall, and F1-score.

The enactment of Republic Act 11106 establishes Filipino Sign Language (FSL) as the primary mode of communication within the deaf-mute community in the Philippines. However, this legal recognition has highlighted a significant communication gap between the deaf-mute and non-deaf-mute populations, as the latter typically does not understand FSL. This study introduces a mobile application designed to bridge this gap by translating FSL gestures into textual sentences. The application leverages a CNN-BiLSTM deep learning architecture integrated with Mistral 7B, a state-of-the-art Large Language Model (LLM), to recognize continuous multi-sign gestures and translate them into coherent text. To evaluate the system’s effectiveness, two gesture recognition models were compared based on Word Error Rate (WER), calculated using the Levenshtein distance to measure word-level discrepancies. The 1080p30 model with a stride of 5 and a window size of 30 frames achieved a WER of 27.02%, while the 720p60 model achieved, with a stride of 5 and a window size of 60 frames, a WER of 43.37%. The superior performance of the 1080p30 model is attributed to its higher spatial resolution. This research addresses the critical need for accessible communication tools, offering a solution that enhances inclusivity for the Filipino deaf community.

Ce document vise a approfondir des questions generales relatives a l'interpretation de Ia Langue des Signes TaYwanaise, et a Ia profession des interpretes en langue des signes, mais se concentre egalement sur certains aspects interpretatifs difficiles, comme le discours figuratif, les metaphores et l'iconicite. En outre, il propose une structure academique pour reconnaitre le statut professionnel des interpretes en langue des signes qui ne differe pas de celui des interpretes oraux, non seulement in theoria, mais aussi de facto. Les donnees recueillies a partir de ces recherches contribueront a ameliorer les connaissances des interpretes sur leur propre profession et leur statut professionnel. Cette etude est egalement le premier article sur l'interpretation en Langue des Signes Taiwanaise.

The purpose of this paper is to examine the pronominal system and reflexives in Korean Sign Language (KSL). In order to do this, I first take a close look at KSL word order. Second, I investigate whether the KSL pronominal system employs number, person, and gender, comparing their use with that of spoken languages and other sign languages. Third, I examine KSL reflexives, comparing their use with that of spoken languages and other sign languages. Although there are several different reflexive forms, I mainly consider two kinds of reflexives: an adverbial reflexive pronoun <honca> ‘self’ derived from an independent word as an adverb, and a nominal reflexive pronoun <casin-ul pota> ‘self’ derived from a complex verb phrase consisting of an active verb and a backward verb. I also examine whether KSL reflexives are subject to locality by presenting the syntactic structure of complex clauses in KSL. Finally, I will give a brief summary of this paper in conclusion

Image-based sign language recognition frequently exhibits high computational costs and is susceptible to background noise. In this study, we introduce an efficient skeleton-based framework that utilizes 3D landmarks and transformers. A pivotal component of this approach is the creation of similar landmarks method, an ensemble technique that extracts and averages landmarks from multiple augmented video views. This approach enhanced the system's resilience to noise and missing coordinates. The model was evaluated using quadruple crossvalidation on a 30-word dataset combining AI-Hub (studio) and KSL (wild) data. The results demonstrated that the technique applying ensembles produced superior results in comparison to the technique not applied.

This study presents an innovative approach to bidirectional sign language translation using an AI-based 3D interactive Unity character. We have developed a system that receives Korean text input, translates it into Korean Sign Language(KSL), and then uses Unity3D to enable a 3D character to perform sign language gestures in real-time. The system aims to deliver natural character movements while accurately reflecting the vocabulary and grammatical structure of KSL. Although the representation of non-manual signals(NMS) remains a significant challenge, the combination of Unity3D characters with AI-based translation plays a crucial role in addressing the issue of information alienation among KSL users.

본 연구의 목적은 우리나라 청인 수어통역사들이 인식하고 있는 수어통역서비스 전달체계의 문제점을 파악함과 동시에 수어통역 업무와 관련된 수어통역사들의 욕구를 도출하여 농인이 만족할 수 있는 질 높은 수어통역을 위한 실제적인 수어통역서비스 전달체계 개선방안을 제언하는 데에 있다. 본 연구는 문헌고찰을 바탕으로 초점집단인터뷰를 실시하고 그에 따른 결과 분석으로 진행되었다. 우리나라의 수어통역 및 수어통역서비스 전달체계에 대한 전반적인 내용은 국내외 문헌고찰을 통해 살펴보았으며 초점집단인터뷰 결과로 도출된 자료는 그 내용을 범주화하여 분석하였다. 문헌고찰 결과, 한국의 수어통역서비스 전달체계에서 가장 큰 비중을 차지하고 있는 것은 ‘수어통역센터’와 ‘통신중계센터’였다. 이외에도 농인들이 많이 재학하고 있는 대학이나 농인들의 이용 욕구가 큰 공공기관, 농인들을 고용한 기업 등이 있었다. 초점집단인터뷰 결과 청인 수어통역사가 인식하는 수어통역서비스 전달체계는 ‘업무 환경’, ‘통역서비스 수행과 연계’, ‘운영체계’, ‘개선을 위한 제안’ 등 총 4개의 상위범주로 분류되었고 16개의 하위범주와 47개의 개념이 도출되었다. 결론으로 수어통역서비스 전달체계 개선방안을 제안하면 다음과 같다. 첫째, 수어통역사의 양적 확충과 더불어 열악한 근무환경을 개선해야 한다. 둘째, 통합과 조정을 통한 획기적인 수어통역서비스 전달체계의 개편을 실행해야 한다. 셋째, 수어통역사 양성 출구와 과정을 혁신적으로 개선하여 수어통역사의 질적 성숙을 기할 수 있는 방안 모색을 고민해야 한다. 넷째, 현장 중심의 수어통역서비스 매뉴얼과 지침이 구체적으로 개발되어야 한다. 다섯째, 효과적이며 효율적인 수어통역사 관리 운영체계를 구체적으로 수립해야 한다. 여섯째, 이용이 편리한 수어통역서비스 체제를 확립해야 한다.

The purpose of this study is to comprehend the issues of sign language interpretation service delivery system recognized by sign language interpreters and draw out their needs in tasks related to sign language interpretation. Through this, practical plans to reform sign language interpretation service delivery system for high quality sign language interpretation, which the people with hearing impairment are satisfied with, are proposed. The research method of literature review and focus group interview was used in this study. The general details on sign language interpretation and sign language interpretation service delivery system of Korea was examined through literature review and the data derived from the results of focus group interview categorized the details for analysis. As the result of literature review the largest portion of sign language interpretation service delivery system of Korea was ‘sign language interpretation center’ and ‘communication transmission center’. And besides these two, there were universities with many people with hearing disability, major public institutions with high level of needs by people with hearing disability, companies employing people with hearing disability. 4 high categories in perception of sign language interpreter on sign language interpretation service delivery system were identified through focus group interview which are ‘work environment’, ‘execution and connection of interpretation service’, ‘operating system’, ‘contents suggestion’. Moreover, 16 sub categories and 47 concepts were derived as well. In conclusion, proposals for reform in sign language interpretation service delivery system are first, expansion of sign language interpreter in number and improvement of their poor work environment. Second, ground-breaking reorganizing of sign language interpretation service delivery system through unification and adjustment should be put in practice. Third, plans that promise enhancement of quality of sign language interpreter should be put under mature consideration through innovative reform of sign language interpreter training system and its process. Fourth, the development of detailed field centered sign language interpretation service manual and guideline should be done. Fifth, effective and efficient sign language interpreter management operating system should be established with specific details. Last, sign language interpretation service system with convenient use should be developed.