Earticle

초록

영어

The pre-training language model, BERT has achieved a great success in natural language processing by transferring knowledge from rich-resource pre-training task to the low-resource downstream tasks. The model was recognized as a breakthrough or an innovative technology that changed the paradigm of natural language processing. In this paper, a number of studies have been analyzed to classify and compare the research directions. We examined a technical challenges after BERT. In the pre-training process, self-supervised learning is performed, which relies entirely on training data. If we introduce the linguistic knowledge in the course of the training, it would be possible to get better result more effectively. Therefore, it is necessary to develop a method to insert external knowledge such as linguistic information in the training process. The mask language model and the next sentence prediction are being used in BERT's pre-training tasks. Though, to get much deeper understanding of the natural language, some other effective methods are to be studied and developed. Lastly, we should aim to develop eXplainable Artificial Intelligence (XAI) technology in natural language processing, helping us look into the transparent processing. The pre-trained language model focuses on the development of skills that can be used for all the tasks of natural language understanding. A lot of researches are focused on how to adapt language model effectively to downstream tasks based on common language models, even with the case with little data. It is also hoped that the technical analysis reviewed in this study will provide linguists and computer researchers with an opportunity to understand recent technological achievements in the field of natural language processing and to seek joint research.

목차

1. 서론
2. 언어모델의 종류
2.1 언어모델의 확률표현
2.2 통계적 N-gram 모델
2.3 단어의 분산표현(Distributive Representation) 언어모델
2.4 부호기 복호기(Encoder-Decoder) 방식의 언어 모델
2.5 주의 기구(Attention Mechanism)를 이용한 모델
3. 전이 학습(Transfer Learning)과 사전 학습(Pre-trained Learning)
3.1 전이 학습 (Transfer Learning)
3.2 자기지도 학습(self-supervised learning)
3.3 사전학습 언어모델
4. 사전학습 언어모델의 응용과 기술 분석
4.1 연구 동향
4.2 사전학습 언어모델에 대한 연구과제
5. 결론
인용문헌
[Abstract]

저자

• 지인영 [ In-Youn Jhee | 한국체육대학교 교양교직학부, 교수 ] 제1저자
• 김희동 [ Hee-Dong Kim | 한국외국어대학교 공과대학 정보통신공학과, 교수 ] 교신저자

참고문헌