Earticle

초록

영어

A Language model with neural networks commonly trained with likelihood loss. Such that the model can learn the sequence of human text. State-of-the-art results achieved in various language generation tasks, e.g., text summarization, dialogue response generation, and text generation, by utilizing the language model’s next token output probabilities. Monotonous and boring outputs are a well-known problem of this model, yet only a few solutions proposed to address this problem. Several decoding techniques proposed to suppress repetitive tokens. Unlikelihood training approached this problem by penalizing candidate tokens probabilities if the tokens already seen in previous steps. While the method successfully showed a less repetitive generated token, the method has a large memory consumption because of the training need a big vocabulary size. We effectively reduced memory footprint by encoding words as sequences of subword units. Finally, we report competitive results with token level unlikelihood training in several automatic evaluations compared to the previous work.

목차

Abstract
1. INTRODUCTION
2. METHODOLOGIES
2.1 Subword Tokenization
2.2 Unlikelihood Training
3. EXPERIMENTS
3.1 Experiment Setups
3.2 Evaluation Metrics
4. RESULTS AND DISCUSSIONS
5. CONCLUSION
Acknowledgment
References

키워드

저자

• Salahuddin Muhammad Iqbal [ Master Student, Department of Computer Engineering, Dongseo University, Korea ]
• Dae-Ki Kang [ Professor, Department of Computer Engineering, Dongseo University, Korea ] Corresponding Author

참고문헌

발행기관

간행물

표지보기 관심저널 등록

• 간행물명

  International Journal of Internet, Broadcasting and Communication

• 간기

  계간

• pISSN

  2288-4920

• eISSN

  2288-4939

• 수록기간

  2009~2025

• 십진분류

  KDC 326 DDC 380