Earticle

초록

영어

As many LLMs have been released, modified network layers based on transformer have been researched to improve performance. However, it is essential to design LLMs in a large size for performance, and as a result, current LLMs can only be executed on large servers, and various attempts have been made to reduce the amount of computation. In this paper, we present a method to reduce the amount of computation by using the data attribute of the SwiGLU layer used by meta and google. Since SwiGLU contains an activation function, it generates a large number of near-zero values, and we try to reduce the amount of computation by skipping unnecessary operations. Our experiments show that our algorithm can reduce the computation by 13.3% when there are 20% zeros from activation function.

목차

Abstract
I. INTRODUCTION
II. BACKGROUND
III. PROPOSED ARCHITECTURE
IV. RESULT & CONCLUSION
ACKNOWLEDGMENT
REFERENCES

키워드

저자

• Yunpyo Hong [ Korea Electronics Technology Institute SoC Platform Research Center ]
• Seokhun Jeon [ Korea Electronics Technology Institute SoC Platform Research Center ]
• Young-Jong Jang [ Korea Electronics Technology Institute SoC Platform Research Center ]
• Byung-Soo Kim [ Korea Electronics Technology Institute SoC Platform Research Center ] Corresponding Author

참고문헌

발행기관

간행물

표지보기 관심저널 등록

• 간행물명

  한국차세대컴퓨팅학회 학술대회

• 간기

  반년간

• 수록기간

  2021~2025

• 십진분류

  KDC 566 DDC 004