Earticle

초록

영어

With the development of mobile internet and social network, the scale of structured data have been increasing to PB level and above rapidly, while the query performance is greatly reduce. The efficiency of query optimization on large-scale datasets is currently a research focus in both academia and industry. In this paper, we present a distributed data management method, designed to improve query performance, called KCSQ. KCSQ analyses historical SQL commands, deduces statistics using frequency and the coupling degree of tables and table columns, and confirms the key column based on statistical evidence. When importing new tables into the HDFS, the data are divided into different blocks according to their key column. Any query on these columns can reduce the amount of data to be queried and the number of working nodes and thus effectively improves the throughput rate of the system.

목차

Abstract
 1. Introduction
 2. Related Work
 3. Key Column-based Split and Query (KCSQ)
 4. Design and Realization of KCSQ
  4.1 Sqoop
  4.2 Key Column-based Data Partition
  4.3 Storage and Application of Metadata
  4.4 The Generation Process of Efficient Query Tasks
 5. Case Verification
 6. Conclusion
 References

키워드

저자

• Xu Tao [ Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China ]
• Zhang Wei [ School of Computer Science, Beijing Information Science & Technology University, Beijing 100101, China, Beijing Key Laboratory of Internet Culture and Digital Dissemination Research, Beijing Information Science & Technology University，Beijing 100101, China ]
• Li Baolu [ School of Computer Science, Beijing Information Science & Technology University, Beijing 100101, China ]

참고문헌

발행기관

간행물

표지보기 관심저널 등록

• 간행물명

  International Journal of Database Theory and Application

• 간기

  격월간

• pISSN

  2005-4270

• 수록기간

  2008~2016

• 십진분류

  KDC 505 DDC 605