Earticle

초록

영어

Hadoop Distributed File System (HDFS) becomes a representative cloud platform, benefiting from its reliable, scalable and low-cost storage capability. However, HDFS does not present good storage and access performance when processing a huge number of small files, because massive small files bring heavy burden on NameNode of HDFS. Meanwhile, HDFS does not provide any optimization solution for storing and accessing small files, as well as no prefetching mechanism to reduce I/O operations. This paper proposes an optimized scheme, Structured Index File Merging-SIFM, using two level file indexes, the structured metadata storage, and prefetching and caching strategy, to reduce the I/O operations and improve the access efficiency. Extensive experiments demonstrate that the proposed SIFM can effectively achieve better performance in the terms of the storing and accessing for a large number of small files on HDFS, compared with native HDFS and HAR.

목차

Abstract
 1. Introduction
 2. Related Work
 3. Analysis of Small File Problem on HDFS
  3.1. Architecture and Access Mechanism of HDFS
  3.2. Impact on HDFS
 4. Optimization Scheme -- SIFM
  4.1. File Merging Strategy
  4.2. Metadata Files Storage
  4.3. Prefetching and Caching Files
 5. Efficiency Analysis
  5.1. Writing Efficiency Analysis
  5.2. Reading Efficiency Analysis
 6. Experiment Evaluation
  6.1. Experimental Settings
  6.2. Experimental Methodology
  6.3. Experimental Results
 7. Conclusion
 Acknowledgments
 References

키워드

저자

• Yingchi Mao [ College of Computer and Information Engineering, Hohai University, Nanjing, 211100, China, Huaian Research Institute of Hohai University, Huaian, China ]
• Bicong Jia [ College of Computer and Information Engineering, Hohai University, Nanjing, 211100, China ]
• Wei Min [ College of Computer and Information Engineering, Hohai University, Nanjing, 211100, China ]
• Jiulong Wang [ College of Computer and Information Engineering, Hohai University, Nanjing, 211100, China ]

참고문헌

발행기관

간행물

표지보기 관심저널 등록

• 간행물명

  International Journal of Database Theory and Application

• 간기

  격월간

• pISSN

  2005-4270

• 수록기간

  2008~2016

• 십진분류

  KDC 505 DDC 605