Earticle

초록

영어

Document similarity is a common task to a variety of problems such as clustering, unsupervised learning and text retrieval. It has been seen that document with the very similar content provides little or no new information to the user. This work tackles this problem focusing on detecting near duplicates documents in large corpora. In this paper, we are presenting a new method to compute pairwise document similarity in a corpus which will reduce the time execution and save space execution resources. Our method group shingles of all documents of a corpus in a relation, with an advantage of efficiently manage up to millions of records and ease counting and aggregating. Three algorithms are introduced to reduce the candidates shingles to be compared: one creates the relation of shingles to be considered, the second one creates the set of triples and the third one gives the similarity of documents by efficiently counting the shared shingles between documents. The experiment results show that our method reduces the number of candidates pairs to be compared from which reduce also the execution time and space compared with existing algorithms which consider the computation of all pairs candidates.

목차

Abstract
 1. Introduction
 2. Proposed Method
  2.1 Reducing Computation Time
  2.2 Create the Relation for Pairwise Computation
  2.3 Similarity Calculation
 3. Experimental Evaluation
  3.1 Reducing Candidates Pairs
  3.2 Time Complexity Evaluation
  3.3 Effectiveness
 4. Conclusion
 Acknowledgements
 References

키워드

저자

• Papias Niyigena [ School of Information Science and Engineering, Central South University, Changsha, 410083, China ]
• Zhang Zuping [ School of Information Science and Engineering, Central South University, Changsha, 410083, China ]
• Weiqi Li [ School of Electronic and Information Engineering, Xi’an Jiaotong University Xian, 710049, China ]
• Jun Long [ School of Information Science and Engineering, Central South University, Changsha, 410083, China ]

참고문헌

발행기관

간행물

표지보기 관심저널 등록

• 간행물명

  International Journal of Database Theory and Application

• 간기

  격월간

• pISSN

  2005-4270

• 수록기간

  2008~2016

• 십진분류

  KDC 505 DDC 605