Earticle

Data is growing day by day all over the world. The word ''Big-data'' refers to a collection of data sets that are huge and complicated. The data is usually measured in Petabyte or Exabyte. Big data is one of the most talked-about topics in the field of IT. Its role in the future will be very considerable. The massive use of the internet, smartphones and social media has resulted in an increase in data. The usage of such gadgets and the internet not only increases the volume of data but also increases market velocity by allowing into to be moved and shared at light speed across optical fiber and wireless networks. Numerous problems arise as a result of the rapid creation of a large amount of data. So, we presented important concepts of Big-data in this paper. The basic purpose of this paper was to explore Big-data, challenges, and different tools related to it. We studied the previous work in which all writers have well-described big data, its challenges, and how to deal with them. And we have achieved our goal by providing a platform that explored big data challenges and tools for big data storage in detail. Moreover, we have compared these tools by identifying their parameters. In the future, we will find out solutions for these challenges. Furthermore, this paper will open new doors for researchers to explore Big-data, and develop solutions for challenges and unsolved research questions.

그래는 생물학, 화학, 사회 계망 데이터 등 복잡한 구조의 데이터를 모델링하는데 리 사용되는 자료구조이다. 이러한 그래 데이터에서 서로 유사한 데이터를 찾아내는 것은 매우 다양한 응용에서 필수으로 요구되는 작업이 다. 본 논문에서는 화학 그래에 을 맞추어, 필터링-검증 임워크 기반의 유사도 기반 그래 질의 처리 기 법을 제안한다. 계산량이 많은 그래 데이터의 특징에 맞는 필터링 방법을 제안하고, 후보 그래들을 효율으로 검증하기 한 기법을 제안한다. 한, 규모 데이터에서 실시간 분석이 가능할 수 있도록 MapReduce를 이용 하는 분산처리 기법을 제안하고, 실 데이터에 한 실험을 통해 본 논문에서 제안하는 방법이 유사도 기반 그래 질의 처리의 성능을 크게 개선함을 입증한다.

Graph is widely used to model complex structured data such as biomedical data, chemical data, and social network data. It is essential to find graphs similar to a given query in many applications. In this paper, we focus on chemical graph data and propose a graph similarity query processing technique based on a filtering and verification framework. We propose a filtering technique considering the heavy computation requirement of graph data and develop a novel technique to efficiently verify candidate graphs. We also propose a distributed processing framework using MapReduce to support large scale graph data. Through the experiments on real chemical data, we show that the proposed technique substantially improves the performance of query processing.

Attempts to hack online game servers and abusing problems in online games have been issues in gameindustry. Mobile games are famous these days thanks to the widespread of smart devices. Unfortunately, mobilegames have very short life cycle. Therefore, analysis of game log data becomes more important to overcome thehacking and abusing problems in online games and extend their life cycles in mobile games. In this paper, wepropose a new game log data analysis technique based on the MapReduce methodology. MapReduce is a widelyused programming model for analyzing and processing Big data. Instead of processing each analysis queryseparately, the proposed technique processes multiple analysis queries together in a batch by a single, optimizedMapReduce job. As a result, the number of queries processed per unit time increases significantly. Experimentresults show that the proposed technique improves the performance significantly compared to a naive method.

This study measured the stream data from the several sensors, and stores the database in MapReduce framework environment, and it aims to design system with the small performance and cluster analysis error rate through the KMSVM algorithm. Through the KM-SVM algorithm, the cluster analysis effective data was used for U-health system. In the results of experiment by using 2003 data sets obtained from 52 test subjects, the k-NN algorithm showed 79.29% cluster analysis accuracy, K-means algorithm showed 87.15 cluster analysis accuracy, and SVM algorithm showed 83.72%, KM-SVM showed 90.72%. As a result, the process speed and cluster analysis effective ratio of KM- SVM algorithm was better.

편집거리는 근사문자열매칭의 대표적인 점수척도로, 길이가 m, n인 두 문자열에 대한 편집거리는 동적프로그래밍 을 이용하여 O(mn) 시간에 계산할 수 있다. 편집거리 계산을 위한 다양한 알고리즘이 연구되고 있으며 그 중에는 병렬 알고리즘에 대한 연구도 포함되어 있다. 본 논문에서는 두 문자열에 대한 편집거리를 계산하는 맵리듀스 알고 리즘을 제시한다. O(mn) 시간에 동작하는 Wagner와 Fischer의 알고리즘과 O(mn/t) 시간에 동작하는 4-러 시안 알고리즘을 각각 맵리듀스 알고리즘으로 변환하고 설명한다. 그리고 맵리듀스 알고리즘들에 대한 이론적인 비용 분석과 함께 제안하는 알고리즘들이 순차 알고리즘보다 시간 효율성을 갖기 위한 조건도 제시한다.

The edit distance metric is one of the most widely used scoring metric for the approximate string matching. Given two strings with lengths m and n, we can compute the edit distance between them in O(mn) time using dynamic programming technique. There are several algorithms for the edit distance problem and some of them are parallel algorithms. In this paper, we present two MapReduce algorithms for the edit distance problem between two strings. We convert the Wagner and Fischer algorithm and the Four-Russians algorithms to the MapReduce algorithms and explain them. In addition, we explain some theoretical analysis for our algorithms using a cost model for MapReduce.

맵리듀스는 하둡의 필수 핵심 기술로 하둡 분산 파일 시스템을 기반으로 빅데이터를 처리하는 가장 보편화되어 사용되고 있다. 그러나 기존 맵리듀스 기반 빅데이터 처리 기법은 하둡 분산 파일 시스템에 정해진 블록의 크기대로 파일 나눠 저장되는 특징으로 인해 인프라 자원의 낭비가 극심하다. 이에 본 논문에서는 효율적인 맵리듀스 기반 빅데이터 처리 기법을 제안한다. 제안하는 기법은 처리할 데이터를 사전에 맵리듀스에서 처리하기 적합한 데이터 형태로 변환 및 압축하여 빅데이터 인프라 환경의 저장 효율성을 증가시킨다. 또한 제안하는 기법은 저장 효율성을 중점으로 구현했을 때 발생할 수 있는 데이터 처리 시간의 지연 문제를 해결한다.

MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.

최근 데이터의 폭발적인 증가로 인해 대규모 데이터의 분석에 대한 요구를 충족할 수 있는 방 법들이 계속 연구되고 있다. 본 논문에서는 맵리듀스를 이용한 분산 병렬 처리를 통해 대규모 데이터 큐브의 효율적인 계산이 가능한 MRIterativeBUC 알고리즘을 제안하였다. MRIterativeBUC 알고 리즘은 기존의 BUC 알고리즘을 맵리듀스의 반복적 단계에 따른 효율적인 동작이 가능하도록 개발되 었고, 기존의 대규모 데이터 큐브 계산에 따른 문제인 데이터 크기와 저장 및 처리 능력의 한계를 해 결하였다. 또한, 분석자의 관심 부분에 대해서만 계산하는 빙산 큐브 개념의 도입과 파티셔닝, 정렬과 같은 큐브 계산을 분산 병렬 처리하는 방법 등의 장점들을 통해 데이터 방출량을 줄여서 네트워크 부 하를 줄이고, 각 노드의 처리량을 줄이며, 궁극적으로 전체 큐브 계산 비용을 줄일 수 있다. 본 연구 결과는 맵리듀스를 이용한 데이터 큐브 계산에 대해서 상향식 처리와 반복적 알고리즘을 통해 다양한 확장이 가능하며, 여러 응용 분야에서 활용이 가능할 것으로 예상된다.

Due to the recent data explosion, methods which can meet the requirement of large data analysis has been studying. This paper proposes MRIterativeBUC algorithm which enables efficient computation of large data cube by distributed parallel processing with MapReduce framework. MRIterativeBUC algorithm is developed for efficient iterative operation of the BUC method with MapReduce, and overcomes the limitations about the storage size and processing ability caused by large data cube computation. It employs the idea from the iceberg cube which computes only the interesting aspect of analysts and the distributed parallel process of cube computation by partitioning and sorting. Thus, it reduces data emission so that it can reduce network overload, processing amount on each node, and eventually the cube computation cost. The bottom-up cube computation and iterative algorithm using MapReduce, proposed in this paper, can be expanded in various way, and will make full use of many applications.

The remote sensing images are massively stored, so it is difficult for the traditional single-node mode to meet the real-time requirement for remote sensing image retrieval. In order to improve remote sensing image retrieval efficiency and accuracy, a kind of feature information MapReduce based remote sensing image retrieval algorithm is proposed in this article. Specifically, the color features and the texture features of the remote sensing image are firstly extracted, and then Map function is adopted to calculate the similarity between the remote sensing image to be retrieved and the image in the feature library according to the color features and the texture features, and finally Reduce function is adopted to collect the intermediate results of various node tasks and the remote sensing images are ranked by a descending order according to the similarity in order to obtain the remote sensing image retrieval result. The test result shows that the proposed algorithm can rapidly and accurately retrieve the remote sensing image, thus not only improving the remote sensing image retrieval efficiency, but also improving the remote sensing image retrieval accuracy.

MapReduce, a large-scale data processing paradigm, is gaining popularity. However, like other distributed computing frameworks, MapReduce suffers from the integrity assurance vulnerability: malicious workers in the MapReduce cluster could tamper with its computation result and thereby render the overall computation result inaccurate. Existing solutions are effective in defeating the malicious behavior of non-collusive workers, but are less effective in detecting collusive workers. In this paper, we propose the Verification-based Integrity Assurance Framework (VIAF). By using task replication and probabilistic result verification, VIAF can detect both non-collusive and collusive workers, even if the malicious workers dominate the environment. We have implemented VIAF on Hadoop, an open source MapReduce implementation. Our theoretical analysis and experimental result show that VIAF can achieve high job accuracy while imposing moderate performance overhead.

MapReduce is an effective programming model for large-scale data-intensive computing applications. Hadoop, an open-source implementation of MapReduce, has been widely used. The communication overhead from the big data sets’ transmission affects the performance of Hadoop greatly. In consideration of data locality, Hadoop schedules tasks to the nodes near the data locations preferentially to decrease data transmission overhead, which works well in homogeneous and dedicated MapReduce environments. However, due to practical considerations about cost and resource utilization, it is common to maintain heterogeneous clusters or share resources by multiple users. Unfortunately, it’s difficult to take advantage of data locality in these heterogeneous or shared environments. To improve the performance of MapReduce in heterogeneous or shared environments, a data prefetching mechanism is proposed in this paper, which can fetch the data to corresponding compute nodes in advance. It is proved that the proposal of this paper reduces data transmission overhead effectively with theoretical analysis. The mechanism is implemented and evaluated on Hadoop-1.0.4. Experiment results on real applications show that the data prefetching mechanism can reduce data transmission time by up to 94%.

As a widely used programming model for the purposes of processing large data sets, MapReduce (MR) becomes inevitable in data clusters or grids, e.g. a Hadoop environment. Load balancing as a key factor affecting the performance of map resource distribution, has recently gained high concerns to optimize. Current MR processes in the realization of distributed tasks to clusters use hashing with random modulo operations, which can lead to uneven data distribution and inclined loads, thereby obstruct the performance of the entire distribution system. In this paper, a virtual partition consistent hashing (VPCH) algorithm is proposed for the reduce stage of MR processes, in order to achieve such a trade-off on job allocation. Besides, experienced programmers are needed to decide the number of reducers used during the reduce phase of the MR, which makes the quality of MR scripts differ. So, an extreme learning method is employed to recommend potential number of reducer a mapped task needs. Execution time is also predicted for user to better arrange their tasks. According to the results, VPCH can lead to load balancing and our prediction model can provide fast prediction than SVM with similar accuracy maintained.

Evolution of IT and computer has made e-books popular day by day. In this paper, we are interested in searching a word in e-books. However, it is impossible to search a word in digitized e-books if they consist of image files such as JPG and PDF. Our solution to this problem is to transform the image file based e-books into text files based e-books to enable searching a word in e-books. We use EPUB, a XML-based text file, which is defined by IDPF(International Digital Publishing Forum). That is, we convert the image file based e-books into EPUB format e-books, so that searching a word in e-books can be done without any problem. The converting job should deal with very big data usually and require a lot of computing power. If we do the conversion in an usual personal computer, it would take a lot of processing time or it might be impossible for us to complete it. We used MapReduce model with a cluster system which enables us to perform the conversion successfully and reduce the processing time. This paper presents our Hadoop-based e-book Conversion System which is a distributed computing framework to transform the image based e-books into EPUB format e-books. Our experimental system consists of up to 15 cluster nodes. This paper evaluates the performance of the experimental system which processes the conversion of up to 2TB(Terra Byte) image files into EPUB files with a 15 nodes cluster system. We analyzed the processing time when the number of nodes in the cluster system was varied. We also analyzed the improvement effect when the dpi of the image file was varied. The performance evaluation confirmed us that the Hadoop-based e-book Conversion System successfully processed the big data for e-book.

Though MapReduce programming model simplifies the development of parallel program, ordinary users have difficulties in setting up the development environment for MapReduce. The online integrated development environment for MapReduce programming can solve this problem, thus users need not build the environment themselves, only need to focus on the logical design of the parallel program. During the software construction, the problem of independent space setting and naming conflict of the file in the multi-user environment, and the problem of online compiling, execution and instant feedback message to client are solved. The software has been deployed and tested in Hadoop cluster, and can meet users’ basic requirements for the development of MapReduce.

The job scheduling for Map Reduce clusters has received significant attention in recent years, because it plays an important role on Map Reduce clusters. Traditional job scheduling performs poorly in assigning a task to appropriate nodes, and can not predict the resource utilization of the unexecuted tasks. To address the problems, an efficient job scheduling for Map Reduce clusters is proposed in this paper. The job scheduling introduces dynamic priority scheduling and real-time prediction model. Dynamic priority scheduling introduces the minimum cost data locality algorithm with a weight to deal with different size jobs, and real-time prediction model can predict the resource utilization of unexecuted tasks by calculating the running tasks. The resource utilization contains CPU, memory, and network. Experimental results prove that the proposed job scheduling is able to perform well in Map Reduce clusters.

MapReduce is a popular framework for processing large datasets in parallel over a cluster. It has gained wide attention for its high scalability, reliability and low cost. However, its performance may be degraded by excessive network traffic when processing jobs, for such two problems as data locality in reduce task scheduling and partitioning skew. We propose a Minimum Transmission Cost Reduce task Scheduler (MTCRS) based on sampling evaluation to solve the two problems. The MTCRS takes the waiting time of each reduce task and the transmission cost set as indicators to decide appropriate launching locations for Reduce tasks. The transmission cost set is computed by a mathematical model, in which the parameters are the sizes and the locations of intermediate data partitions generated by Average Reservoir Sampling (ARS) algorithm. The experiments show that the MTCRS reduces network traffic by 8.4% compared with Fair scheduler.

Data analyzing and processing are important tasks in cloud computing. In this field, the MapReduce framework has become a more and more popular tool to analyze large-scale data over large clusters. Compared with the parallel relational database, it has the advantages of excellent scalability and good fault tolerance. However, the performance of join operation using MapReduce is not as good as that of parallel relational database. Thus, how to optimize theta-join operations using MapReduce is an attractive point to which researchers have been paying attention. In this paper, a randomized algorithm named Strict-Even-Join(SEJ) is designed to solve the multi-way theta-joins in a single MapReduce job. Moreover, a dynamic programming algorithm is elaborated to optimize the multi-way theta-joins by calling the SEJ algorithm. The results of experiments show that our approach is feasible and effective.

With manufacturing technology developing persistently, hardware manufacturing cost becomes lower and lower. More and more computers equipped with multiple CPUs and enormous data disk emerge. Existing programming modes make people unable to make effective use of growing computational resources. Hence cloud computing appears. With the utilization of Map Reduce parallelized model, existing computing and storage capabilities are effectively integrated and powerful distributed computing ability is provided. Firstly, transform Apriori algorithm to Map Reduce model; realize Apriori parallel transformation; then use the way of compressing original transaction sets to improve the performance of Apriori algorithm in Hadoop framework; lastly, Map Reduce-Apriori algorithm is realized which is highly scalable for running in cloud computing environment.

K-Means is a famous partition based clustering algorithm. Various extensions of K-Means have been proposed depending on the type of datasets being handled. Popular ones include K-Modes for categorical data and K-Prototype for mixed numerical and categorical data. The K-Means and its extensions suffer from one major limitation that is dependency on prior input of number of clusters K. Sometimes it becomes practically impossible to correctly estimate the optimum number of clusters in advance. Various ways have been suggested in literature to overcome this limitation for numerical data. But for categorical and mixed data work is still in progress. In this paper, we introduce a new algorithm based on the K-Means that takes mixed dataset as an input and generates appropriate number of clusters on the run using MapReduce programming style. The new algorithm not only overcomes the limitation of providing the value of K initially but also reduces the computation time using MapReduce framework.

Spatial Cluster analysis is another important technique in the field of spatial data mining, especially the K-Means spatial clustering method, which can deal with spatial objects with geographical location and attribute. However, with the development of the information society, the spatial data grows explosively, but the serial algorithm has low computing efficiency and is difficult to process massive spatial data. Aiming at spatial with a double meaning of location and attribute, the paper designed and implemented K-Means spatial clustering parallel algorithm on Hadoop. Using Yahoo Weibo user data is to do clustering analysis. Finally, the visualization of clustering results was implemented by Google Map.