Earticle

There has been a snappy forward advancement in cloud. With the developing measures of affiliations, turning number of affiliations are being utilized as assets in the cloud. Issues arise regarding the certainties of various clients utilizing these assets like round measure of room associations which helps in keeping from the cost. Putting away these associations evades the high cost while posting information during the building of machine orders. This collection of information gives better execution whilst less “far point cost” and ready to make prepared modifications. Cloud can yield better possibilities through net where security weaknesses are restricted. Despite of “wellbeing” being one of the brimming with risk weakness, it adjusts unmistakable connection to go into assumed control handling general condition. This paper uses “MapReduce” library that parallelizes the figuring, and handles perplexed issues like data spread, stack modifying and adjustment to non-basic disappointment. Enormous information, spread across finished many machines, need to parallelize. Moves the data, and gives booking, adjustment to non-basic disappointment. In this paper, a graph of MapReduce programming model along with the applications are explored. The maker has delineated here the work procedure of MapReduce process. Some fundamental issues, like adjustment to non-basic disappointment, are considered in more detail. Without a doubt, even the outline of working of Map Reduce is given.

본 논문에서는 모바일 컴퓨팅 환경에서의 디바이스 연계 통합을 위한 스토리지 가상화 기술로서, PosCFS+ 분산파일 서비스를 소개한다. PosCFS+ 분산 파일 서비스는 사용자 영역 네트워크(PAN) 상의 장치별 스토리지 공간을 하나의 가상 스토리지 공간으로 자동 구성하며, 구성된 가상 스토리지 공간의 파일 입출력을 위하여 파일 메타데이터를 기준으로 가상 디렉토리를 동적으로 구성함으로써, 사용자의 데이터 접근성 개선 및 데이터 통합 관리를 지원한다.

In this paper, we present the design and implementation of a distributed file service, PosCFS+, for device ensembles. All the storages in the network could be integrated automatically into a virtual storage by the file service and then the service creates virtual directories to provide a semantic file addressing based on the file metadata on the virtual storage. Through the virtualization technique, the file service is able to enhance the data accessibility.

Caching strategies and the resulting cache coherence control technique have become the key techniques for the system development. Based on the study of the cache technology of the home and abroad distributed file systems, and combined with the design requirements and characteristics for our multi-user parallel file system Cappella, this paper proposes a new meta-data cache management strategy. And this strategy not only improves the performance of metadata services, reducing the user access latency, but also controls metadata cache consistency very well.

The growth of websites and the Internet has opened up new research, social, entertainment, education and business opportunities. With the fast growth of the Internet, the digital data generated by the websites is becoming so massive that the traditional text software and relational database technology faces a bottleneck while processing such massive data and the results generated by these technologies are not satisfactory. Cloud computing offers a good solution for this problem. Cloud computing is not only capable of storing such massive data but also capable of processing and analyzing such voluminous data faster, by making use of distributed storage and distributed computing technology. A weblog is a group of connected web pages that consists of a log or daily record of information, particular fields or views which is altered, every now and then, by owner of site, other websites or by website users. An enterprise weblog analysis system based on Hadoop architecture with Hadoop Distributed File System (HDFS), Hadoop MapReduce Software Framework and Pig Latin Language aids the business decision-making process of the system administrators and helps them to collect and identify the potential value which is hidden within such huge data generated by the websites. Such a weblog analysis includes the analysis of an Internet site’s entry log as well as provides information about the amount of visitors, days of week and rush hours, views, hits, very often accessed pages, application server traffic trends, performance reports at varying intervals and statistical reports which indicate the performance of program.

There have two effective ways to improve the overall efficiency of the object-based distributed storage system, one is network caching technology and the other is metadata server cluster load balancing strategy. Under the heterogeneous storage environments, object-based distributed file system face the key challenge of lacking caching mechanism for the high-frequency access files, metadata server load imbalance also reduce the overall efficiency. To address these challenges, we discuss the design and implementation of an object-based distributed file storage caching system, and propose a weight-based metadata server cluster load balancing strategy. It effectively improve the metadata processing capacity. Base

Cloud storage is the development direction of data storage nowadays, and MooseFS provides a good solution for cloud storage. But as a kind of typical distributed file system, the problem of load balancing among chunk servers affects the use of MooseFS. Though MooseFS has provided certain load balancing capability, but it takes into account only the consumed space, and which causes the chunk servers with larger space were overburdened. In this study, we analyzed the load balancing algorithm among chunk servers in MooseFS, and proposed an improved load balancing algorithm. Through the experiment and comparative analysis, the improved algorithm enhances the load balancing performance of MooseFS obviously.

With the rapid development of cloud computing technology, it’s the cloud rendering that cloud computing is applied to render the job in CG (Computer Graphic) industry. The cloud rendering can handle a large number of rendering requests which are enormous pressure for back-end servers in system. Facing with massive data and computing resources, the bottleneck of original HDFS (Hadoop Distributed File System) based on cloud computing has become more and more prominent, such as the failure of single Namenode, scalability issues. Therefore, the paper proposed an improved HDFS which evolved a single Namenode into multi-Namenode. In HDFS, Metadata management is very important. So this paper presented a two-level Metadata distribution algorithm. The two-level algorithm was based on the principle that different distribution strategies were used to different categories of Metadata. The experiments verified that the improved HDFS effectively improved the performance of the system.

The mobile computing paradigm is becoming a reality due to the advent of mobile devices such as, PDA and Smart Phones having wireless interface to WWW. As a result, the high-end applications related to mobile multimedia and distributed digital contents have gained much attention. However, the existing distributed and mobile file systems are not adequately suitable to support such applications handling the challenges offered by mobile computing systems. In this paper, the concept, design architecture and consistency model of Virtual memory based Mobile Distributed File System (VMDFS) are proposed based on thin-mobileclient/ fat-server model. VMDFS creates the migrate-able virtual memory based mobile file system on top of the disk file systems of remote servers employing dynamic frame-lock to reduce network latency and to attain high-performance. An abstract mathematical model of VMDFS is constructed and associated properties are evaluated. The kernel-level implementation framework of VMDFS is illustrated based on Linux 2.4.22. The network paging latencies for LAN, Wireless VPN and 2.5G GPRS are measured experimentally. The results indicate that VMDFS is realizable.

Hadoop distributed file system (HDFS) is designed to store huge data set reliably, has been widely used for processing massive-scale data in parallel. In HDFS, the data locality problem is one of critical problem that causes the performance decrement of a file system. To solve the data locality problem, we propose an efficient data replication scheme based on access count prediction in a Hadoop framework. By the previous data access count, the existing data replication scheme predicts the next access count of data files using Lagrange’s interpolation. Then, the proposed data replication scheme determines the replication factor with the predicted data access count, whether it generates a new replica or it uses the loaded data as cache selectively. Finally, the proposed scheme provides improvement of data locality. By performance evaluation, proposed efficient data replication scheme is compared with default data replication setting of Hadoop that shows proposed scheme reduces averagely 8.9% of the task completion time in the map phase. Regarding the data locality, proposed scheme provides the increase of node locality by 6.6% and the decrease of rack and rack-off locality by 38.9% and 56.5%.

Hadoop Distributed File System (HDFS) becomes a representative cloud platform, benefiting from its reliable, scalable and low-cost storage capability. However, HDFS does not present good storage and access performance when processing a huge number of small files, because massive small files bring heavy burden on NameNode of HDFS. Meanwhile, HDFS does not provide any optimization solution for storing and accessing small files, as well as no prefetching mechanism to reduce I/O operations. This paper proposes an optimized scheme, Structured Index File Merging-SIFM, using two level file indexes, the structured metadata storage, and prefetching and caching strategy, to reduce the I/O operations and improve the access efficiency. Extensive experiments demonstrate that the proposed SIFM can effectively achieve better performance in the terms of the storing and accessing for a large number of small files on HDFS, compared with native HDFS and HAR.

최근 렌더링을 위한 플랫폼으로 멀티 클라우드 환경이 주목받고 있다. 이는 렌더링의 연산량이 시간에 따라 변동 폭이 큰 반면 각 렌더링 작업은 독립적으로 수행될 수 있기 때문이다. 그러나, 멀티 클라우드 렌더링은 대용량의 렌더링 입력 데이터에 대한 일관성을 유지하면서 실시간으로 데이터를 전송해야 하는 어려운 점이 존재한다. 본 논문에서는 멀 티 클라우드 렌더링을 위한 새로운 분산 파일 시스템을 개발하였다. 개발된 파일 시스템은 로컬 머신에 파일 서버를 두어 렌더링 입력 파일에 대한 버전을 관리하고, 클라우드에 캐쉬 관리자를 두어 파일의 버전을 고려한 분산 협력 캐슁 을 수행한다. 렌더링 워크로드를 이용한 실측 실험을 통해 개발된 파일 시스템이 NFS 대비 745%의 I/O 처리율을 나타 내는 것을 확인했으며, 업로드 방식과 비교할 때 평균 56%의 실행시간 개선이 있는 것으로 확인되었다.

Multi-cloud rendering has been attracting attention recently as the computational load of rendering fluctuates over time and each rendering process can be performed independently. However, it is challenging in multi-cloud rendering to deliver large amounts of input data instantly with consistency constraints. In this paper, we develop a new distributed file system for multi-cloud rendering. In our file system, a local machine maintains a file server that manages versions of rendering input files, and each cloud node maintains a rendering cache manager, which performs distributed cooperative caching by considering file versions. Measurement studies with rendering workloads show that the proposed file system performs better than NFS and the uploading schemes by 745% and 56%, respectively, in terms of I/O throughput and execution time.

최근 인터넷 사용자가 늘어나면서 클러스터링 기반의 대형 서버가 등장하게 되었고 또한 분산파일시스템(DFS)에 대한 관심이 커지게 되었다. 기존 분산파일 시스템들은 파일시스템 측면의 많은 기능들이 제공되고 뛰어난 성능을 발휘하지만 사용자 인터페이스 및 관리 측면에서 본다면 미흡한 점이 많다. 본 논문에서는 분산파일 시스템의 종류와 플랫폼에 상관없이, 사용하기 쉬운 DFS Management System을 설계하고 구축한다. 전체 시스템 구조를 파악 할 수 있도록 GUI 환경을 제공하며, 모듈화/계층화 구조로 설계하는데 기존 DFS와 연동하기 위한 하부 통신모듈이 포함된다. 본 논문에서는 Java기반으로 DFS Management System을 구축하였으며, Coda DFS가 설치된 클러스터링 서버와 연동하여 시험하였다.

Gadget-2 is a scientific simulation code has been used for many different types of simulations like, Colliding Galaxies, Cluster Formation and the popular Millennium Simulation. The code is parallelized with Message Passing Interface (MPI) and is written in C language. There is also a Java adaptation of the original code written using MPJ Express called Java Gadget. Java Gadget writes a lot of checkpoint data which may or may not use the HDF-5 file format. Since, HDF-5 is MPI-IO compliant, we can use our MPJ-IO library to perform parallel reading and writing of the checkpoint files and improve I/O performance. Additionally, to add reliability to the code execution, we propose the usage of Hadoop Distributed File System (HDFS) for writing the intermediate (checkpoint files) and final data (output files). The current code writes and reads the input, output and checkpoint files sequentially which can easily become bottleneck for large scale simulations. In this paper, we propose Sim-Hadoop, a framework to leverage HDFS and MPJ-IO for improving the I/O performance of Java Gadget code.

대량의 데이터를 효과적으로 저장하고 관리하기 위해서 클라우드 스토리지 시스템에서는 분산 파일 시스템 기술이 이용되고 있다. 그러나 데이터가 증가함에 따라 분산 파일 시스템을 이용함에도 스토리지 확장 비용이 증가하게 된다. 본 논문에서는 분산 파일 시스템의 스토리지 확장 비용을 줄이기 위해서 우리는 중복제거 파일 시스템을 적용한 분산 파일 시스템에서의 청크 배치 기법을 제안한다. 오픈 소스 기반의 분산 파일 시스템인 MooseFS 에 중복제거 파일 시스템인 lessfs 를 적용함으로써 스토리지의 가용공간을 늘릴 수 있으며, 이는 스토리지 확장 비용을 줄이는 효과를 가져온다. 또한, 동일한 청크는 같은 청크 서버에 배치 시킴으로써 중복제거 기회를 높인다. 실험을 통해서 제안 시스템의 중복제거량과 성능에 대해서 평가한다.

클러스터 시스템의 응용 분야가 다양화되고 복잡해짐에 따라, 대규모 클러스터 시스템을 보다 효율적으로 사용하기 위해서 실제 사용자의 이용 패턴을 예측할 수 있는 워크로드 분석의 필요성이 높아지고 있다. 이에 본 논문에서는 현재 가동중인 188개의 계산 노드, 3008개 CPU 자원을 보유한 대규모 클러스터 시스템에서 병렬 분산 파일 시스템에 대한 워크로드를 분석하였다.

클러스터 시스템의 응용 분야가 다양화되고 복잡해짐에 따라, 대규모 클러스터 시스템을 보다 효율적으로 사용하기 위해서 실제 사용자의 이용 패턴을 예측할 수 있는 워크로드 분석의 필요성이 높아지고 있다. 이에 본 논문에서는 현재 가동중인 188개의 계산 노드, 3008개 CPU 자원을 보유한 대규모 클러스터 시스템에서 병렬 분산 파일 시스템에 대한 워크로드를 분석하였다.

본 논문에서는 빅 데이터 처리를 위한 플랫폼인 하둡이 가지고 있는 분산 노드 관리 기법의 문제점을 분석하고 하둡 분산 파일시스템에서 노드 관리를 유연하게 처리하기 위한 기법을 제안한다. 기존의 방법은 클러스터에 포함된 노드가 일시적으로 연결되지 않는 경우 이를 즉시 고장으로 판정함으로 인해 클러스터를 동적으로 구성하지는 못하는 문제가 있다. 본 논문에서 제안하는 지연된 블록 복제 기법은 연결이 끊어진 노드가 추후 클러스터에 쉽게 편입될 수 있도록 노드의 제거를 최대한 지연함으로써 노드 관리의 유연성을 제공한다. 실험을 통해 제안하는 기법이 클러스터의 규모가 변화하는 환경에서 분산 처리 성능에 영향을 거의 미치지 않으면서도 노드 관리의 유연성을 증대시키는 것을 입증한다.

This paper discusses management problems of Hadoop distributed node, which is a platform for big data processing, and proposes a novel technique for enabling flexible node management of Hadoop Distributed File System. Hadoop cannot configure Hadoop cluster dynamically because it judges temporarily unavailable nodes as a failure. Delayed block replication scheme proposed in this paper delays the removal of unavailable node as much as possible so as to be easily rejoined. Experimental results show that the proposed scheme increases flexibility of node management with little impact on distributed processing performance when the cluster size changes.