Earticle

Distributed computing systems have evolved over decades to support various types of scientific applications and overall computing paradigms have been categorized into HTC (High-Throughput Computing) to support bags of tasks which are usually long running, HPC (High-Performance Computing) for processing tightly-coupled communication-intensive tasks on top of dedicated clusters of workstations or Supercomputers, and Data-intensive Computing leveraging distributed storage systems and parallel processing frameworks. Many-Task Computing (MTC) aims to bridge the gap between traditional HTC and HPC by building efficient middleware systems throughout employing lightweight task dispatching mechanisms, minimizing data movements, leveraging data-aware scheduling, and proposing of next generation Exascale storage systems. Recent emerging applications requiring millions or even billions of tasks to be processed with relatively short per task execution times have driven this new computing paradigm. In this paper, we investigate concepts and technologies of MTC and propose guidelines for building an efficient and effective middleware system to fully support MTC applications. Throughout our short survey about challenges, systems and applications of MTC, we argue that a next generation distributed middleware system must effectively leverage distributed file systems, parallel processing frameworks, decentralized data/compute management systems, and dynamic load balancing techniques to solve the most challenging and complex scientific problems.

본 연구팀에서는 YARN 기반의 하둡 플랫폼에서 대규모의 태스크들로 구성된 Many-Task Computing(MTC) 응용들을 효율적으로 지원할 수 있는 신규 프레임워크로서 MOHA(Mtc On HAdoop)를 연구/개발해왔다. MTC 응용들은 수십만 개에서 수백만 개 이상의 대규모 태스크들로 구성되고 각 응용별로 자원의 사용 패턴이 다를 수 있기 때문에, 전체적인 시스템 성능 향상을 위해 MOHA-TaskExecutor(MTC 응용 태스크를 실행하는 주체)의 Adaptive Parallel Computability 기술 연구를 수행하였다. 이는 한 번에 하나의 태스크를 실행하던 기존의 처리 모델을 고도화하여 하나의 TaskExecutor가 동시에 여러 개의 태스크들을 실행함으로써 YARN Container의 병렬 컴퓨팅 능력을 극대화하기 위함이다. 이를 위해 각각의 TaskExecutor들이 "독립적이고, 동적으로" 동시에 실행시키는 MTC 응용 태스크들을 조정할 수 있도록 하였으며, 최적의 동시 실행 태스크 숫자를 찾기 위해서 Hill-Climbing 알고리즘을 활용하였다.

We have designed and implemented a new data processing framework called MOHA(Mtc On HAdoop) which can effectively support Many-Task Computing(MTC) applications in a YARN-based Hadoop platform. MTC applications can be composed of a very large number of computational tasks ranging from hundreds of thousands to millions of tasks, and each MTC application may have different resource usage patterns. Therefore, we have implemented MOHA-TaskExecutor(a pilot-job that executes real MTC application tasks)'s Adaptive Parallel Computability which can adaptively execute multiple tasks simultaneously, in order to improve the parallel computability of a YARN container and the overall system throughput. We have implemented multi-threaded version of TaskExecutor which can "independently and dynamically" adjust the number of concurrently running tasks, and in order to find the optimal number of concurrent tasks, we have employed Hill-Climbing algorithm.