Earticle

스마트 씨티, 스마트 팩토리 등의 어플리케이션에서 실시간 처리는 매우 중요하다. 많은 데이터의 실시간 처리를 위 해 스트림 프로세싱 플랫폼을 이용하는데, 5G 네트워크 등 고속 대용량의 네트워크를 이용하는 수요가 증가함에 따 라 부하 분산 및 로컬에서의 실시간 응답 속도 향상을 위해 엣지에서의 프로세싱이 강조되고 있다. 따라서, 엣지에 서의 실시간 처리를 위해 스트림 프로세싱이 중요해질 것이다. 이 글에서는 공개되어 있는 스트림 프로세싱 플랫폼 에 대해 알아보고, Raspberry Pi 4와 같이 엣지 디바이스로 쓰일 수 있는 임베디드 시스템에서 스트림 프로세싱 플랫폼들의 성능을 실험을 통해 비교해 보았다.

Real time processing is very important for many applications such as Smart city and Smart factory. In order to handle real time big data processing, stream processing platforms are being used. Due to the rapid increase of processing demand from the usage of high speed networks such as 5G networks, edge processing can give us benefits such as off-loading of data processing from clouds or servers, and improving response time if the processing result is asked at local devices. Therefore, the stream processing will be important on the edge devices. In this work, we surveyed available open source streaming platforms, and carried out performance comparisons on the embedded system such as the Raspberry Pi 4 which can be used for edge devices.

대량의 데이터를 실시간으로 처리하는 스트리밍 처리에서는 데이터가 발생한 순서에 따라 처리하는 것이 중요하다. 그러나 데이터 스트림은 다양한 원인으로 인해 입력 순서가 뒤바뀔 수 있으며, 이런 무질서 데이터는 데이터 처리 시간을 지연시키거나, 처리 정확도를 떨어트리는 문제를 초래한다. 본 논문에서는 무질서 데이터 처리결과의 정확성 을 높이면서 이로 인해 발생하는 데이터 처리 지연을 줄일 수 있는 데이터 스트림 분리 처리 방법인 DSSP(Data Stream Split Processing) 모델을 제안한다. DSSP 모델은 지연 데이터로 인한 전체 처리시간의 지연을 방지하 기 위해 입력 데이터 스트림을 처리시간 내에 입력되는 정상 데이터 스트림과 처리시간 내에 입력되지 못한 지연 데 이터 스트림으로 분리하여 처리한다. 정상적으로 입력된 데이터는 처리결과를 우선 신속히 제공하고, 지연 입력된 데이터는 별도로 저장하였다가 그 처리결과를 추가로 반영하여 데이터의 정확성을 보장할 수 있도록 한다. 본 논문 에서는 DSSP 시스템의 실험적 구현을 통해 그 효율과 성능을 분석한다.

It is crucial to process the data stream in event-time ordering when real-time processing a great amount of data. Data stream however, through various causes may create disorientation of the input order. These out-of-order data may induce data latency and reduce the correctness of the result. We propose a one such approach, a DSSP(Data Stream Split Processing) model that can reduce the latency of the data processing while increasing the accuracy of out-of-order data process results. To prevent the delay of response time caused by late data, our DSSP model splits input data stream into normal data stream that arrives on time and late data stream that does not. This model provides the result of normal data stream first, then applies the late data result that has been separately cached to assure the correctness of the data processing. We analyze the efficiency and performance of our DSSP model through experimental implementation.

Broad deployment of IoT solutions has resulted in the emergence of a large number of applications that generate value from processing the resulting high-frequency data streams. To meet the temporal requirements of those arbitrary simple applications, practitioners commonly employ monolithic approaches. However, the need for low cost and reusable solutions have rendered those approaches obsolete. This paper addresses this issue by discussing how microservice architectures (MSA) can deliver robust and low-cost solutions to manage the data load and real-time requirements. The motivation is to provide a methodology (stream-MSA) that will enable the easy and effective migration of suchlike applications to an MSA approach.

분산 스트림 처리 시스템을 위한 고가용성 알고리즘에는 Passive Standby, Active Standby, Upstream Backup 알고리즘 등이 있다. 기존의 고가용성 알고리즘은 복구 시 필요한 데이터의 백업을 위한 bandwidth overhead가 크며 노드의 연산결과를 다수의 downstream 노드들이 공유하며 데이터의 유입률이 폭발적으로 증가하는 경우에 출력 큐의 오버플로우로 인한 데이터의 손실 문제가 발생 할 수 있다. 본 논문은 이러한 문제들을 해결하기 위해 데이터 스트림의 유입량과 노드들의 연산처리율 모니터링을 통해 백업 방법을 유동적으로 변경시키는 적응적 Upstream Backup 알고리즘을 제안한다.

There have been High-Availability algorithms for Distributed Stream Processing System such as Passive Standby, Active Standby and Upstream Backup. Existing High Availability Algorithms have high bandwidth overhead when they backup data needed for restoring from the system failure and data can be lost by overflow of output queue in case of explosive increasing of input data rates. In this paper, we suggest adaptive Upstream Backup algorithm which changes fluidly the way to backup using monitor input rates of data stream and throughput of operation to solve those problems.

GPU는 높은 연산 성능에 비해 그래픽 처리를 하지 않을 경우에는 유휴상태에 놓여있다. 최근 유휴상태인 GPU를 이용하여 그래프 문제, 대용량 탐색 및 정렬, 행렬 연산 등 다양한 문제들을 병렬처리 하는 GPGPU가 각광받고 있고 NVIDIA사의 CUDA플랫폼의 등장으로 이러한 흐름은 더욱 가속화 되고 있다. GPGPU의 발전에 맞추어 GPU상에서 몇몇의 암호 알고리즘들이 구현되었지만, 스트림 암호의 경우에는 최근에서야 ECRYPT에 의해 표준이 제정되었기 때문에 아직 연구 결과가 미비하다. 이에 본 논문에서는 ECRYPT의 소프트웨어 기반 암호인 Salsa20을 GPU상에서 구현하고 이를 최적화하였으며, 파이프라이닝 기법을 사용하여 성능을 극대화하였다. 본 논문의 실험 결과에 따르면 CUDA를 사용한 Salsa20은 6.2Gbps의 암호화 속도를 보여주어 138.7Mbps의 성능을 보인 CPU상에서의 구현에 비해 무려 45배 이상의 성능 향상을 보여주었다.

Recent performance improvements in GPU enabled optimized parallel implementation of various algorithmsincluding graph algorithms, searching and sorting of massive data, matrix operations, and so on. After theadvent of CUDA platform of NVIDIA, more algorithms including several cryptographic algorithms are beingimplemented over GPUs. However, little work has been done on the parallel implementation of stream ciphersbecause it hasn’t been long since the first standard for stream cipher were published by ECRYPT. In thispaper, we provide a parallel implementation of Salsa20 stream cipher, one of the four software-based streamciphers in the ECRYPT standard, over a GPU, and optimize its performance using a pipelining technique.According to our experimental results, the throughput of Salsa20 on the CUDA platform is up to 6.2Gbps,which is approximately 45 times faster than the implementation over a CPU with 138.7Mbps.

Estimating the locations of workers in a shipyard is beneficial for a variety of applications such as selecting potential forwarders for transferring data in IoT services and quickly rescuing workers in the event of industrial disasters or accidents. In this work, we propose a human movement stream processing system for estimating worker locations in shipyards based on Apache Spark and TensorFlow serving. First, we use Apache Spark to process location data streams. Then, we design a worker location prediction model to estimate the locations of workers. TensorFlow serving manages and executes the worker location prediction model. When there are requirements from clients, Apache Spark extracts input data from the processed data for the prediction model and then sends it to TensorFlow serving for estimating workers’ locations. The worker movement data is needed to evaluate the proposed system but there are no available worker movement traces in shipyards. Therefore, we also develop a mobility model for generating the workers' movements in shipyards. Based on synthetic data, the proposed system is evaluated. It obtains a high performance and could be used for a variety of tasks in shipyards.

This paper first describes how to process SoC (State of Charge) streams made up of temporal and spatial stamps, altitude, SoC reading, velocity, and the like. Collected by a probe vehicle on the same road in Jeju city, Republic of Korea, multiple times, each stream has its own initial SoC and speed. The driving distance is calculated by adding up the line segment connecting two consecutive sensor points. For each stream, we plot 1) the absolute SoC change to figure out the SoC dynamics, 2) the consumed SoC change to find the similarity between different streams, and 3) the normalized curves to build a common SoC change model. Next, the artificial neural network is exploited to trace a common pattern out of them, aiming at providing an essential basis for navigation applications supporting electric vehicles. The analysis result reveals that the model can find reasonable representative values out of different streams for each subsection, even with a simple model having just one input variable denoting the distance from the start point and one output variable denoting the SoC reading. Our model keeps the trace error less than 5 % for the duration in which the vehicle speed is the same for each stream.

In the IOT environment, sensor data stream consists of event data from heterogeneous multi-sensors. One type of sensor may have quite a different event frequency from those other kinds of sensors, which makes most sensor data sets imbalanced. To classify an imbalanced data effectively, it is necessary to preprocess it for converting into a balanced data. This process may unify heterogeneous attributes in the imbalanced data and alleviate the difficulties for data mining on it. Mass function plays an important role in the fuzzy theory and Dempster-Shafer Theory. In this paper, using a mass function is suggested to process imbalanced data stream. A mass function is developed to compute mass values for imbalanced data sets, and an experiment is performed to investigate the validity to apply the mass function to the sensor data stream.

Recently, we have witnessed the emergence of new data stream management systems that can scale up to a large number of machines for the real-time processing of big data. These systems typically provide a procedural programming interface with their own APIs. However, to enable the rapid development of applications, it is desirable to support a declarative interface with clear processing semantics, such as window SQL. This paper examines the programming interfaces of the state-of-the-art data stream management systems and discusses the necessity of SQL support to help users write stream queries easily and in an integrated manner.

In this paper, we present a novel feedback control-based load shedding scheme for data stream processing. Firstly we apply system identification to establish a dynamic model to describe data stream management system (DSMS), which enables us to analyze DSMS quantitatively. Then, based on the model, we use the Root Locus method to design the PI controller with proven performance guarantees. Theoretic analysis and experimental results demonstrate that our approach is robust even when system load changes frequently. Comparing to existing strategies, our approach also achieves significantly better performance.

Detecting anomalies in human movement is an important task in industrial applications, such as monitoring industrial disasters or accidents and recognizing unauthorized factory intruders. In this paper, we propose an abnormal worker movement detection system based on data stream processing and hierarchical clustering. In the proposed system, Apache Spark is used for streaming the location data of people. A hierarchical clusteringbased anomalous trajectory detection algorithm is designed for detecting anomalies in human movement. The algorithm is integrated into Apache Spark for detecting anomalies from location data. Specifically, the location information is streamed to Apache Spark using the message queuing telemetry transport protocol. Then, Apache Spark processes and stores location data in a data frame. When there is a request from a client, the processed data in the data frame is taken and put into the proposed algorithm for detecting anomalies. A real mobility trace of people is used to evaluate the proposed system. The obtained results show that the system has high performance and can be used for a wide range of industrial applications.

In many real-world applications, data streams are usually collected in a decentralized manner such like sensor network, ubiquities sensor network, internet traffic analysis, and so on. In particularly, requirements for continuous, fast, high-volumes, adaptability, costly streaming data, an approximated analysis is needful for fast response to users on forward predicates. Distance estimate for both of “continuously” queries and streams is still a more challenge area because of a smaller or larger threshold selected is very easily to lead to a wrong result for continuously k-nearest neighbor queries. Therefore, we proposed a required filtering method to help to choose a well threshold of distance estimate in order to control error rates of approximated answers.

XML message brokers have a lot of importance because XML has become a practical standard for data exchange in many applications. Message brokers covered in this document store many users. This paper is a study of the processing of twig pattern queries in XML documents using branching node streams in XML message broker structures. This work is about query processing in XML documents, especially for query processing with XML twig patterns in the XML message broker structure and proposed a method to reduce query processing time when parsing documents with XML twig patterns by processing information. In this paper, the twig pattern query processing method of documents using the branching node stream removes the twigging value of the branch node that does not include the labeling value of the branch node stream when it receives a twig query from the client. In this paper, the leaf node discovery time can be reduced by reducing the navigation time of nodes in XML documents that are matched to leaf nodes in twig queries for client twig queries. Overall, the overall processing time to respond to queries is reduced, allowing for rapid questionanswer processing.

OSGi는 로컬 네트워크에서 하드웨어 및 미들웨어에 대한 상호 호환성 및 안전한 접근을 위한 보안성을 보장한다. 홈 네트워크 서비스가 복잡해짐에 따라 홈 네트워크 내에 실시간 데이터를 생성하는 센서가 추가되고 이 센서들을 활용한 OSGi 응용 서비스들이 증가하고 있다. 하지만, OSGi 응용에서는 이러한 센서에서 획득 가능한 실시간 센서 데이터를 효과적으로 처리하는 방법에 관한 연구는 미진하다. 또한, 실시간 데이터를 처리하기 위하여 대부분의 OSGi 기반 응용에서는 응용에 종속적인 미들웨어를 사용하거나 실시간 데이터의 처리와 가공에 대해서는 고려하지 않는다. 따라서 본 논문에서는 OSGi 응용에서 실시간 데이터를 활용하여 서비스를 효율적으로 제공할 수 있는 OSGi 응용 프레임워크를 제안한다. 이를 위하여 OSGi 응용으로 하여금 표준화된 방법으로 실시간 데이터를 처리하도록 지원하는 ALE 기반 실시간 센서 스트림 처리 미들웨어와의 연동 방법에 대하여 제안한다.

OSGi provides interoperability between hardwares and softwares and security for safe access in local network environments. According to the complexity of services in a home network environment, the usage of newly constructed sensors which generate real-time sensor data stream are steadily growing in the network and OSGi application services are also increasing using those sensors. However, the researches to effectively process real-time sensor data stream generated by those sensors are not actively going on, nowadays. Also, almost of OSGi-based applications use application-dependant middleware to process real-time data and do not even consider the process and manipulation of real-time data stream in most cases. In this paper, we propose OSGi application framework to effectively provide home network services by using real-time data stream in OSGi applications. For this, the method of interoperability with ALE-compliant middleware is proposed to process sensor data stream in a standardized method.

마이크로서비스 아키텍쳐(Microservice Architecture)는 실시간 실감 미디어 방송시스템과 같이 대규모 분산 시스템에 적합한 서비스 아키텍쳐의 하나이다. 스케일-아웃(Scale-Out)기법 과 같은 수평적 성능 확장이 쉽기 때문에 최근 넷플릭스나 트위터와 같은 서비스 플랫폼 업체들이 앞다투어 이와 같은 시스템을 도입하고 있다. 또한 마이크로 서비스 아키텍쳐는 기존의 REST와 같은 웹 API에서 처리하기 어려운 영상처리나 실시간 데이터 분석 등을 비동기 기반의 프로세싱를 이용하여 처리 가능하게 하고 있다. 본 논문은 IoT 센서 데이터 분석이나 대용량 실감미디어를 실 시간으로 편집하는 클라우드 기반 영상편집과 같은 다수의 이벤트들이 스트림으로 발생하며 플랫폼 내에서 비동기로 처리하는 상황에서 이벤트의 처리 순서가 보장되지 않음을 실험으로 증명하고 이에 알맞은 비동기 기반 마이크로서비 스에 적용 가능한 이벤트 스트림 처리 프레임워크를 제안한다.

Micro-service Architecture is a service architecture optimized for large-scale distributed systems such as real-time realistic broadcasting systems, so that are fiercely adopted by Global leading service platform vendors such as Netflix and Twitter due to the merit of horizontal performance scalability enabling the scale-out technique. In addition, micro-service architecture makes it possible to execute image processing and real-time data analysis using an asynchronous-based processing that are difficult to handle in Web API such as REST. In this paper, an event stream processing framework applicable to asynchronous based micro services is proposed in the sense that the accountability of event processing order is not guaranteed in the events such as IoT sensor data analysis or cloud-based image editing because these are the situations where the real-time media editing generates multiple event streams and asynchronous processes in the platform.

본 논문은 대량의 stream data를 처리하는 어플리케이션에서 하드웨어 가속기들이 접근하는 메모리가 non-cacheable에서 cacheable으로 변경됨에 따라 발생할 수 있는 데이터 일관성 문제를 고려하여 시스템 최적화를 진행하였다. 이를 위해 상위 수준 시뮬레이션을 통한 프로파일링 결과를 토대로 분석식을 만들어 활용하였다. 실험한 결과 여러 이미지 크기에서 메모리가 cacheable로 변경됨에 따라 평균 1.40배의 성능 향상을 보였다. 분석식의 주요 파라미터 최적화를 통해 최종적으로 3.88배의 성능 이득이 발생했으며, 항상 메모리가 cacheable인 경우의 성능이 항상 우월한 것은 아님을 확인할 수 있었다.

Performance optimization of applications with massive stream data processing has been performed by considering I/O data coherency problem where a memory is shared between processors and hardware accelerators. A formula for performance analyses is derived based on profiling results of system-level simulations. Our experimental results show that overall performance was improved by 1.40 times on average for various image sizes. Also, further optimization has been performed based on the parameters appeared in the derived formula. The final performance gain was 3.88 times comparing to the original design and we can find that the performance of the design with cacheable shared memory is not always.

In this letter, we present an event stream processing system that can evaluate a pattern query for a data sequence with predicates. We propose a pattern query language and develop a pattern query processing system. In our system, we propose novel techniques for run-time aggregation and negation processing and apply our system to stream data generated from vehicles to monitor unusual driving patterns.

With the advent of the Internet-of-Things paradigm, the amount of data production has grown exponentially and the user demand for responsive consumption of data has increased significantly. Herein, we present DART, a fast and lightweight stream processing framework for the IoT environment. Because the DART framework targets a geospatially distributed environment of heterogeneous devices, the framework provides (1) an end-user tool for device registration and application authoring, (2) automatic worker node monitoring and task allocations, and (3) runtime management of user applications with fault tolerance. To maximize performance, the DART framework adopts an actor model in which applications are segmented into microtasks and assigned to an actor following a single responsibility. To prove the feasibility of the proposed framework, we implemented the DART system. We also conducted experiments to show that the system can significantly reduce computing burdens and alleviate network load by utilizing the idle resources of intermediate edge devices.

The data generated at a very high rate by sensors and RFIDs are required to be handled by continuous queries keeping real time response. Because of its purpose, DSMSs are used in several cases of these large scale systems. On the other hand, sensor terminal systems include lightweight RDBMSs generally in many cases. So if lightweight RDBMSs can handle the high rate data directly, it is convenient for several applications. This paper proposes a speed-up method of stream processing by using the lightweight RDBMS SQLite without any special modifications.

Many continuous queries are important to be process efficiently in a data stream environment. It is applied a query index technique that takes linear performance irrespective of the number and width of intervals for processing many continuous queries. Previous researches are not able to support the dynamic insertion and deletion to arrange intervals for constructing an index previously. It shows that the insertion and search performance is slowed by the number and width of interval inserted. Many intervals have to be inserted and searched linearly in a data stream environment. Therefore, we propose Hashed Multiple Lists in order to process continuous queries linearly. Proposed technique shows fast linear search performance. It can be utilized the systems applying a sensor network, and preprocessing technique of spatiotemporal data mining.