Earticle

본 연구는 월(Month), 시간(Hour), 휴일(Day off) 여부를 종합한 복합 시간적 요소(Temporal Factors)를 활용하여, 실시간 교통 데이터의 이상치와 결측치를 정밀하게 처리하는 보정 모델을 제안한다. 모델은 이 시간적 요소로 데이터를 그룹화 후 그룹 내 Z-score로 이상치를 탐지하며, 결측치는 시간적 요소 그룹 내 평균 기반 단계적 보간 방식을 결합한 파이프라인을 구성한다. 모델의 성능을 검증하기 위해 인천시 1,569개 도로의 교통량 데이터를 기반으로, 실무에서 널 리 쓰이는 기법들과 비교 평가를 수행했다. 그 결과, 복원 및 예측 정확도 실험 모두에서 제안 모델이 다른 기법 조합들보다 통계적으로 유의미하게 우수한 성능을 보이는 것을 확인했다. 이는 계절성, 일별 주기, 휴일 등 복합적 시간 요소를 반영하는 것이 예측 정확도 향상에 매우 효과적임을 입증하며, 실시간 데이터 전처리를 위한 본 모델의 높은 실용적 가치를 시사한다.

This study proposes an integrated correction method that effectively handles outliers and missing values in real-time traffic data, using data from 1,569 roads in Incheon between 2022 and 2024. The proposed method first removes outliers empirically, then constructs an integrated pipeline by combining "hourly Z-score" with "hourly average imputation." To validate this approach, we assembled 35 models by combining seven outlier-detection techniques and five missing-value imputation methods, including those commonly used in practice. We then conducted experiments involving artificially generated outliers and missing values, as well as performance comparisons using an LSTM prediction model. The results demonstrate that the proposed method outperforms all other combinations in both verification tests. This suggests that a simple, statistically based preprocessing strategy incorporating hourly characteristics is highly effective for improving urban traffic flow forecasts and has significant potential for real-time environments.

물 환경과 관련된 정보는 다양할 뿐만 아니라 그 규모가 방대하기 때문에 사용자가 정보를 이 용하기 위해서는 적절한 처리가 필요하다. 특히 수질자동측정망의 이상데이터는 측정장비 이상, 전원 문제, 부정확한 보정, 데이터처리상의 오류 등에 의해 나타난다. 이러한 데이터를 이용하여 계산된 통계량은 실제값과 편차를 가지게 되 며 따라서 수질자동측정망의 신뢰도 향상을 위한 장비개선과 더불어 데이터를 스크리닝하고 검증할 수 있는 방안이 필요하 다. 본 연구는 수질자동측 정자료 검증에 적절한 국소회귀모형(LOESS)을 이용한 이상치 선별 방안을 제안하고자 한다.

Water quality related informations are various and the amount of it is also huge, It requires the proper processing to provide information for users. Specially, The outlier in the AWQMS (Automatic Water Quality Monitoring System) might be originated from the inner mal-function, power problem, in-accurate calibration, etc. These in-correct data with embodied error leads the bias between the real value and the calculated value. Therefore techniques are required to screen and validate the data as well as to improve the reliability of AWQMS by equipment enhancement. In this study, we use LOESS (Local regression) model to detect outlier, so we propose the new technology for outlier detection method and treatment of water quality data.

본 논문은 다수의 전자전 센서에서 추출된 시간지연 차이정보와 도플러주파수 차이정보를 이용하는 Two-step weighted least-squares 기반의 이동 신호원 위치 및 속도 추정 기법에서, 수집 정보의 이상치를 검출 하는 알고리즘을 제안하고자 한다. 다수의 전자전 센서에서 추출되는 정보는 다양한 요인에 의해 정보에 이상치가 발생할 수 있으며, 이를 효과적으로 검출하고 데이터 융합과정에서 이상치를 배제하여 이동 신호원의 위치와 속도 추정의 정확도를 높이고자 한다. 본 논문에서는 이상치를 제외한 최소의 정상치 정보 집합을 추출하고, 이를 기반 으로 나머지 정보의 이상치 여부를 확률적으로 판단하는 알고리즘을 제안하였으며, 이를 모의실험을 통해, 정보의 이상치가 효과적으로 제거되어 위치 및 속도 추정의 정확도를 향상시킬 수 있음을 확인하였다. 정상치 거리정보 잡음이 20dB 이하인 경우, 이상치 정보를 효과적으로 제거하여, Cramér-Rao lower bound에 근접한 위치 및 속도 추정 정확도를 얻음을 확인하였다.

This paper presents the outlier detection algorithm in the estimation method of a source location and velocity based on two-step weighted least-squares method using time difference of arrival(TDOA) and frequency difference of arrival(FDOA) data. Since the accuracy of the estimated location and velocity of a moving source can be reduced by the outliers of TDOA and FDOA data, it is important to detect and remove the outliers. In this paper, the method to find the minimum inlier data and the method to determine whether TDOA and FDOA data are included in inliers or outliers are presented. The results of numerical simulations show that the accuracy of the estimated location and velocity is improved by removing the outliers of TDOA and FDOA data.

본 논문에서는 무선 센서 네트워크에서 이상치를 포함한 수신 신호 강도와 신호의 도달 각도 측정치 기반의 표적 위치추정 성능 저하를 방지하기 위한 이상치 검출 알고리즘 C-SCGP를 제안한다. 센서 오작동, 재밍, 심한 잡음과 같은 다양한 이상치 원인으로 인해 표적 위치추정 정확도가 크게 떨어질 수 있어, 모든 이상치를 탐지하고 제거하는 것이 중요 하다. 이러한 이상치를 제거하기 위해 single cluster graph partitioning (SCGP) 알고리즘이 널리 사용되고 있다. 기존 의 SCGP 알고리즘은 hyperparameter 최적화를 통한 threshold 설정과 이상치 확률 계산이 필수적이므로 다양한 분야 에 효율적인 적용이 제한되어왔다. 본 논문에서 제안된 continuous-SCGP (C-SCGP) 알고리즘은 이러한 SCGP의 약점 을 극복한다. 다양한 잡음 환경에서 threshold 설정과 이상치 확률 계산이 필요 없는 제안된 C-SCGP 알고리즘과 threshold 설정과 이상치 확률 계산을 요구하는 SCGP 알고리즘의 이상치 제거 성능이 같음을 최종 추정된 표적의 RMSE 성능을 통하여 검증하였다.

In this paper, we propose an outlier detection algorithm called C-SCGP to prevent the degradation of localization performance based on RSS (Received Signal Strength) and AOA (Angle of Arrival) in the presence of outliers in wireless sensor networks. Since the accuracy of target estimation can significantly deteriorate due to various cause of outliers such as malfunction of sensor, jamming, and severe noise, it is important to detect and filter out all outliers. The single cluster graph partitioning (SCGP) algorithm has been widely used to remove such outliers. The proposed continuous-SCGP (C-SCGP) algorithm overcomes the weakness of the SCGP that requires the threshold and computing probability of outliers, which are impratical in many applications. The results of numerical simulations show that the performance of C-SCGP without setting threshold and probability computation is the same performance of SCGP.

Special Complex non-Gaussian processes may have dynamic operation scenario shifts so that the traditional Outlier detection approaches become ill-suited. This paper proposes a new outlier detection approach based on using subspace learning and Gaussian mixture model(GMM) in energy disaggregation. Locality preserving projections(LPP) of subspace learning can optimally preserve the neighborhood structure, reveal the intrinsic manifold structure of the data and keep outliers far away from the normal sample compared with the principal component analysis (PCA). The results show proposed approach can significantly improve performance of outlier detection in energy disaggregation, increase the fraction true-positive from 93.8% to 97%, decrease the fraction false-positive from 35.48% to 25.8%.

Subspace outlier mining has a very important significance in big data analysis. To a large extent, subspace clustering algorithm has impact on the efficiency of mining outliers in subspaces. To solve the problem that CMI method selects best clustering subspaces unstably and complexly, formulas of chain rule of Cumulative Entropy, Cumulative Total Correlation and Cumulative Holoentropy were given. Cumulative Holoentropy was used to mine the best clustering subspaces on continuous data sets in which outliers were detected. Subspace outlier detection algorithm based on Cumulative Holoentropy was then proposed. Finally, the validity and scalability of proposed method were tested on real datasets and virtual datasets. Experiment shows that the efficiency of mining outliers in subspaces is enhanced by the proposed algorithm.

An outlier detection method using mixed prediction model has been described in this paper. The mixed prediction model consists of time-series model and regression model. The parameter estimation of the prediction model was performed using supervised learning and a genetic algorithm is adopted for a learning method. The experiments were performed in artificial and real data set. The prediction performance is compared with the existing prediction methods using artificial data. Outlier detection is conducted using the real sensor measurements in a dam. The validity of the proposed method was shown in the experiments.

In wireless sensor networks (WSNs), localization is one of the most important topics because the location information is typically useful for many applications. The primary data used in a localization process include the locations of anchor nodes and the distances between neighboring nodes. However, these data may contain outliers that deviate from their true values. The existence of the outliers might make the estimated positions not accurate. Thus, it is important to detect and handle outliers in order to achieve high localization accuracy. In this paper, we survey the existing outlier detection techniques for localization in wireless sensor networks. We provide taxonomy for classifying outlier detection techniques in WSNs localization based on different features. In addition, we present comparisons of these techniques. Finally, we discuss the future research directions in this area.

Outlier detection and clustering technologies are an important branch of data mining, such as combining the two technologies can improve the mining significance. In this paper, both clustering and outlier detection can be the starting point, proposed a DBSCAN-LOF algorithm is the core idea is to use k_ neighbors thought, DBSCAN redefine the core of the object, making the only non-core objects LOF The operation, thereby reducing the original LOF algorithm is computing the number of global objects, and makes no DBSCAN algorithm input parameters Eps. Real and simulated data sets by experimental results confirm that the algorithm to improve the operating efficiency and the LOF algorithm DBSCAN clustering effect, and while producing clustering and outlier detection results.

Accurate location information is critical to many applications in wireless sensor networks (WSNs) such as target tracking, environmental monitoring and geographical routing. Localization aims to figure out the locations of unknown nodes based on global locations of anchors and inter-node distance measurements. However, the existence of outlier anchors and outlier distances degrade localization accuracy in many localization algorithms. Most existing outlier detection approaches focus on distance outlier detection; few efforts have been devoted to anchor outlier detection. In this paper, we propose a mobile-assisted approach to detect outlier anchors and mitigate their negative effects in localization to achieve high localization accuracy. The proposed approach, namely Mobile-Assisted Anchor Outlier detection (MAAO),employs a mobile element to traverse the wireless sensor network several times to collect position information from static anchors in the network. For every static anchor, the mobile element computes the average of the anchor’s positions acquired from all tours, and compare it with the position acquired from the last mobile tour to detect whether the anchor is an outlier or not. The evaluation results show that MAAO can effectively detect outlier anchors, which consequently results in remarkable improvement in localization accuracy by not using outlier anchors in the localization process.

Most of the existing outlier detection methods aim at numerical data, but there will be a large number of categorical data in real life. Some outlier detection algorithms have been designed for categorical data. There are two main problems of outlier detection for categorical data, which are the similarity measure between categorical data objects and the detection efficiency problem. A cloud model-based outlier detection algorithm for categorical data is proposed in this paper. The algorithm is based on data driven idea and does not require the user to specify parameters. We utilize the synthetic data set and real data set to verify, compare our algorithm with the existing outlier detection algorithms for categorical data, and the experimental result demonstrates that our proposed algorithm has a higher detection rate and lower false alarm rate, while the time complexity is also more competitive.

In this study, in order to efficiently detect data patterns and outliers in time series data, outlier detection processing is performed for each section based on a smart contract in the data preprocessing process, and parameters for the ARIMA model are determined by generating and reflecting the significance and outlierrelated parameters of the data. It was created and applied to the modified arithmetic expression to lower the data abnormality. To evaluate the performance of this study, the normality of the data was compared and evaluated when the parameters of the general ARIMA model and the ARIMA model through this study were applied, and a performance improvement of more than 6% was confirmed.

In this study, in order to efficiently detect data patterns and outliers in time series data, outlier detection processing is performed for each section based on a smart contract in the data preprocessing process, and parameters for the ARIMA model are determined by generating and reflecting the significance and outlierrelated parameters of the data. It was created and applied to the modified arithmetic expression to lower the data abnormality. To evaluate the performance of this study, the normality of the data was compared and evaluated when the parameters of the general ARIMA model and the ARIMA model through this study were applied, and a performance improvement of more than 6% was confirmed.

In current years, data streams have been gradually turn into most important research area in the field of computer science. Data streams are defined as fast, limitless, unbounded, river flow, continuous, stop less, massive, tremendous unremitting, immediate, stream flow, arrival of ordered and unordered data. Data streams are divided into two types, they are online and offline streams. Online data streams are mainly used for real world applications like face book, twitter, network traffic monitoring, intrusion detection and credit card processes. Offline data streams are mainly used for manipulating the information which is based on web log streams. In data streams, data size is extremely huge and potentially infinite and it is not possible to lay up all the data, so it leads to a mining challenge where shortage of limitations has occur in hardware and software. Data mining techniques such as clustering, load shedding, classification and frequent pattern mining are to be applied in data streams to get useful knowledge. But, the existing algorithms are not suitable for performing the data mining process in data streams; hence there is a need for new techniques and algorithms. The main objective of this research work is to perform the clustering process in data streams and detecting the outliers in data streams. New hybrid approach is proposed which combines the hierarchical clustering algorithm and partitioning clustering algorithm. In hierarchical clustering, CURE algorithm is used and enhanced (E-CURE) and in partitioning clustering, CLARANS algorithm is used and enhanced (E-CLARANS). In this research work, the two algorithms E-CURE and E-CLARANS are combined (Hybrid) for performing a clustering process and finding the outliers in data streams. The performance of this hybrid clustering algorithm is compared with the existing hybrid clustering algorithms namely BIRCH with CLARANS and CURE with CLARANS. The performance factors used in this analysis are clustering accuracy and outlier detection accuracy. By analyzing the experimental results, it is observed that the proposed hybrid clustering approach E-CURE with E-CLARANS performance is more accurate than the existing hybrid clustering algorithms.

In data mining one of the challenging problems is how to handle high dimensional and complex datasets. Decision trees when applied to high dimensional and complex datasets produce decision trees which are very complex in nature and thereby reducing generalization. To address this issue we propose an algorithm know as Radom Matrix Projection with Outlier Detection (RMPOD). The proposed algorithm is validated on 24 UCI datasets against accuracy and tree size metrics. The results of the proposed algorithm with compared algorithm suggest an improvement in accuracy and tree size for better generalization.

최근 금융권에서는 스마트 기기 사용자의 급증으로 모바일 POS(point of sale) 기술에 대한 관심이 높아지고 있다. 본 논문은 모바일 POS 이상 서명을 빠르고 정확하게 탐지할 수 있는 기술을 제시한 다. 제안하는 분류기는 k-근접 이웃 분류기의 연산 효율 문제를 해결함으로써, 결제 관련 이상 서명 탐지를 빠르게 할 수 있게 해준다. 제안하는 모델에 대한 성능 평가는 기존 k-근접 이웃 분류기와의 분류 속도 비교를 통해 이뤄진다. 사용하는 데이터는 타블렛 기기 상에서 전자 펜을 사용하여 입력한 영문자 데이터이다. 본 실험의 결과는, 제안하는 방법으로 구현된 k-근접 이웃 분류기가 전자 수기 서명 정보 인식에 대한 연구 뿐 아니라 k-근접 이웃 분류기를 사용하는 여러 정보 보호 연구들을 한 층 더 효율적으로 만들어 줄 수 있다는 것을 보여준다.

Mobile POS technology is being magnified in financial industry as smart device has been popular. This paper proposes the technique to detect the mobile POS(point of sale) outlier signature in quickness and accuracy. The proposed classifier makes it possible to recognize the electronic signature concerned with payment and search out abnormal signatures. The performance evaluation was conducted, comparing the classification speed of the existing nearest neighborhood classifier and incremental k-nearest neighbor classifier. The experiment data is the English data handwritten on tablet devices with an electronic pen. The result of the experiment shows incremental k-nearest neighbor classifier can improve not only the quality of research on electronic The handwritten signature information recognition but also other information security study by reducing the quantity of arithmetic and operation time.

In this paper, we developed a forest fire detection algorithm which uses a regression function between NDVI and land surface temperature. Previous detection algorithms use the land surface temperature as a main factor to discriminate fire pixels from non-fire pixels. These algorithms assume that the surface temperatures of non-fire pixels are intrinsically analogous and obey Gaussian normal distribution, regardless of land surface types and conditions. And the temperature thresholds for detecting fire pixels are derived from the statistical distribution of non-fire pixels’ temperature using heuristic methods. This assumption makes the temperature distribution of non-fire pixels very diverse and sometimes slightly overlapped with that of fire pixel. So, sometimes there occur omission errors in the cases of small fires. To ease such problem somewhat, we separated non-fire pixels into each land cover type by clustering algorithm and calculated the residuals between the temperature of a pixel under examination whether fire pixel or not and estimated temperature of the pixel using the linear regression between surface temperature and NDVI. As a result, this algorithm could modify the temperature threshold considering land types and conditions and showed improved detection accuracy.

In order to synchronize a remote system time to the reference time like Coordinated Universal Time (UTC), it is required to compare the time difference between the two clocks. The time comparison data may have some outliers and the time synchronization performance can be significantly degraded if the outliers are not removed. Therefore, it is required to employ an effective outlier detection algorithm for keeping high accurate system time. In this paper, an outlier detection method is presented for the time difference data of GNSS time transfer receivers. The time difference data between the system time and the GNSS usually have slopes because the remote system clock is under free running until synchronized to the reference clock time. For investigating the outlier detection performance of the proposed algorithm, simulations are performed by using the time difference data of a GNSS time transfer receiver corrected to a free running Cesium clock with intentionally inserted outliers. From the simulation, it is investigated that the proposed algorithm can effectively detect the inserted outliers while conventional methods such as modified Z-score and adjusted boxplot cannot. Furthermore, it is also observed that the synchronization performance can be degraded to more than 15% with 20 outliers compared to that of original data without outliers.

A general method of detecting outliers based on a change of likelihood by using the influence function is suggested. It can be applied to all kinds of distributions that are specified by parameters. For the multivariate normal case, specific computations are made to get the corresponding conditional influence function. A numerical example is provided for illustration.