Earticle

디지털 카메라, 스마트폰, 타블렛과 같은 스마트 기기의 대중화와 소셜 네트워크 서비스를 통해서 사진과 같은 멀티미디어 데이터의 양이 빠르고, 급격하게 확산되고 있다. 사진 검색 방법은 키워드 기반의 검색 방법, 예제 기반의 검색 방법, 시각화 질의 기반의 검색 방법의 세 가지 분류될 수 있다. 이전에 연구된 사진 검색 기법은 일반 PC 환경에 최적화되었기 때문에 최근에 등장한 스마트 TV 환경에서 사진 검색하기 위한 방법으로 사용하는 것은 적합하지 않은 상황이다. 본 논문에서는 스마트 TV 환경에서 키넥트를 이용한 소셜 네트워크에 존재하는 사진 검색 시스템을 제안하였다. 이를 위해서 키넥트 센서를 사용하여 마우스의 컨트롤을 제어할 수 있도록 구현하였으며, 제안하는 시스템의 검색 결과는 임계값이 0.7일 때, 평균 재현율과 평균 정확도는 각각 81%, 80%의 성능을 보였다.

Advances of digital device technology such as digital cameras, smart phones and tablets, provide convenience way for people to take pictures during his/her life. Photo data is being spread rapidly throughout the social network, causing the excessive amount of data available on the internet. Photo retrieval is categorized into three types, which are: keyword-based search, example-based search, visualize query-based search. The commonly used multimedia search methods which are implemented on Smart TV are adapting the previous methods that were optimized for PC environment. That causes some features of the method becoming irrelevant to be implemented on Smart TV. This paper proposes a novel Visual Query-based Photo Retrieval Method in Smart TV Environment using a motion sensing input device known as Kinect Sensor. We detected hand gestures using kinect sensor and used the information to mimic the control function of a mouse. The average precision and recall of the proposed system are 81% and 80%, respectively, with threshold value was set to 0.7

Creating avatar animations are tedious and time-consuming task since the desired avatar poses should be specified for each of a large number of keyframes. This paper proposes a fast and handy method to create game character animation contents using the motion data captured from the Kinect sensor. A Kinect sensor captures and saves the human motion. The Kinect sensor provides the motion information in a simple form of coordinates of joint positions. Using the captured motion data we determine the set of bone transforms that makes up the human skeletal animation data. The animation data is utilized to determine the position of all the bones at the current time in the animation. For experimental purpose we create a simple avatar character. We express the character model by the MD5 format, in which the mesh data and animation data are separated. A set of twenty joint positions reflect a snapshot of the character pose. The sets are used to evaluate the bone transform matrices and construct our skeletal animation scheme. We verified our method by appling the captured Kinect motion data to character animation. Our approach provides an easy method for creating avatar animations.

This study proposes a novel context-aware recommender system, which is designed to recommend the items according to the customer’s responses to the previously recommended item. In specific, our proposed system predicts the user’s emotional state from his or her responses (such as facial expressions and movements) to the previous recommended item, and then it recommends the items that are similar to the previous one when his or her emotional state is estimated as positive. If the customer’s emotional state on the previously recommended item is regarded as negative, the system recommends the items that have characteristics opposite to the previous item. Our proposed system consists of two sub modules-(1) emotion prediction module, and (2) responsive recommendation module. Emotion prediction module contains the emotion prediction model that predicts a customer’s arousal level-a physiological and psychological state of being awake or reactive to stimuli-using the customer’s reaction data including facial expressions and body movements, which can be measured using Microsoft’s Kinect Sensor. Responsive recommendation module generates a recommendation list by using the results from the first module-emotion prediction module. If a customer shows a high level of arousal on the previously recommended item, the module recommends the items that are most similar to the previous item. Otherwise, it recommends the items that are most dissimilar to the previous one. In order to validate the performance and usefulness of the proposed recommender system, we conducted empirical validation. In total, 30 undergraduate students participated in the experiment. We used 100 trailers of Korean movies that had been released from 2009 to 2012 as the items for recommendation. For the experiment, we manually constructed Korean movie trailer DB which contains the fields such as release date, genre, director, writer, and actors. In order to check if the recommendation using customers’ responses outperforms the recommendation using their demographic information, we compared them. The performance of the recommendation was measured using two metrics-satisfaction and arousal levels. Experimental results showed that the recommendation using customers’ responses (i.e. our proposed system) outperformed the recommendation using their demographic information with statistical significance.

최근 카메라, 캠코더 및 CCTV 등의 사용이 활발해지면서 영상 처리 기술의 수요가 급증하고 있다. 특히 키넥트 센서와 같은 깊이(Depth) 카메라를 사용한 3D 영상 기술에 대한 연구개발이 더욱더 활성화되고 있다. 키넥트 센서는 RGB, 골격(Skeleton) 및 깊이(Depth) 영상을 통해 인체의 3D 골격 구조를 실시간 프레임 단위로 획득할 수 있는 고성능 카메라이다. 본 논문에서는 키넥트 센서를 사용하여 인체의 3D 골격 구조를 모션 캡처하고 범용으로 사용되고 있는 모션 파일 포맷(*.trc 및 *.bvh)으로 선택하여 저장할 수 있는 시스템을 개발한다. 또한 본 시스템은 광학식 모션 캡처 파일 포맷(*.trc)을 자기식 모션 캡처 파일 포맷(*.bvh)으로 변환할 수 있도록 하는 기능을 가진다. 마지막으로 본 논문에서는 키넥트 센서를 사용하여 캡처한 모션 데이터가 제대로 캡처되어졌는지 모션 캡처 데이터 뷰어를 통하여 확인한다.

Recently, there is increasing demand for image processing technology while activated the use of equipments such as camera, camcorder and CCTV. In particular, research and development related to 3D image technology using the depth camera such as Kinect sensor has been more activated. Kinect sensor is a high-performance camera that can acquire a 3D human skeleton structure via a RGB, skeleton and depth image in real-time frame-by-frame. In this paper, we develop a system. This system captures the motion of a 3D human skeleton structure using the Kinect sensor. And this system can be stored by selecting the motion file format as trc and bvh that is used for general purposes. The system also has a function that converts TRC motion captured format file into BVH format. Finally, this paper confirms visually through the motion capture data viewer that motion data captured using the Kinect sensor is captured correctly.

The necessity of automation gradually increases by increasing industrialization at the present. Needed workforce is supplied on occasion of being short of manpower or threatening man’s health. It brings about a rapid improvement in the robotics world. All the used robot technologies in automatic systems performed by this time, fulfil the given commands as they are prearranged, or they can be momentarily controlled by the help of remote controls that are designed for the control of the system. In this thesis study, it is purposed that designed robot arms can be commanded simultaneously and one-to one with articulation motions of man’s arms by using Kinect technology. In this performed study, prearranged robot’s arms can be momentarily controlled by Kinect sensor. Motions of arm sensed by Kinect are transformed into mathematical data in software which is fixed for Kinect and then the process of leading servo motor, composed robot’s articulations, is achieved by sending the data to software fixed for Arduino. The ability to control Kinect as a software is supplied with WPF (Windows Presentation Foundation) and programming of Arduino is supplied with C++ program language.

본 연구에서는 마이크로소프트 키넥트 센서와 굽힘 감지 데이터 글러브를 이용하여 광업 분야 소프트웨어를제어할 수 있는 3차원 사용자 인터페이스를 개발하였다. 개발된 인터페이스는 키넥트 센서로 추적되는 사용자의 손위치에 따라 윈도우즈 환경에서 마우스 커서를 제어한다. 또한 굽힘 감지 데이터 글러브로 측정한 사용자 손가락의구부러짐 정도에 따라 윈도우즈 환경에서 마우스 클릭 이벤트를 처리한다. 인터페이스의 개발에는 오픈소스 하드웨어인 아두이노 Lilypad 보드와 National Instrument의 LabVIEW 소프트웨어가 사용되었다. 3차원 사용자 인터페이스를 광업 분야에서 사용되고 있는 몇 가지 3차원 소프트웨어에 적용해본 결과, 마우스나 키보드와 같은 전통적인 인터페이스보다 더 직관적으로 3차원 소프트웨어를 제어할 수 있음을 확인할 수 있었다.

In this study, we developed a 3D user interface (UI) using the Microsoft’s Kinect sensor and bend-sensing data glove to control software in the mining industry. The developed UI controls the mouse cursor in Windows according to the user’s hand position tracked by the Kinect sensor. In addition, it controls the mouse click events in Windows by measuring the user’s finger flexion using the bend-sensing data glove. An open-source hardware, Arduino Lilypad Board, and National Instrument’s LabVIEW software were used to develop the UI. Applications of the 3D UI to several 3D software utilized in the mining industry revealed that it can control the 3D software in more intuitive way than traditional UIs such as mouse and keyboard.

Many new concepts have recently been developed for human–computer interfaces. Xbox, introduced by Microsoft, is a game console. Kinect is a low-cost device that supports multiple speakers, a microphone, a red-green-blue camera, and a depth sensor camera for motion recognition. In the beginning, Kinect was developed for Xbox, but then supported various functions for the Windows operating system (OS). Similar to a Smart TV interface, in this study, we propose an intelligent interface for the Windows OS using Microsoft Kinect. Instead of using a mouse or keyboard, the interface helps humans interact with the computer using voice and hand gesture recognition. To interact with our system, users first register their voice commands in our recognition system, and then utilize those registered commands control the Windows OS. Our proposed system also provides scalability and flexibility and was assessed with a high level of recognition rate and user’s satisfaction.

Tracking multiple objects in real-time videos represents a challenging area in the era of computer vision. This paper proposes a new method to track the multiple objects under different environment conditions such as rotation, illumination, blurred, occlusion, and many others. In addition, the kinect color depth image processing is used to estimate the distance of the objects. The tracking of multiple objects is formulated as classification task which competitively use the object features in the different video frames of the video sequences. To obtain the optimal configuration of feature classification, a neural network based framework is presented to make a global influence based on winner pixel estimation between the video frames. The objects are tracked efficiently in less time as compared with SIFT techniques and distance of objects is calculated with kinect based depth image processing. Experimental results are given for real-time scenes, and many experiments are conducted to examine the performance of the proposed approach. The proposed method resulted into efficient tracking of multiple objects in various conditions including rotation, scaling, occlusion, etc. The distance of multiple tracked objects is estimated using the kinect depth processing.

The purpose of this study is to suggest that the movement expression activity of intellectually disabled people is effective in the learning process of LMA motion recognition based on Kinect sensor. We performed an ICT motion recognition games for intellectually disabled based on movement learning of LMA. The characteristics of the movement through Laban's LMA include the change of time in which movement occurs through the human body that recognizes space and the tension or relaxation of emotion expression. The design and implementation of the motion recognition model will be described, and the possibility of using the proposed motion recognition model is verified through a simple experiment. As a result of the experiment, 24 movement expression activities conducted through 10 learning sessions of 5 participants showed a concordance rate of 53.4% or more of the total average. Learning motion games that appear in response to changes in motion had a good effect on positive learning emotions. As a result of study, learning motion games that appear in response to changes in motion had a good effect on positive learning emotions

Kinect is a motion-sensing device which was originally developed for the Xbox 360 gaming console. This recently developed low-cost sensor detects the body position, motion, and voice; it consists of a microphone, a RGB camera, and a depth sensor. Kinect is PC-centric sensor which allows developers to develop real-life applications with human gestures and body motions. This paper presents an approach to interpret the indoor room objects in order to match the objects features in depth images captured from an RGBD video database. The dataset consists of color and depth image pairs gathered in real-time indoor home environment. The objects features are matched in depth image pairs with the feature association method to detect stable features at different time instances.

As the Microsoft’s Kinect sensor can generate a real-time dense depth map with relatively commercial available, it is widely used in depth map capturing. However, there are some artifacts like holes, instability of the raw input data, which seriously affect the application. To solve this problem, in this paper, we propose a novel depth map refinement method based on by GMM and CS theory which enable the kinect sensor generate a dense depth map, the background large holes are filled without blurring, and the edges of the objects are sharpened, median filter is used to remove noise. Experiments on captured indoor data demonstrate the effectiveness of the method especially in the edge area and occlusion area that our method can obtain better results.

It is widely considered that it is very difficult to use a personal computer for people with disabilities. The mouse and keyboard based interfaces assumes that all people have hands and arms. System interface that will allow handicapped users without the hand and arm having a user experience similar to the general users is required. In this paper, by using the Kinect sensor, which is supplied with the MS X-box, with speech and motion recognition, author will offer a new computer interface for handicapped users with no hands and arms to be able to control personal computers at a level similar to the general user. It is shown that implemented application with the new interface system enables users to input user data, information and easily control the execution of the program.

Kinect v2 sensor does not provide depth information and skeletal traction function in near distance from the sensor. That is why many researches, to recognize hand gestures, are focused on the skeletal tracking only inside the range of detection. This paper proposes a method which can recognize hand gestures in the distance less than 0.5 meter without conventional skeletal tracking when Kinect v2 sensor is used. The proposed method does not use the information of depth sensor and infrared sensor, but detect hand area and count the number of isolated areas which are generated by drawing a circle in the center of the hand area. This method introduces new detectable gestures without high cost, so that it can be a substitute for the existing mouse-movement controlling and dynamic gesture recognition method such as clicking a mouse, clicking and dragging, rotating an image with two hands, and scaling an image with two hands in near distance. The gestures are appropriate for the user interface of smart devices which employ the interactions based on hand gestures in near distance.

In modern life, human–computer interaction has received the most interest from researchers, especially in developing new interaction methods. Microsoft’s Kinect sensor, which has integrated red-green-blue and depth (RGB+D) cameras, opens up new possibilities As well, many three-dimensional (3D) applications have been developed fashioning the computer world more natural and real for the user. In this paper, we propose and develop a 3D viewer application for heritage, which we call an HT3DViewer, using information received from the RGB+D camera of the Microsoft Kinect sensor. This viewer provides a hand gesture recognition library for users to control the object in the viewer by hands. Hand gestures are detected and defined using tracked information from the Kinect depth camera. The prototype of our hand gesture recognition library and 3D viewer application was built using the Microsoft Kinect software developer’s kit with C# programming language on the Microsoft .NET platform. Moreover, we investigate and select a simple and fast process to generate 3D models of heritage items from 2D images captured by cameras.

[PURPOSE] This study is a pilot study to investigate the possibility of kinect sensor based muscle strength test. [METHODS] Ten male subjects (age: 26.30±1.70 years, height: 176.88±2.89cm, weight: 73.40±4.03kg) were selected for the study. The exercise specialist and kinect sensor based muscle strength test program were evaluated at the same time. The items of evaluation were upper strength test (30 sec. arm curls) and lower strength test (30 sec. chair stands). Intraclass correlation coefficient (ICC) was calculated using a two-way mixed model to determine the reliability between the results measured by the exercise instructor and the results measured by the kinect sensor-based muscle strength test method. Bland-Altman plots were used to assess the differences between the two methods and to evaluate the agreement. To obtain the maximum and minimum values of the error intervals, mean + 1.96 × standard deviation and mean -1.96 × standard deviation were calculated. [RESULTS] As a result of upper strength test, ICC was 0.904 (95% CI: .614-.976), Bland-Altman analysis showed that the mean value of the result was 0.6 and the error interval was -3.44 to 4.64. As a result of the test of lower strength test, ICC was found to be 0.895 (95% CI: .576-.974), Bland-Altman analysis showed that the mean value of the result was -0.3 and the error interval was -3.50 to 2.90. [CONCLUSIONS] The ICC of the test showed very high inter - test reliability, showing no difference between the two methods. Therefore, kinect sensor-based muscle strength test may be recommended as a new objective method for testing the strength of the elderly.

스마트폰에서 시작한 혁신적인 사용자 인터페이스는 사람의 자연스러운 동작을 인식하는 NUI(Natural User Interface)로 발전하여 앞으로 모든 가전제품의 기본적인 사용자 경험으로 자리 잡을 것으로 예상된다. 본 연구는 교육용 Java 개발도구인 Greenfoot과 마이크로소프트 Kinect 센서를 이용하여 NUI 프로그래밍을 지도하는 방법을 사례 중심으로 모색하였다. 본 연 구에 참여한 학생들을 대상으로 총 30차시를 지도한 후 NUI에 대한 이해와 분석, 한계점 인 식, 그리고 NUI 프로그램 제작에 대한 자기효능감에 대해 1:1 심층 면담을 실시하였다. 연구 결과, 중･고등학생들도 어렵지 않게 고품질의 NUI 프로그램을 제작할 수 있고, 이 과정에서 객체지향 프로그래밍의 개념과 혁신적인 사용자 인터페이스에 대한 가능성을 탐색하였으며, NUI 프로그래밍에 대해 긍정적으로 인식하고 있음을 확인할 수 있었다.

This case study is on the teaching method of NUI programming using Kinect sensor and Greenfoot. A touch based user interface of smartphone contributed to popularity of smart devices and was opportunity to confirm how important the user experience is in terms of innovation of the product. Furthermore, it’s significant that Microsoft Kinect sensor gives us applicability of human’s motion recognition technology to everyday life. But we have to program in C++, C# to implement NUI software using Kinect sensor. Especially it is very difficult for middle and high school students to program. So we used the Java educational tool, Greenfoot. Greenfoot is an educational development environment highly specialized for the development of interactive, graphical applications. It is based on the Java programming language. Using Greenfoot and Kinect sensor, the students from high school age could develop engaging and interesting programs, such as games and simulations with NUI quickly and easily while learning fundamental object-oriented programming concepts.

본 논문에서는 키넥트 센서 기반의 운동 자세 교정 애플리케이션를 설계하고 구현한다. 이 애플리케이션은 사용자의 운동 자세를 실시간으로 감지하고 분석하여, 잘못된 자세를 교정하는 기능을 제공한다. 키넥트 센서는 사용자의 움직임을 3D로 캡처하여 자세의 정확도를 평가하며, 개선이 필요한 부분에 대한 피드백을 제공한다. 또한, 사용자가 올바른 운동 자세를 유지할 수 있도록 지원하며, 장기적으로는 운동 효과를 극대화하고 부상 위험을 줄이는 데 기여한다. 또한, 이 애플리케이션은 개인 트레이너의 필요성을 줄이고, 사용자가 스스로 운동 자세를 교정할 수 있도록 도와준다.

본 논문에서는 키넥트 센서 기반의 치매 예방 애플리케이션을 설계하고 구현한다. 이 애플리케이션은 노년층의 치매 예방을 위해 기억력을 향상시키고 간단한 동작으로 운동을 촉진할 수 있는 햄버거 만들기 게임을 구현한다. 햄버거 만들기 게임은 키넥트 센서 기반으로 조인트와 스켈레톤 기능을 활용하여 하늘에서 떨어지는 재료들을 순서에 맞게 획득하여 햄버거를 완성함으로써 점수를 얻는다. 사용자들은 제한 시간이 끝날 때까지 계속 진행하며 순서를 기억해 내는 과정을 통해 기억력을 향상시키고, 재료를 잡기 위한 활동적인 움직임으로 치매 예방에 도움이 되는 운동 기능을 제공한다.

본 논문에서는 키넥트 센서 기반의 쓰레기 분리수거 게임을 설계하고 구현한다. 이 애플리케이션은 키넥트 센서의 스켈레톤 인식 기능을 통해 사용자를 게임 속 분리 수거통으로 표현하게 하여 분리 수거 대상 품목에 해당하는 개체에 닿으면 점수를 증가시킨다. 만일 일반 쓰레기에 닿을 시 3회의 기회가 카운트 되어 전부 소진히게 되면 게임이 종료되도록 구현한다. 해당 게임을 시행함으로써 분리수거에 적극적인 참여를 도모하여 이에 따른 행동을 취함으로 쓰레기 문제에 대한 인식을 높이고, 분리수거에 대한 재미와 도전을 제공한다.