Earticle

본 논문에서는 손가락 마디 추정을 이용한 비전 및 깊이 정보 기반 손 인터페이스 방법을 제안한다. 먼저 비주얼 영상 및 깊이 정보 영상을 매핑한 후 왼손과 오른손의 영역의 레이블링 및 윤곽선 잡음 보정 후 각 손 영역에 대하여 손 중심점 및 회전각을 구현한다. 그리고 손 중심점에서 일정간격의 원을 확장하여 손 경계 교차점의 중간 지점을 계산하여 손가락 끝점과 마디를 추정하여 사용자의 손가락 동작을 인식한다. 본 방법을 실험한 결과 손의 회전 및 손가락 시작점 및 끝점을 정확하게 추정하여 다양한 손동작 인식 및 제어가 가능함을 보였다. 왼손과 오른손을 사용하여 다양한 손 포즈에 대해 실험한 결과, 본 논문의 제안 방법은 평균 90% 이상의 정확도로 초당 25프레임 이상의 처리 성능을 보였다. 제안 방법은 컴퓨터간의 HCI 제어, 게임, 교육 등의 비접촉식 인터페이스 응용분야에 적용될 수 있다.

In this paper, we propose a vision and depth information based real-time hand gesture interface method using finger joint estimation. For this, the areas of left and right hands are segmented after mapping of the visual image and depth information image, and labeling and boundary noise removal is performed. Then, the centroid point and rotation angle of each hand area are calculated. Afterwards, a circle is expanded at following pattern from a centroid point of the hand to detect joint points and end points of the finger by obtaining the midway points of the hand boundary crossing and the hand model is recognized. Experimental results that our method enabled fingertip distinction and recognized various hand gestures fast and accurately. As a result of the experiment on various hand poses with the hidden fingers using both hands, the accuracy showed over 90% and the performance indicated over 25 fps. The proposed method can be used as a without contacts input interface in HCI control, education, and game applications.

본 연구는 Kinect Sensor 기반의 개인 맞춤형 운동 처방 시스템을 개발하고, 개발 시스템의 사용성을 평가하는 것이다. 개발 시스템을 개인 맞춤형 운동 처방 시스템을 모션캡쳐 도구로서 이용할 수 있을지에 대한 가능성 검증을 위해서 스마트 거울 시스템에 부착된 키넥트 센서(Kinect sensor)에서 측정된 인체 움직임 데이터와 적외선 모션캡쳐 장비에서 측정된 인체 움직임 데이터를 비교하여 타당성과 신뢰성을 분석하였다. 타당성 검증 결과 상관계수 r=0.871~0.919로 높은 양의 상관성을 보였고, 예측가능정도가 88%로 높게 나타났다. 신뢰성 검증 결과 r=0.743~0.916 높은 양의 상관성을 보였고, 반복 측정에 대한 일관성도 ICC=0.937로 매우 높게 나타났다. 결론적으로 본 연구에서 개발한 키넥트 센서기반의 운동 처방 시스템에서 인간 골격에 대한 특징 벡터를 통한 관절의 가동범위 평가 및 자세측정평가가 운동 처방을 제공하는데 있어서 하나의 기준이 될 수 있다는 가능성을 보여주었다. 향후 병원, 임상시험센터, 스포츠센터 등의 운동처방사 혹은 물리치료사, 퍼스널 트레이너들에게 전문성 제고에 도움을 줄 수 있을 것으로 사료된다.

The purpose of this study is to investigate the personalized treacmill exercise analysis using a smart mirror based on Kinect sensor. To evaluate the performance of the development system, 10 health males were used to measure the range of the hip joint, knee joint, and ankle joint using a smart mirror when walking on a treadmill. For the validity and reliability of the development system, the validity and reliability were analyzed by comparing the human movement data measured by the Kinect sensor with the human movement data measured by the infrared motion capture device. As a result of validity verification, the correlation coefficient r=0.871~0.919 showed a high positive correlation, and through linear regression analysis, the validity of the smart mirror system was 88%. Reliability verification was conducted by ICC analysis. As a result of reliability verification, the correlation coefficient r=0.743~0.916 showed high correlation between subjects, and the consistency for repeated measurement was also very high at ICC=0.937. In conclusion, despite the disadvantage that Kinect sensor is less accurate than the motion capture system, Kinect is it has the advantage of low price and real-time information feedback. This means that the Kinect sensor is likely to be used as a tool for evaluating exercise prescription through human motion measurement and analysis.

Purpose: The purpose of this study was to investigate the availability of software for rehabilitation with the Kinect sensor by presenting an efficient algorithm based on machine learning when classifying the motion data of the PNF pattern if the subjects were wearing a patient gown. Methods: The motion data of the PNF pattern for upper extremities were collected by Kinect sensor. The data were obtained from 8 normal university students without the limitation of upper extremities. The subjects, wearing a T-shirt, performed the PNF patterns, D1 and D2 flexion, extensions, 30 times; the same protocol was repeated while wearing a patient gown to compare the classification performance of algorithms. For comparison of performance, we chose four algorithms, Naive Bayes Classifier, C4.5, Multilayer Perceptron, and Hidden Markov Model. The motion data for wearing a T-shirt were used for the training set, and 10 fold cross-validation test was performed. The motion data for wearing a gown were used for the test set. Results: The results showed that all of the algorithms performed well with 10 fold cross-validation test. However, when classifying the data with a hospital gown, Hidden Markov model (HMM) was the best algorithm for classifying the motion of PNF. Conclusion: We showed that HMM is the most efficient algorithm that could handle the sequence data related to time. Thus, we suggested that the algorithm which considered the sequence of motion, such as HMM, would be selected when developing software for rehabilitation which required determining the correctness of the motion.

최근 카메라, 캠코더 및 CCTV 등의 사용이 활발해지면서 영상 처리 기술의 수요가 급증하고 있다. 특히 키넥트 센서와 같은 깊이(Depth) 카메라를 사용한 3D 영상 기술에 대한 연구개발이 더욱더 활성화되고 있다. 키넥트 센서는 RGB, 골격(Skeleton) 및 깊이(Depth) 영상을 통해 인체의 3D 골격 구조를 실시간 프레임 단위로 획득할 수 있는 고성능 카메라이다. 본 논문에서는 키넥트 센서를 사용하여 인체의 3D 골격 구조를 모션 캡처하고 범용으로 사용되고 있는 모션 파일 포맷(*.trc 및 *.bvh)으로 선택하여 저장할 수 있는 시스템을 개발한다. 또한 본 시스템은 광학식 모션 캡처 파일 포맷(*.trc)을 자기식 모션 캡처 파일 포맷(*.bvh)으로 변환할 수 있도록 하는 기능을 가진다. 마지막으로 본 논문에서는 키넥트 센서를 사용하여 캡처한 모션 데이터가 제대로 캡처되어졌는지 모션 캡처 데이터 뷰어를 통하여 확인한다.

Recently, there is increasing demand for image processing technology while activated the use of equipments such as camera, camcorder and CCTV. In particular, research and development related to 3D image technology using the depth camera such as Kinect sensor has been more activated. Kinect sensor is a high-performance camera that can acquire a 3D human skeleton structure via a RGB, skeleton and depth image in real-time frame-by-frame. In this paper, we develop a system. This system captures the motion of a 3D human skeleton structure using the Kinect sensor. And this system can be stored by selecting the motion file format as trc and bvh that is used for general purposes. The system also has a function that converts TRC motion captured format file into BVH format. Finally, this paper confirms visually through the motion capture data viewer that motion data captured using the Kinect sensor is captured correctly.

The necessity of automation gradually increases by increasing industrialization at the present. Needed workforce is supplied on occasion of being short of manpower or threatening man’s health. It brings about a rapid improvement in the robotics world. All the used robot technologies in automatic systems performed by this time, fulfil the given commands as they are prearranged, or they can be momentarily controlled by the help of remote controls that are designed for the control of the system. In this thesis study, it is purposed that designed robot arms can be commanded simultaneously and one-to one with articulation motions of man’s arms by using Kinect technology. In this performed study, prearranged robot’s arms can be momentarily controlled by Kinect sensor. Motions of arm sensed by Kinect are transformed into mathematical data in software which is fixed for Kinect and then the process of leading servo motor, composed robot’s articulations, is achieved by sending the data to software fixed for Arduino. The ability to control Kinect as a software is supplied with WPF (Windows Presentation Foundation) and programming of Arduino is supplied with C++ program language.

본 연구에서는 마이크로소프트 키넥트 센서와 굽힘 감지 데이터 글러브를 이용하여 광업 분야 소프트웨어를제어할 수 있는 3차원 사용자 인터페이스를 개발하였다. 개발된 인터페이스는 키넥트 센서로 추적되는 사용자의 손위치에 따라 윈도우즈 환경에서 마우스 커서를 제어한다. 또한 굽힘 감지 데이터 글러브로 측정한 사용자 손가락의구부러짐 정도에 따라 윈도우즈 환경에서 마우스 클릭 이벤트를 처리한다. 인터페이스의 개발에는 오픈소스 하드웨어인 아두이노 Lilypad 보드와 National Instrument의 LabVIEW 소프트웨어가 사용되었다. 3차원 사용자 인터페이스를 광업 분야에서 사용되고 있는 몇 가지 3차원 소프트웨어에 적용해본 결과, 마우스나 키보드와 같은 전통적인 인터페이스보다 더 직관적으로 3차원 소프트웨어를 제어할 수 있음을 확인할 수 있었다.

In this study, we developed a 3D user interface (UI) using the Microsoft’s Kinect sensor and bend-sensing data glove to control software in the mining industry. The developed UI controls the mouse cursor in Windows according to the user’s hand position tracked by the Kinect sensor. In addition, it controls the mouse click events in Windows by measuring the user’s finger flexion using the bend-sensing data glove. An open-source hardware, Arduino Lilypad Board, and National Instrument’s LabVIEW software were used to develop the UI. Applications of the 3D UI to several 3D software utilized in the mining industry revealed that it can control the 3D software in more intuitive way than traditional UIs such as mouse and keyboard.

Background: The Microsoft Kinect which is a low-cost gaming device has been studied as a promise clinical gait analysis tool having satisfactory reliability and validity. However, its accuracy is only guaranteed when it is properly positioned in front of a subject. Objects: The purpose of this study was to identify the error when the Kinect was positioned at a $45^{\circ}$ angle to the longitudinal walking plane compare with those when the Kinect was positioned in front of a subject. Methods: Sixteen healthy adults performed two testing sessions consisting of walking toward and $45^{\circ}$ obliquely the Kinect. Spatiotemporal outcome measures related to stride length, stride time, step length, step time and walking speed were examined. To assess the error between Kinect and 3D motion analysis systems, mean absolute errors (MAE) were determined and compared. Results: MAE of stride length, stride time, step time and walking speed when the Kinect set in front of subjects were investigated as .36, .04, .20 and .32 respectively. MAE of those when the Kinect placed obliquely were investigated as .67, .09, .37, and .58 respectively. There were significant differences in spatiotemporal outcomes between the two conditions. Conclusion: Based on our study experience, positioning the Kinect directly in front of the person walking towards it provides the optimal spatiotemporal data. Therefore, we concluded that the Kinect should be placed carefully and adequately in clinical settings.

디지털 카메라, 스마트폰, 타블렛과 같은 스마트 기기의 대중화와 소셜 네트워크 서비스를 통해서 사진과 같은 멀티미디어 데이터의 양이 빠르고, 급격하게 확산되고 있다. 사진 검색 방법은 키워드 기반의 검색 방법, 예제 기반의 검색 방법, 시각화 질의 기반의 검색 방법의 세 가지 분류될 수 있다. 이전에 연구된 사진 검색 기법은 일반 PC 환경에 최적화되었기 때문에 최근에 등장한 스마트 TV 환경에서 사진 검색하기 위한 방법으로 사용하는 것은 적합하지 않은 상황이다. 본 논문에서는 스마트 TV 환경에서 키넥트를 이용한 소셜 네트워크에 존재하는 사진 검색 시스템을 제안하였다. 이를 위해서 키넥트 센서를 사용하여 마우스의 컨트롤을 제어할 수 있도록 구현하였으며, 제안하는 시스템의 검색 결과는 임계값이 0.7일 때, 평균 재현율과 평균 정확도는 각각 81%, 80%의 성능을 보였다.

Advances of digital device technology such as digital cameras, smart phones and tablets, provide convenience way for people to take pictures during his/her life. Photo data is being spread rapidly throughout the social network, causing the excessive amount of data available on the internet. Photo retrieval is categorized into three types, which are: keyword-based search, example-based search, visualize query-based search. The commonly used multimedia search methods which are implemented on Smart TV are adapting the previous methods that were optimized for PC environment. That causes some features of the method becoming irrelevant to be implemented on Smart TV. This paper proposes a novel Visual Query-based Photo Retrieval Method in Smart TV Environment using a motion sensing input device known as Kinect Sensor. We detected hand gestures using kinect sensor and used the information to mimic the control function of a mouse. The average precision and recall of the proposed system are 81% and 80%, respectively, with threshold value was set to 0.7

Creating avatar animations are tedious and time-consuming task since the desired avatar poses should be specified for each of a large number of keyframes. This paper proposes a fast and handy method to create game character animation contents using the motion data captured from the Kinect sensor. A Kinect sensor captures and saves the human motion. The Kinect sensor provides the motion information in a simple form of coordinates of joint positions. Using the captured motion data we determine the set of bone transforms that makes up the human skeletal animation data. The animation data is utilized to determine the position of all the bones at the current time in the animation. For experimental purpose we create a simple avatar character. We express the character model by the MD5 format, in which the mesh data and animation data are separated. A set of twenty joint positions reflect a snapshot of the character pose. The sets are used to evaluate the bone transform matrices and construct our skeletal animation scheme. We verified our method by appling the captured Kinect motion data to character animation. Our approach provides an easy method for creating avatar animations.

Kinect is a motion-sensing device which was originally developed for the Xbox 360 gaming console. This recently developed low-cost sensor detects the body position, motion, and voice; it consists of a microphone, a RGB camera, and a depth sensor. Kinect is PC-centric sensor which allows developers to develop real-life applications with human gestures and body motions. This paper presents an approach to interpret the indoor room objects in order to match the objects features in depth images captured from an RGBD video database. The dataset consists of color and depth image pairs gathered in real-time indoor home environment. The objects features are matched in depth image pairs with the feature association method to detect stable features at different time instances.

[PURPOSE] This study is a pilot study to investigate the possibility of kinect sensor based muscle strength test. [METHODS] Ten male subjects (age: 26.30±1.70 years, height: 176.88±2.89cm, weight: 73.40±4.03kg) were selected for the study. The exercise specialist and kinect sensor based muscle strength test program were evaluated at the same time. The items of evaluation were upper strength test (30 sec. arm curls) and lower strength test (30 sec. chair stands). Intraclass correlation coefficient (ICC) was calculated using a two-way mixed model to determine the reliability between the results measured by the exercise instructor and the results measured by the kinect sensor-based muscle strength test method. Bland-Altman plots were used to assess the differences between the two methods and to evaluate the agreement. To obtain the maximum and minimum values of the error intervals, mean + 1.96 × standard deviation and mean -1.96 × standard deviation were calculated. [RESULTS] As a result of upper strength test, ICC was 0.904 (95% CI: .614-.976), Bland-Altman analysis showed that the mean value of the result was 0.6 and the error interval was -3.44 to 4.64. As a result of the test of lower strength test, ICC was found to be 0.895 (95% CI: .576-.974), Bland-Altman analysis showed that the mean value of the result was -0.3 and the error interval was -3.50 to 2.90. [CONCLUSIONS] The ICC of the test showed very high inter - test reliability, showing no difference between the two methods. Therefore, kinect sensor-based muscle strength test may be recommended as a new objective method for testing the strength of the elderly.

The purpose of this study is to suggest that the movement expression activity of intellectually disabled people is effective in the learning process of LMA motion recognition based on Kinect sensor. We performed an ICT motion recognition games for intellectually disabled based on movement learning of LMA. The characteristics of the movement through Laban's LMA include the change of time in which movement occurs through the human body that recognizes space and the tension or relaxation of emotion expression. The design and implementation of the motion recognition model will be described, and the possibility of using the proposed motion recognition model is verified through a simple experiment. As a result of the experiment, 24 movement expression activities conducted through 10 learning sessions of 5 participants showed a concordance rate of 53.4% or more of the total average. Learning motion games that appear in response to changes in motion had a good effect on positive learning emotions. As a result of study, learning motion games that appear in response to changes in motion had a good effect on positive learning emotions

Many new concepts have recently been developed for human–computer interfaces. Xbox, introduced by Microsoft, is a game console. Kinect is a low-cost device that supports multiple speakers, a microphone, a red-green-blue camera, and a depth sensor camera for motion recognition. In the beginning, Kinect was developed for Xbox, but then supported various functions for the Windows operating system (OS). Similar to a Smart TV interface, in this study, we propose an intelligent interface for the Windows OS using Microsoft Kinect. Instead of using a mouse or keyboard, the interface helps humans interact with the computer using voice and hand gesture recognition. To interact with our system, users first register their voice commands in our recognition system, and then utilize those registered commands control the Windows OS. Our proposed system also provides scalability and flexibility and was assessed with a high level of recognition rate and user’s satisfaction.

As the Microsoft’s Kinect sensor can generate a real-time dense depth map with relatively commercial available, it is widely used in depth map capturing. However, there are some artifacts like holes, instability of the raw input data, which seriously affect the application. To solve this problem, in this paper, we propose a novel depth map refinement method based on by GMM and CS theory which enable the kinect sensor generate a dense depth map, the background large holes are filled without blurring, and the edges of the objects are sharpened, median filter is used to remove noise. Experiments on captured indoor data demonstrate the effectiveness of the method especially in the edge area and occlusion area that our method can obtain better results.

It is widely considered that it is very difficult to use a personal computer for people with disabilities. The mouse and keyboard based interfaces assumes that all people have hands and arms. System interface that will allow handicapped users without the hand and arm having a user experience similar to the general users is required. In this paper, by using the Kinect sensor, which is supplied with the MS X-box, with speech and motion recognition, author will offer a new computer interface for handicapped users with no hands and arms to be able to control personal computers at a level similar to the general user. It is shown that implemented application with the new interface system enables users to input user data, information and easily control the execution of the program.

Kinect v2 sensor does not provide depth information and skeletal traction function in near distance from the sensor. That is why many researches, to recognize hand gestures, are focused on the skeletal tracking only inside the range of detection. This paper proposes a method which can recognize hand gestures in the distance less than 0.5 meter without conventional skeletal tracking when Kinect v2 sensor is used. The proposed method does not use the information of depth sensor and infrared sensor, but detect hand area and count the number of isolated areas which are generated by drawing a circle in the center of the hand area. This method introduces new detectable gestures without high cost, so that it can be a substitute for the existing mouse-movement controlling and dynamic gesture recognition method such as clicking a mouse, clicking and dragging, rotating an image with two hands, and scaling an image with two hands in near distance. The gestures are appropriate for the user interface of smart devices which employ the interactions based on hand gestures in near distance.

In modern life, human–computer interaction has received the most interest from researchers, especially in developing new interaction methods. Microsoft’s Kinect sensor, which has integrated red-green-blue and depth (RGB+D) cameras, opens up new possibilities As well, many three-dimensional (3D) applications have been developed fashioning the computer world more natural and real for the user. In this paper, we propose and develop a 3D viewer application for heritage, which we call an HT3DViewer, using information received from the RGB+D camera of the Microsoft Kinect sensor. This viewer provides a hand gesture recognition library for users to control the object in the viewer by hands. Hand gestures are detected and defined using tracked information from the Kinect depth camera. The prototype of our hand gesture recognition library and 3D viewer application was built using the Microsoft Kinect software developer’s kit with C# programming language on the Microsoft .NET platform. Moreover, we investigate and select a simple and fast process to generate 3D models of heritage items from 2D images captured by cameras.

스마트폰에서 시작한 혁신적인 사용자 인터페이스는 사람의 자연스러운 동작을 인식하는 NUI(Natural User Interface)로 발전하여 앞으로 모든 가전제품의 기본적인 사용자 경험으로 자리 잡을 것으로 예상된다. 본 연구는 교육용 Java 개발도구인 Greenfoot과 마이크로소프트 Kinect 센서를 이용하여 NUI 프로그래밍을 지도하는 방법을 사례 중심으로 모색하였다. 본 연 구에 참여한 학생들을 대상으로 총 30차시를 지도한 후 NUI에 대한 이해와 분석, 한계점 인 식, 그리고 NUI 프로그램 제작에 대한 자기효능감에 대해 1:1 심층 면담을 실시하였다. 연구 결과, 중･고등학생들도 어렵지 않게 고품질의 NUI 프로그램을 제작할 수 있고, 이 과정에서 객체지향 프로그래밍의 개념과 혁신적인 사용자 인터페이스에 대한 가능성을 탐색하였으며, NUI 프로그래밍에 대해 긍정적으로 인식하고 있음을 확인할 수 있었다.

This case study is on the teaching method of NUI programming using Kinect sensor and Greenfoot. A touch based user interface of smartphone contributed to popularity of smart devices and was opportunity to confirm how important the user experience is in terms of innovation of the product. Furthermore, it’s significant that Microsoft Kinect sensor gives us applicability of human’s motion recognition technology to everyday life. But we have to program in C++, C# to implement NUI software using Kinect sensor. Especially it is very difficult for middle and high school students to program. So we used the Java educational tool, Greenfoot. Greenfoot is an educational development environment highly specialized for the development of interactive, graphical applications. It is based on the Java programming language. Using Greenfoot and Kinect sensor, the students from high school age could develop engaging and interesting programs, such as games and simulations with NUI quickly and easily while learning fundamental object-oriented programming concepts.

본 논문에서는 키넥트 센서 기반의 운동 자세 교정 애플리케이션를 설계하고 구현한다. 이 애플리케이션은 사용자의 운동 자세를 실시간으로 감지하고 분석하여, 잘못된 자세를 교정하는 기능을 제공한다. 키넥트 센서는 사용자의 움직임을 3D로 캡처하여 자세의 정확도를 평가하며, 개선이 필요한 부분에 대한 피드백을 제공한다. 또한, 사용자가 올바른 운동 자세를 유지할 수 있도록 지원하며, 장기적으로는 운동 효과를 극대화하고 부상 위험을 줄이는 데 기여한다. 또한, 이 애플리케이션은 개인 트레이너의 필요성을 줄이고, 사용자가 스스로 운동 자세를 교정할 수 있도록 도와준다.