그래프 합성곱 LSTM과 HMM을 이용한 사람 행동 영상 분석

Abstract

We propose a part-based graph convolutional network and long short term memory(PBGCN-LSTM) encoder and hidden Markov model(HMM) model for activity analysis. PBGCN-LSTM is for extracting spatiotemporal features in a human skeleton activity video. HMM analyzes the spatiotemporal features for classifying postures in an activity. First, PBGCN extracts spatial features in each frame. PBGCN divides human skeleton parts and extracts graph convolution features in a skeleton. The human skeleton can be represented as a graph, in which we view the nodes as the joints and the edges as the bones. And we give prior knowledge to GCN to divide parts. Second, LSTM extracts temporal features using the spatial features of each frame. Third, we obtain the spatiotemporal features represented as a 128 dimensional vector which is represented an activity video from PBGCN-LSTM. Finally, we input these vectors to HMMs. We train HMMs in each activity class. We experiment PBGCN-LSTM and HMM with NTU-RGB+D60 dataset which provides the human skeleton activity video in 60 activity classes. We measure this model for action recognition tasks and show several visualizations for analyzing the postures in a video.