Segment differential aggregation representation and supervised compensation learning of ConvNets for human action recognition

Ren ZL, Zhang QS, Cheng Q, Xu ZY, Yuan S, Luo DL (2024)

Publication Type: Journal article

Publication year: 2024

Journal

Science China-Technological Sciences Springer Verlag (Germany)

Book Volume: 67

Pages Range: 197-208

Journal Issue: 1

DOI: 10.1007/s11431-023-2491-4

Abstract

With more multi-modal data available for visual classification tasks, human action recognition has become an increasingly attractive topic. However, one of the main challenges is to effectively extract complementary features from different modalities for action recognition. In this work, a novel multimodal supervised learning framework based on convolution neural networks (ConvNets) is proposed to facilitate extracting the compensation features from different modalities for human action recognition. Built on information aggregation mechanism and deep ConvNets, our recognition framework represents spatial-temporal information from the base modalities by a designed frame difference aggregation spatial-temporal module (FDA-STM), that the networks bridges information from skeleton data through a multimodal supervised compensation block (SCB) to supervise the extraction of compensation features. We evaluate the proposed recognition framework on three human action datasets, including NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD. The results demonstrate that our model with FDA-STM and SCB achieves the state-of-the-art recognition performance on three benchmark datasets.

Involved external institutions

Dongguan University of Technology (DGUT) / 东莞理工学院 CN China (CN) Shenzhen Institutes of Advanced Technology (SIAT) CN China (CN) Xiamen University CN China (CN)

How to cite

APA:

Ren, Z.L., Zhang, Q.S., Cheng, Q., Xu, Z.Y., Yuan, S., & Luo, D.L. (2024). Segment differential aggregation representation and supervised compensation learning of ConvNets for human action recognition. Science China-Technological Sciences, 67(1), 197-208. https://dx.doi.org/10.1007/s11431-023-2491-4

MLA:

Ren, Zi Liang, et al. "Segment differential aggregation representation and supervised compensation learning of ConvNets for human action recognition." Science China-Technological Sciences 67.1 (2024): 197-208.

BibTeX: Download