TDSN-GCN: Transformerify Overall Structure Decaying Static Graph Embedding NAS-Guided GCN for Skeleton Action Recognition

Skeleton action recognition is becoming a representative of video motion recognition based on GCNs or Transformers. However, GCN-based works suffer from the overall structure with low learning cap, over-smoothing of dynamic graphs, and limited temporal learning of fixed receptive fields. Transformer-based works are encumbered by high computational resources, lack of artificial priors, and shallow temporal features over-aggregation. Thus, we propose TDSN-GCN with Transformerify architecture, Decaying Static Graph Embedding, and NAS-guided temporal receptive field strategy. First, we constructed the GCN architecture following the Transformer style with the info-decreased staged strategy, effectively raising learning capacity. Then, we theorize that the spatial graph matrix over-smooths by row as the depth increases. For this issue, a decaying static topology embedding with multi-topological hypergraphs is proposed with effective artificial priors. Finally, we design a NAS with the linear interpolation expansion receptive field search to explore the temporal receptive field preferences in depth. With the guidance of NAS, the temporal receptive field stage expansion strategy is proposed. Extensive experiments show that TDSN-GCN achieves the highest single-stream accuracy and state-of-the-art accuracy in 2-stream and 4-stream fusion compared to previous work with more streams. The code is available at https://github.com/vvhj/TDSN-GCN.