Qian Xiaohua's Medical Image and Health Informatics Lab, in cooperation with the Department of Functional Neurosurgery of Ruijin Hospital, published three research articles on Intelligent assessment of motion video for Parkinson's diseases(PD). They sytematically proposed the spatiotemporal fine-grained feature mining technology, which solved the key challenges of fine-grained analysis of medical video actions. Based on this, an intelligent model was constructed to objectively evaluate the fist clenching test, standing balance test and toe slapping motion in the Parkinson Disease Rating Scale based on video.

Research results

1. A Tree-Structure-Guided Graph Convolutional Network with Contrastive Learning for the Assessment of Parkinsonian Hand Movements

In this paper, we propose a tree-structure-guided graph convolutional network with contrastive learning scheme to solve the challenge of difficulty in fine-grained feature extraction and insufficient model stability, finally achieving the video-based automated assessment of Parkinsonian hand movements, which represent a vital MDS-UPDRS component for examining upper-limb bradykinesia. Specifically, a tri-directional skeleton tree scheme is proposed to achieve effective fine-grained modeling of spatial hand dependencies. In this scheme, hand skeletons are extracted from videos, and then the spatial structures of these skeletons are constructed through depth-first tree traversal. Afterwards, a tree max-pooling module is employed to establish remote exchange between outer and inner nodes, hierarchically gather the most salient motion features, and hence achieve fine-grained mining. Finally, a group-sparsity-induced momentum contrast is also developed to learn similar motion patterns under different interference through contrastive learning. This can promote stable learning of discriminative spatial-temporal features with invariant motion semantics. Comprehensive experiments on a large clinical video dataset reveal that our method achieves competitive results, and outperforms other sensor-based and RGB-depth methods. The proposed method leads to accurate assessment of PD bradykinesia through videos collected by low-cost consumer cameras of limited capabilities. Hence, our work provides a convenient tool for PD telemedicine applications with modest hardware requirements.

Graphical abstract

Rui Guo, Hao Li, Chencheng Zhang, Xiaohua Qian*. A Tree-Structure-Guided Graph Convolutional Network with Contrastive Learning for the Assessment of Parkinsonian Hand Movements. Medical Image Analysis, v.81, 2022.

Available at https://www.sciencedirect.com/science/article/pii/S1361841522002055

2. A Contrastive Graph Convolutional Network for Toe-Tapping Assessment in Parkinson’s Disease

we propose a novel contrastive graph convolutional network for automated and objective toe-tapping assessment, which is one of the most important tests of lower-extremity bradykinesia. Specifically, based on joint sequences extracted from videos, a supervised contrastive learning strategy was followed to cluster together the features of each class, thereby enhancing the specificity of the learnt class-specific features. Subsequently, a multi-stream joint sparse learning mechanism was designed to eliminate potentially similar redundant features of joint position and motion, hence strengthening the discriminability of features extracted from different streams. Finally, a spatial-temporal interaction graph convolutional module was developed to explicitly model remote dependencies across time and space, and hence boost the mining of fine-grained motion features. Comprehensive experimental results demonstrate that this method achieved remarkable classification performance on a clinical video dataset, with an accuracy of 70.04% and an acceptable accuracy of 98.70%. These results obviously outperformed other existing sensor- and video-based methods. The proposed video-based scheme provides a reliable and objective tool for automated quantitative toe-tapping assessment, and is expected to be a viable method for remote medical assessment and diagnosis.

Graphical abstract

Rui Guo, Jie Sun, Chencheng Zhang, Xiaohua Qian*. A Contrastive Graph Convolutional Network for Toe-Tapping Assessment in Parkinson’s Disease. IEEE Transactions on Circuits and Systems for Video Technology, 2022-08-01 (online).

Available at http://ieeexplore.ieee.org/document/9847262

3. A Self-Supervised Metric Learning Framework for the Arising-from-Chair Assessment of Parkinsonians with Graph Convolutional Networks

This paper proposes a novel vision-based method for automated assessment of the arising-from-chair task, which is one of the key MDS-UPDRS components. The proposed method is based on a self-supervised metric learning scheme with a graph convolutional network (SSM-GCN). Specifically, for human skeleton sequences extracted from videos, a self-supervised intra-video quadruplet learning strategy is proposed to construct a metric learning formulation with prior knowledge, for improving the spatial-temporal representations. Afterwards, a vertex-specific convolution operation is designed to achieve effective aggregation of all skeletal joint features, where each joint or feature is weighted differently based on its relative factor of importance. Finally, a graph representation supervised mechanism is developed to maximize the potential consistency between the joint and bone information streams. Experimental results on a clinical dataset demonstrate the superiority of the proposed method over the existing sensor-based methods, with an accuracy of 70.60% and an acceptable accuracy of 98.65%. The analysis of discriminative spatial connections makes our predictions more clinically interpretable. This method can achieve reliable automated PD assessment using only easily-obtainable videos, thus providing an effective tool for real-time PD diagnosis or remote continuous monitoring.

Rui Guo#, Jie Sun#, Chencheng Zhang, Xiaohua Qian*. A Self-Supervised Metric Learning Framework for the Arising-from-Chair Assessment of Parkinsonians with Graph Convolutional Networks. IEEE Transactions on Circuits and Systems for Video Technology, 2022-03-31 (online).

Available at http://ieeexplore.ieee.org/document/9745580