Please wait a minute...
Big Data Mining and Analytics  2021, Vol. 4 Issue (4): 223-232    DOI: 10.26599/BDMA.2021.9020006
Multimodal Adaptive Identity-Recognition Algorithm Fused with Gait Perception
Changjie Wang(),Zhihua Li*(),Benjamin Sarpong()
Department of Computer Science and Technology, School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China
Download: PDF (5424 KB)      HTML  
Export: BibTeX | EndNote (RIS)      


Identity-recognition technologies require assistive equipment, whereas they are poor in recognition accuracy and expensive. To overcome this deficiency, this paper proposes several gait feature identification algorithms. First, in combination with the collected gait information of individuals from triaxial accelerometers on smartphones, the collected information is preprocessed, and multimodal fusion is used with the existing standard datasets to yield a multimodal synthetic dataset; then, with the multimodal characteristics of the collected biological gait information, a Convolutional Neural Network based Gait Recognition (CNN-GR) model and the related scheme for the multimodal features are developed; at last, regarding the proposed CNN-GR model and scheme, a unimodal gait feature identity single-gait feature identification algorithm and a multimodal gait feature fusion identity multimodal gait information algorithm are proposed. Experimental results show that the proposed algorithms perform well in recognition accuracy, the confusion matrix, and the kappa statistic, and they have better recognition scores and robustness than the compared algorithms; thus, the proposed algorithm has prominent promise in practice.

Key wordsgait recognition      person identification      deep learning      multimodal feature fusion     
Received: 22 October 2020      Published: 30 August 2021
Fund:  Smart Manufacturing New Model Application Project Ministry of Industry and Information Technology(ZH-XZ-18004);Future Research Projects Funds for Science and Technology Department of Jiangsu Province(BY2013015-23);Fundamental Research Funds for the Ministry of Education(JUSRP211A41);Fundamental Research Funds for the Central Universities(JUSRP42003);the 111 Project(B2018)
Corresponding Authors: Zhihua Li     E-mail:;;
About author: Changjie Wang received the BEng degree from Changsha University of Science & Technology, China in 2018. He is currently a master student at Jiangnan University, China. His research interests include the Internet of Things (IoTs) and cyber security.|Zhihua Li received the BEng degree from Wuxi Light Industry University, China in 1992, and the MEng and PhD degrees from Jiangnan University, China in 2002 and 2009, respectively. He is currently an associate professor at Jiangnan University. His research interests include network technology, parallel/distributed computing, information security, edge computing, and mobile computing.|Benjamin Sarpong received the BEng degree from Kwame Nkrumah University of Science and Technology, Ghana in 2014, and the MEng degree from Jiangnan University, China in 2020. He is currently an experimentalist at Jiangnan University. His research interests include sensor network, mobile computing, and their information security.
Cite this article:

Changjie Wang,Zhihua Li,Benjamin Sarpong. Multimodal Adaptive Identity-Recognition Algorithm Fused with Gait Perception. Big Data Mining and Analytics, 2021, 4(4): 223-232.

URL:     OR

Fig. 1 Line chart of individual gait information demo.
Fig. 2 ReLU function graph.
Fig. 3 Max pooling demo.
Fig. 4 CNN-GR infrastructure.
Fig. 5 Unimodal gait feature recognition model.
Fig. 6 Gait feature fusion-based identity-recognition scheme.
Fig. 7 Recognition rate varies with the number of trainings.
Fig. 8 Comparison of average recognition rates.
Fig. 9 Confusion matrix comparison of different algorithms.
DatasetAlgorithmKappa indicator
Table 1 Kappa indicator comparison.
[1]   Haghighat M., Abdel-Mottaleb M., and Alhalabi W., Discriminant correlation analysis: Real-time feature level fusion for multimodal biometric recognition, IEEE Transactions on Information Forensics and Security, vol. 11, no. 9, pp. 1984-1996, 2016.
[2]   Murray M. P., Drought A. B., and Kory R. C., Walking patterns of normal men, J. Bone Joint Surg. Am., vol. 46, pp. 335-360, 1964.
[3]   Johansson G., Visual perception of biological motion and a model for its analysis, Perception & Psychophysics, vol. 14, no. 2, pp. 201-211, 1973.
[4]   LeCun Y. and Bengio Y., Convolutional networks for images, speech, and time-series, in The Handbook of Brain Theory and Neural Networks, Arbib M. A., ed. Cambridge, MA, UK: The MIT Press, 1995, pp. 276-278.
[5]   Niyogi S. A. and Adelson E. H., Analyzing and recognizing walking figures in XYT, in Proc. IEEE Conf. Computer Vision & Pattern Recognition, Seattle, WA, USA, 1994, pp. 469-474.
[6]   Ailisto H. J., Lindholm M., Mantyjarvi J., Vildjiounaite E., and Makela S. M., Identifying people from gait pattern with accelerometers, in Proc. Volume 5779, Biometric Technology for Human Identification II, Orlando, FL, USA, 2005, pp. 7-14.
[7]   Tafazzoli F. and Safabakhsh R., Model-based human gait recognition using leg and arm movements, Engineering Applications of Artificial Intelligence, vol. 23, no. 8, pp. 1237-1246, 2010.
[8]   Mogan J. N., Lee C. P., and Tan A. W. C., Gait recognition using temporal gradient patterns, in Proc. 5th Int. Conf. Information and Communication Technology, Melaka, Malaysia, 2017, pp. 1-4.
[9]   Bigün E. S., Biüun J., Duc B., and Fischer S., Expert conciliation for multi modal person authentication systems by Bayesian statistics, in Proc. of Int. Conf. Audio- and Video-Based Biometric Person Authentication, Crans-Montana, Switzerland, 1997, pp. 291-300.
[10]   Jain A. K., Prabhakar S., and Chen S. Y., Combining multiple matchers for a high security fingerprint verification system, Pattern Recognition Letters, vol. 20, nos. 11-13, pp. 1371-1379, 1999.
[11]   Jain A. K., Prabhakar S., and Hong L., A multichannel approach to fingerprint classification, IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 4, pp. 348-359, 1999.
[12]   Baig A., Bouridane A., Kurugollu F., and Qu G., Fingerprint-iris fusion based identification system using a single hamming distance matcher, in Proc. of 2009 Symp. Bio-Inspired Learning and Intelligent Systems for Security, Edinburgh, UK, 2009, pp. 9-12.
[13]   Li Y., Ming D., Wang L., Qi H. Z., and Wan B. K., Gait and face profile fusion based human identification at a distance in video, (in Chinese), Chinese Journal of Scientific Instrument, vol. 32, no. 2, pp. 264-270, 2011.
[14]   Bazin A. I. and Nixon M. S., Probabilistic combination of static and dynamic gait features for verification, in Proc. Volume 5779, Biometric Technology for Human Identification II, Orlando, FL, USA, 2005, pp. 23-30.
[15]   Wagg D. K. and Nixon M. S., Automated markerless extraction of walking people using deformable contour models, Computer Animation and Virtual Worlds, vol. 15, nos. 3&4, pp. 399-406, 2004.
[16]   Mantyjarvi J., Lindholm M., Vildjiounaite E., Makela S. M., and Ailisto H. A., Identifying users of portable devices from gait pattern with accelerometers, in Proc. of IEEE Int. Conf. Acoustics, Speech, and Signal Processing, Philadelphia, PA, USA, 2005, pp. ii/973-ii/976.
[17]   Bruton L. and Ramamoorthy P., Maximally flat low-pass transfer function synthesis using continuous and discrete filters, IEEE Transactions on Circuits & Systems, vol. 23, no. 8, pp. 510-513, 1976.
[18]   Stojanovi? N., Krsti? I., Stamenkovi? N., and Pereni? G., Butterworth transfer function with the equalised group delay response in the maximally flat sense, Electronics Letters, vol. 54, no. 25, pp. 1436-1438, 2018.
[19]   Eagle N., Pentland A., and Lazer D., Inferring friendship network structure by using mobile phone data, Proceedings of the National Academy of Sciences of the United States of America, vol. 106, no. 36, pp. 15274-15278, 2009.
[20]   Scott J., Gass R., Crowcroft J., Hui P., Diot C., and Chaintreau A., CRAWDAD dataset cambridge/haggle (v. 2009-05-29), , 2021.
[21]   Kwapisz J. R., Weiss G. M., and Moore S. A., Activity recognition using cell phone accelerometers, ACM SIGKDD Explorations Newsletter, vol. 12, no. 2, pp. 74-82, 2011.
[22]   Cohen J., A coefficient of agreement for nominal scales, Educational and Psychological Measurement, vol. 20, no. 1, pp. 37-46, 1960.
[23]   Liu L. L., Yin Y. L., Qin W., and Li Y., Gait recognition based on outermost contour, International Journal of Computational Intelligence Systems, vol. 4, no. 5, pp. 1090-1099, 2011.
[24]   Hecht-Nielsen R., Theory of the backpropagation neural network, in Neural Networks for Perception. San Diego, CA, USA: Academic Press, 1992, pp. 65-93.
[25]   Wu Z. F., Huang Y. Z., Wang L., Wang X. G., and Tan T. N., A comprehensive study on cross-view gait based human identification with deep CNNs, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 2, pp. 209-226, 2017.
[26]   Ebert P., An extension of rate-distortion theory to confusion matrices, IEEE Transactions on Information Theory, vol. 14, no. 1, pp. 6-11, 1968.
[27]   Simon D. and Simon D. L., Analytic confusion matrix bounds for fault detection and isolation using a sum-of-squared-residuals approach, IEEE Transactions on Reliability, vol. 59, no. 2, pp. 287-296, 2010.
[1] Shuai Zhang,Hongyan Liu,Jun He,Sanpu Han,Xiaoyong Du. Deep Sequential Model for Anchor Recommendation on Live Streaming Platforms[J]. Big Data Mining and Analytics, 2021, 4(3): 173-182.
[2] Yong Bie,Yan Yang. A Multitask Multiview Neural Network for End-to-End Aspect-Based Sentiment Analysis[J]. Big Data Mining and Analytics, 2021, 4(3): 195-207.
[3] Krishna Kant Singh,Akansha Singh. Diagnosis of COVID-19 from Chest X-Ray Images Using Wavelets-Based Depthwise Convolution Network[J]. Big Data Mining and Analytics, 2021, 4(2): 84-93.
[4] Natarajan Yuvaraj,Kannan Srihari,Selvaraj Chandragandhi,Rajan Arshath Raja,Gaurav Dhiman,Amandeep Kaur. Analysis of Protein-Ligand Interactions of SARS-CoV-2 Against Selective Drug Using Deep Neural Networks[J]. Big Data Mining and Analytics, 2021, 4(2): 76-83.
[5] Youssef Nait Malek,Mehdi Najib,Mohamed Bakhouya,Mohammed Essaaidi. Multivariate Deep Learning Approach for Electric Vehicle Speed Forecasting[J]. Big Data Mining and Analytics, 2021, 4(1): 56-64.
[6] Wei Zhong, Ning Yu, Chunyu Ai. Applying Big Data Based Deep Learning System to Intrusion Detection[J]. Big Data Mining and Analytics, 2020, 3(3): 181-195.
[7] Sunitha Basodi, Chunyan Ji, Haiping Zhang, Yi Pan. Gradient Amplification: An Efficient Way to Train Deep Neural Networks[J]. Big Data Mining and Analytics, 2020, 3(3): 196-207.
[8] Chaity Banerjee, Tathagata Mukherjee, Eduardo Pasiliao Jr.. Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation[J]. Big Data Mining and Analytics, 2020, 3(2): 102-120.
[9] Zhenxing Guo, Shihua Zhang. Sparse Deep Nonnegative Matrix Factorization[J]. Big Data Mining and Analytics, 2020, 03(01): 13-28.
[10] Qile Zhu, Xiyao Ma, Xiaolin Li. Statistical Learning for Semantic Parsing: A Survey[J]. Big Data Mining and Analytics, 2019, 2(4): 217-239.
[11] Ying Yu, Min Li, Liangliang Liu, Yaohang Li, Jianxin Wang. Clinical Big Data and Deep Learning: Applications, Challenges, and Future Outlooks[J]. Big Data Mining and Analytics, 2019, 2(4): 288-305.
[12] Wenmao Wu, Zhizhou Yu, Jieyue He. A Semi-Supervised Deep Network Embedding Approach Based on the Neighborhood Structure[J]. Big Data Mining and Analytics, 2019, 2(3): 205-216.
[13] Jiangcheng Zhu, Shuang Hu, Rossella Arcucci, Chao Xu, Jihong Zhu, Yi-ke Guo. Model Error Correction in Data Assimilation by Integrating Neural Networks[J]. Big Data Mining and Analytics, 2019, 2(2): 83-91.
[14] Jin Liu, Yi Pan, Min Li, Ziyue Chen, Lu Tang, Chengqian Lu, Jianxin Wang. Applications of Deep Learning to MRI Images: A Survey[J]. Big Data Mining and Analytics, 2018, 1(1): 1-18.
[15] Ning Yu, Zhihua Li, Zeng Yu. Survey on Encoding Schemes for Genomic Data Representation and Feature Learning—From Signal Processing to Machine Learning[J]. Big Data Mining and Analytics, 2018, 01(03): 191-210.