Please wait a minute...
Big Data Mining and Analytics  2020, Vol. 3 Issue (3): 181-195    DOI: 10.26599/BDMA.2020.9020003
Applying Big Data Based Deep Learning System to Intrusion Detection
Wei Zhong*, Ning Yu, Chunyu Ai
Wei Zhong and Chunyu Ai are with the Division of Math and Computer Science, University of South Carolina Upstate, Spartanburg, SC 29303, USA. E-mail:
Ning Yu is with the Department of Computing Sciences, State University of New York College at Brockport, Brockport, NY 14420, USA. E-mail:
Download: PDF (5676 KB)      HTML  
Export: BibTeX | EndNote (RIS)      


With vast amounts of data being generated daily and the ever increasing interconnectivity of the world’s internet infrastructures, a machine learning based Intrusion Detection Systems (IDS) has become a vital component to protect our economic and national security. Previous shallow learning and deep learning strategies adopt the single learning model approach for intrusion detection. The single learning model approach may experience problems to understand increasingly complicated data distribution of intrusion patterns. Particularly, the single deep learning model may not be effective to capture unique patterns from intrusive attacks having a small number of samples. In order to further enhance the performance of machine learning based IDS, we propose the Big Data based Hierarchical Deep Learning System (BDHDLS). BDHDLS utilizes behavioral features and content features to understand both network traffic characteristics and information stored in the payload. Each deep learning model in the BDHDLS concentrates its efforts to learn the unique data distribution in one cluster. This strategy can increase the detection rate of intrusive attacks as compared to the previous single learning model approaches. Based on parallel training strategy and big data techniques, the model construction time of BDHDLS is reduced substantially when multiple machines are deployed.

Key wordsintrusion detection      deep learning      convolution neural network      fully connected feedforward neural network      multi-level clustering algorithm     
Received: 08 March 2020      Published: 15 September 2020
Corresponding Authors: Wei Zhong   
Cite this article:

Wei Zhong, Ning Yu, Chunyu Ai. Applying Big Data Based Deep Learning System to Intrusion Detection. Big Data Mining and Analytics, 2020, 3(3): 181-195.

URL:     OR

Fig. 1 Flow chart for building BDHDLS, where FC represents Full Connected feed forward neural network.
Fig. 2 Major steps to extract important content-based features using the Spark framework.
Fig. 3 Diagram for CNN.
Fig. 4 Diagram for RNN.
Fig. 5 Five phases of BDHDLS.
Model configuration 1Model configuration 2Model configuration 3
1 Conv layer (64 5 × 5 filters)3 Conv layers (64 5 × 5 filters)5 Conv layers (64 5 × 5 filters)
Max-pooling layerMax-pooling layerMax-pooling layer
1 Conv layer (128 5 × 5 filters)3 Conv layers (128 5 × 5 filters)5 Conv layers (128 5 × 5 filters)
Max-pooling layerMax-pooling layerMax-pooling layer
1 Conv layer (256 5 × 5 filters)3 Conv layers (256 5 × 5 filters)5 Conv layers (256 5 × 5 filters)
Max-pooling layerMax-pooling layerMax-pooling layer
1 FC layer (1024 neurons)3 FC layers (1024 neurons)5 FC layers (1024 neurons)
Sigmoid output layerSigmoid output layerSigmoid output layer
Table 1 Model configurations of convolutional neural network.
Number of layersNumber of neurons
8[128, 128, 128, 128, 64, 64, 64, 64]
8[64, 64, 64, 64, 128, 128, 128, 128]
12[256, 256, 256, 256, 128, 128, 128, 128, 64, 64, 64, 64]
12[64, 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256]
16[256, 256, 256, 256, 128, 128, 128, 128, 64, 64, 64, 64, 32, 32, 32, 32]
16[32, 32, 32, 32, 64, 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256]
Table 2 Model configurations of FC.
Hyper parameterDepthNumber of neurons
Table 3 Model configurations of recurrent neural network.
Fig. 6 TPR and ACC for different feature sets in the ISCX2012 dataset.
Fig. 7 FPR for different feature sets in the ISCX2012 dataset.
Fig. 8 Number of samples of different intrusive attacks for ISCX2012 dataset.
Fig. 9 TPR and ACC for ISCX2012 dataset.
Fig. 10 FPR for ISCX2012 dataset.
Table 4 "p value by F test" for binary classification in the ISCX2012 dataset. (%)
Fig. 11 Number of samples for different intrusive attacks in the CICIDS2017 dataset.
Fig. 12 TPR and ACC for CICIDS2017 dataset.
Fig. 13 FPR for CICIDS2017 dataset.
Table 5 "p value by F test" for binary classification in the CICIDS2017 dataset. (%)
Fig. 14 Number of samples for different intrusive attacks in the DARPA1998 dataset.
Fig. 15 TPR and ACC for DARPA1998 dataset.
Fig. 16 FPR for the DARPA1998 dataset.
× 2 CV F test in the ISCX2012 dataset.">
Fig. 17 Average construction time of BDHDLS for 5 <inline-formula><math xmlns:mml="" display="inline" id="MA95"><mml:mo>×</mml:mo></math></inline-formula> 2 CV F test in the ISCX2012 dataset.
[1]   Homeland Security Council, National strategy for homeland security, , 2007.
[2]   Dua S. and Du X, Data Mining and Machine Learning in Cybersecurity. Boston, MA, USA: Auerbach Publications, 2011.
[3]   Kim K. and Aminanto M. E., Deep learning in intrusion detection perspective: Overview and further challenges, in Proc. 2017 Int. Workshop on Big Data and Information Security (IWBIS), Jakarta, Indonesia, 2017, pp. 5-10.
[4]   Buczak A. L. and Guven E., A survey of data mining and machine learning methods for cyber security intrusion detection, IEEE Commun. Surv. Tutor., vol. 18, no. 2, pp. 1153-1176, 2016.
[5]   Catania C. A. and Garino C. G., Automatic network intrusion detection: Current techniques and open issues, Comput. Electr. Eng., vol. 38, no. 5, pp. 1062-1072, 2012.
[6]   Litjens G., Kooi T., Bejnordi B. E., Setio A. A. A., Ciompi F., Ghafoorian M., Van Der Laak J. A. W. M., Van Ginneken B., and Sánchez C. I., A survey on deep learning in medical image analysis, Med. Image Anal., vol. 42, pp. 60-88, 2017.
[7]   Hodo E., Bellekens X., Hamilton A., Tachtatzis C., and Atkinson R., Shallow and deep networks intrusion detection system: A taxonomy and survey, arXiv preprint arXiv: 1701.02145, 2017.
[8]   Chandra B. and Sharma R. K., Deep learning with adaptive learning rate using laplacian score, Exp. Syst. Appl., vol. 63, pp. 1-7, 2016.
[9]   Li Y. C., Nie X. Q., and Huang R., Web spam classification method based on deep belief networks, Exp. Syst. Appl., vol. 96, pp. 261-270, 2018.
[10]   LeCun Y., Bengio Y., and Hinton G., Deep learning, Nature, vol. 521, no. 7553, pp. 436-444, 2015.
[11]   Papakostas M. and Giannakopoulos T., Speech-music discrimination using deep visual feature extractors, Exp. Syst. Appl., vol. 114, pp. 334-344, 2018.
[12]   Yu Y., Long J., and Cai Z. P., Network intrusion detection through stacking dilated convolutional autoencoders, Secur. Commun. Networks, vol. 2017, p. 4184196, 2017.
[13]   Le T. T. H., Kim J., and Kim H., An effective intrusion detection classifier using long short-term memory with gradient descent optimization, in Proc. 2017 Int. Conf. Platform Technology and Service (PlatCon), Busan, South Korea, 2017, pp. 1-6.
[14]   Agarap A. F. M., A neural network architecture combining gated recurrent unit (GRU) and support vector machine (SVM) for intrusion detection in network traffic data, in Proc. 10th Int. Conf. Machine Learning and Computing, Macau, China, 2018, pp. 26-30.
[15]   Krizhevsky A., Sutskever I., and Hinton G. E., Imagenet classification with deep convolutional neural networks, in Proc. 25th Int. Conf. Neural Information Processing Systems, Lake Tahoe, NV, USA, 2012, pp. 1097-1105.
[16]   Shiravi A., Shiravi H., Tavallaee M., and Ghorbani A. A., Toward developing a systematic approach to generate benchmark datasets for intrusion detection, Comput. Secur., vol. 31, no. 3, pp. 357-374, 2012.
[17]   Wang W., Sheng Y. Q., Wang J. L., Zeng X. W., Ye X. Z., Huang Y. Z., and Zhu M., HAST-IDS: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection, IEEE Access, vol. 6, pp. 1792-1806, 2017.
[18]   Alpaydm E., Combined 5 × 2 cv F test for comparing supervised classification learning algorithms, Neural Comput., vol. 11, no. 8, pp. 1885-1892, 1999.
[19]   Baldi P., Brunak S., Chauvin Y., Andersen C. A. F., and Nielsen H., Assessing the accuracy of prediction algorithms for classification: An overview, Bioinformatics, vol. 16, no. 5, pp. 412-424, 2000.
[20]   Shone N., Ngoc T. N., Phai V. D., and Shi Q., A deep learning approach to network intrusion detection, IEEE Trans. Emerg. Top. Comput. Intell., vol. 2, no. 1, pp. 41-50, 2018.
[21]   Fiore U., Palmieri F., Castiglione A., and De Santis A., Network anomaly detection with the restricted boltzmann machine, Neurocomputing, vol. 122, pp. 13-23, 2013.
[22]   Schmidhuber J., Deep learning in neural networks: An overview, Neural Networks, vol. 61, pp. 85-117, 2015.
[23]   Vinayakumar R., Alazab M., Soman K. P., Poornachandran P., Al-Nemrat A., and Venkatraman S., Deep learning approach for intelligent intrusion detection system, IEEE Access, vol. 7, pp. 41525-41550, 2019.
[24]   Kasongo S. M. and Sun Y. X., A deep learning method with filter based feature engineering for wireless intrusion detection system, IEEE Access, vol. 7, pp. 38597-38607, 2019.
[25]   Nagar P., Menaria H. K., and Tiwari M., Novel approach of intrusion detection classification deeplearning using SVM, presented at First International Conference on Sustainable Technologies for Computational Intelligence, Singapore, 2020, pp. 365-381.
[26]   Akter M., Dip G. D., Mira M. S., Hamid M. A., and Mridha M., Construing attacks of internet of things (IoT) and a prehensile intrusion detection system for anomaly detection using deep learning approach, presented at International Conference on Innovative Computing and Communications: Proceedings of ICICC 2019, Singapore, 2020, pp. 427-438.
[27]   Liu Z. Q., Ghulam M. U. D., Zhu Y., Yan X. L., Wang L. F., Jiang Z. J., and Luo J. C., Deep learning approach for ids, presented at Fourth International Congress on Information and Communication Technology: ICICT 2019, Singapore, 2020, pp. 471-479.
[28]   Sekhar C. and Rao K. V., A study: Machine learning and deep learning approaches for intrusion detection system, presented at Int. Conf. Computer Networks and Inventive Communication Technologies, Coimbatore, India, 2019, pp. 845-849.
[29]   Nguyen G., Dlugolinsky S., Tran V., and García A. L., Deep learning for proactive network monitoring and security protection, IEEE Access, vol. 8, pp. 19696-19716, 2020.
[30]   Abusitta A., Bellaiche M., Dagenais M., and Halabi T., A deep learning approach for proactive multi-cloud cooperative intrusion detection system, Future Generation Comput. Syst., vol. 98, pp. 308-318, 2019.
[31]   Liu A. and Sun B., An intrusion detection system based on a quantitative model of interaction mode between ports, IEEE Access, vol. 7, pp. 161725-161740, 2019.
[32]   Aldwairi T., Perera D., and Novotny M. A., An evaluation of the performance of restricted boltzmann machines as a model for anomaly network intrusion detection, Comput. Networks, vol. 144, pp. 111-119, 2018.
[33]   Alliance C., Big data analytics for security intelligence, , 2013.
[34]   Zhong W. and Gu F., A multi-level deep learning system for malware detection, Exp. Syst. Appl., vol. 133, pp. 151-162, 2019.
[35]   Han J. W. and Kamber M., Data Mining: Concepts and Techniques. San Francisco, CA, USA: Elsevier, 2011.
[36]   Gupta S. K., Rao K. S., and Bhatnagar V., K-means clustering algorithm for categorical attributes, in Proc. 1st Int. Conf. Data Warehousing and Knowledge Discovery, Berlin, Germany: Springer, 1999, pp. 203-208.
[37]   Owen S., Anil R., Dunning T., and Friedman E., Mahout in Action. Shelter Island, NY, USA: Manning Publications, 2011.
[38]   Zhong W., Altun G., Harrison R., Tai P. C., and Pan Y., Improved K-means clustering algorithm for exploring local protein sequence motifs representing common structural property, IEEE Trans. Nanobioscience, vol. 4, no. 3, pp. 255-265, 2005.
[39]   Gibert L. D., Convolutional neural networks for malware classification, Master dissertation, Universitat Politècnica de Catalunya, Tarragona, Spain, 2016.
[40]   Tavallaee M., Bagheri E., Lu W., and Ghorbani A. A., A detailed analysis of the KDD CUP 99 data set, in Proc. 2009 IEEE Symp. Computational Intelligence for Security and Defense Applications, Ottawa, Canada, 2009, pp. 1-6.
[41]   Song J., Takakura H., and Okabe Y., Description of Kyoto University benchmark data, , 2006.
[42]   Lippmann R., Cunningham R. K., Fried D. J., Graf I., Kendall K. R., Webster S. E., and Zissman M. A., Results of the DARPA 1998 offline intrusion detection evaluation, presented at Recent Advances in Intrusion Detection: 4th International Symposium, New York, NY, USA, 1999, pp. 829-835.
[43]   Sharafaldin I., Lashkari A. H., and Ghorbani A. A., Toward generating a new intrusion detection dataset and intrusion traffic characterization, in Proc. 4th Int. Conf. Information Systems Security and Privacy (ICISSP), Funchal, Portugal, 2018, pp. 108-116.
[44]   Chen X., A simple utility to classify packets into flows, , 2017.
[45]   Bhuyan M. H., Bhattacharyya D. K., and Kalita J. K., Network anomaly detection: Methods, systems and tools, IEEE Commun. Surv. Tutor., vol. 16, no. 1, pp. 303-336, 2014.
[1] Zhenxing Guo, Shihua Zhang. Sparse Deep Nonnegative Matrix Factorization[J]. Big Data Mining and Analytics, 2020, 03(01): 13-28.
[2] Qile Zhu, Xiyao Ma, Xiaolin Li. Statistical Learning for Semantic Parsing: A Survey[J]. Big Data Mining and Analytics, 2019, 2(4): 217-239.
[3] Ying Yu, Min Li, Liangliang Liu, Yaohang Li, Jianxin Wang. Clinical Big Data and Deep Learning: Applications, Challenges, and Future Outlooks[J]. Big Data Mining and Analytics, 2019, 2(4): 288-305.
[4] Wenmao Wu, Zhizhou Yu, Jieyue He. A Semi-Supervised Deep Network Embedding Approach Based on the Neighborhood Structure[J]. Big Data Mining and Analytics, 2019, 2(3): 205-216.
[5] Jiangcheng Zhu, Shuang Hu, Rossella Arcucci, Chao Xu, Jihong Zhu, Yi-ke Guo. Model Error Correction in Data Assimilation by Integrating Neural Networks[J]. Big Data Mining and Analytics, 2019, 2(2): 83-91.
[6] Jin Liu, Yi Pan, Min Li, Ziyue Chen, Lu Tang, Chengqian Lu, Jianxin Wang. Applications of Deep Learning to MRI Images: A Survey[J]. Big Data Mining and Analytics, 2018, 1(1): 1-18.
[7] Ning Yu, Zhihua Li, Zeng Yu. Survey on Encoding Schemes for Genomic Data Representation and Feature Learning—From Signal Processing to Machine Learning[J]. Big Data Mining and Analytics, 2018, 01(03): 191-210.
[8] Qianyu Meng, Kun Wang, Xiaoming He, Minyi Guo. QoE-Driven Big Data Management in Pervasive Edge Computing Environment[J]. Big Data Mining and Analytics, 2018, 01(03): 222-233.