Please wait a minute...
Big Data Mining and Analytics  2021, Vol. 4 Issue (4): 233-241    DOI: 10.26599/BDMA.2021.9020008
    
Attention-Aware Heterogeneous Graph Neural Network
Jintao Zhang(),Quan Xu*()
College of Sciences, Northeastern University, Shenyang 110004, China
State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang 110819, China
Download: PDF (648 KB)      HTML  
Export: BibTeX | EndNote (RIS)      

Abstract  

As a powerful tool for elucidating the embedding representation of graph-structured data, Graph Neural Networks (GNNs), which are a series of powerful tools built on homogeneous networks, have been widely used in various data mining tasks. It is a huge challenge to apply a GNN to an embedding Heterogeneous Information Network (HIN). The main reason for this challenge is that HINs contain many different types of nodes and different types of relationships between nodes. HIN contains rich semantic and structural information, which requires a specially designed graph neural network. However, the existing HIN-based graph neural network models rarely consider the interactive information hidden between the meta-paths of HIN in the poor embedding of nodes in the HIN. In this paper, we propose an Attention-aware Heterogeneous graph Neural Network (AHNN) model to effectively extract useful information from HIN and use it to learn the embedding representation of nodes. Specifically, we first use node-level attention to aggregate and update the embedding representation of nodes, and then concatenate the embedding representation of the nodes on different meta-paths. Finally, the semantic-level neural network is proposed to extract the feature interaction relationships on different meta-paths and learn the final embedding of nodes. Experimental results on three widely used datasets showed that the AHNN model could significantly outperform the state-of-the-art models.



Key wordsGraph Neural Network (GNN)      Heterogeneous Information Network (HIN)      embedding     
Received: 21 February 2021      Published: 30 August 2021
Fund:  Key Scientific Guiding Project for the Central Universities Research Funds(N2008005);Major Science and Technology Project of Liaoning Province of China(2020JH1/10100008);National Key Research and Development Program of China(2018YFB1701104)
Corresponding Authors: Quan Xu     E-mail: 20201825@stu.neu.edu.cn;quanxu@mail.neu.edu.cn
About author: Jintao Zhang is an undergraduate student at the College of Sciences, Northeastern University, China. His current research interests include big data analytics and recommender systems.|Quan Xu received the PhD degree from University of Lille, France in 2011. He is an associate professor at the State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, China. His research interests include industrial Internet, cloud services, and big data analytics and visualization.
Cite this article:

Jintao Zhang,Quan Xu. Attention-Aware Heterogeneous Graph Neural Network. Big Data Mining and Analytics, 2021, 4(4): 233-241.

URL:

http://bigdata.tsinghuajournals.com/10.26599/BDMA.2021.9020008     OR     http://bigdata.tsinghuajournals.com/Y2021/V4/I4/233

Fig. 1 A toy example of an HIN.
Fig. 2 Semantic-level attention of HAN.
Fig. 3 Framework of the AHNN.
𝚽 and target node u, calculate the attention between the target node and its neighbors, and then aggregate the feature information of the neighboring nodes through attention.
">
Fig. 4 Node-level attention. For a given meta-path 𝚽 and target node u, calculate the attention between the target node and its neighbors, and then aggregate the feature information of the neighboring nodes through attention.
DatasetRelation (A-B)# A# B# A-B#Feature#Training#Validation#TestMeta-path
DBLPPaper-Author14 328405719 6453348004002857APA
Paper-Conference14 3282014 328APCPA
Paper-Term14 327878988 420APTPA
IMDBMovie-Actor4780584114 34012323003002687MAM
Movie-Director478022694780MDM
ACMPaper-Author30255835974418306003002125MAM
Paper-Subject3025563025MDM
Table 1 Statistical information of three datasets.
DatasetMetricTrainingDeepworkHERecGCNGATHANAHNN
ACMMacro-F12077.2566.1785.2286.7087.3590.47
4080.4770.8986.1187.1187.7390.01
6082.5572.3887.4488.7988.8289.11
8084.1773.9288.9789.8289.8090.37
Micro-F12076.9266.0385.1486.7087.3590.42
4079.9970.7385.8887.0287.5989.79
6082.1172.2487.3088.7388.6988.85
8083.8873.8488.9089.7589.7590.18
DBLPMacro-F12077.4391.6889.8792.4292.4492.68
4081.0292.1690.2492.6092.7892.33
6083.6792.8091.2793.0893.6693.66
8084.8192.3492.9994.6995.3395.72
Micro-F12079.3792.6990.8293.3293.3593.69
4082.7393.1891.2693.6093.7393.29
6085.2793.7092.4993.9994.5294.43
8086.2693.2793.9495.2795.8996.23
IMDBMacro-F12040.7241.6538.2643.2745.8844.93
4045.1943.8638.3644.2846.4047.39
6048.1346.2739.3543.5948.8150.50
8050.3547.6441.0644.7349.0652.33
Micro-F12046.3845.8141.6846.3049.5948.81
4049.9947.5941.2747.0449.7350.92
6052.2149.8842.1346.3151.8453.37
8054.3350.9943.4546.4552.3554.60
Table 2 Results of the classification. (%)
DatasetMetricDeepworkHERecGCNGATHANAHNN
ACMNMI41.6140.7055.7055.1056.6962.73
ARI35.1037.1359.3058.7560.5066.39
DBLPNMI76.5376.7376.1677.8278.2078.58
ARI81.3580.9881.5082.8184.2984.64
IMDBNMI1.451.203.476.569.498.92
ARI2.151.653.818.198.247.53
Table 3 Results of the clustering. (%)
Fig. 5 Impact of the number of attention heads.
Fig. 6 Impact of the number of hidden dimensions.
Fig. 7 Impact of the number of embedding dimensions.
[1]   Duvenaud D., Maclaurin D., Aguilera-Iparraguirre J., Gómez-Bombarelli R., Hirzel T., Aspuru-Guzik A., and Adams R. P., Convolutional networks on graphs for learning molecular fingerprints, in Proc. 28th Int. Conf. Neural Information Processing Systems, Montreal, Canada, 2015, pp. 2224-2232.
[2]   Gilmer J., Schoenholz S. S., Riley P. F., Vinyals O., and Dahl G. E., Neural message passing for quantum chemistry, in Proc. 34th Int. Conf. Machine Learning, Sydney, Australia, 2017, pp. 1263-1277.
[3]   Cranmer M. D., Xu R., Battaglia P., and Ho S., Learning symbolic physics with graph networks, arXiv preprint arXiv: 1909.05862, 2019.
[4]   Sanchez-Gonzalez A., Heess N., Springenberg J. T., Merel J., Riedmiller M., Hadsell R., and Battaglia P., Graph networks as learnable physics engines for inference and control, in Proc. 35th Int. Conf. Machine Learning, Stockholm, Sweden, 2018, pp. 4470-4479.
[5]   Kipf T. N. and Welling M., Semi-supervised classification with graph convolutional networks, in Proc. 5th Int. Conf. Learning Representations, Toulon, France, 2017, .
[6]   He X. N., Deng K., Wang X., Li Y., Zhang Y. D., and Wang M., LightGCN: Simplifying and powering graph convolution network for recommendation, in Proc. 43rdInt. ACM SIGIR Conf. Research and Development in Information Retrieval, .
doi: 10.1145/3397271.3401063
[7]   Dwivedi V. P., Joshi C. K., Laurent T., Bengio Y., and Breson X., Benchmarking graph neural networks, arXiv preprint arXiv: 2003.00982, 2020.
[8]   Zhang Z. W., Cui P., and Zhu W. W., Deep learning on graphs: A survey, IEEE Transactions on Knowledge and Data Engineering, .
doi: 10.1109/TKDE.2020.2981333
[9]   Velickovie P., Cucurull G., Casanova A., Romero A., Liò P., and Bengio Y., Graph attention networks, in Proc. 6th Int. Conf. Learning Representations, Vancouver, Canada, 2018, .
[10]   Xu K., Hu W. H., Leskovec J., and Jegelka S., How powerful are graph neural networks? in Proc. 7th Int. Conf. Learning Representations, New Orleans, LA, USA, 2019, .
[11]   Zhu H. M., Feng F. L., He X. N., Wang X., Li Y., Zheng K., and Zhang Y. D., Bilinear graph neural network with neighbor interactions, in Proc. 29th Int. Joint Conf. Artificial Intelligence, Yokohama, Japan, 2020, pp. 1452-1458.
[12]   Shi C., Hu B. B., Zhao W. X., and Yu P. S., Heterogeneous information network embedding for recommendation, IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 2, pp. 357-370, 2019.
[13]   Wang X., Zhu M. Q., Bo D. Y., Cui P., Shi C., and Pei J., AM-GCN: Adaptive multi-channel graph convolutional networks, in Proc. 26th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, San Diego, CA, USA, 2020, pp. 1243-1253.
[14]   Wang X., Ji H. Y., Shi C., Wang B., Ye Y. F., Cui P., and Yu P. S., Heterogeneous graph attention network, in Proc. of the World Wide Web Conf., San Francisco, CA, USA, 2019, pp. 2022-2032.
[15]   Chen W. J., Gu Y. L., Ren Z. C., He X. N., Xie H. T., Guo T., Yin D. W., and Zhang Y. D., Semi-supervised user profiling with heterogeneous graph attention networks, in Proc. 28th Int. Joint Conf. Artificial Intelligence, Macao, China, 2019, pp. 2116-2122.
[16]   Hong H. T., Guo H. T., Lin Y. C., Yang X. Q., Li Z., and Ye J. P., An attention-based graph neural network for heterogeneous structural learning, in Proc. 34th AAAI Conf. Artificial Intelligence, New York, NY, USA, 2020, pp. 4132-4139.
[17]   Zhang S. and Xie L., Improving attention mechanism in graph neural networks via cardinality preservation, in Proc. 29th Int. Joint Conf. Artificial Intelligence, New York, NY, USA, 2020, pp. 1395-1402.
[18]   Perozzi B., Al-Rfou R., and Skiena S., DeepWalk: Online learning of social representations, in Proc. 20th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, New York, NY, USA, 2014, pp. 701-710.
[19]   Tang J., Qu M., Wang M. Z., Zhang M., Yan J., and Mei Q. Z., LINE: Large-scale information network embedding, in Proc. 24th Int. Conf. World Wide Web, Florence, Italy, 2015, pp. 1067-1077.
[20]   Dong Y. X., Chawla N. V., and Swami A., Metapath2vec: Scalable representation learning for heterogeneous networks, in Proc. 23rd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, Halifax, Canada, 2017, pp. 135-144.
[21]   Hu Z. N., Dong Y. X., Wang K. S., and Sun Y. Z., Heterogeneous graph transformer, in Proc. Web Conf., Taipei, China, 2020, pp. 2704-2710.
[22]   Fu X. Y., Zhang J. N., Meng Z. Q., and King I., MAGNN: Metapath aggregated graph neural network for heterogeneous graph embedding, in Proc. Web Conf., Taipei, China, 2020, pp. 2331-2341.
[1] Lujia Shen, Qianjun Liu, Gong Chen, Shouling Ji. Text-Based Price Recommendation System for Online Rental Houses[J]. Big Data Mining and Analytics, 2020, 3(2): 143-152.
[2] Zhonglin Ye, Haixing Zhao, Ke Zhang, Zhaoyang Wang, Yu Zhu. Network Representation Based on the Joint Learning of Three Feature Views[J]. Big Data Mining and Analytics, 2019, 2(4): 248-260.
[3] Wenmao Wu, Zhizhou Yu, Jieyue He. A Semi-Supervised Deep Network Embedding Approach Based on the Neighborhood Structure[J]. Big Data Mining and Analytics, 2019, 2(3): 205-216.
[4] Thosini Bamunu Mudiyanselage, Yanqing Zhang. Feature Selection with Graph Mining Technology[J]. Big Data Mining and Analytics, 2019, 2(2): 73-82.
[5] Yaojing Wang, Yuan Yao, Hanghang Tong, Feng Xu, Jian Lu. A Brief Review of Network Embedding[J]. Big Data Mining and Analytics, 2019, 2(1): 35-47.