Conference Paper2021Cited by 6

Node embedding using mutual information and self-supervision based bi-level aggregation

Kashob Kumar Roy, Amit Roy, AKM Mahbubur Rahman, M Ashraful Amin, Amin Ahsan Ali

2021 International Joint Conference on Neural Networks (IJCNN)

IEEE, pp. 1–8

DOI: 10.1109/ijcnn52387.2021.9533715Open AccessURL

Abstract

Graph Neural Networks (GNNs) learn low dimensional representations of nodes by aggregating information from their neighborhood in graphs. However, traditional GNNs suffer from two fundamental shortcomings due to their local (l-hop neighborhood) aggregation scheme. First, not all nodes in the neighborhood carry relevant information for the target node. Since GNNs do not exclude noisy nodes in their neighborhood, irrelevant information gets aggregated, which reduces the quality of the representation. Second, traditional GNNs also fail to capture long-range non-local dependencies between nodes. To address these limitations, we exploit mutual information (MI) to define two types of neighborhood, 1) Local Neighborhood where nodes are densely connected within a community and each node would share higher MI with its neighbors, and 2) Non-Local Neighborhood where MI-based node clustering is introduced to assemble informative but graphically distant nodes in the same cluster. To generate node presentations, we combine the embeddings generated by bi-level aggregation - local aggregation to aggregate features from local neighborhoods to avoid noisy information and non-local aggregation to aggregate features from non-local neighborhoods. Furthermore, we leverage self-supervision learning to estimate MI with few labeled data. Finally, we show that our model significantly outperforms the state-of-the-art methods in a wide range of assortative and disassortative graphs11Source Code at: https://github.com/forkkr/LnL-GNN.

Keywords

  • graph neural networks
  • Mutual information
  • node embeddings
  • self-supervised learning
  • long-range dependencies
  • node clustering

CCDS Authors

Kashob Kumar Roy
Kashob Kumar Roy

Kashob Kumar Roy

  • Alumni, CCDS

  • PhD Student

    University of Illinois Urbana-Champaign, USA

  • BSc

    University of Dhaka, Bangladesh

Published at

PAKDD, IJCNN

Amit Roy
Amit Roy

Amit Roy

  • Alumni, CCDS

  • PhD Student

    Purdue University, USA

  • BSc

    University of Dhaka, Bangladesh

Published at

PAKDD, IJCNN

AKM Mahbubur Rahman, PhD
AKM Mahbubur Rhman, PhD

AKM Mahbubur Rahman, PhD

  • Director, DS & IPT Wings

  • Associate Professor

    Department of Computer Science and Engineering

    Independent University, Bangladesh (IUB)

Published at

CHI, ECCV, WACV, PAKDD, LREC, IJCNN, ICPR, ICIP, HCII, BEA, Scientific Reports, Information Sciences

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, Pattern Recognition, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Prof. Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Prof. Amin Ahsan Ali, PhD

  • Director, AI/ML Wing
  • Supervisor, ComBio Wing

  • Professor

    Department of Computer Science and Engineering

    Independent University, Bangladesh (IUB)

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

References

  1. 1.Vincent D. Blondel, Jean‐Loup Guillaume, Renaud Lambiotte, Etienne Lefebvre. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics Theory and Experiment, 2008(10), P10008[10.1088/1742-5468/2008/10/p10008]
  2. 2.Chen, Tianyu, Wang, Zeyu, Li, Lin, Li, Ding, Li, Zongyang, Chang, Xiaoning, Bian, Pan, Liang, Guangtai, Wang, Qianxiang, Xie, Tao. (2025). Detecting Functionality-Specific Vulnerabilities via Retrieving Individual Functionality-Equivalent APIs in Open-Source Repositories. Dagstuhl Research Online Publication Server[10.4230/lipics.ecoop.2025.6]
  3. 3.M. E. J. Newman. (2006). Modularity and community structure in networks. Proceedings of the National Academy of Sciences, 103(23), 8577–8582[10.1073/pnas.0601602103]
  4. 4.Qimai Li, Zhichao Han, Xiao-Ming Wu. (2018). Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1)[10.1609/aaai.v32i1.11604]
  5. 5.M. E. J. Newman. (2004). Analysis of weighted networks. Physical Review E, 70(5), 056131[10.1103/physreve.70.056131]
  6. 6.Guohao Li, Matthias Müller, Ali Thabet, Bernard Ghanem. (2019). DeepGCNs: Can GCNs Go As Deep As CNNs?. , 9266–9275[10.1109/iccv.2019.00936]
  7. 7.Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, Xu Sun. (2020). Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 3438–3445[10.1609/aaai.v34i04.5747]
  8. 8.Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, Shiqiang Yang. (2017). Community Preserving Network Embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1)[10.1609/aaai.v31i1.10488]
  9. 9.Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, Devon Hjelm. (2018). Mutual Information Neural Estimation.. International Conference on Machine Learning, 531–540
  10. 10.Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken‐ichi Kawarabayashi, Stefanie Jegelka. (2018). Representation Learning on Graphs with Jumping Knowledge Networks. International Conference on Machine Learning, 5453–5462
  11. 11.Di Jin, Zhizhi Yu, Pengfei Jiao, Shirui Pan, Dongxiao He, Jia Wu, Philip L. H. Yu, Weixiong Zhang. (2021). A Survey of Community Detection Approaches: From Statistical Modeling to Deep Learning. IEEE Transactions on Knowledge and Data Engineering, 1[10.1109/tkde.2021.3104155]
  12. 12.Yuning You, Tianlong Chen, Zhangyang Wang, Yang Shen. (2020). When Does Self-Supervision Help Graph Convolutional Networks?. PubMed, 119, 10871–10880[link]
  13. 13.Meng Liu, Zhengyang Wang, Shuiwang Ji. (2021). Non-Local Graph Neural Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12), 10270–10276[10.1109/tpami.2021.3134200]
  14. 14.Pengyang Wang, Yanjie Fu, Yuanchun Zhou, Kunpeng Liu, Xiaolin Li, Kien A. Hua. (2020). Exploiting Mutual Information for Substructure-aware Graph Representation Learning. , 3415–3421[10.24963/ijcai.2020/472]
  15. 15.R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio. (2018). Learning deep representations by mutual information estimation and maximization. arXiv (Cornell University)[10.48550/arxiv.1808.06670]
  16. 16.Yu Rong, Wenbing Huang, Tingyang Xu, Junzhou Huang. (2019). DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. arXiv (Cornell University)[10.48550/arxiv.1907.10903]
  17. 17.Petar Veličković, William Fedus, William L. Hamilton, Píetro Lió, Yoshua Bengio, R Devon Hjelm. (2018). Deep Graph Infomax. Apollo (University of Cambridge)[10.17863/cam.40744]
  18. 18.Hongbin Pei, Bingzhe Wei, Kevin Chen–Chuan Chang, Lei Yu, Bo Yang. (2020). Geom-GCN: Geometric Graph Convolutional Networks. arXiv (Cornell University)[10.48550/arxiv.2002.05287]
  19. 19.Di Jin, Zhizhi Yu, Pengfei Jiao, Shirui Pan, Dongxiao He, Jia Wu, Philip S. Yu, Weixiong Zhang. (2021). A Survey of Community Detection Approaches: From Statistical Modeling to Deep Learning. arXiv (Cornell University)[10.48550/arxiv.2101.01669]

Cited By

  1. 1.Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, Philip Yu. (2022). Graph Self-Supervised Learning: A Survey. IEEE Transactions on Knowledge and Data Engineering, 1[10.1109/tkde.2022.3172903]
  2. 2.Bisheng Tang, Xiaojun Chen, Shaopu Wang, Yuexin Xuan, Zhendong Zhao. (2023). Generalized heterophily graph data augmentation for node classification. Neural Networks, 168, 339–349[10.1016/j.neunet.2023.09.021]
  3. 3.Aoran Bao, Cheng Ding. (2026). Autonomic Signature-Driven Anesthesia Depth Monitoring with Biomimetic Wearable ECG and Knowledge Graph-Augmented Deep Networks. Sensors, 26(11), 3498[10.3390/s26113498]
  4. 4.Kang Liu, Wenqing Dai, Xunyuan Liu, Mengtao Kang, Runshi Ji. (2025). NHSH: Graph Hybrid Learning with Node Homophily and Spectral Heterophily for Node Classification. Symmetry, 17(1), 115[10.3390/sym17010115]
  5. 5.Xiaofeng Zeng, Tang Jin. (2023). 3D reconstruction of buildings based on transformer-MVSNet. International Conference on Applied Mathematics, Modelling and Intelligent Computing (CAMMIC), 17, 191[10.1117/12.2686276]
  6. 6.Sun, Feng, Kumar, Ajith, Yang, Guanci, Zhu, Qikui, Zhang, Yiyun, Zhang, Ansi, Makwana, Dhruv. (2021). RRLFSOR: An Efficient Self-Supervised Learning Strategy of Graph Convolutional Networks. arXiv (Cornell University)[10.48550/arxiv.2108.07481]
← All publications