Motivation
Learning powerful data embeddings has become a center piece in machine learning for producing superior results. This new trend of learning embeddings from data can be attributed to the huge success of word2vec with unprecedented real-world performance in natural language processing (NLP). The importance of extracting high quality embeddings has now been realized in many other domains such as computer vision (CV) and recommendation systems. The crux of these embeddings is that they are pretrained in an unsupervised fashion on huge amount of data and thus can potentially capture all kind of contextual information. Such embedding learning is further advanced by the incorporation of multi-task learning and transfer learning which allow more generalized embeddings to be learned across different datasets.
Besides natural languages ("sequence" or 1D data) and images (2D or 3D) which are well-structured, the idea of embedding learning has been applied to (irregular) "graph-structured" data for various graph learning tasks, such as node classification or link prediction. Unlike word embeddings where vocabulary is typically finite, graph "vocabulary" is potentially infinite (i.e, count of non-isomorphic graphs). Hence learning contextual based embeddings is crucial. Building upon the success of deep learning in images and words, graph neural networks (GNNs) have been recently developed for various graph learning tasks on graph-structured datasets. Most of existing GNNs are task-specific in the sense that they are trained on datasets via supervised (or semi-supervised) learning with task-specific labels, and thus the trained models cannot be directly applied to other tasks. Figure 1 shows some sample graphs in four different real-world datasets from fields as diverse as bioinformatics and quantum mechanics. While the node features (and their meanings) can differ vastly but the underlying graphs governing them contain isomorphic graph structures. This suggests that learning universal graph embeddings across diverse datasets is not only possible, but can potentially offer the benefits of transfer learning.
Deep Universal Graph Neural Network
We envision a deep universal graph embedding neural network (DUGNN) (see Figure 2), which is capable of the following: 1) It can be trained on diverse datasets (e.g., with different node features) for a variety of tasks in an unsupervised fashion to learn task-independent graph embeddings; 2) The learned graph embedding model can be shared across different datasets, thus enjoying the benefits of transfer learning; 3) The learned model can further be adapted and improved for specific tasks using adaptive supervised learning. From theoretical point of view, we establishes the generalization guarantee of DUGNN model for graph classification task and discuss the role of transfer learning in helping towards reducing the generalization gap. To best of our knowledge, we are the first to propose doing transfer learning in the graph neural network domain.
Publications
-
Learning Universal Graph Neural Network Embeddings With Aid Of Transfer Learning.
Saurabh Verma and Zhi-Li Zhang. (arXiv), 2019 -
Stability and Generalization of Graph Convolutional Neural Networks.
Saurabh Verma and Zhi-Li Zhang. ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2019 -
Graph Capsule Convolutional Neural Networks.
Saurabh Verma and Zhi-Li Zhang. Joint ICML and IJCAI Workshop on Computational Biology (WCB), 2018 -
Hunt For The Unique, Stable, Sparse And Fast Feature Learning On Graphs.
Saurabh Verma and Zhi-Li Zhang. Advances in Neural Information Processing Systems (NIPS), 2017