How Graph Neural Networks Are Transforming Industries

Company
AssemblyAI

Date published
Feb. 8, 2024

While AI systems like chat, GPT, or diffusion models have been in the limelight recently, graph neural networks, or GNNs, have been rapidly advancing in the last few years. GNNs have quietly become the dark horse behind many exciting achievements that have made their way from research breakthroughs to actual solutions with large scale deployment. Companies like Uber, Alibaba, Pinterest, and Google have already shifted to GNN based approaches in some of their core products, motivated by the substantial performance improvements exhibited by this method compared to previous state of the art architectures. But what are the actual advantages of graph machine learning? And why do graph neural networks matter in 2024? First of all, what are GNLs at a high level? While traditional machine learning architectures are trained on rectilinear structures or euclidean structured data, such as sequences of tokens like strings of text, tabular data, or images, which are two dimensional grids of pixel values, graph neural networks are designed to learn on graph data. A graph is a representation of items or nodes linked by relations or edges between them. There is good reason to study graphs. Concrete examples range from molecules, social networks, knowledge graphs 3D meshes the Internet all the way to the human brain with its neurons and synapses. In many ways, graphs are a natural and intuitive way to organize the data we receive from nature. But rather than going into the details of how a GNN works, which may be the content for another video, add your requests below. I will now survey a number of cool applications recently made possible by graph neural networks. One of the largest applications of machine learning in general is arguably recommendation systems, which are usually trained on sparse tabular data. Somewhat surprisingly, GNNs have proven to be very effective in modeling certain types of recommendation systems. For example, developers at Uber Eats, the food delivery app, have implemented GNN based methods into their recommendation system, which aims to surface the foods most likely to appeal to an individual user. They reported a performance boost of over 20% compared to the previous production model, and their analysis of impact revealed that the GNN based feature was by far the most influential one in the recommendation model as a whole. Pinterest is a visual discovery engine that operates as a social network where users interact with visual bookmarks called pins that link to web based resources. The pinterest graph, which encompasses 3 billion nodes and over 18 billion edges, represents a massive graph ecosystem with potential implications for understanding a great deal about human preferences. Pinterest actively deploys pinsage, a GNN powered recommendation system able to predict in novel ways which visual concepts that users have found interesting can map to new things that might appeal to them. Researchers reported how pinsage yields a 150% improvement in the hit rate over the previous baseline. Hit rate measures the probability that recommendations made by the algorithm contain the items related to the query. Another highly impactful application of graph neural networks came from a team of researchers at DeepMind who showed how GNNs can be applied to transportation maps to improve the accuracy of the estimated time of arrival, or ETA. The idea is to use GNNs to learn representations of the transportation network that capture its dynamics. This system is actively deployed at scale by Google Maps in several major cities around the world, and the new graph based approach led to up to 50% accuracy improvements compared to the prior approach deployed in production. Last November, Google DeepMind published a blog post where they introduced graphcast, a new weather forecasting model, and open sourced the model's code. The architecture is based on a graph neural network with an encoder processor decoder configuration where the earth's surface is modeled as a homogeneous spatial graph. By terratively refining a regular ecosiaedron six times. Graphcast is now considered to be the most accurate ten day global forecasting system in the world and can predict extreme weather events further into the future than was previously possible. The model is also highly efficient and can make ten day forecasts in less than a minute on a single Google TPU. For comparison, a ten day forecast using a conventional approach can take hours of computation on a supercomputer with hundreds of machines. Another exciting application area for graph neural networks is data mining. Most organizations store their key business data in relational databases, where information is spread across multiple linked tables. Traditionally, machine learning on this data required manual feature engineering to first aggregate the data from all relevant tables into one single table before modeling a process that is time consuming and easily loses information. Recently, researchers proposed a new approach called relational deep learning that leverages GNNs to learn useful patterns and embeddings directly from a relational database without any feature engineering. Let's explain how it works with a concrete example. Consider an ecommerce company with three databases, products, transactions, and customers. The company is interested in predicting future spending, for example, how much customers with some given characteristics will purchase over the next 90 days. Traditionally, this would require joining the multiple tables into one aggregated data set for training a machine learning model with relational deep learning, the database schema is directly converted into a heterogeneous graph representation with node types for products, transactions, and customers, and edges linking related entries across tables. A graph neural network can then be trained on this graph to solve the purchase prediction task end to end, the model takes as input the task along with the target values from the training set, the actual 90 day purchase amounts for past customers. By directly operating on the relational graph, the GNN approach exploits all available information in the databases relevant to the task. Given the ubiquity of relational databases, this technology has immense potential to enable new applications in countless industries. In a paper published in Nature last November, the team at Google DeepMind introduced graph networks for material exploration, or genome, a new deep learning tool that can discover new materials and predict their stability at scale. It is hard to understate the scale of impact on the field as genome expanded overnight the number of stable materials known to humanity by an order of magnitude. Genome leverages GNNs to model materials at the atomic level. The atoms and their bonds are represented as graphs, with nodes denoting individual atoms and edges capturing interatomic interactions. By operating on these graphs, the GNNs can effectively learn to predict the energetic properties of the molecules. Crucially, genome employs active learning in conjunction with density functional theory DFT calculations to iteratively expand its knowledge. Here's what this means. The framework alternates between using its GNNs to screen candidate materials with DfT simulations in the loop that are basically used to verify the model's most uncertain predictions. This creates an automatic feedback loop where the model is continuously retrained on the expanded data set. Remarkably, genome demonstrates some emergent abilities in that it can generalize to entirely new compositions beyond its training distribution. The authors reported how, without ever seeing a five element crystal during training, the model can still make reliable predictions on such structures. Perhaps one of the most famous applications of AI methods in the pharmaceutical domain came out of a research project from MIT that was published in Cell. The goal was to use GNNs to predict the antibiotic activity of molecules by learning their graph representation this way, capturing their potential antibiotic activity. The choice of encoding the information with graphs is very natural in this setting, since antibiotics can be represented as small molecular graphs where the nodes are atoms and the edges correspond to their chemical bonds. This led to a major breakthrough in antibiotic discovery research, which hit the big news. The model identified a new compound named halicin, found to be a highly potent antibiotic and effective against antibiotic resistant bacteria. Interestingly, the researchers reported how the other state of the art models that were also tested against hylicin failed to output a high prediction rank, contrary to the Gnn based approach. Of course, such models are only used to create numerical predictions that then need to be tested in the lab. The model itself is not able a priori to explain its own results. This situation is common to most AI models, and it's the reason why many people refer to them as black boxes. But a breakthrough paper published in Nature at the end of last year from MIT and Harvard seeks to demonstrate something different on this front. The researchers were not only able to predict antibiotic activity of over 12,000 compounds, but did so by using explainable graph algorithms, which by design are able to explain the rationale behind their predictions. Now one thing is to use a model for making predictions on a set of preexisting structures, maybe generated combinatorially, and a different one is to leverage generative methods, for example GPT like architectures or diffusion models, to have a model entirely design new structures from scratch. This brings us to discuss the last area of application of GNNs in this video, when these are combined with generative frameworks for protein design. Protein design seeks to create novel protein structures with desired properties. This is traditionally an experimental process done in the lab through trial and error, and obviously has high costs. Recently, researchers at the Baker lab combined GNNs and diffusion models into a system called Rosetta fold diffusion, or RF diffusion, which can generate protein structures that conform to specified constraints at scale. By conditioning the graph generation process, the user can dictate properties like binding sites or component shapes. It operates via an en equivalent graph neural network, a special kind of GNN expressly designed to process data structures with rigid motion symmetries. These are translations, rotations and reflections in space. Results show RF diffusion solving twice as many protein design challenges as prior state of the art. For example, it achieved an 18% success rate in designing novel protein binders, a task considered the grand challenge in the field. Generate biomedicines went a step further and introduced chroma, an equivalent conditional diffusion model for generating proteins. It was introduced in Nature in late 2023. It's fully open sourced, and the authors also provide an API for creating your own conditioners in a few lines of code. The conditional part is pretty cool. Chroma allows to impose both functional and geometric constraints, and even uses natural language queries like generate a protein with chat domain thanks to a small GPT neo trained on protein captioning I hope this video gave you a general idea of how graph neural networks are impacting a number of very different domains at this point. If you have questions, feel free to leave a comment below and see you in the next video.