Mixture of Experts: How an Ensemble of AI Models Decide As One

Mixture-of-Experts (MoE) is a technique in artificial neural networks that allows for efficient scaling of model capabilities without introducing significant computational overhead. Proposed in 1991, MoE adopts a conditional computation paradigm by selectively activating parts of an ensemble, or "experts," based on the data at hand. In recent years, MoE has gained popularity with the rise of large language models and transformer-based models due to their ability to handle complex datasets. The MoE architecture consists of dividing a dataset into local subsets, training expert models for each subset, using a gating model to interpret predictions from each expert and decide which expert to trust for a given input, and employing a pooling method to make a prediction based on the output from the gating network and the experts. In 2017, an extension of MoE suited for deep learning was proposed by Noam Shazeer et al., introducing the Sparsely-Gated Mixture-of-Experts Layer, which consists of numerous expert networks and a trainable gating network that dynamically selects a sparse combination of these experts to process each input. MoE has shown impressive results in domains like NLP and computer vision, but there is still much room for exploration and improvement in its design and application across various fields.