Soft Merging of Experts with Adaptive Routing
Mixture of Experts model where within the network blocks of layers are weighted averages of different experts
Read more →
Blog - network architecture
← All Posts
Unified Scaling Laws for Routed Language Models
Presentation for: Unified Scaling Laws for Routed Language Models. Shows how MoE models scale wrt parameter size & number of experts
Read more →
Review of Sparse Expert Models in Deep Learning
Survey paper of MoE (Mixture of Expert) Models from 2022. Overview of variations of MoE's, strengths, & future research.
Read more →
Adaptive Mixtures of Local Experts
Notes on the classic: Adaptive Mixtures of Local Experts from 1991 by Hinton
Read more →