Mixture of Experts - AI Visual Lab

y = Σ_i g(x)_i · E_i(x), g(x) = TopK(softmax(W_g·x))

⚙ Router Configuration

Number of Experts 4

Temperature 1.00

Load Balance Loss 0.10

▶ Simulation

Click canvas to add input points

🎵 Audio

Volume 30%

⚖ Expert Load Balance

Balance Entropy-

Max Load-

📊 Metrics

Input Points0

Train Steps0

Router Loss0.000

Expert Loss0.000

Total Loss0.000

📚 Mixture of Experts

A Mixture of Experts layer routes each input to a subset of specialized expert networks. A learned gating router decides which experts process each input.

Router: g(x) = TopK(softmax(W_g * x / T))
Output: y = sum of g_i * E_i(x)

Top-1 sends each input to one expert (sparse). Top-2 blends two experts (denser). A load balance loss prevents all inputs from collapsing to a single expert.