Spot the odd detail in this week's biggest AI story: Z.ai's new GLM-5.3-Flash is described as a "320B-A18B" model. That reads like a typo — is it 320 billion parameters or 18 billion?
It's neither a typo nor a compromise. It's Mixture of Experts (MoE) — the architecture behind most of the models you've heard of lately, including GLM, DeepSeek, Qwen, Grok, and (reportedly) the biggest closed models too. Understanding MoE explains why AI got so cheap, so fast, over the last two years.
This article explains it from zero, with no math prerequisites.
The Problem MoE Solves
A language model's "parameters" are the learned numbers it uses to make predictions. More parameters generally means more capability — a bigger brain can hold more knowledge and handle more nuanced reasoning.
But there's a brutal catch: in a traditional ("dense") model, every parameter participates in processing every token.
Imagine a library where, to answer any question, a librarian must physically walk past and touch every single book — all 320 billion of them — before answering. More books means better answers, but every question gets slower and more expensive in exact proportion to the library's size.
That's a dense model. Capability and cost are chained together. Want a smarter model? Pay more per token. Forever.
The industry asked: what if the librarian only needed to consult the relevant sections for each question?
The Core Idea: Specialists Instead of Generalists
Mixture of Experts splits the model's heavy lifting layer (the feed-forward network, which makes up the bulk of a transformer's parameters) into multiple parallel sub-networks called experts.
Each expert, over training, tends to specialize — not because anyone assigns it a topic, but because gradient descent naturally finds divisions of labor. One expert might get better at code syntax, another at idiomatic phrasing, another at factual recall patterns.
Then comes the crucial addition: a router.
For every single token, the router looks at what's coming through and decides, in real time, which small handful of experts should handle it. The rest of the experts — the vast majority — sit completely idle for that token.
Back to the library: instead of one librarian touching every book, you have dozens of specialist librarians, and a receptionist who instantly routes each question to the 2 most relevant specialists. The library's total knowledge (all 320B books) is available, but each question only ever involves a couple of librarians' worth of work.
Sparse Activation: Where "320B-A18B" Comes From
This selective engagement has a name: sparse activation. And it's the source of that strange notation:
- 320B = total parameters that exist in the model (the "book count")
- A18B = active parameters per token — the "A" literally stands for active
So GLM-5.3-Flash is a 320-billion-parameter model where only about 18 billion parameters fire for any given token. Roughly 6% of the brain works at a time; the other 94% is asleep until a later token needs it.
The consequences are dramatic:
- Capability of a big model. All that knowledge is in the weights, reachable whenever the router sends work its way.
- Cost and speed of a small model. Each token only pays the compute bill for ~18B parameters, not 320B.
- The two are decoupled. Labs can now grow capability without growing per-token cost proportionally. This decoupling is the structural reason API prices fell so hard in 2025–2026.
For contrast, a dense model shows something like "70B" with no "A" — meaning all 70B parameters run for every token.
A Concrete Walkthrough
Let's trace the sentence: "def calculate_total(price):"
- The token
defarrives at an MoE layer. The router scores it against all experts and picks the top 2 — say, experts #3 and #17, which training has shaped for code-structure patterns. - The token
calculate_totalarrives. The router might pick #3 again (still code) plus #41, which handles identifier-style tokens. - Later, if the docstring says "Returns the total price including tax," those tokens route to a completely different pair — maybe #12 and #88, which see more natural-language duty.
Every token gets a tailored team. Across a whole response, nearly every expert contributes somewhere — but never all at once. The system behaves like a fully-staffed hospital where each patient only sees the relevant specialists, not every doctor in the building.
Why Training MoE Models Is Hard (and Why Labs Do It Anyway)
MoE isn't free lunch. Two classic problems shape how these models are built:
Load balancing. If the router sends 80% of traffic to the same few popular experts, you've rebuilt a dense model with extra steps — those experts become bottlenecks while others collect dust. Training includes explicit incentives to spread work evenly.
Expert collapse. Left alone, a router can fall into a winner-takes-all pattern where it always picks the same experts, making the rest dead weight. Modern training recipes use noise, auxiliary losses, and router capacity limits to keep every expert earning its keep.
There's also a hardware reality: MoE serving works best when the experts fit cleanly across GPU memory in a serving cluster. That's engineering rather than theory, but it's part of why the number of experts tends to be a power-of-two-friendly count like 64, 128, or 256.
Labs accept all this complexity for one reason: the economics are transformative. You get frontier-scale knowledge at efficiency-tier serving cost.
MoE Is Why Budget Models Stopped Being Bad
Two years ago, "cheap model" meant "small dense model that forgets the question halfway through." MoE changed the meaning of cheap:
- DeepSeek's V3/R1 line — MoE, and the reason its prices shocked the industry
- Qwen's open-weight flagships — MoE at multiple tiers
- Grok 4 family — MoE
- GLM 5.x, including the new 5.3 Flash — MoE, with GLM-5.3-Flash at 320B-A18B
When you look at a model spec sheet now, the "A-number" is arguably more important than the total for predicting your bill. A 400B-A10B model will often cost less per token than a 32B dense model despite ten times the total parameters.
You can see this pattern across the live catalog on qubax.ai/models — sort by price and notice how many of the cheapest capable models carry an MoE-style active-parameter count.
Dense vs MoE at a Glance
| Dense | Mixture of Experts | |
|---|---|---|
| Parameters used per token | All of them | A small fraction (the "A" number) |
| Cost as capability grows | Rises proportionally | Rises much slower |
| Speed | Slower at equal total size | Faster — less compute per token |
| Total knowledge ceiling | Limited by affordable size | Huge — knowledge stored across many experts |
| Example | Llama 3.3 70B, Mistral Small | GLM 5.3 Flash (320B-A18B), DeepSeek V4, Qwen 3 Max |
What This Means When You Pick a Model
Three practical rules of thumb:
- Total parameters tell you the knowledge ceiling; active parameters predict your cost. Check both numbers.
- MoE budget models punch above their price. An efficiency-tier MoE often matches a much more expensive dense model on routine work — coding assistance, summarization, extraction, classification.
- Routing across models is the real superpower. Because the cheap tier is now genuinely capable, mature setups send 80–90% of traffic to MoE budget models and escalate only the genuinely hard requests to premium tiers.
If you want to experiment with exactly that pattern, Qubax gives you one OpenAI-compatible endpoint across the GLM, DeepSeek, Qwen, Claude, GPT, and Gemini families with usage-based billing — swap models by changing one string: qubax.ai/models
The Bottom Line
Mixture of Experts is the quiet architectural shift that broke the "smarter = more expensive" law. By installing a router that assigns each token to a handful of specialist sub-networks, labs get models that store frontier-scale knowledge (320B parameters) while only paying an 18B-parameter compute bill per token.
The next time you see "320B-A18B" or "2.4T-A95B" in a model announcement, you'll know exactly what you're looking at: a big brain that's frugal by design — and a price war powered by architecture, not just marketing.
FAQ
What does Mixture of Experts (MoE) mean?
It's a model architecture where the network contains many specialist sub-networks ("experts"), and a router selects only a few of them to process each token. Most of the model stays idle at any moment.
What does "320B-A18B" mean?
The model contains 320 billion total parameters, but only about 18 billion are active for any given token. The "A" stands for active.
Is MoE better than dense models?
Neither is strictly better. MoE delivers far better cost-to-capability at scale, which is why most large modern models use it. Dense models remain simpler to train and serve, and dominate at smaller sizes.
Does MoE make models cheaper to run?
Yes — dramatically. Since each token only activates a fraction of the parameters, compute per token (your main cost driver) is a fraction of an equally-sized dense model.
Which models use Mixture of Experts?
Most recent large models: the GLM 5.x family (including GLM-5.3-Flash), DeepSeek's V3/V4 line, Qwen's flagship tiers, and Grok 4 family, among others.
Do the "experts" have named specialties?
Not officially. Specialization emerges naturally during training — some experts gravitate toward code-like patterns, others toward prose — but no one assigns labels. It's an emergent division of labor.
How do I try an MoE model cheaply?
Browse the catalog at qubax.ai/models — GLM 5.3 Flash and DeepSeek V4 Flash are among the cheapest capable models listed, with usage-based billing and no separate accounts per lab.
