Where Candidates Come From
Thunder, Phoenix retrieval, and SimClusters — the three sourcing systems
Three Sourcing Systems
Thunder — In-Network
thunder/ keeps recent posts from accounts you follow in memory. It's fast because it doesn't need to search — it already has the posts. Thunder has a max results cap (default: 1200) and uses a clustering algorithm to organize posts.
Phoenix — Out-of-Network Retrieval
phoenix/ is the ML retrieval system. It embeds you (based on your recent engagement) and every post as vectors, then finds the nearest neighbors. Think of it as: "what would this person likely engage with, even from accounts they don't follow?"
Phoenix has two modes:
- Inference — uses your real-time engagement sequence
- Retrieval — uses a separate retrieval model for broader discovery
SimClusters — Community Detection
simclusters/ groups accounts and posts into clusters based on who engages with what. If you engage with AI/ML content, you'll be associated with the AI cluster, and posts from that cluster will surface in your out-of-network candidates.
The key insight: Phoenix doesn't just find posts similar to what you've liked — it predicts what you'll do next. The embedding is forward-looking, not just a similarity match.
Check Your Understanding
1. What does Thunder store?
- A. All posts on X
- B. Recent posts from accounts you follow, in memory✓ correct
- C. Trending topics
- D. Your DM history
Why: Thunder is the in-network source — it holds recent posts from people you follow in memory for fast retrieval.
2. How does Phoenix find out-of-network posts?
- A. By checking what's trending
- B. By embedding you and posts as vectors and finding nearest neighbors✓ correct
- C. By looking at what your friends liked
- D. By random sampling
Why: Phoenix embeds the viewer and every post as vectors, then finds nearest neighbors — ML-based retrieval.
3. What does SimClusters do?
- A. Ranks posts by engagement
- B. Groups accounts/posts by engagement patterns into clusters✓ correct
- C. Filters spam
- D. Generates post recommendations
Why: SimClusters groups accounts and posts into clusters based on who engages with what — community detection.
Exercise
Think about the last 5 posts you engaged with. Were they from accounts you follow (Thunder) or accounts you don't (Phoenix/SimClusters)? This tells you which sourcing system is driving your feed.
Hint: If most are from people you don't follow, Phoenix is doing heavy lifting for you.