← Back to the library
System designDeep dive · 9 min

One big account posts and everyone's feed lags

“A user with millions of followers posts, the fan-out queue backs up for minutes, everyone else's posts show up late, and the feed cache cluster runs hot.”

THE PRINCIPLE

Pay for a post where it is cheapest: at write time for small audiences, at read time for huge ones. Decide per author, not for the whole system.

FIRST MOVES

  1. Graph fan-out queue age and per-post fan-out size; one post with millions of inserts confirms the cause.
  2. Mark authors above a follower threshold (and very high-rate posters) as pull-only; skip their fan-out entirely.
  3. At read time, merge the precomputed feed with recent posts from the pull-only authors the reader follows; cache those author timelines hard, they are read by everyone.
  4. Run fan-out in batched, low-priority workers separate from the post write path, so a backlog delays feeds but never posting.
  5. Cap materialized feeds (keep the last N ids, not bodies) and skip fan-out to inactive followers; rebuild their feed on next login.

TOOLS: THEN → NOW

Fan-out on write for every authorHybrid: push for most authors, pull and merge for high-follower ones
Copy full post bodies into each feedStore post ids in capped per-user lists (e.g. Redis lists or sorted sets) and hydrate on read

PATTERN SNAPSHOT

const PUSH_LIMIT = 10_000; // followers; tune from fan-out cost vs read cost
async function onPost(post: Post) {
  await authorTimeline.append(post.authorId, post.id);
  if ((await followers.count(post.authorId)) > PUSH_LIMIT) return; // pulled at read time
  for await (const ids of followers.batches(post.authorId, 1_000)) {
    await fanoutQueue.send({ postId: post.id, followerIds: ids });
  }
}

async function homeFeed(userId: string, limit = 50) {
  const pushed = await feedStore.latest(userId, limit);
  const stars = await follows.pullOnlyAuthors(userId);
  const pulled = await Promise.all(stars.map((a) => authorTimeline.latest(a, limit)));
  return mergeNewestFirst([pushed, ...pulled]).slice(0, limit);
}

CLOSE THE AI. EXPLAIN THIS.

Why is fan-out on read the right choice for an account with 50 million followers but the wrong one for a user who follows 2,000 small accounts?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed