Scaling Systems to 150 Billion Messages: Lessons from Large-Scale Infrastructure

Designing software that operates reliably when delivering billions of events requires shifting how you think about failure modes, state management, and backpressure.

During my work at Google and Amazon, I have spearheaded initiatives ranging from scaling user bases by 20x to coordinating campaigns dispatched more than 150 billion times. Here are key takeaways and patterns that keep systems resilient at this order of magnitude.

1. Asynchronous Decoupling & Queue Topologies

Direct synchronous RPC calls fail catastrophically when downstream services encounter transient spikes. In high-volume systems, every event should be ingested into a distributed append-only log or message streaming cluster (such as Kafka, AWS Kinesis, or Pub/Sub) before any processing begins.

[Event Ingestion] 
       │
       ▼
 [Partitioned Queue] ──────► [Worker Pool A: Legal Validation]
                                      │
                                      ▼
                             [Worker Pool B: Localization]
                                      │
                                      ▼
                             [Worker Pool C: Dispatch Engine]

Key Considerations:

• Partition Keys: Ensure partition keys are evenly distributed across tenant or user hashes to prevent hot partitions.
• Dead Letter Queues (DLQ): Categorize errors into transient (rate-limiting, timeout) and fatal (schema mismatch, invalid payload). Never retry poison-pill messages indefinitely without backoff and maximum retry thresholds.

2. Managing Legal Compliance at Scale

In modern cloud systems, legal compliance is not an afterthought—it dictates architecture. For instance, user consent storage must guarantee:

1. Immediate Revocation Propagation: If a user updates their privacy settings, ongoing dispatches must abort in real-time.
2. Storage Efficiency: Storing billions of consent state changes requires tiered compression. By utilizing columnar storage and compact delta-encoding, we saved thousands of petabytes in consent infrastructure.

interface ConsentRecord {
  userId: string;
  scope: 'marketing' | 'analytics' | 'mandatory';
  grantedAt: number; // Unix timestamp
  version: number;
}

function shouldDispatch(userConsent: ConsentRecord, campaignScope: string): boolean {
  if (campaignScope === 'mandatory') return true;
  return userConsent.scope === campaignScope && userConsent.version >= REQUIRED_VERSION;
}

3. Graceful Degradation & Throttling

When pushing 150 billion dispatches across international boundaries, downstream internet service providers (ISPs) and internal mail exchangers enforce strict rate quotas.

• Leaky Bucket Rate Limiting: Enforce global and regional token buckets to smooth out traffic curves.
• Circuit Breakers: If downstream error rates exceed 0.5%, back off exponentially and dump pending payloads into intermediate archival buffers.

Summary

Scaling to billions of events isn’t just about throwing more compute at the problem. It is about understanding the weakest link in your dependency chain, respecting legal constraints natively in your data models, and designing for idempotency from day one.