How Pub/Sub systems work.

I'm currently learning System Design - and writing notes and blogs the explain concepts in simpler terms (which also helps me learn better). This article is a part of this learning series.

I was reading about the Pub-Sub model and went looking for an intuitive way to picture it. This write-up contains an intutitve example and interactive diagram for the same.

The problem

Imagine your app has a hundred services — order management, notifications, analytics, inventory. Every time a user places an order, all of them need to know.

A naive solution — have each service call every other service directly.

Real World Solution — You decouple producers from consumers using a message queue. So, by design, any service (e.g. Order Service) can fire an event and forget, and the consumer services can read at their own pace.

This is the Pub/Sub — publish / subscribe — model

How Pub/Sub works

The system has three actors:

• Producer — Publishes messages to a topic.
• Consumer — Subscribes to a topic and processes messages.
• Broker / pub-sub system — The middleman. Receives messages from producers and serves them to consumers.

// The basic flow

Producer  →  publishes to  →  Topic  →  stored in  →  pub-sub system
                                                          ↓
                                          Consumer A  reads independently
                                          Consumer B  reads independently
                                          Consumer C  reads independently

— Topic (concept) — A named channel for a category of messages: order-placed, payment-processed, user-signed-up. Producers write to it and consumers read from it.

The internals — a library you can walk through

Here is the analogy I used —

Picture a large library — which stores books(events) about food delivery orders.

1. **Topic — all books on "order placed" — are placed in one section of the library. This sections is the topic.
2. Broker — the bookcase. Books need physical space. A broker is a bookcase that physically holds events/data on disk.
3. Partition — a volume. Usually an event is far too big to store in one book, so it's split into numbered volumes. Each volume is an independent, ordered stream. Different volumes sit on different bookcases — so ten readers can read ten volumes at once instead of queueing at one shelf.
4. Segment — a chapter. Each volume is split into chapters. In real life — a chapter is made up of fixed range of pages (say 100-200). So through the "contents table" you can jump directly to the pg 100 to read that chapter. Similarly, dividing a volume into chapter allows us to jump to the right page without reading from page 1.
5. Offset — a page number. Every chapter(segment) has a page number inside it. This allows a reader to remember where it was and picks up from there after a break.
6. Producer — the scribe. Writes new pages into the active chapter and moves on. The scribe never waits to see who reads them.
7. Consumer — the reader. Opens a volume at a remembered page, reads at their own pace, and updates their bookmark as they go. One reader falling behind never slows the scribe.
8. Application — the reading department. A single reader is rarely the whole story. A department — say Delivery Dispatch — sends a team of readers in together and splits the volumes between them, so no two read the same page and the work finishes faster. That department is an application (a consumer group); the wider library may have many of them — Dispatch, Analytics, Fraud — each reading the same section for its own reasons, none aware of the others.
9. Replication — Photocopied volumes — Each volume has copies in other wings. If one bookcase burns, the copy survives.

A concrete walkthrough: the order-placed topic

The library makes the parts memorable. Now let's learn with an exmaple — a food-delivery app with one topic — order-placed. Every time a customer orders, the producer publishes a message — and the broker has to decide which partition it lands in.

Topics, partitions, and offsets

There are three partitions, on its independent broker. Interact by using Send order. You'll see orders being send from producer and notice where and how they're placed in partitions.

Segments on disk

Now we look into single partitions. You descide a range for segments, and then the offsets stick to it. Once a segment is full, it's sealed and a new active segment is created. This helps in maintaining ordered data.

That's why offsets stay cheap to look up (even when a partition holds billions of messages) — because the broker never have to scan from offset 0, it automatically indexes to the right segment.

Putting it all together

Producer  (Swiggy app)
    │   publishes to "order-placed"
    ▼
┌─────────── weighted round robin / round robin ───────────┐
▼                    ▼                        ▼
P0  Broker 1         P1  Broker 2             P2  Broker 3
│
├── Segment 1   offsets 0–999      (sealed)
├── Segment 2   offsets 1000–1999  (sealed)
└── Segment 3   offsets 2000–now   (active)  ◀── writes land here
│
▼
Consumer  (delivery-assignment service)
reads from offset 2001 onward, commits as it goes

*The elegance of Pub/Sub is in what it removes — producers never wait for consumers, consumers never block producers.