Pub/Sub Topics and Subscriptions: Designing PulseCart's Event Backbone

In Day 1 we defined PulseCart's event taxonomy — what events exist, what they mean, and what triggers them. Now we need somewhere to send them. That's Pub/Sub's job: a fully managed message broker that decouples producers from consumers, handles delivery guarantees, and scales without you managing a single broker node.

Before writing any producer code, it's worth spending time on Pub/Sub's design — specifically topics, subscriptions, and the decisions around them. Getting this right early saves you from painful restructuring once consumers are in production.

---

How Pub/Sub Works

The model is straightforward:

• A topic is a named channel. Producers publish messages to a topic.

• A subscription is an attachment to a topic. Consumers read from subscriptions, not directly from topics.

• One topic can have multiple subscriptions. Each subscription gets its own independent copy of every message published to that topic.

This means you can add a new consumer (a new subscription) without changing the producer or any existing consumers. The producer doesn't know or care who's listening.

Producer (FastAPI)
      │
      ▼
  [Topic: pulsecart.commerce.events]
      │
      ├──▶ [Subscription: sub-realtime-consumer]   → Cloud Run
      ├──▶ [Subscription: sub-cloud-tasks-router]  → Cloud Tasks
      └──▶ [Subscription: sub-airflow-ingestion]   → Cloud Composer DAG

---

PulseCart's Topic Structure

A common mistake is creating one topic for everything. It feels simpler upfront but creates problems: consumers that only care about order.placed still receive every product.viewed event and have to filter client-side. Routing gets messy. Access control becomes coarse.

PulseCart uses three topics, aligned to the event categories from Day 1:

Topic	Event Types	Purpose
pulsecart.user-actions	cart.item_added, product.viewed, search.performed	High-volume, low-criticality user signals
pulsecart.commerce-events	order.placed, cart.abandoned, payment.failed	Business-critical state changes
pulsecart.system-events	message.personalized_send, inventory.low_stock_alert	Internal automation triggers

Separating commerce-events from user-actions is particularly important. Commerce events are higher-stakes — they need tighter retry policies, stricter monitoring, and potentially different IAM permissions. Mixing them with high-volume user action events makes both harder to manage.

---

Push vs Pull Subscriptions

This is the decision that shapes how your consumers are built. Pub/Sub supports two delivery modes:

Pull Subscriptions

Your consumer service calls Pub/Sub's API to fetch messages. It controls the polling rate and acknowledges each message after processing.

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    "your-project-id", "sub-airflow-ingestion"
)

response = subscriber.pull(
    request={"subscription": subscription_path, "max_messages": 100}
)

for msg in response.received_messages:
    print(f"Received: {msg.message.data.decode()}")
    subscriber.acknowledge(
        request={
            "subscription": subscription_path,
            "ack_ids": [msg.ack_id]
        }
    )

Use pull when: the consumer controls its own processing pace (Airflow DAGs pulling batches on a schedule), or when you're processing large volumes and want to batch messages explicitly.

Push Subscriptions

Pub/Sub delivers messages to an HTTPS endpoint — your Cloud Run service URL. No polling needed; messages arrive as HTTP POST requests.

{
  "subscription": "projects/your-project/subscriptions/sub-realtime-consumer",
  "message": {
    "data": "eyJldmVudF90eXBlIjogIm9yZGVyLnBsYWNlZCIsIC4uLn0=",
    "messageId": "136969346945",
    "publishTime": "2025-06-18T10:31:05Z",
    "attributes": {
      "event_type": "order.placed"
    }
  }
}

Your Cloud Run service receives this, processes it, and returns a 2xx to acknowledge. No ack_id management needed.

Use push when: you have a serverless consumer (Cloud Run) that should react immediately to each message, and you want Pub/Sub to handle the delivery rather than polling.

PulseCart's Subscription Breakdown

Subscription	Mode	Consumer	Reasoning
sub-realtime-consumer	Push	Cloud Run	Immediate reaction to commerce events
sub-cloud-tasks-router	Push	Cloud Run (router service)	Creates Cloud Tasks for delayed work
sub-airflow-ingestion	Pull	Cloud Composer	DAGs pull batches on their own schedule

---

Message Ordering

By default, Pub/Sub does not guarantee message ordering across a subscription. For most of PulseCart's events, this is fine — a product.viewed event being delivered slightly out of order has no consequence.

But for commerce events, order can matter. A cart.abandoned event being processed before cart.item_added would trigger an abandonment email for a cart the system doesn't know exists yet.

Pub/Sub handles this with ordering keys. Messages published with the same ordering key are delivered in order to a given subscriber.

from google.cloud import pubsub_v1
import json

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("your-project-id", "pulsecart.commerce-events")

event = {
    "event_type": "cart.abandoned",
    "user_id": "usr_8821",
    "payload": { "cart_id": "cart_99012", "total": 89.99 }
}

future = publisher.publish(
    topic_path,
    data=json.dumps(event).encode("utf-8"),
    ordering_key="usr_8821",       # ordered per user
    event_type="cart.abandoned"    # message attribute for filtering
)

Using user_id as the ordering key ensures all events for a given user are delivered in publish order. This is the right scope for PulseCart — user-level ordering, not global ordering (which would be a performance bottleneck at scale).

Note: ordering keys require the subscription to have message ordering enabled. You can't enable it retroactively on a live subscription without downtime.

---

Dead-Letter Topics

Even with retries, some messages will never be successfully processed — a malformed payload, a consumer bug, a dependency that's permanently unavailable. Without a dead-letter topic, those messages either retry indefinitely (blocking the subscription) or get dropped silently.

PulseCart uses a dead-letter topic per main topic:

pulsecart.commerce-events       →  pulsecart.commerce-events.dead-letter
pulsecart.user-actions          →  pulsecart.user-actions.dead-letter
pulsecart.system-events         →  pulsecart.system-events.dead-letter

When a message exceeds its maximum delivery attempt count (we use 5), Pub/Sub automatically forwards it to the dead-letter topic. A separate monitoring subscription on each dead-letter topic triggers an alert, and a periodic Airflow DAG (Day 5) reconciles and reprocesses eligible dead-lettered messages.

Configuring this in Terraform looks like this:

resource "google_pubsub_subscription" "realtime_consumer" {
  name  = "sub-realtime-consumer"
  topic = google_pubsub_topic.commerce_events.name

  ack_deadline_seconds    = 30
  enable_message_ordering = true

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.commerce_events_dead_letter.id
    max_delivery_attempts = 5
  }

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "300s"
  }
}

The exponential backoff retry policy (10s to 300s) prevents a struggling consumer from hammering a downstream dependency in a tight retry loop.

---

Acknowledgement Deadlines

Every Pub/Sub message has an acknowledgement deadline — the window within which your consumer must ack the message before Pub/Sub considers it undelivered and retries. The default is 10 seconds.

For Cloud Run consumers doing lightweight work (validate, route, trigger), 10–30 seconds is fine. For consumers doing heavier processing (calling an external API, writing to Cloud SQL), you'll want to either extend the deadline or design the consumer to ack fast and handle the heavy work asynchronously via Cloud Tasks (exactly the pattern we cover in Day 4).

---

What's Next

Day 3 builds the FastAPI producer service that validates events against our Pydantic schemas and publishes them to the Pub/Sub topics we've just designed — including how to handle batching, message attributes, and at-least-once delivery in practice.