It lived on their laptop. Which IT had wiped.

Re-importing 23 GCP resources manually — Cloud Run services, Pub/Sub topics, Cloud SQL instances, IAM bindings — from a project with no consistent naming convention or resource tags took two days. Two days of terraform import, gcloud commands, and guessing which resource in the console matched which block in the config.

State, secrets, and environment configuration are the parts of Terraform that feel like infrastructure concerns and get deferred. They're the decisions that quietly determine whether your setup survives the team scaling, an engineer leaving, or a CI pipeline running on the wrong environment.

State

Terraform state is the source of truth for what exists. It maps your HCL config to real cloud resources. Without it, Terraform doesn't know what's deployed — it can't plan changes, detect drift, or destroy resources cleanly.

Local state (terraform.tfstate in your project directory) is a single point of failure. It can't be shared across a team. It disappears when a laptop dies or gets wiped. Never use it for anything beyond a personal experiment.

Remote state in GCS is the minimum viable setup:

# versions.tf
terraform {
  backend "gcs" {
    bucket = "pulsecart-terraform-state"
    prefix = "terraform/state"
  }
}

Create the bucket before running terraform init:

gsutil mb -l us-central1 gs://pulsecart-terraform-state
gsutil versioning set on gs://pulsecart-terraform-state

Versioning is what gives you state history. If a bad terraform apply corrupts state, you can restore a previous version from the bucket. Without versioning, that option doesn't exist.

GCS also handles state locking natively — when one engineer or CI job is running terraform apply, the state file is locked and other applies are blocked. No two applies can corrupt state simultaneously.

The three commands you'll need when something goes wrong:

# List all resources Terraform knows about
terraform state list

# Inspect a specific resource's state
terraform state show google_cloud_run_v2_service.producer

# Import an existing resource into state (when recovering from lost state)
terraform import google_cloud_run_v2_service.producer \
  projects/your-project/locations/us-central1/services/pulsecart-producer

terraform import is how you recover from the laptop situation — but it requires knowing the exact resource ID for every resource, which is why tagging and consistent naming conventions are worth enforcing from day one.

Secrets

Terraform state stores resource attributes in plaintext JSON. If you pass a database password as a variable and Terraform writes it to a resource, it's in your state file. In plaintext. In a GCS bucket that whoever has GCS access can read.

Two rules:

Never put secrets in tfvars files. prod.tfvars gets committed to Git. Even if it's in .gitignore today, it gets committed by accident eventually.

Read secrets from GCP Secret Manager at apply time:

# Read the DB password from Secret Manager during terraform apply
data "google_secret_manager_secret_version" "db_password" {
  secret  = "pulsecart-db-password"
  version = "latest"
}

resource "google_sql_database_instance" "pulsecart" {
  name             = "pulsecart-postgres"
  database_version = "POSTGRES_15"
  region           = var.gcp_region

  settings {
    tier = var.db_tier
  }
}

resource "google_sql_user" "app" {
  name     = "pulsecart_app"
  instance = google_sql_database_instance.pulsecart.name
  password = data.google_secret_manager_secret_version.db_password.secret_data
}

The secret value is fetched at apply time from Secret Manager — it never lives in your tfvars, your repo, or your CI environment variables. The only thing in Git is the secret name.

For variables that contain sensitive values, mark them explicitly:

# variables.tf
variable "db_password" {
  type      = string
  sensitive = true   # redacted from plan output and logs
}

sensitive = true prevents the value from appearing in terraform plan output or in CI logs. It doesn't prevent the value from being stored in state — which is why reading from Secret Manager rather than passing as a variable is the better pattern for anything genuinely sensitive.

Environments

Terraform Workspaces are the built-in answer to environments. They feel right — one codebase, multiple workspaces, each with isolated state. In practice they have a footgun: all workspaces share the same backend bucket prefix by default, just with a workspace name injected. It's easy to accidentally run terraform apply against the wrong workspace, especially in CI where the workspace is set by an environment variable.

Use tfvars files per environment instead:

environments/
├── dev.tfvars
├── staging.tfvars
└── prod.tfvars

# environments/dev.tfvars
gcp_project_id        = "pulsecart-dev"
environment           = "dev"
db_tier               = "db-g1-small"
db_availability_type  = "ZONAL"
min_instances         = 0
max_instances         = 5
deletion_protection   = false

# environments/prod.tfvars
gcp_project_id        = "pulsecart-prod"
environment           = "prod"
db_tier               = "db-custom-4-15360"
db_availability_type  = "REGIONAL"
min_instances         = 1
max_instances         = 20
deletion_protection   = true

Apply explicitly with the right var file:

# Dev
terraform apply -var-file="environments/dev.tfvars"

# Prod
terraform apply -var-file="environments/prod.tfvars"

In CI, the var file is determined by the branch or environment the pipeline runs against — explicit, auditable, hard to accidentally misapply.

Use workspaces when you have genuinely identical environments that differ only by region or account — multi-region deployments, for example. For dev/staging/prod, tfvars is clearer and safer.

Variable Validation

Catch misconfiguration at plan time, not after a broken apply:

# variables.tf
variable "environment" {
  type = string
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of: dev, staging, prod"
  }
}

variable "db_tier" {
  type = string
  validation {
    condition     = can(regex("^db-", var.db_tier))
    error_message = "db_tier must be a valid Cloud SQL tier starting with 'db-'"
  }
}

variable "min_instances" {
  type = number
  validation {
    condition     = var.min_instances >= 0 && var.min_instances <= var.max_instances
    error_message = "min_instances must be >= 0 and <= max_instances"
  }
}

terraform plan fails with a clear error message if validation fails. No apply, no partial deployment, no guessing what went wrong. One line of validation per constraint costs nothing and prevents a category of misconfiguration that's otherwise only caught in production.

The laptop incident was avoidable at every step — remote state from day one, a tagging policy, a runbook for what to do when someone leaves the team. None of it is complex. It's just the kind of setup that gets skipped when you're moving fast and the project is small.

It stops being small eventually.

What nobody noticed: the cache TTL was set to 7 days. When the API came back up, the service kept serving cached results for users whose inputs had changed — because the cache key was too coarse and the invalidation logic had a bug. No errors, no alerts. Clients were seeing stale AI-generated content for days.

The fallback worked exactly as coded. The design was wrong.

Building LLM integrations that hold up in production means getting three things right: how you call the model, how you retry when it fails, and what you actually do when it's unavailable. All three are worth more thought than they usually get.

Structuring the LLM Client

Don't scatter openai.chat.completions.create() calls across your service layer. Wrap the client once, in one place, with all the configuration and error handling it needs:

# services/llm_client.py
from openai import AsyncOpenAI
from config import settings
import logging

logger = logging.getLogger(__name__)

client = AsyncOpenAI(
    api_key=settings.openai_api_key,
    timeout=30.0,        # hard timeout — don't let slow responses hang indefinitely
    max_retries=0,       # handle retries yourself, not via the SDK's built-in retry
)

async def call_model(
    prompt: str,
    system: str,
    model: str = "gpt-4o-mini",
    max_tokens: int = 1000,
    temperature: float = 0.2,
) -> str:
    response = await client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": prompt},
        ],
        max_tokens=max_tokens,
        temperature=temperature,
    )
    content = response.choices[0].message.content
    logger.info(
        "LLM call complete",
        extra={
            "model": model,
            "prompt_tokens": response.usage.prompt_tokens,
            "completion_tokens": response.usage.completion_tokens,
            "total_tokens": response.usage.total_tokens,
        }
    )
    return content

max_retries=0 is intentional — the SDK's built-in retry is a black box. You want explicit control over when, how many times, and with what backoff you retry. More on that below.

timeout=30.0 is a hard limit on how long a single API call can take. Without it, a slow model response holds your Cloud Run instance's concurrency slot indefinitely.

For Gemini, the structure is the same pattern with a different client:

import google.generativeai as genai
from config import settings

genai.configure(api_key=settings.gemini_api_key)
gemini_model = genai.GenerativeModel("gemini-1.5-flash")

async def call_gemini(prompt: str, system: str) -> str:
    response = await gemini_model.generate_content_async(
        f"{system}\n\n{prompt}"
    )
    return response.text

The key difference: Gemini doesn't have a separate system role in the same way — you prepend system instructions to the user prompt. If you're abstracting across both providers, account for this in your prompt builder.

Prompt Management

Prompts hardcoded as string constants in service files are a maintenance problem. They can't be iterated without a code deployment. They can't be versioned or rolled back independently. They can't be tested in isolation.

Store prompts as versioned config, loaded at startup:

# prompts/store.py
import json
from pathlib import Path
from config import settings

_prompts: dict = {}

def load_prompts():
    prompt_file = Path(f"prompts/{settings.environment}.json")
    with open(prompt_file) as f:
        global _prompts
        _prompts = json.load(f)

def get_prompt(key: str, version: str = "latest") -> dict:
    prompt_versions = _prompts.get(key, {})
    if version == "latest":
        version = max(prompt_versions.keys())
    return prompt_versions[version]

// prompts/prod.json
{
  "document_extraction": {
    "v1": {
      "system": "You are a document extraction assistant. Extract structured data from the provided document. Return only valid JSON matching the schema provided.",
      "user_template": "Extract the following fields from this document:\n{fields}\n\nDocument:\n{document}"
    },
    "v2": {
      "system": "You are a precise document extraction assistant. Extract only the explicitly stated fields. If a field is not present, return null. Return only valid JSON.",
      "user_template": "Extract these fields: {fields}\n\nDocument content:\n{document}\n\nReturn JSON only, no explanation."
    }
  }
}

# Usage in service
from prompts.store import get_prompt

prompt_config = get_prompt("document_extraction", version="v2")
system = prompt_config["system"]
user = prompt_config["user_template"].format(
    fields=", ".join(required_fields),
    document=document_text
)
result = await call_model(prompt=user, system=system)

Rolling back a bad prompt is now a config change, not a code deployment. A/B testing two prompt versions is straightforward. The service code doesn't change when prompts do.

Output Validation

Model output is untrusted input. Validate it before using it:

# services/extraction.py
from pydantic import BaseModel, ValidationError
from services.llm_client import call_model
from prompts.store import get_prompt
import json
import logging

logger = logging.getLogger(__name__)

class ExtractionResult(BaseModel):
    invoice_number: str
    total_amount: float
    vendor_name: str
    date: str
    line_items: list[dict]

async def extract_document(document_text: str) -> ExtractionResult:
    prompt_config = get_prompt("document_extraction")
    raw = await call_model(
        prompt=prompt_config["user_template"].format(document=document_text),
        system=prompt_config["system"],
    )

    # Strip markdown fences if model wraps output in ```json ... ```
    clean = raw.strip().removeprefix("```json").removesuffix("```").strip()

    try:
        data = json.loads(clean)
        return ExtractionResult(**data)
    except (json.JSONDecodeError, ValidationError) as e:
        logger.error("Output validation failed", extra={"raw_output": raw, "error": str(e)})
        raise ValueError(f"Model returned unparseable output: {e}")

The removeprefix/removesuffix for markdown fences is not optional — models frequently wrap JSON output in code fences even when instructed not to. Strip them before parsing.

Retry Logic

Retries on LLM APIs need explicit backoff. The failure modes worth retrying are rate limit errors (429) and server errors (500, 503). Don't retry on invalid request errors (400) or auth errors (401) — those won't resolve with a retry.

# services/llm_client.py (extended)
import asyncio
from openai import RateLimitError, APIStatusError

async def call_model_with_retry(
    prompt: str,
    system: str,
    max_attempts: int = 3,
    base_delay: float = 1.0,
    **kwargs,
) -> str:
    last_error = None

    for attempt in range(max_attempts):
        try:
            return await call_model(prompt, system, **kwargs)

        except RateLimitError as e:
            last_error = e
            delay = base_delay * (2 ** attempt)   # exponential backoff
            logger.warning(f"Rate limited, retrying in {delay}s (attempt {attempt + 1}/{max_attempts})")
            await asyncio.sleep(delay)

        except APIStatusError as e:
            if e.status_code in (500, 503):
                last_error = e
                delay = base_delay * (2 ** attempt)
                logger.warning(f"API error {e.status_code}, retrying in {delay}s")
                await asyncio.sleep(delay)
            else:
                raise   # don't retry 400, 401, 422

    raise last_error

Cap retries at 3. Add jitter if you're making many concurrent calls (multiply delay by random.uniform(0.8, 1.2)) to avoid thundering herd when multiple instances hit a rate limit simultaneously.

Fallbacks That Don't Silently Mislead

The opening story failed because the fallback served data that was no longer valid without telling anyone. A good fallback is explicit about what it's doing:

# services/extraction.py (with fallback)
from db.queries.extractions import get_cached_extraction
from datetime import datetime, timedelta

MAX_CACHE_AGE = timedelta(hours=1)   # not 7 days

async def extract_document_with_fallback(
    document_id: str,
    document_text: str,
) -> tuple[ExtractionResult, bool]:
    """Returns (result, is_fresh) — callers decide what to do with stale results."""

    try:
        result = await call_model_with_retry(
            prompt=..., system=...
        )
        fresh_result = ExtractionResult(**json.loads(result))
        await cache_extraction(document_id, fresh_result)
        return fresh_result, True

    except Exception as e:
        logger.error(f"LLM call failed, attempting cache fallback: {e}")
        cached = await get_cached_extraction(document_id)

        if cached and (datetime.utcnow() - cached.created_at) < MAX_CACHE_AGE:
            return cached.result, False   # is_fresh=False signals to the caller

        raise   # no usable cache — fail explicitly

The caller receives is_fresh=False and decides what to do — surface a "results may be outdated" indicator to the user, skip the result entirely, or queue a refresh job. The fallback doesn't hide that it's a fallback.

Wiring It Into FastAPI

# routers/documents.py
@router.post("/documents/{document_id}/extract")
async def extract(document_id: str, request: Request):
    document = await get_document(document_id)

    result, is_fresh = await extract_document_with_fallback(
        document_id=document_id,
        document_text=document.text,
    )

    return {
        "result": result.model_dump(),
        "fresh": is_fresh,
        "extracted_at": datetime.utcnow().isoformat(),
    }

fresh in the response gives the client full information. A dashboard can show "results from 47 minutes ago" instead of presenting stale data as current.

Here's what that looks like in numbers.

You configure your SQLAlchemy pool with pool_size=5 and max_overflow=5 — ten connections per instance. Sounds reasonable. At 10 Cloud Run instances under moderate load, that's 100 connections. At 50 instances during a traffic spike, that's 500. PostgreSQL's default max_connections is 100. At 20 instances you've already exceeded it. New connections get rejected. Requests fail.

This isn't a Cloud Run bug. It's the expected behaviour of two systems — one that scales horizontally without limit, one that has a hard connection ceiling — combined without accounting for the interaction.

Why It's Worse Than It Looks

The math above assumes every instance is using its full pool. In practice it's more unpredictable. Cloud Run instances don't scale down immediately. Idle instances sit around holding open connections they're not using. Your effective connection count at any given moment is max_connections_per_instance × live_instances — and live instances includes the idle ones.

Cloud SQL's connection limits vary by machine tier. A db-g1-small (the default for dev environments that get accidentally promoted to prod) caps at around 25 connections. A db-custom-4-15360 caps at around 4,000. If you're not sure what tier your instance is, you're probably not sure what your connection ceiling is.

-- Check your current connection usage and limit
SELECT
  count(*) as active_connections,
  max_conn as connection_limit
FROM pg_stat_activity, (SELECT setting::int AS max_conn FROM pg_settings WHERE name = 'max_connections') AS limits
GROUP BY max_conn;

Run this during a load test. The number is usually higher than expected.

The Fix: Pool Sizing That Accounts for Horizontal Scale

The formula isn't pool_size = how_many_feels_right. It's:

max_pool_size_per_instance = floor(db_max_connections / max_cloud_run_instances)

Leave headroom for Cloud SQL's internal processes (~3 connections) and for admin access during incidents. A practical formula:

usable_connections = db_max_connections - 5
max_pool_per_instance = floor(usable_connections / max_cloud_run_instances)

For a db-custom-2-7680 with 500 max connections and 20 max Cloud Run instances:

usable_connections = 495
max_pool_per_instance = floor(495 / 20) = 24

Set pool_size below that ceiling with headroom for overflow:

# config.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    db_pool_size: int = 5
    db_max_overflow: int = 10
    db_pool_timeout: int = 30
    db_pool_recycle: int = 1800   # recycle connections every 30 minutes

settings = Settings()

# db/pool.py
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession, async_sessionmaker
from config import settings

engine = create_async_engine(
    settings.database_url,
    pool_size=settings.db_pool_size,
    max_overflow=settings.db_max_overflow,
    pool_timeout=settings.db_pool_timeout,
    pool_recycle=settings.db_pool_recycle,
    pool_pre_ping=True,   # verify connections before using them
)

AsyncSessionLocal = async_sessionmaker(
    engine,
    class_=AsyncSession,
    expire_on_commit=False,
)

# dependencies.py
from db.pool import AsyncSessionLocal
from typing import AsyncGenerator
from sqlalchemy.ext.asyncio import AsyncSession

async def get_db() -> AsyncGenerator[AsyncSession, None]:
    async with AsyncSessionLocal() as session:
        yield session

pool_pre_ping=True adds a lightweight SELECT 1 before each connection is used. It catches stale connections that Cloud SQL dropped on its end — without it, a connection that's been idle for Cloud SQL's timeout period returns an error on the first query instead of reconnecting cleanly.

pool_recycle=1800 proactively replaces connections older than 30 minutes. Cloud SQL drops idle connections after a configurable timeout (default 10 minutes). Recycling before that point prevents the errors that pool_pre_ping catches — it's the belt to pre_ping's suspenders.

The Better Fix: PgBouncer

Pool sizing helps. It doesn't solve the fundamental problem — Cloud Run's horizontal scaling means your connection count grows with your instance count, and at high scale, even a small pool per instance adds up.

PgBouncer sits between your application and Cloud SQL and maintains a fixed pool of server-side connections regardless of how many application instances connect to it. A hundred Cloud Run instances can each open 10 connections to PgBouncer; PgBouncer maintains 50 actual connections to PostgreSQL.

On GCP, the cleanest way to run PgBouncer is as a Cloud Run sidecar or a dedicated Cloud Run service:

# cloud-run-service.yaml (simplified)
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: pulsecart-api
spec:
  template:
    spec:
      containers:
        - image: gcr.io/your-project/pulsecart-api
          env:
            - name: DATABASE_URL
              value: postgresql+asyncpg://user:pass@localhost:5432/pulsecart
        - image: edoburu/pgbouncer
          env:
            - name: DATABASE_URL
              value: postgresql://user:pass@/cloudsql/your-project:region:instance/pulsecart
            - name: POOL_MODE
              value: transaction
            - name: MAX_CLIENT_CONN
              value: "1000"
            - name: DEFAULT_POOL_SIZE
              value: "50"

POOL_MODE=transaction is the right mode for stateless APIs — PgBouncer assigns a server connection for the duration of a transaction only, not the entire client session. This gives the highest multiplexing efficiency for request-per-transaction workloads like FastAPI.

The tradeoff: transaction mode doesn't support session-level features like SET LOCAL, advisory locks, or LISTEN/NOTIFY. If you use any of these, use session mode instead and accept lower multiplexing efficiency.

Cloud SQL Auth Proxy

If you're connecting to Cloud SQL via the Auth Proxy (which you should be — it handles IAM auth and TLS termination), the proxy itself doesn't pool connections. It's a pass-through. All the pooling logic above still applies; the proxy just sits between your application and Cloud SQL handling the authentication layer.

Run the Auth Proxy as a sidecar alongside your application container, not as a separate service — the connection overhead between containers on the same Cloud Run instance is negligible, and it avoids the network hop of a separate service.

The Monitoring You Need

Set this up before you need it:

# Add to your /health endpoint
from sqlalchemy import text

@app.get("/health")
async def health(db: AsyncSession = Depends(get_db)):
    try:
        result = await db.execute(text(
            "SELECT count(*) FROM pg_stat_activity WHERE datname = current_database()"
        ))
        active_connections = result.scalar()
        return {"status": "ok", "db_connections": active_connections}
    except Exception:
        return JSONResponse(status_code=503, content={"status": "degraded"})

Expose connection count in your health endpoint. Set a Cloud Monitoring alert when it approaches 80% of your max_connections. At 80% you have time to react. At 100% you're already dropping requests.

Summary

Cloud Run + PostgreSQL without connection management is a scaling time bomb. The failure doesn't happen at low traffic — it happens exactly when you can least afford it.

Size your pool to account for horizontal scaling. Add pool_pre_ping and pool_recycle. Monitor connection count as a first-class metric. Add PgBouncer when the math stops working in your favour.

The problem is predictable. The fix is straightforward. The only reason it surprises teams is that nobody mentions it until production is on fire.

Three days after going live, client reports started coming in. Extracted data was missing fields. Some records had partial output. A handful had nothing at all. The service wasn't throwing errors. Logs showed successful API calls. The model was returning responses — they just weren't what we expected.

The root cause: in testing, we'd used short, simple documents. In production, clients uploaded complex multi-page PDFs. The model's output structure drifted — fields appeared in different orders, optional sections were omitted, the JSON we were parsing sometimes contained prose where we expected a value. We'd built the parsing layer assuming the model always returned a consistent structure. It didn't.

No error. No alert. Silent data loss.

Why AI Integrations Are Different

A REST API call either succeeds or fails in a way you can detect. Status codes, error bodies, timeouts — the failure modes are well-defined. You write a retry, you write an error handler, and you know what you're dealing with.

LLM API calls fail differently. The HTTP request can return 200 with a well-formed response body that contains output you can't use. The model might:

• Return valid JSON that doesn't match your expected schema

• Truncate output mid-sentence when it hits a token limit

• Interpret an ambiguous prompt differently than it did in testing

• Produce structurally correct output with factually wrong values

• Respond slowly enough to exceed your downstream timeout, but fast enough that you don't configure a sensible one

None of these show up as errors in your API client. They show up as data quality issues, silent failures, or confused end users — often days after the fact.

The Five Reasons Most Integrations Break

1. No output validation

The most common failure. You call the model, get a response, and pass it downstream without checking whether it matches what you need. In testing, the model is consistent enough that this works. In production, edge cases in real data expose inconsistencies you didn't anticipate.

The fix isn't clever prompt engineering — it's treating model output like untrusted external input and validating it the same way you'd validate an API response from a third party. Pydantic, schema validation, explicit checks before any downstream write.

2. Prompts treated as static strings

A prompt that works today may not work after a model update, a context window change, or a shift in the input data. Teams that hardcode prompts as string literals in their service code have no way to iterate, version, or roll back a prompt change without a code deployment.

Prompts are configuration, not code. They belong in a versioned store, not string constants.

3. No fallback when the model fails

LLM APIs have rate limits, occasional downtime, and elevated latency during high-demand periods. Services that make synchronous blocking calls to an LLM API in the request path are one OpenAI outage away from a full service degradation.

Every AI integration needs a defined behaviour for when the model is unavailable: queue the work and process it later, return a graceful degraded response, or fail fast with a meaningful error rather than hanging.

4. Token limits treated as someone else's problem

Context windows have limits. Large inputs get truncated, sometimes silently, sometimes with a clear error, depending on how you're calling the API. Teams that don't explicitly manage token counts end up with truncated outputs that look complete until someone looks closely.

This is especially common with document processing — a service built on short test documents works fine until a client uploads a 40-page PDF.

5. Cost left unmonitored until the bill arrives

LLM API costs scale with usage in ways that aren't always intuitive. A feature that costs pennies in testing can cost hundreds of dollars in production if usage is higher than expected, if prompts are larger than necessary, or if a retry loop is accidentally calling the API in a tight loop.

Without per-endpoint cost tracking and usage alerts, the first signal is a monthly bill that's ten times what you expected.

The document extraction service wasn't badly built. It just treated the LLM as a reliable structured data source rather than a probabilistic system that needs the same defensive patterns you'd apply to any external dependency.

We restructured twice before landing on something that actually scaled with the team. Here's what we learned.

The Mistakes

Everything in main.py. Feels clean at the start. Becomes a 1,500-line file that nobody wants to open. Every feature touches it. Every PR conflicts with the one before it.

Models mixed with business logic. Pydantic schemas sitting next to database query functions sitting next to helper utilities. No clear layer boundaries. A junior engineer joining the team has no idea where to put a new thing.

No separation between internal and external schemas. Using the same Pydantic model for API input, database writes, and inter-service events. One schema change breaks three things at once.

Config scattered across files. Environment variables read directly in route handlers, service files, and utility modules. No single place to look at what the application needs to run.

The Structure That Fixed It

pulsecart-api/
├── main.py                  # app factory only — no routes, no logic
├── config.py                # all env vars in one place
├── dependencies.py          # shared FastAPI dependencies (db pool, auth, etc.)
│
├── routers/
│   ├── __init__.py
│   ├── events.py            # POST /events/ingest
│   ├── orders.py            # GET/POST /orders
│   └── health.py            # GET /health, GET /ready
│
├── models/
│   ├── __init__.py
│   ├── base.py              # PulseCartEvent base schema
│   ├── commerce.py          # OrderPlaced, CartAbandoned, etc.
│   └── internal.py          # schemas for inter-service communication
│
├── services/
│   ├── __init__.py
│   ├── publisher.py         # Pub/Sub publishing logic
│   ├── idempotency.py       # Redis deduplication
│   └── messaging.py         # personalized message triggering
│
├── db/
│   ├── __init__.py
│   ├── pool.py              # connection pool setup
│   └── queries/
│       ├── orders.py        # SQL for order operations
│       └── events.py        # SQL for event logging
│
└── tests/
    ├── conftest.py
    ├── unit/
    └── integration/

Five folders, clear ownership. A new engineer can look at this and know exactly where to find something and where to put something new.

main.py — App Factory Only

# main.py
from contextlib import asynccontextmanager
from fastapi import FastAPI
from routers import events, orders, health
from db.pool import init_pool, close_pool
from config import settings
import logging

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.db = await init_pool(settings.database_url)
    yield
    await close_pool(app.state.db)

app = FastAPI(
    title="PulseCart API",
    version="1.0.0",
    lifespan=lifespan
)

app.include_router(events.router)
app.include_router(orders.router)
app.include_router(health.router)

main.py does three things: lifespan management, router registration, app config. Nothing else. If you're adding business logic here, it belongs in services/.

config.py — All Env Vars in One Place

# config.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    redis_url: str
    gcp_project_id: str
    pubsub_topic_commerce: str = "pulsecart.commerce-events"
    environment: str = "dev"
    log_level: str = "INFO"

    class Config:
        env_file = ".env"

settings = Settings()

Every environment variable the application needs is declared here with a type and an optional default. If a required variable is missing, the app fails at startup with a clear error — not at runtime when the first request hits the code path that needs it.

No os.getenv() anywhere else in the codebase. If an engineer needs a new env var, it goes in config.py first.

dependencies.py — Shared FastAPI Dependencies

# dependencies.py
from fastapi import Request, HTTPException, status
from typing import AsyncGenerator
import asyncpg

async def get_db(request: Request) -> AsyncGenerator[asyncpg.Connection, None]:
    async with request.app.state.db.acquire() as conn:
        yield conn

async def get_current_service(request: Request) -> str:
    api_key = request.headers.get("X-Service-Key")
    if not api_key or api_key not in VALID_SERVICE_KEYS:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Invalid or missing service key"
        )
    return api_key

Anything shared across routes — database connections, auth checks, request context — lives here and gets injected via FastAPI's dependency system. Routes stay clean. Tests swap these out via dependency_overrides.

routers/ — One File Per Domain

# routers/events.py
from fastapi import APIRouter, Depends, status
from models.commerce import EVENT_REGISTRY
from services.publisher import publish_event
from dependencies import get_db
import asyncpg

router = APIRouter(prefix="/events", tags=["events"])

@router.post("/ingest", status_code=status.HTTP_202_ACCEPTED)
async def ingest_event(
    raw_event: dict,
    db: asyncpg.Connection = Depends(get_db)
):
    event_type = raw_event.get("event_type")
    model_class = EVENT_REGISTRY.get(event_type)
    event = model_class(**raw_event)
    message_id = await publish_event(event.model_dump())
    return {"status": "accepted", "event_id": event.event_id, "message_id": message_id}

Each router owns one domain. events.py owns event ingestion. orders.py owns order operations. When an engineer is working on orders, they touch routers/orders.py, services/ if needed, and db/queries/orders.py. They don't touch anything else.

services/ — Business Logic, No HTTP

Services contain the actual work. No Request objects, no Response objects, no FastAPI imports. Pure Python functions that take inputs and return outputs.

This matters for testing — services can be tested directly without spinning up an HTTP server, and without mocking the FastAPI layer.

# services/publisher.py
from google.cloud import pubsub_v1
from config import settings
import json
import logging

logger = logging.getLogger(__name__)
publisher = pubsub_v1.PublisherClient()

async def publish_event(event: dict) -> str:
    topic_path = resolve_topic(event["event_type"])
    future = publisher.publish(
        topic_path,
        data=json.dumps(event, default=str).encode(),
        ordering_key=event["user_id"],
        event_type=event["event_type"],
    )
    return future.result(timeout=10)

The Rules the Team Follows

Routes call services. Services call db queries. Services don't call routes. The dependency direction is one-way. A router can call a service. A service can call a db query. Nothing calls back up the chain.

No business logic in routes. A route handler should be readable in 10 lines. If it's longer, something belongs in services/.

No direct DB access outside db/. Every SQL query lives in db/queries/. Routes and services never construct SQL strings directly.

One model per concern. API input schema, database write schema, and event payload schema are separate models — even if they look similar today. They'll diverge, and when they do, having them separate costs nothing. Having them merged costs a refactor.

The structure above isn't the only valid one. It's the one that reduced merge conflicts, made onboarding faster, and gave every engineer on a five-person team a clear answer to "where does this go?"

That's the bar worth optimising for.

Every resource — Cloud Run services, Pub/Sub topics, Cloud SQL, Redis, IAM bindings, secrets — lived in one file. Nobody wanted to touch it. Adding a new Cloud Run service meant scrolling through hundreds of lines to find the right place, hoping you didn't accidentally modify something adjacent. Reviewing a PR meant reading a diff that changed lines 47, 312, and 651 with no obvious relationship between them.

The file worked. It was just impossible to reason about.

Modules are the fix. Not because they're the Terraform-approved way to do things, but because they enforce the same principle that makes application code maintainable: one thing, one place, clear boundaries.

What a Module Is

A module is a directory with Terraform files. That's it. You call it with module block, pass in variables, and get outputs back. It's a function for infrastructure.

# Without modules — everything inline in main.tf
resource "google_cloud_run_v2_service" "producer" { ... }
resource "google_cloud_run_v2_service" "consumer" { ... }
resource "google_pubsub_topic" "commerce_events" { ... }
resource "google_pubsub_subscription" "realtime" { ... }
resource "google_sql_database_instance" "main" { ... }
# ... 750 more lines

# With modules — main.tf becomes a composition
module "pubsub" {
  source         = "./modules/pubsub"
  project_id     = var.gcp_project_id
  environment    = var.environment
}

module "cloud_run" {
  source               = "./modules/cloud_run"
  project_id           = var.gcp_project_id
  producer_image       = var.producer_image
  consumer_image       = var.consumer_image
  consumer_url         = module.pubsub.consumer_push_url
}

module "database" {
  source      = "./modules/database"
  project_id  = var.gcp_project_id
  environment = var.environment
  tier        = var.db_tier
}

The top-level main.tf is now a readable description of what the system is. The details live in the modules.

Module Structure That Scales

infra/
├── main.tf           # module composition only
├── variables.tf      # top-level inputs
├── outputs.tf        # top-level outputs
├── versions.tf       # provider + backend config
├── environments/
│   ├── dev.tfvars
│   └── prod.tfvars
└── modules/
    ├── pubsub/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── outputs.tf
    ├── cloud_run/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── outputs.tf
    ├── database/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── outputs.tf
    └── redis/
        ├── main.tf
        ├── variables.tf
        └── outputs.tf

Every module has three files. main.tf declares resources. variables.tf declares inputs. outputs.tf declares what the module exposes to the outside. Nothing else.

The rule: if a resource belongs to a specific service or concern, it lives in that module. If you're not sure which module a resource belongs to, that's a signal the module boundaries need rethinking.

Writing a Module That's Actually Reusable

A module that works only for one exact configuration isn't a module — it's just a file split. The variables interface is what makes a module reusable.

# modules/cloud_run/variables.tf
variable "project_id" {
  type        = string
  description = "GCP project ID"
}

variable "region" {
  type        = string
  default     = "us-central1"
}

variable "producer_image" {
  type        = string
  description = "Docker image URI for the producer service"
}

variable "min_instances" {
  type        = number
  default     = 0
  description = "Minimum Cloud Run instances. Set to 1+ in prod to avoid cold starts."
}

variable "max_instances" {
  type        = number
  default     = 10
}

variable "service_account_email" {
  type        = string
}

# modules/cloud_run/main.tf
resource "google_cloud_run_v2_service" "producer" {
  name     = "pulsecart-producer"
  location = var.region

  template {
    scaling {
      min_instance_count = var.min_instances
      max_instance_count = var.max_instances
    }

    containers {
      image = var.producer_image
    }

    service_account = var.service_account_email
  }
}

# modules/cloud_run/outputs.tf
output "producer_url" {
  value       = google_cloud_run_v2_service.producer.uri
  description = "URL of the deployed producer service"
}

# environments/prod.tfvars
min_instances = 1
max_instances = 20
db_tier       = "db-custom-4-15360"

# environments/dev.tfvars
min_instances = 0
max_instances = 5
db_tier       = "db-g1-small"

The same module, two environments, zero duplication. Changing the prod scaling config is one line in prod.tfvars — not a search through 800 lines of main.tf.

Module Outputs and Cross-Module Dependencies

Modules need to talk to each other. The Cloud Run module needs the Pub/Sub push endpoint. The Pub/Sub module needs the Cloud Run service URL for the push subscription. Outputs wire them together cleanly:

# main.tf — passing outputs between modules
module "cloud_run" {
  source    = "./modules/cloud_run"
  # Pass the consumer URL to Pub/Sub so it knows where to push
  consumer_url = module.cloud_run.consumer_url  # ← circular?
}

module "pubsub" {
  source         = "./modules/pubsub"
  consumer_push_url = module.cloud_run.consumer_url  # ← resolved at plan time
}

Terraform resolves inter-module dependencies automatically at plan time — it builds a dependency graph and applies resources in the right order. You don't need to manage this manually.

Watch for genuine circular dependencies (module A needs an output from module B, which needs an output from module A). These require restructuring — usually by extracting the shared resource into a third module that both depend on.

The One Rule

Each module owns exactly one concern. Pub/Sub owns topics and subscriptions. Cloud Run owns services and revisions. Database owns the Cloud SQL instance and its config. When a module starts owning two unrelated things, split it.

This rule is what keeps modules reviewable. A PR that touches only modules/pubsub/ tells a reviewer exactly what changed and why. A PR that touches main.tf line 47, 312, and 651 tells them nothing.

It took two hours to find the problem. Someone had manually updated a firewall rule in the GCP Console three days earlier to test something locally. The change had never been reverted. It had been sitting there silently breaking a specific traffic path until load shifted to expose it.

Nobody on the team knew the rule existed. It wasn't in code. It wasn't in version control. It wasn't in any PR or deployment log. It was a click in a console that lived only in one person's memory — and that person was asleep.

This is what infrastructure without ownership looks like.

The "Ops Team Handles It" Assumption

There's a common assumption in engineering teams — especially small ones — that infrastructure is someone else's problem. Developers write the application. Someone else (an ops engineer, a DevOps team, a more senior engineer) handles the cloud config.

This assumption has a cost that's invisible until it isn't.

When infrastructure lives outside the application team's ownership, nobody knows what's actually running. Resources get created manually and never documented. Configuration drifts from what was intended. When something breaks at 11pm, the engineers who are on-call don't have the context to debug it because they never owned the thing that broke.

At a small team of four or five engineers — where there is no separate ops function — this problem is worse. If the lead engineer is the only one who knows how the infrastructure is configured, the team has a single point of failure that isn't visible in any architecture diagram.

Ownership doesn't mean everyone needs to be an infrastructure expert. It means the infrastructure is legible to the people who depend on it.

What "Owning" Infrastructure Actually Means

Owning your infrastructure doesn't mean clicking around in the GCP or AWS Console. It means the infrastructure is defined as code, version-controlled alongside the application, reviewable in a PR, and reproducible from scratch.

When infrastructure is code:

It's auditable. The firewall rule incident above couldn't happen with Terraform. Any change to infrastructure goes through a PR. The PR shows exactly what changes, who approved it, and why. The history is in Git, not in someone's memory.

It's reproducible. Spinning up a staging environment means running terraform apply against a different variable file. It doesn't mean spending two days manually recreating what you think production looks like.

It's shared context. A new engineer who joins the team can read the Terraform code and understand exactly what's running — what services exist, how they're connected, what the scaling config is. They don't need to ask.

It fails loudly. terraform plan shows you what will change before anything changes. The console doesn't.

Why Application Engineers Specifically

Infrastructure-as-code tools like Terraform are often positioned as a DevOps or platform engineering concern. That framing is part of the problem.

Application engineers are the ones who understand what the application needs — how it scales, what it connects to, what its failure modes are. A DevOps engineer who writes Terraform for a service they don't understand will provision something technically correct but architecturally wrong. They'll set a max instance count that's too low. They'll miss a dependency. They'll configure a health check that doesn't reflect what the service actually needs to be healthy.

The engineer who built the Cloud Run service is the right person to write the Terraform for it. Not because DevOps engineers aren't capable, but because application context and infrastructure config belong together.

On PulseCart — the event-driven pipeline built across this blog — every Terraform resource was written by the same engineers who wrote the FastAPI services and the Pub/Sub consumers. The Cloud Run autoscaling config reflected what we knew about traffic patterns. The dead-letter topic retry policy reflected what we knew about consumer failure modes. The infrastructure made sense because the people who wrote it understood what it was for.

The Objection: "I Don't Have Time to Learn Terraform"

Fair. Terraform has a learning curve. The HCL syntax is unfamiliar at first. State management has rough edges. Debugging a plan that's doing something unexpected takes practice.

But the alternative isn't "no infrastructure cost." The alternative is debugging production incidents caused by config nobody understands, rebuilding environments from memory when something goes wrong, and onboarding new engineers into a system that exists only in the console and in the heads of whoever set it up.

The time you spend learning Terraform is paid back the first time you spin up a staging environment in 20 minutes instead of two days. It's paid back the second time you can point at a PR and say "that's the change that caused it." It's paid back every time you hand a new engineer a codebase that includes the infrastructure and they don't need to ask where anything is.

Start Small, Start Now

You don't need to Terraform everything at once. Pick one service — the one you understand best — and write the infrastructure for it. Get it into version control. Run terraform plan before you apply anything. See what it feels like to have a change history for your infrastructure the way you have one for your code.

The firewall rule that broke our service at 11pm would have been a three-line PR. Someone would have asked why it was needed. It would have been reverted when the test was done, or at minimum documented when it wasn't.

That's the whole argument. Infrastructure in code is infrastructure you can reason about.

This is what I wish I'd known in the first six months.

Mistake 1: I Kept Writing Code Like an IC

The first instinct when you become a lead is to keep doing what made you good — writing code. It feels productive. It's measurable. And for the first few weeks, it's fine.

Then sprint planning gets skipped because you're heads-down on a feature. A junior engineer is blocked for two days because you're in flow and didn't check in. A client requirement gets misinterpreted because nobody translated it properly into a ticket, and you weren't in the room when it was discussed.

I was individually productive and collectively a bottleneck.

The mindset shift that fixed it: your output is no longer lines of code. It's the velocity and quality of the team. A day spent unblocking three engineers, clarifying requirements, and reviewing PRs carefully is a more productive day than a day spent shipping a feature yourself — even if it doesn't feel that way.

I still write code. I own specific modules, handle architectural spikes, and stay close enough to the codebase to do meaningful reviews. But I no longer treat it as the primary metric of a good day.

Mistake 2: I Assumed Requirements Were Clear When They Weren't

Client requirements arrive in many forms — a Notion doc, a Slack message, a 45-minute call, a Figma file with comments. Early on, I'd take whatever arrived, mentally fill in the gaps, and write tickets based on my interpretation.

Engineers would build against those tickets. Then the client would see the result and say "that's not quite what we meant." Not because the engineers built it wrong — they built exactly what the ticket said. The ticket was wrong.

The gap between what a client says and what they mean is almost always larger than it looks. Now before any ticket gets written, I ask three questions: What does success look like for the user? What's explicitly out of scope? What's the edge case nobody's thought about yet? The answers to those three questions change almost every requirement I've ever received.

Writing tickets well is an underrated engineering skill. A well-written ticket — clear acceptance criteria, explicit non-goals, a definition of done — reduces back-and-forth, reduces misbuilds, and makes code review faster because the reviewer knows exactly what to evaluate against.

Mistake 3: I Gave Feedback Only in Code Review

For the first few months, my feedback to engineers lived almost entirely in PR comments. Inline notes on code quality, architecture choices, naming, test coverage. Detailed, sometimes lengthy, delivered asynchronously after the fact.

This is a bad feedback loop. By the time someone gets a PR comment, they've already spent hours building in the wrong direction. Fixing it means rework, which is demoralising and wastes time. And async text-based feedback strips all the nuance — what reads as a sharp critique in a comment lands completely differently in a five-minute conversation.

Two things changed this. First, I started having brief design conversations before implementation, not after. Five minutes discussing the approach before someone writes 200 lines of code saves hours of rework. Second, I made feedback in code review additive rather than corrective — pointing out what's working as explicitly as what isn't, so engineers understood their own strengths and weren't only receiving criticism.

PR reviews got shorter. Rework dropped significantly. The quality of the code that arrived for review improved because engineers were better calibrated before they started.

Mistake 4: I Didn't Protect the Team's Focus

Clients communicate directly with engineers in Slack. A question here, a small change request there, a "quick clarification" that turns into a 30-minute tangent. Each one feels minor. Collectively they destroy deep work.

I underestimated how much this mattered early on. An engineer context-switching three times in a morning has effectively lost that morning. The cost isn't the interruption — it's the recovery time after each one.

Now client-facing communication routes through me first. Engineers aren't hidden from clients — they're in calls, they demo features, they answer technical questions. But they're not on the receiving end of every ad-hoc request. I take the interrupt, translate it into a ticket if it needs work, and the engineer gets a clean, scoped ask instead of a half-formed request with missing context.

This also benefits clients. They get faster, clearer responses than they did when questions landed in a channel and waited for whoever was least busy to notice.

Mistake 5: I Waited Too Long to Have Hard Conversations

Two situations in the first six months where I noticed a problem — a deadline at risk, a quality issue on a specific engineer's output, a communication pattern that was creating friction — and said nothing for too long. I told myself I was gathering more data. I was avoiding discomfort.

Both situations got worse before I addressed them. The conversations I eventually had were harder because the issues had compounded. And both engineers later told me, separately, that they'd have preferred to hear it earlier.

Feedback given early is a gift. Feedback withheld until it becomes an incident is a problem. I learned to say something within a day or two of noticing an issue, directly and without softening it into ambiguity. "I noticed the last three PRs have had recurring issues with error handling — let's talk about it" is better than waiting until a client flags a bug.

The conversation doesn't have to be long. It doesn't have to be formal. It just has to happen.

Mistake 6: I Didn't Invest in Onboarding

When a new engineer joined the team mid-year, I handed them a GitHub repo link, a Notion doc with some context, and told them to ping me with questions. I was in the middle of a delivery crunch and didn't have time to do it properly.

They were unproductive for three weeks. Three weeks of context-gathering through trial and error, confusion about conventions, and questions that interrupted everyone else on the team.

A proper onboarding doc — architecture overview, environment setup, coding conventions, how we write tickets, what good looks like on this team — takes half a day to write and pays back immediately. Every engineer who joins after spends less time confused and less time interrupting others. I wrote one after that experience. It's now the first thing a new engineer gets.

What the Job Actually Is

Engineering leadership at a small team isn't management in the traditional sense. There's no org chart to navigate, no performance review cycles, no HR processes. It's closer to: make sure the right things get built correctly, by people who have what they need to do their best work.

The mistakes above all share a root cause — I was optimising for my own productivity and comfort rather than the team's. Staying in code because it felt familiar. Avoiding hard conversations because they were uncomfortable. Skipping onboarding because I was busy.

The job got easier when I accepted that my discomfort is often the signal that I'm doing it right.

Six months in, I was a better lead than I was at the start. Twelve months in, I was significantly better than six months. The gap between where I started and where I am now is almost entirely made up of mistakes I don't make twice.

That's probably how it works for everyone.

The bug was a one-line fix. The 140,000 retries took two days to drain.

The real problem wasn't the bug. It was that nobody had set up queue depth alerting, because nobody had explicitly decided that Cloud Tasks was the right tool and what "healthy" looked like for it. The choice had been made by default.

Don't choose your async pattern by default.

The Decision Framework

Three questions determine which tool to use:

1. Can you afford to lose this work if the process restarts? If yes → FastAPI background tasks. If no → Pub/Sub or Cloud Tasks.

2. Does the work need to happen at a specific time in the future? If yes → Cloud Tasks. If no → Pub/Sub.

3. Do multiple independent consumers need to react to the same event? If yes → Pub/Sub. If no → Cloud Tasks or background tasks.

FastAPI Background Tasks — Use Sparingly

FastAPI's BackgroundTasks runs work in the same process as the request, after the response is sent. It's the simplest option and the most dangerous one to overuse.

from fastapi import BackgroundTasks

def send_welcome_email(user_id: str):
    # runs after response is returned
    email_service.send(user_id, template="welcome")

@app.post("/users", status_code=201)
async def create_user(payload: UserCreate, background_tasks: BackgroundTasks):
    user = await db.create_user(payload)
    background_tasks.add_task(send_welcome_email, user.id)
    return user

What it's good for: fire-and-forget work where loss is acceptable — sending a welcome email, logging an analytics event, invalidating a cache. Low volume, low stakes.

Where it breaks: Cloud Run scales to zero. When an instance shuts down, any in-flight background tasks die with it. There's no queue, no retry, no visibility. At moderate traffic, you won't notice. At scale, you will.

Never use background tasks for anything that must complete — payment confirmations, order receipts, inventory updates.

Pub/Sub — Use for Event-Driven Fan-Out

Pub/Sub is the right choice when an event needs to reach multiple independent consumers, or when you want the producer completely decoupled from what happens next.

# Producer — doesn't know or care who's consuming
async def publish_order_event(order: Order):
    await publisher.publish(
        topic_path,
        data=order.model_dump_json().encode(),
        event_type="order.placed",
        ordering_key=order.user_id,
    )

# Consumer 1 — sends confirmation email
# Consumer 2 — updates inventory
# Consumer 3 — feeds personalization score
# All subscribe independently, all receive the same message

What it's good for: anything event-driven where multiple things need to react — the PulseCart pipeline is built entirely on this. Also the right choice when consumers are owned by different teams and should be independently deployable.

Where it breaks: Pub/Sub is not a scheduler. You can't say "deliver this message in 2 hours." You can't guarantee exactly-once delivery without a Redis deduplication layer (covered in PulseCart Day 4). And dead-letter queues need explicit setup — messages don't automatically surface when they fail.

Cloud Tasks — Use for Delayed and Retryable Work

Cloud Tasks is the right choice when work needs to happen at a specific time, needs explicit retry control, or needs to check state at execution time rather than at event time.

from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime

async def schedule_abandonment_reminder(user_id: str, cart_id: str, event_id: str):
    scheduled_time = datetime.datetime.utcnow() + datetime.timedelta(hours=2)

    timestamp = timestamp_pb2.Timestamp()
    timestamp.FromDatetime(scheduled_time)

    task = {
        "http_request": {
            "http_method": tasks_v2.HttpMethod.POST,
            "url": f"{HANDLER_URL}/tasks/cart-reminder",
            "body": json.dumps({
                "user_id": user_id,
                "cart_id": cart_id,
                "event_id": event_id,
            }).encode(),
            "oidc_token": {"service_account_email": SERVICE_ACCOUNT},
        },
        "schedule_time": timestamp,
        "name": f"{QUEUE_PATH}/tasks/reminder-{event_id}",  # deterministic name = free dedup
    }

    tasks_client.create_task(request={"parent": QUEUE_PATH, "task": task})

The handler re-validates state at execution time — if the cart was purchased in the two hours since scheduling, the task exits cleanly without sending anything. This is the pattern you can't replicate with Pub/Sub alone.

What it's good for: delayed workflows (cart reminders, payment retry after N minutes), work that needs idempotency without a Redis layer, and any job where you want per-task retry configuration rather than subscription-level policy.

Where it breaks: Cloud Tasks is one consumer per task. If you need fan-out, it's the wrong tool. And as the story at the top shows — queue depth alerting is not optional. Set it up before the queue goes live, not after.

The Alerting You Need for Cloud Tasks

The lesson from the opening story:

# Terraform — Cloud Tasks queue depth alert
resource "google_monitoring_alert_policy" "tasks_queue_depth" {
  display_name = "Cart Reminders Queue Depth High"

  conditions {
    display_name = "Queue depth > 5000 for 10 minutes"
    condition_threshold {
      filter     = "resource.type=\"cloudtasks.googleapis.com/Queue\" AND metric.type=\"cloudtasks.googleapis.com/queue/depth\""
      threshold_value = 5000
      duration        = "600s"
      comparison      = "COMPARISON_GT"
    }
  }

  notification_channels = [var.alert_channel_id]
}

Add this at provisioning time, not after an incident.

The One-Line Summary

Use background tasks for low-stakes fire-and-forget. Use Pub/Sub when multiple consumers need the same event. Use Cloud Tasks when timing, retries, or state re-validation matter.

When in doubt, Cloud Tasks over background tasks — the observability alone is worth it.

It went well enough. But "well enough" means we made mistakes we didn't have to make. Here's what actually happened, in order, and what I'd do differently.

Step 1: We Audited the AWS Stack First

What we did: before touching GCP, we spent a week mapping every AWS service in use — ECS Fargate, RDS Postgres, S3, CloudFront, SES, Secrets Manager, Route 53 — and found the GCP equivalent for each.

This was the right call. The mapping exercise surfaced two services we'd forgotten were even running (a legacy SQS queue and a Lambda that ran once a week), and forced us to make explicit decisions about what to migrate vs. what to retire.

What I'd do the same: the audit. Non-negotiable. You will find things you forgot existed.

What I'd do differently: involve the whole team earlier. I did most of the audit myself, which meant others were context-free when implementation started. A shared doc with the mapping, built collaboratively, would have halved the onboarding time later.

Step 2: We Wrote Terraform Before Touching the Console

What we did: we committed to 100% Infrastructure as Code from day one. No GCP Console clicks that weren't codified in Terraform. Every resource — Cloud Run services, Cloud SQL instance, Redis Memorystore, Pub/Sub topics, Cloud Tasks queues — was provisioned via Terraform with remote state in GCS.

This was the best decision we made. When we needed to spin up a staging environment mid-migration, it took 20 minutes, not two days.

What I'd do the same: everything here. IaC from day one, no exceptions.

What I'd do differently: set up the Terraform module structure earlier. We started with a flat main.tf and refactored into modules halfway through when it got unwieldy. Starting modular costs nothing and saves a painful refactor.

Step 3: We Migrated Compute Before Data

What we did: we deployed the application services to Cloud Run first, kept them pointed at the existing AWS RDS Postgres instance, and validated the full stack was working before touching the database.

In theory this was smart — decouple the compute migration from the data migration. In practice it meant our Cloud Run services in GCP were making cross-cloud calls to RDS in AWS for three weeks, adding ~40ms of latency on every database call. Clients noticed.

What I'd do differently: plan the database migration timeline before starting compute. Three weeks of cross-cloud latency was avoidable. A tighter migration window — compute and database moved within the same week — would have been uncomfortable but faster.

Step 4: The Database Migration Was the Hardest Part

What we did: we used AWS DMS (Database Migration Service) to stream changes from RDS Postgres to Cloud SQL during a live migration window. The plan was to run both databases in sync, then do a cutover with a short maintenance window.

DMS had opinions about our schema we hadn't anticipated. Several constraints didn't replicate cleanly, and we spent two days debugging replication lag before the cutover. The actual maintenance window was 47 minutes — longer than the 15 we'd planned.

What I'd do differently: test the full DMS pipeline on a production-sized data copy at least two weeks before the actual cutover. We tested on a small subset, which didn't surface the schema issues. Test at scale, with real data volume.

Also: pg_dump / pg_restore for smaller databases (under ~50GB) is less exciting than DMS but dramatically less surprising. We overcomplicated it.

Step 5: We Underestimated Secret Migration

What we did: we had ~30 secrets in AWS Secrets Manager — API keys, DB credentials, service tokens. We manually recreated them in GCP Secret Manager. It took half a day and introduced two errors we caught in staging (wrong value copy-pasted, one secret missed entirely).

What I'd do differently: script it. Even a simple Python script that reads from AWS Secrets Manager and writes to GCP Secret Manager would have been faster, auditable, and error-free. Manual secret migration at any meaningful scale is asking for problems.

Step 6: CI/CD Was the Easiest Part

What we did: replaced GitLab CI pipelines pointing at ECS Fargate with GitHub Actions pipelines pointing at Cloud Run. The logic was near-identical — build Docker image, push to registry, deploy.

Switching to Workload Identity Federation instead of long-lived service account keys added a day of setup but was worth it. No credentials to rotate, no keys to accidentally commit.

What I'd do the same: Workload Identity Federation from the start. Not an afterthought.

What I'd Tell Myself Before Starting

Audit everything, including the things you think you know. You will find forgotten services.

IaC from day one, modular from day one. Refactoring Terraform mid-migration is pain you don't need.

Test the database migration at production scale, weeks early. Schema surprises at cutover time are expensive.

Script the boring parts. Secret migration, DNS cutover checklists, smoke test sequences — anything manual and repetitive should be a script.

Set a tighter timeline for cross-cloud transitional states. Three weeks of cross-cloud DB latency was a self-inflicted wound.

The migration took eight weeks end to end. With these changes, I think six was achievable — and the last two weeks would have been less stressful.

A test that was supposed to mock the database wasn't. The mock was patched at the wrong path — the module where the function was defined, not where it was imported and used. Real DB calls went through. Real rows got written. Green tests, real side effects.

This is the most common testing mistake in FastAPI services, and it's entirely avoidable.

Project Setup

tests/
├── conftest.py          # shared fixtures: app, client, db, mocks
├── unit/
│   └── test_models.py   # Pydantic validation, pure logic
├── integration/
│   └── test_routes.py   # full request → response against real dependencies
└── contract/
    └── test_contracts.py # schema compatibility between producer and consumer

One conftest.py at the root. Fixtures flow down — integration tests use the same client fixture as unit tests, just with different dependency overrides.

The Mock Path Problem

The bug from the story above:

# services/orders.py
from db import get_pool  # defined here

async def create_order(payload): ...

# ❌ Wrong — patches where it's defined, not where it's used
@patch("db.get_pool")
async def test_create_order(mock_pool): ...

# ✅ Correct — patch where the function is imported and called
@patch("services.orders.get_pool")
async def test_create_order(mock_pool): ...

Always patch at the import site, not the definition site. If services/orders.py imports get_pool from db, patch services.orders.get_pool.

FastAPI's dependency injection makes this cleaner — use app.dependency_overrides instead of @patch wherever possible:

# conftest.py
import pytest
from httpx import AsyncClient, ASGITransport
from main import app
from db import get_db

async def override_get_db():
    # return a test session or mock
    yield MockDB()

@pytest.fixture
def client():
    app.dependency_overrides[get_db] = override_get_db
    yield  # tests run here
    app.dependency_overrides.clear()  # always clean up

@pytest.fixture
async def async_client(client):
    async with AsyncClient(
        transport=ASGITransport(app=app),
        base_url="http://test"
    ) as ac:
        yield ac

dependency_overrides swaps the real dependency for the test version at the FastAPI layer — no patching required, no wrong-path bugs.

Unit Tests — Pure Logic Only

Unit tests should have zero I/O. No DB, no HTTP, no Redis. Test Pydantic models, business logic, and utility functions:

# tests/unit/test_models.py
import pytest
from models.commerce import OrderPlacedEvent

def test_order_event_requires_amount():
    with pytest.raises(ValueError):
        OrderPlacedEvent(order_id="ord_1", user_id="usr_1", payload={})
        # amount is required — missing field raises

def test_order_event_legacy_field_accepted():
    event = OrderPlacedEvent(
        order_id="ord_1",
        user_id="usr_1",
        payload={"total_amount": 99.99}  # old field name
    )
    assert event.payload["amount"] == 99.99  # transition validator fired

Fast, no fixtures, no setup. These should run in milliseconds.

Integration Tests — Full Request Stack

Integration tests hit the actual FastAPI routes with real request/response cycles, but with controlled dependencies:

# tests/integration/test_routes.py
import pytest
from httpx import AsyncClient

@pytest.mark.anyio
async def test_ingest_valid_event(async_client: AsyncClient, mock_publisher):
    response = await async_client.post("/events/ingest", json={
        "event_type": "order.placed",
        "user_id": "usr_8821",
        "order_id": "ord_77234",
        "payload": {"amount": 89.99, "currency": "USD"}
    })

    assert response.status_code == 202
    assert "event_id" in response.json()
    mock_publisher.assert_called_once()

@pytest.mark.anyio
async def test_ingest_unknown_event_type(async_client: AsyncClient):
    response = await async_client.post("/events/ingest", json={
        "event_type": "nonexistent.event",
        "user_id": "usr_8821",
        "payload": {}
    })

    assert response.status_code == 422

@pytest.mark.anyio
async def test_health_check_degraded_when_db_down(async_client: AsyncClient, broken_db):
    response = await async_client.get("/health")
    assert response.status_code == 503

The mock_publisher and broken_db fixtures live in conftest.py and swap real dependencies via dependency_overrides. The routes, validation, exception handlers, and response shapes all get exercised — just not the real infrastructure.

Add anyio mode to pytest.ini or pyproject.toml:

[tool.pytest.ini_options]
anyio_mode = "auto"

Contract Tests — Schema Compatibility

Contract tests answer one question: if the producer changes its schema, does the consumer still work?

# tests/contract/test_contracts.py
from models.commerce import OrderPlacedEvent   # producer model
from consumer.models import IncomingOrderEvent  # consumer model
import json

def test_producer_output_satisfies_consumer_contract():
    # Simulate what the producer publishes
    producer_event = OrderPlacedEvent(
        order_id="ord_77234",
        user_id="usr_8821",
        payload={"amount": 89.99, "currency": "USD"}
    )

    # Serialize as the producer would (to Pub/Sub)
    published = json.loads(producer_event.model_dump_json())

    # Deserialize as the consumer would
    try:
        IncomingOrderEvent(**published)
    except Exception as e:
        pytest.fail(f"Consumer cannot parse producer output: {e}")

This test runs in CI on every PR. If a producer field rename would break the consumer, this catches it before merge — not after 3,000 rows of bad data.

For teams using a shared models package (covered in S01E02), this test is even simpler — the same class is imported on both sides, and the contract is enforced by the package version.

CI Configuration

# .github/workflows/test.yml
name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: pulsecart_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - run: pip install -r requirements.txt

      - run: pytest tests/ -v --tb=short
        env:
          DATABASE_URL: postgresql://postgres:test@localhost/pulsecart_test
          ENVIRONMENT: test

Real Postgres in CI via GitHub Actions services — no mocking the DB layer in integration tests, just a throwaway test database that gets created fresh on every run. Unit and contract tests run against it too, but don't touch it.

The Three Rules

Mock at the import site, not the definition site. Or better — use dependency_overrides and skip @patch entirely.

Unit tests have zero I/O. If a unit test needs a database or an HTTP call, it's an integration test.

Contract tests run in CI on every PR. Schema mismatches are a merge-time problem, not a production problem.

Next: S01E04 — FastAPI Background Tasks vs Pub/Sub vs Cloud Tasks: when each one breaks.

No exception. No 5xx. Just missing data.

The producer had shipped a field rename — total_amount to amount — in a patch release. The consumer's Pydantic model still expected total_amount. FastAPI validated the incoming payload against the consumer's model, found no total_amount, set it to None (it was Optional), and continued. No error. No alert. Downstream, every order record had a null value where revenue should be.

This is what a schema mismatch looks like in production — not a crash, but silent data corruption.

Pydantic as a Contract, Not Just a Validator

Most engineers use Pydantic to validate inputs. That's half its value. The other half is using it as an explicit contract between services — a shared definition of what data looks like, enforced at the boundary.

The mistake in the story above was Optional without a default that makes the problem visible. Here's the pattern that would have caught it:

# ❌ Silent failure — None slips through
class OrderEvent(BaseModel):
    order_id: str
    user_id: str
    total_amount: Optional[float] = None  # missing field → None, no error

# ✅ Loud failure — missing field raises immediately
class OrderEvent(BaseModel):
    order_id: str
    user_id: str
    amount: float  # required, no default — missing field → ValidationError

Required fields with no default make schema mismatches loud. If a field is genuinely optional in your domain, make it optional. If it's required, don't give it a default just to suppress validation errors.

Strict Mode for Cross-Service Boundaries

At internal service boundaries, you want Pydantic to be strict about what it accepts. Extra fields should be rejected, not silently ignored:

from pydantic import BaseModel, ConfigDict

class OrderEvent(BaseModel):
    model_config = ConfigDict(extra="forbid")  # reject unknown fields

    order_id: str
    user_id: str
    amount: float

With extra="forbid", if the producer sends a field the consumer doesn't recognise, it raises immediately — forcing the teams to align rather than silently diverging. Use this on consumer models at service boundaries. Don't use it on models that accept user input, where extra fields should be ignored cleanly.

Backward-Compatible Schema Changes

Schema changes are inevitable. The question is whether they're breaking or backward-compatible. Here's the decision tree:

Adding a new optional field — backward-compatible. Existing consumers ignore it, new consumers can use it.

# v1
class OrderEvent(BaseModel):
    order_id: str
    amount: float

# v2 — safe, existing consumers unaffected
class OrderEvent(BaseModel):
    order_id: str
    amount: float
    currency: str = "USD"  # optional with default

Renaming a field — breaking. Handle it with a validator that accepts both during a transition window:

from pydantic import model_validator

class OrderEvent(BaseModel):
    order_id: str
    amount: float

    @model_validator(mode="before")
    @classmethod
    def handle_legacy_field(cls, data):
        # Accept old field name during transition
        if "total_amount" in data and "amount" not in data:
            data["amount"] = data.pop("total_amount")
        return data

Once all producers are on the new field name, remove the validator. This gives you a controlled migration window without a hard cutover.

Removing a field — deprecate first, remove later. Mark it in the model with a comment and log a warning when it's present, so you can see when producers stop sending it before you drop it.

API Versioning Strategy

Pydantic handles schema evolution within a version. For breaking changes that can't be backward-compatible, you need versioning at the API level.

Two approaches worth knowing:

URL prefix versioning — explicit, simple, the default choice for most teams:

from fastapi import APIRouter

v1_router = APIRouter(prefix="/v1")
v2_router = APIRouter(prefix="/v2")

@v1_router.post("/orders")
async def create_order_v1(payload: OrderEventV1): ...

@v2_router.post("/orders")
async def create_order_v2(payload: OrderEventV2): ...

app.include_router(v1_router)
app.include_router(v2_router)

Run both versions simultaneously during the transition. Deprecate v1 with a response header (Deprecation: true, Sunset: <date>) so consumers know the clock is ticking.

Header-based versioning — cleaner URLs, slightly more complex routing:

from fastapi import Header, HTTPException

@app.post("/orders")
async def create_order(
    payload: dict,
    api_version: str = Header(default="1", alias="X-API-Version")
):
    if api_version == "2":
        event = OrderEventV2(**payload)
    else:
        event = OrderEventV1(**payload)
    ...

URL versioning is easier to reason about, easier to document, and easier to deprecate. Use header versioning only if you have a specific reason — cleaner client URLs, or a gateway that routes by header. Otherwise, URL prefix is the default.

Shared Models Across Services

If you control both the producer and consumer (common in a small team), consider a shared models package rather than duplicating Pydantic schemas:

pulsecart-shared/
├── pyproject.toml
└── pulsecart_shared/
    └── models/
        ├── __init__.py
        └── events.py   # OrderEvent, CartAbandonedEvent, etc.

Both services install pulsecart-shared as a dependency. Schema changes happen in one place, and a version bump forces consumers to update explicitly. For a team of 4–5 engineers on a shared codebase, this is worth the overhead. For teams with independent deployment cycles, it can create coupling — weigh it against your release cadence.

The Rule

Make schema mismatches loud at the boundary, not silent in the data. Required fields stay required. Extra fields get rejected at service boundaries. Breaking changes get a transition window with a validator, not a hard cutover.

Next: S01E03 — Testing FastAPI Services: pytest setup, mocking dependencies, and integration tests in CI.

No 5xx spike. No memory pressure. No CPU ceiling. Just slow.

Two things were wrong. Both came directly from patterns copied out of the FastAPI docs.

Problem 1: A New DB Connection on Every Request

The FastAPI quickstart doesn't show you how to manage shared resources. So most services end up doing this:

# ❌ What most people write first
@app.get("/orders/{order_id}")
async def get_order(order_id: str):
    conn = await asyncpg.connect(DATABASE_URL)  # new connection every request
    order = await conn.fetchrow("SELECT * FROM orders WHERE id = $1", order_id)
    await conn.close()
    return order

Fine locally. Under 50 concurrent requests on Cloud Run hitting Cloud SQL, this means 50 simultaneous TCP handshakes + authentication attempts. PostgreSQL has a finite connection limit. Requests queue waiting for a slot. Latency compounds.

The fix is a connection pool initialised once at startup via FastAPI's lifespan hook — not per-request, not as a module-level global:

# ✅ The right way
from contextlib import asynccontextmanager
from fastapi import FastAPI
import asyncpg

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.db = await asyncpg.create_pool(DATABASE_URL, min_size=2, max_size=10)
    yield
    await app.state.db.close()

app = FastAPI(lifespan=lifespan)

@app.get("/orders/{order_id}")
async def get_order(order_id: str, request: Request):
    async with request.app.state.db.acquire() as conn:
        return await conn.fetchrow("SELECT * FROM orders WHERE id = $1", order_id)

Same pattern applies to Redis clients, HTTP sessions, and any SDK that's expensive to initialise. If it's shared, it belongs in lifespan.

Problem 2: Stack Traces Leaking to Clients

While diagnosing the latency issue, we found something else in Cloud Logging. Clients were receiving this:

{
  "detail": "500: Internal Server Error\nTraceback (most recent call last):\n  File \"/app/routers/orders.py\", line 34...\nasyncpg.exceptions.TooManyConnectionsError: ..."
}

Internal file paths. Dependency names. Exception types. Leaking silently for weeks.

FastAPI's default behaviour on an unhandled exception exposes more than you want in production. The fix is a single catch-all exception handler:

# ✅ Global exception handler
import logging
from fastapi import Request
from fastapi.responses import JSONResponse

logger = logging.getLogger(__name__)

@app.exception_handler(Exception)
async def unhandled_exception_handler(request: Request, exc: Exception):
    logger.exception(
        "Unhandled exception",
        extra={"path": request.url.path, "method": request.method}
    )
    return JSONResponse(
        status_code=500,
        content={"error": "internal_server_error", "message": "Something went wrong."}
    )

Full traceback goes to Cloud Logging — where only your team sees it. Clients get a safe, consistent error shape. One file, set it once, never think about it again.

Three More Things the Docs Skip

1. Structured logging

Default uvicorn logs are human-readable text. Cloud Logging expects JSON. Replace the default logger before your first deploy:

import logging, json, sys

class JSONFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "severity": record.levelname,
            "message": record.getMessage(),
            "logger": record.name,
        })

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JSONFormatter())
logging.root.handlers = [handler]
logging.root.setLevel(logging.INFO)

2. Uvicorn config for Cloud Run

Don't copy --reload from the quickstart into your Dockerfile. For Cloud Run:

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "1", "--loop", "uvloop", "--timeout-keep-alive", "30"]

Single worker per instance (Cloud Run scales horizontally, not vertically), uvloop for async performance, no --reload.

3. A health check that actually checks something

@app.get("/health")
async def health(request: Request):
    try:
        async with request.app.state.db.acquire() as conn:
            await conn.fetchval("SELECT 1")
        return {"status": "ok"}
    except Exception:
        return JSONResponse(status_code=503, content={"status": "degraded"})

A health endpoint that just returns {"status": "ok"} is a lie. Cloud Run routes traffic based on this endpoint — a lying health check sends requests to a broken instance.

The framework didn't fail. The configuration did.

Next: S01E02 — FastAPI + Pydantic in Production: Contracts, Validation, and Versioning.

Observability in an event-driven system isn't optional, and it isn't just application logging. It's knowing the health of every layer — the broker, the consumers, the task queues, and the orchestration layer — before users are affected. This final post covers how PulseCart monitors each layer, what alerts matter, and what actually breaks at 10x current scale.

Layer 1: Pub/Sub Subscription Backlog

The most important metric in a Pub/Sub-based pipeline is subscription/num_undelivered_messages — the number of messages waiting to be delivered on a subscription. A healthy consumer processes messages as fast as they arrive. A growing backlog means either the consumer is too slow, crashing, or not running at all.

Setting Up a Backlog Alert

# monitoring/alerts.py — create alerts programmatically via Cloud Monitoring API
from google.cloud import monitoring_v3
from google.protobuf.duration_pb2 import Duration

client = monitoring_v3.AlertPolicyServiceClient()
project_name = f"projects/your-gcp-project-id"

backlog_alert = monitoring_v3.AlertPolicy(
    display_name="PulseCart: Commerce Events Backlog High",
    conditions=[
        monitoring_v3.AlertPolicy.Condition(
            display_name="Undelivered messages > 1000",
            condition_threshold=monitoring_v3.AlertPolicy.Condition.MetricThreshold(
                filter='resource.type="pubsub_subscription" '
                       'AND resource.labels.subscription_id="sub-realtime-consumer" '
                       'AND metric.type="pubsub.googleapis.com/subscription/num_undelivered_messages"',
                comparison=monitoring_v3.ComparisonType.COMPARISON_GT,
                threshold_value=1000,
                duration=Duration(seconds=300),   # sustained for 5 minutes before alerting
                aggregations=[
                    monitoring_v3.Aggregation(
                        alignment_period=Duration(seconds=60),
                        per_series_aligner=monitoring_v3.Aggregation.Aligner.ALIGN_MAX,
                    )
                ],
            ),
        )
    ],
    notification_channels=["projects/your-gcp-project-id/notificationChannels/YOUR_CHANNEL_ID"],
    alert_strategy=monitoring_v3.AlertPolicy.AlertStrategy(
        auto_close=Duration(seconds=1800)
    ),
)

client.create_alert_policy(name=project_name, alert_policy=backlog_alert)

The 5-minute duration window prevents alert noise from brief spikes — a surge of order.placed events during a flash sale will temporarily spike the backlog, but if the consumer is healthy it clears within minutes. You want alerts for sustained backlogs, not momentary ones.

Track these metrics per subscription, not per topic:

oldest_unacked_message_age is often more useful than raw backlog size. A backlog of 5000 messages being processed quickly is fine. A backlog of 50 messages where the oldest is 30 minutes old means something is stuck.

Layer 2: Cloud Run Consumer Health

Cloud Run surfaces two categories of metrics that matter for PulseCart's consumers:

Request metrics — latency, error rate, request count. These tell you how the consumer is performing per invocation.

Instance metrics — active instances, startup latency. These tell you whether autoscaling is keeping up with load.

# Cloud Monitoring dashboard config (as code via Terraform)
# modules/monitoring/main.tf

resource "google_monitoring_dashboard" "pulsecart" {
  dashboard_json = jsonencode({
    displayName = "PulseCart Pipeline Health"
    gridLayout = {
      columns = 2
      widgets = [
        {
          title = "Consumer Error Rate"
          xyChart = {
            dataSets = [{
              timeSeriesQuery = {
                timeSeriesFilter = {
                  filter = join(" AND ", [
                    "resource.type=\"cloud_run_revision\"",
                    "resource.labels.service_name=\"pulsecart-consumer\"",
                    "metric.type=\"run.googleapis.com/request_count\"",
                    "metric.labels.response_code_class!=\"2xx\""
                  ])
                }
              }
            }]
          }
        },
        {
          title = "Consumer P99 Latency"
          xyChart = {
            dataSets = [{
              timeSeriesQuery = {
                timeSeriesFilter = {
                  filter = join(" AND ", [
                    "resource.type=\"cloud_run_revision\"",
                    "resource.labels.service_name=\"pulsecart-consumer\"",
                    "metric.type=\"run.googleapis.com/request_latencies\""
                  ])
                }
              }
            }]
          }
        }
      ]
    }
  })
}

Alert thresholds for Cloud Run:

P99 latency approaching the Pub/Sub ack deadline is a leading indicator of trouble. If the consumer takes 28 seconds on a 30-second deadline, any variance causes missed acks and redelivery — which compounds the backlog problem.

Layer 3: Cloud Tasks Queue Depth

Cloud Tasks exposes queue-level metrics that tell you whether delayed work is executing on schedule or backing up.

# Cloud Monitoring alert for Cloud Tasks — in Terraform alerting config
filter: >
  resource.type="cloudtasks.googleapis.com/Queue"
  AND resource.labels.queue_id="pulsecart-cart-reminders"
  AND metric.type="cloudtasks.googleapis.com/queue/depth"
threshold_value: 10000
duration: 600s  # 10 minutes

For PulseCart's cart reminder queue, a depth of 10,000 tasks sustained for 10 minutes means either the handler is failing (Cloud Tasks retrying everything) or there was a genuine spike in cart abandonments. Check the task dispatch error rate alongside queue depth to distinguish between the two.

Key Cloud Tasks metrics:

Layer 4: Airflow DAG Monitoring

Cloud Composer exposes Airflow metrics via Cloud Monitoring. The ones that matter for PulseCart:

# dags/monitoring_check.py — a lightweight DAG that validates upstream DAGs ran successfully

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.models import DagRun
from airflow.utils.state import State
from datetime import datetime, timedelta
import pendulum
import logging

logger = logging.getLogger(__name__)

def check_critical_dags(**context):
    critical_dags = [
        "pulsecart_nightly_aggregation",
        "pulsecart_dead_letter_reconciliation",
    ]

    execution_date = context["data_interval_start"]
    lookback = execution_date - timedelta(hours=25)  # give 1-hour buffer

    failed = []
    for dag_id in critical_dags:
        runs = DagRun.find(
            dag_id=dag_id,
            execution_start_date=lookback,
            execution_end_date=execution_date,
        )
        if not runs:
            failed.append(f"{dag_id}: no run found in last 25 hours")
        elif runs[-1].state != State.SUCCESS:
            failed.append(f"{dag_id}: last run state={runs[-1].state}")

    if failed:
        raise ValueError(f"Critical DAG failures detected:\n" + "\n".join(failed))

    logger.info("All critical DAGs ran successfully.")


with DAG(
    dag_id="pulsecart_dag_health_check",
    schedule_interval="0 6 * * *",   # runs at 06:00 UTC, after nightly DAGs complete
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    catchup=False,
    tags=["pulsecart", "monitoring"],
) as dag:

    PythonOperator(
        task_id="check_critical_dags",
        python_callable=check_critical_dags,
    )

This DAG runs at 06:00 UTC — after the nightly aggregation and metrics DAGs should have completed — and raises if either didn't succeed. Airflow's built-in alerting fires on DAG failure, which triggers the notification channel.

Beyond this, configure SLA misses directly on your critical DAGs:

# Add to nightly_aggregation DAG definition
with DAG(
    dag_id="pulsecart_nightly_aggregation",
    sla_miss_callback=notify_sla_miss,   # custom callback that pages on-call
    ...
) as dag:

Layer 5: Structured Logging Across the Pipeline

Individual metric alerts tell you something is wrong. Structured logs tell you why. Every service in PulseCart logs in JSON with consistent fields so Cloud Logging can correlate events across the pipeline by event_id.

# services/logging.py
import logging
import json
import sys

class StructuredLogger:
    def __init__(self, service_name: str):
        self.service_name = service_name
        self.logger = logging.getLogger(service_name)

    def info(self, message: str, **kwargs):
        self._log("INFO", message, **kwargs)

    def error(self, message: str, **kwargs):
        self._log("ERROR", message, **kwargs)

    def warning(self, message: str, **kwargs):
        self._log("WARNING", message, **kwargs)

    def _log(self, severity: str, message: str, **kwargs):
        entry = {
            "severity": severity,
            "message": message,
            "service": self.service_name,
            **kwargs
        }
        print(json.dumps(entry), file=sys.stdout)


# Usage in consumer
logger = StructuredLogger("pulsecart-consumer")

logger.info(
    "Event processed",
    event_id="evt_a3f9b21c4d8e",
    event_type="order.placed",
    user_id="usr_8821",
    latency_ms=42,
)

With event_id on every log entry, a Cloud Logging query like jsonPayload.event_id="evt_a3f9b21c4d8e" traces a single event from ingestion through the producer, across Pub/Sub, into the consumer, and through any downstream Cloud Tasks — across services, without distributed tracing infrastructure.

What Breaks at 10x Scale

PulseCart currently processes roughly 1 million events per day across all topics. At 10x — 10 million events — here's what changes:

Pub/Sub ordering keys become a bottleneck. Ordering keys force sequential delivery per key value within a subscription. At 10x user volume, popular users (or bots) generate enough events to create ordering-key hot spots that slow delivery for the entire subscription. The fix is to shard high-volume subscriptions or relax ordering requirements on non-critical event types.

The Redis idempotency store needs sharding. A single Redis instance has throughput limits. At 10x event volume, the idempotency check (GET processed:{event_id}) becomes a bottleneck. Cloud Memorystore for Redis Cluster (available in GCP) handles this via automatic sharding, or you can partition the key space across multiple Redis instances at the application layer.

Cloud SQL connections exhaust. Cloud Run scales to many instances under load, each holding a connection pool. At 10x scale, the total connection count across all consumer instances can exceed PostgreSQL's max_connections. The fix is PgBouncer as a connection pooler in front of Cloud SQL, or Cloud SQL's built-in connection pooling via the Auth Proxy with a --max-connections flag.

Airflow worker concurrency caps out. The dead-letter reconciliation DAG pulls up to 500 messages per run. At 10x dead-letter volume, a single worker takes too long per topic. The fix is dynamic task mapping — Airflow 2.3+ lets you generate task instances at runtime, so each dead-letter topic gets its own worker in parallel rather than sequential processing.

Cloud Composer cost grows non-linearly. At 10x DAG runs with more complex dependencies, Composer's GKE cluster costs scale with you. At that point, re-evaluate whether managed Airflow is the right fit or whether a lighter orchestrator (Prefect, Dagster, or a self-hosted Airflow on GKE Autopilot) makes more economic sense.

The Observability Stack, Summarised

Closing Thoughts

This series set out to build PulseCart the way a production system actually gets built — not a hello-world demo, but a full pipeline with real tradeoffs at every layer. Here's what we covered across the eight posts:

• Day 0 — What PulseCart is and what to expect

• Day 1 — Why request-response breaks at scale and how to design an event taxonomy

• Day 2 — Pub/Sub topics, subscriptions, ordering keys, and dead-letter topics

• Day 3 — FastAPI producer with Pydantic validation and at-least-once delivery semantics

• Day 4 — Cloud Run consumers, Redis idempotency, and Cloud Tasks for delayed workflows

• Day 5 — Airflow DAGs for batch aggregation, dead-letter reconciliation, and metrics

• Day 6 — Full Terraform infrastructure and zero-downtime GitHub Actions CI/CD

• Day 7 — Observability, alerting, and what breaks at 10x scale

The GCP-native stack — Pub/Sub, Cloud Tasks, Cloud Run, Cloud Composer — won't be the right fit for every team. If you're already running Kafka, the producer/consumer patterns from Days 3 and 4 translate directly. If you're on AWS, SNS/SQS maps closely to what Pub/Sub does here. The infrastructure changes; the architecture doesn't.

Thanks for following along. If you found something useful, wrong, or worth pushing back on — the comments are open.

Manual infrastructure is a liability. It can't be reviewed, versioned, or reproduced reliably. When you need to spin up a staging environment or recover from a misconfiguration, you're guessing. Terraform eliminates the guesswork — every resource is declared in code, version-controlled alongside the application, and reproducible across environments.

This post provisions PulseCart's complete infrastructure and wires it to a GitHub Actions pipeline for zero-downtime deploys.

Project Structure

pulsecart-infra/
├── main.tf
├── variables.tf
├── outputs.tf
├── versions.tf
├── modules/
│   ├── pubsub/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── cloud_run/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── cloud_tasks/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── cloud_sql/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── redis/
│   │   ├── main.tf
│   │   └── variables.tf
│   └── composer/
│       ├── main.tf
│       └── variables.tf
└── environments/
    ├── dev.tfvars
    └── prod.tfvars

Splitting into modules keeps each service's resources self-contained and reusable across environments. The environments/ directory holds variable overrides — dev uses smaller machine types and lower min-instance counts; prod uses production-grade sizing.

versions.tf

terraform {
  required_version = ">= 1.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }

  backend "gcs" {
    bucket = "pulsecart-terraform-state"
    prefix = "terraform/state"
  }
}

provider "google" {
  project = var.gcp_project_id
  region  = var.gcp_region
}

Remote state in GCS means the state file is shared across the team and won't get lost when someone's laptop dies. Enable state locking (GCS does this automatically via object versioning) to prevent concurrent applies from corrupting state.

Module: Pub/Sub

# modules/pubsub/main.tf

locals {
  topics = ["commerce-events", "user-actions", "system-events"]
}

resource "google_pubsub_topic" "topics" {
  for_each = toset(local.topics)
  name     = "pulsecart.${each.key}"

  message_retention_duration = "604800s"  # 7 days
}

resource "google_pubsub_topic" "dead_letter_topics" {
  for_each = toset(local.topics)
  name     = "pulsecart.${each.key}.dead-letter"
}

resource "google_pubsub_subscription" "realtime_consumer" {
  name  = "sub-realtime-consumer"
  topic = google_pubsub_topic.topics["commerce-events"].name

  ack_deadline_seconds    = 30
  enable_message_ordering = true

  push_config {
    push_endpoint = var.cloud_run_consumer_url

    oidc_token {
      service_account_email = var.service_account_email
    }
  }

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter_topics["commerce-events"].id
    max_delivery_attempts = 5
  }

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "300s"
  }

  depends_on = [google_pubsub_topic.topics, google_pubsub_topic.dead_letter_topics]
}

resource "google_pubsub_subscription" "airflow_ingestion" {
  name  = "sub-airflow-ingestion"
  topic = google_pubsub_topic.topics["commerce-events"].name

  ack_deadline_seconds    = 60
  enable_message_ordering = true

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter_topics["commerce-events"].id
    max_delivery_attempts = 5
  }
}

The for_each loop over local.topics creates all three main topics and their dead-letter counterparts without repetition. Adding a new topic means adding one string to the locals block.

Module: Cloud Run

# modules/cloud_run/main.tf

resource "google_cloud_run_v2_service" "producer" {
  name     = "pulsecart-producer"
  location = var.gcp_region

  template {
    scaling {
      min_instance_count = var.producer_min_instances
      max_instance_count = 10
    }

    containers {
      image = var.producer_image

      resources {
        limits = {
          cpu    = "1"
          memory = "512Mi"
        }
      }

      env {
        name  = "GCP_PROJECT_ID"
        value = var.gcp_project_id
      }

      env {
        name = "REDIS_URL"
        value_source {
          secret_key_ref {
            secret  = google_secret_manager_secret.redis_url.secret_id
            version = "latest"
          }
        }
      }
    }

    service_account = var.service_account_email
  }

  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
}

resource "google_cloud_run_v2_service" "consumer" {
  name     = "pulsecart-consumer"
  location = var.gcp_region

  template {
    scaling {
      min_instance_count = var.consumer_min_instances
      max_instance_count = 20
    }

    containers {
      image = var.consumer_image

      resources {
        limits = {
          cpu    = "2"
          memory = "1Gi"
        }
      }

      env {
        name  = "GCP_PROJECT_ID"
        value = var.gcp_project_id
      }

      env {
        name = "REDIS_URL"
        value_source {
          secret_key_ref {
            secret  = google_secret_manager_secret.redis_url.secret_id
            version = "latest"
          }
        }
      }
    }

    service_account = var.service_account_email
  }

  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
}

The traffic block set to TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST at 100% is what makes Cloud Run deployments zero-downtime by default — new revisions receive traffic only after they pass health checks. If a new revision fails its health check, traffic stays on the previous revision automatically.

For the consumer, min_instance_count is set higher in prod than dev (typically 2–3 for the consumer) to avoid cold starts on push subscription deliveries.

Module: Cloud Tasks

# modules/cloud_tasks/main.tf

resource "google_cloud_tasks_queue" "cart_reminders" {
  name     = "pulsecart-cart-reminders"
  location = var.gcp_region

  rate_limits {
    max_concurrent_dispatches = 100
    max_dispatches_per_second = 50
  }

  retry_config {
    max_attempts       = 5
    max_retry_duration = "3600s"
    min_backoff        = "10s"
    max_backoff        = "300s"
    max_doublings      = 4
  }

  stackdriver_logging_config {
    sampling_ratio = 1.0
  }
}

max_concurrent_dispatches prevents the cart reminder handler from being overwhelmed during a backlog catch-up. stackdriver_logging_config at 1.0 logs every task dispatch — useful for debugging, though you'd lower this in prod once things are stable to reduce logging costs.

Module: Cloud SQL

# modules/cloud_sql/main.tf

resource "google_sql_database_instance" "pulsecart" {
  name             = "pulsecart-postgres"
  database_version = "POSTGRES_15"
  region           = var.gcp_region

  settings {
    tier              = var.db_tier   # db-g1-small for dev, db-custom-4-15360 for prod
    availability_type = var.db_availability_type  # ZONAL for dev, REGIONAL for prod

    backup_configuration {
      enabled                        = true
      start_time                     = "03:00"
      point_in_time_recovery_enabled = true
      transaction_log_retention_days = 7
    }

    ip_configuration {
      ipv4_enabled    = false
      private_network = var.vpc_network_id
    }

    database_flags {
      name  = "max_connections"
      value = "200"
    }
  }

  deletion_protection = var.deletion_protection  # true in prod, false in dev
}

resource "google_sql_database" "pulsecart" {
  name     = "pulsecart"
  instance = google_sql_database_instance.pulsecart.name
}

REGIONAL availability type in prod means Cloud SQL automatically fails over to a standby instance in another zone if the primary goes down. For PulseCart's commerce-events pipeline, this matters — a database outage that takes down the consumer would cause messages to pile up in Pub/Sub until the subscription's retry window expires.

deletion_protection = true in prod is non-negotiable. It prevents terraform destroy from accidentally dropping your production database.

Module: Redis (Memorystore)

# modules/redis/main.tf

resource "google_redis_instance" "pulsecart" {
  name           = "pulsecart-redis"
  tier           = var.redis_tier   # BASIC for dev, STANDARD_HA for prod
  memory_size_gb = var.redis_memory_gb
  region         = var.gcp_region

  redis_version      = "REDIS_7_0"
  authorized_network = var.vpc_network_id

  redis_configs = {
    maxmemory-policy = "allkeys-lru"
  }
}

STANDARD_HA in prod gives Redis a read replica and automatic failover. The idempotency layer we built in Day 4 depends on Redis being available — if Redis is down, the is_duplicate check fails and the consumer falls back to processing without deduplication. Whether that's acceptable depends on your tolerance for occasional duplicate sends; for PulseCart's transactional emails, we'd rather fail closed and have the consumer return a 5xx (triggering Pub/Sub retry) than silently process without idempotency.

Module: Cloud Composer

# modules/composer/main.tf

resource "google_composer_environment" "pulsecart" {
  name   = "pulsecart-composer"
  region = var.gcp_region

  config {
    software_config {
      image_version = "composer-2-airflow-2"

      pypi_packages = {
        "apache-airflow-providers-google" = ">=10.0.0"
        "apache-airflow-providers-postgres" = ">=5.0.0"
      }

      env_variables = {
        PULSECART_PROJECT_ID = var.gcp_project_id
        PULSECART_ENV        = var.environment
      }
    }

    workloads_config {
      scheduler {
        cpu        = 0.5
        memory_gb  = 1.875
        storage_gb = 1
        count      = 1
      }
      web_server {
        cpu       = 0.5
        memory_gb = 1.875
      }
      worker {
        cpu        = 2
        memory_gb  = 7.5
        storage_gb = 10
        min_count  = 1
        max_count  = 4
      }
    }

    node_config {
      service_account = var.service_account_email
      network         = var.vpc_network_id
    }
  }
}

Worker autoscaling (min_count = 1, max_count = 4) means Composer adds workers during DAG runs and scales back down when idle. For PulseCart's three DAGs running nightly, this keeps costs reasonable without under-provisioning during peak DAG execution.

GitHub Actions CI/CD Pipeline

Two workflows: one for the application (build, push Docker image, deploy to Cloud Run), one for infrastructure (terraform plan on PR, terraform apply on merge).

Application Deploy

# .github/workflows/deploy.yml
name: Deploy to Cloud Run

on:
  push:
    branches: [main]
    paths:
      - 'pulsecart-producer/**'
      - 'pulsecart-consumer/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write   # required for Workload Identity Federation

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v2

      - name: Build and push producer image
        run: |
          docker build -t gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }} \
            ./pulsecart-producer
          docker push gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }}

      - name: Deploy producer to Cloud Run
        run: |
          gcloud run deploy pulsecart-producer \
            --image gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }} \
            --region ${{ secrets.GCP_REGION }} \
            --platform managed \
            --no-traffic   # deploy revision without routing traffic yet

      - name: Run smoke tests against new revision
        run: |
          NEW_URL=$(gcloud run revisions list \
            --service pulsecart-producer \
            --region ${{ secrets.GCP_REGION }} \
            --format 'value(status.url)' \
            --limit 1)
          curl --fail "$NEW_URL/health"

      - name: Shift traffic to new revision
        run: |
          gcloud run services update-traffic pulsecart-producer \
            --to-latest \
            --region ${{ secrets.GCP_REGION }}

The --no-traffic flag deploys the new revision without routing any traffic to it. Smoke tests run against the new revision's URL directly. Only if those pass does the final step shift traffic. If smoke tests fail, the pipeline stops and the previous revision continues serving 100% of traffic — this is the zero-downtime guarantee.

Workload Identity Federation (id-token: write) replaces long-lived service account keys in GitHub secrets. GCP verifies the GitHub Actions OIDC token directly, eliminating a credential rotation concern entirely.

Infrastructure Plan and Apply

# .github/workflows/terraform.yml
name: Terraform

on:
  pull_request:
    paths: ['pulsecart-infra/**']
  push:
    branches: [main]
    paths: ['pulsecart-infra/**']

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.5.7"

      - name: Terraform Init
        working-directory: pulsecart-infra
        run: terraform init

      - name: Terraform Plan
        working-directory: pulsecart-infra
        run: |
          terraform plan \
            -var-file="environments/${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}.tfvars" \
            -out=tfplan

      - name: Post plan to PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const plan = require('fs').readFileSync('pulsecart-infra/tfplan.txt', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '```terraform\n' + plan + '\n```'
            });

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        working-directory: pulsecart-infra
        run: terraform apply -auto-approve tfplan

Terraform plan output is posted as a PR comment — reviewers see exactly what infrastructure changes the PR introduces before it merges. Apply runs only on merge to main. This pattern prevents infrastructure drift and keeps changes reviewable.

Environment Variable Matrix

Dev can scale to zero to keep costs low during development. Prod keeps minimum instances alive to eliminate cold starts on critical paths.

What's Next

Day 7 closes the series with observability — monitoring Pub/Sub subscription backlog, Cloud Tasks queue depth, Airflow DAG failures, and what actually breaks when PulseCart processes at 10x its current scale.

But not everything in PulseCart is reactive. Some work is inherently scheduled — it doesn't make sense to trigger it per event. Nightly aggregation of user behavior into personalization scores, reconciling messages that landed in dead-letter topics, computing daily business metrics — these are batch jobs. They run on a schedule, depend on a window of data being complete, and often have multi-step dependencies between tasks.

This is Airflow's domain. And it's explicitly not a duplication of what Pub/Sub and Cloud Tasks do — it's the complement.

Why Airflow, Not More Cloud Tasks

Cloud Tasks is excellent for individual delayed tasks. It's not designed for multi-step workflows with dependencies, branching logic, retries at the DAG level, or visibility into which step failed and why.

Airflow gives you:

• DAGs (Directed Acyclic Graphs) — workflows defined as code, with explicit task dependencies

• Scheduling — cron-based or data-interval-driven execution

• Observability — a UI showing every DAG run, task status, logs, and retry history

• Backfill — re-run historical DAG runs if something failed or data was corrected

• Sensors — tasks that wait for an external condition before proceeding

For PulseCart's nightly batch jobs, none of this is achievable cleanly with Cloud Tasks alone.

Cloud Composer vs Self-Hosted Airflow

Cloud Composer is GCP's managed Airflow service. It handles the scheduler, webserver, worker pool, and underlying GKE infrastructure. You deploy DAGs by uploading Python files to a GCS bucket — no Airflow internals to manage.

The tradeoff is cost. Cloud Composer is significantly more expensive than a self-hosted Airflow instance on a single VM or a small GKE cluster. For PulseCart at production scale — where DAG failures have real business consequences and on-call engineers shouldn't be debugging Airflow infrastructure at 2am — Cloud Composer is worth it. For a smaller team or lower stakes, self-hosted on Cloud Run or GKE is a legitimate alternative.

This series uses Cloud Composer, but the DAG code is identical either way.

PulseCart's Airflow DAGs

PulseCart runs three scheduled DAGs:

DAG 1: Nightly Aggregation

This DAG pulls the prior day's events from Cloud SQL (written by the Cloud Run consumer as it processes Pub/Sub messages), aggregates them into per-user behavior signals, and writes the results back as personalization scores.

# dags/nightly_aggregation.py
from airflow import DAG
from airflow.providers.google.cloud.operators.cloud_sql import CloudSQLExecuteQueryOperator
from airflow.providers.google.cloud.transfers.sql_to_gcs import CloudSQLToGCSOperator
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from datetime import datetime, timedelta
import pendulum

default_args = {
    "owner": "pulsecart",
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "email_on_failure": True,
    "email": ["engineering@pulsecart.io"],
}

with DAG(
    dag_id="pulsecart_nightly_aggregation",
    default_args=default_args,
    description="Aggregate prior day events into personalization scores",
    schedule_interval="0 2 * * *",   # 02:00 UTC daily
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    catchup=False,
    tags=["pulsecart", "personalization"],
) as dag:

    extract_events = CloudSQLExecuteQueryOperator(
        task_id="extract_prior_day_events",
        gcp_cloudsql_conn_id="pulsecart_cloudsql",
        sql="""
            INSERT INTO event_aggregates (user_id, date, event_type, event_count)
            SELECT
                user_id,
                DATE(timestamp) AS date,
                event_type,
                COUNT(*) AS event_count
            FROM events
            WHERE DATE(timestamp) = CURRENT_DATE - INTERVAL '1 day'
            GROUP BY user_id, DATE(timestamp), event_type
            ON CONFLICT (user_id, date, event_type)
            DO UPDATE SET event_count = EXCLUDED.event_count;
        """,
    )

    compute_scores = CloudSQLExecuteQueryOperator(
        task_id="compute_personalization_scores",
        gcp_cloudsql_conn_id="pulsecart_cloudsql",
        sql="""
            INSERT INTO personalization_scores (user_id, score, computed_at)
            SELECT
                user_id,
                -- Weighted score: recency, frequency, purchase intent
                (
                    SUM(CASE WHEN event_type = 'order.placed'   THEN event_count * 10 ELSE 0 END) +
                    SUM(CASE WHEN event_type = 'cart.item_added' THEN event_count * 3  ELSE 0 END) +
                    SUM(CASE WHEN event_type = 'product.viewed'  THEN event_count * 1  ELSE 0 END)
                ) AS score,
                NOW() AS computed_at
            FROM event_aggregates
            WHERE date >= CURRENT_DATE - INTERVAL '7 days'
            GROUP BY user_id
            ON CONFLICT (user_id)
            DO UPDATE SET score = EXCLUDED.score, computed_at = EXCLUDED.computed_at;
        """,
    )

    extract_events >> compute_scores

The task dependency (extract_events >> compute_scores) ensures scores are never computed against stale data. If extract_events fails, compute_scores never runs — and Airflow retries the whole DAG run up to 2 times before alerting.

DAG 2: Dead-Letter Reconciliation

Every Pub/Sub subscription has a dead-letter topic (configured in Day 2). Messages end up there when they exceed the maximum delivery attempt count — typically due to a consumer bug, a malformed payload, or a dependency outage during the retry window.

Without reconciliation, dead-lettered messages just sit there. With this DAG, they're triaged every 6 hours: reprocessable messages are re-published to the original topic, and genuinely malformed ones are logged for manual review.

# dags/dead_letter_reconciliation.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from datetime import timedelta
from google.cloud import pubsub_v1
import json
import logging
import pendulum

logger = logging.getLogger(__name__)

DEAD_LETTER_TOPICS = [
    ("pulsecart.commerce-events.dead-letter", "pulsecart.commerce-events"),
    ("pulsecart.user-actions.dead-letter",    "pulsecart.user-actions"),
    ("pulsecart.system-events.dead-letter",   "pulsecart.system-events"),
]

PROJECT_ID = "your-gcp-project-id"
MAX_MESSAGES_PER_RUN = 500


def reconcile_dead_letters(dead_letter_topic: str, original_topic: str, **context):
    subscriber = pubsub_v1.SubscriberClient()
    publisher = pubsub_v1.PublisherClient()

    subscription_path = subscriber.subscription_path(
        PROJECT_ID, f"sub-{dead_letter_topic}-reconciler"
    )
    original_topic_path = publisher.topic_path(PROJECT_ID, original_topic)

    reprocessed = 0
    skipped = 0

    response = subscriber.pull(
        request={"subscription": subscription_path, "max_messages": MAX_MESSAGES_PER_RUN}
    )

    for msg in response.received_messages:
        try:
            event = json.loads(msg.message.data.decode("utf-8"))
            event_type = event.get("event_type")

            if not event_type:
                logger.warning(f"Dead-lettered message has no event_type, skipping: {msg.message.message_id}")
                skipped += 1
            else:
                publisher.publish(
                    original_topic_path,
                    data=msg.message.data,
                    ordering_key=event.get("user_id", ""),
                    event_type=event_type,
                    event_id=event.get("event_id", ""),
                    reconciled="true",        # attribute flag so consumers can log it
                )
                reprocessed += 1

            subscriber.acknowledge(
                request={"subscription": subscription_path, "ack_ids": [msg.ack_id]}
            )

        except Exception as e:
            logger.error(f"Failed to reconcile message {msg.message.message_id}: {e}")

    logger.info(f"{dead_letter_topic}: reprocessed={reprocessed}, skipped={skipped}")


default_args = {
    "owner": "pulsecart",
    "retries": 1,
    "retry_delay": timedelta(minutes=2),
}

with DAG(
    dag_id="pulsecart_dead_letter_reconciliation",
    default_args=default_args,
    description="Triage and reprocess dead-lettered Pub/Sub messages",
    schedule_interval="0 */6 * * *",   # every 6 hours
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    catchup=False,
    tags=["pulsecart", "reliability"],
) as dag:

    for dead_letter_topic, original_topic in DEAD_LETTER_TOPICS:
        PythonOperator(
            task_id=f"reconcile_{dead_letter_topic.replace('.', '_')}",
            python_callable=reconcile_dead_letters,
            op_kwargs={
                "dead_letter_topic": dead_letter_topic,
                "original_topic": original_topic,
            },
        )

The three reconciliation tasks run in parallel — each topic's dead-letter queue is independent. Note the reconciled="true" message attribute: downstream consumers can detect and log reconciled messages separately, which is useful for tracking how often dead-lettered messages are genuinely recoverable vs. indicative of a deeper bug.

DAG 3: Daily Metrics

A simpler DAG that computes the previous day's business metrics into a daily_metrics table in Cloud SQL. Downstream, a dashboard reads from this table.

# dags/daily_metrics.py
from airflow import DAG
from airflow.providers.google.cloud.operators.cloud_sql import CloudSQLExecuteQueryOperator
from datetime import timedelta
import pendulum

default_args = {
    "owner": "pulsecart",
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
}

with DAG(
    dag_id="pulsecart_daily_metrics",
    default_args=default_args,
    description="Compute prior day business metrics",
    schedule_interval="0 3 * * *",   # 03:00 UTC daily
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    catchup=False,
    tags=["pulsecart", "metrics"],
) as dag:

    compute_metrics = CloudSQLExecuteQueryOperator(
        task_id="compute_daily_metrics",
        gcp_cloudsql_conn_id="pulsecart_cloudsql",
        sql="""
            INSERT INTO daily_metrics (date, orders_placed, carts_abandoned, abandonment_rate, revenue)
            SELECT
                CURRENT_DATE - INTERVAL '1 day'                                    AS date,
                COUNT(*) FILTER (WHERE event_type = 'order.placed')                AS orders_placed,
                COUNT(*) FILTER (WHERE event_type = 'cart.abandoned')              AS carts_abandoned,
                ROUND(
                    COUNT(*) FILTER (WHERE event_type = 'cart.abandoned')::numeric /
                    NULLIF(COUNT(*) FILTER (WHERE event_type = 'cart.item_added'), 0) * 100,
                    2
                )                                                                   AS abandonment_rate,
                SUM((payload->>'total')::numeric) FILTER (WHERE event_type = 'order.placed') AS revenue
            FROM events
            WHERE DATE(timestamp) = CURRENT_DATE - INTERVAL '1 day'
            ON CONFLICT (date)
            DO UPDATE SET
                orders_placed    = EXCLUDED.orders_placed,
                carts_abandoned  = EXCLUDED.carts_abandoned,
                abandonment_rate = EXCLUDED.abandonment_rate,
                revenue          = EXCLUDED.revenue;
        """,
    )

How Airflow Complements the Reactive Layer

The distinction is worth being explicit about:

These two layers don't overlap in PulseCart — they handle genuinely different categories of work. Trying to do nightly aggregation with Cloud Tasks, or trying to do per-event reactions with Airflow, would be fighting the tool.

Deploying DAGs to Cloud Composer

Cloud Composer exposes a GCS bucket for DAG storage. Deploying is straightforward:

# Upload a DAG file to the Composer environment's DAG bucket
gsutil cp dags/nightly_aggregation.py \
  gs://your-composer-bucket/dags/nightly_aggregation.py

In practice, this step is part of the GitHub Actions CI/CD pipeline (Day 6) — DAG files are synced to the GCS bucket on every merge to main, so the Airflow scheduler picks up changes automatically within a few minutes.

What's Next

Day 6 covers the full production infrastructure: Terraform modules for every service we've built across this series — Pub/Sub topics, Cloud Run, Cloud Tasks queues, Cloud SQL, Redis Memorystore, and Cloud Composer — plus GitHub Actions CI/CD for zero-downtime deploys.

Day 4 covers two distinct patterns that serve different needs in PulseCart:

• Cloud Run + Pub/Sub push for immediate, reactive work

• Cloud Tasks for delayed, scheduled, and explicitly retryable work

Understanding when to use each — and why they're complementary rather than interchangeable — is one of the more practical decisions in this stack.

Pattern 1: Cloud Run Consumer via Pub/Sub Push

When an order.placed event hits pulsecart.commerce-events, we want to react immediately — trigger a confirmation message, update the user's order history, notify the inventory service. This is Cloud Run's job.

A push subscription delivers each Pub/Sub message as an HTTP POST to your Cloud Run service URL. The service processes it and returns a 2xx to acknowledge. No polling, no ack_id management — Pub/Sub handles delivery, Cloud Run handles processing.

Decoding the Push Payload

Pub/Sub wraps the message in an envelope before posting it:

# models/pubsub.py
from pydantic import BaseModel
from typing import Dict, Optional
import base64
import json

class PubSubMessage(BaseModel):
    data: str                          # base64-encoded event JSON
    messageId: str
    publishTime: str
    attributes: Optional[Dict[str, str]] = {}

class PubSubPushEnvelope(BaseModel):
    message: PubSubMessage
    subscription: str

    def decode_event(self) -> dict:
        decoded = base64.b64decode(self.message.data).decode("utf-8")
        return json.loads(decoded)

The Consumer Service

# routers/consumer.py
from fastapi import APIRouter, HTTPException, status, Request
from models.pubsub import PubSubPushEnvelope
from services.idempotency import is_duplicate, mark_processed
from services.tasks import schedule_cart_reminder
from services.messaging import trigger_personalized_message
import logging

logger = logging.getLogger(__name__)
router = APIRouter(prefix="/consumer", tags=["consumer"])


@router.post("/push", status_code=status.HTTP_204_NO_CONTENT)
async def handle_push(envelope: PubSubPushEnvelope):
    event = envelope.decode_event()
    event_id = event.get("event_id")
    event_type = event.get("event_type")

    if not event_id or not event_type:
        # Malformed message — ack it anyway to avoid infinite retry
        logger.error(f"Malformed event, missing event_id or event_type: {event}")
        return

    # Idempotency check — deduplicate before any processing
    if await is_duplicate(event_id):
        logger.info(f"Duplicate event skipped: {event_id}")
        return

    try:
        if event_type == "order.placed":
            await trigger_personalized_message(event, template="order_confirmation")

        elif event_type == "cart.abandoned":
            # Don't send immediately — schedule a delayed reminder via Cloud Tasks
            await schedule_cart_reminder(event, delay_seconds=7200)  # 2 hours

        elif event_type == "payment.failed":
            await trigger_personalized_message(event, template="payment_retry")

        # Mark as processed only after successful handling
        await mark_processed(event_id)

    except Exception as e:
        logger.error(f"Processing failed for {event_id}: {e}")
        # Raise so Pub/Sub retries delivery (don't return 2xx on failure)
        raise HTTPException(
            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
            detail=f"Processing failed: {str(e)}"
        )

Two things to notice here: the cart.abandoned case doesn't process immediately — it hands off to Cloud Tasks for a delayed reminder. And any unhandled exception returns a 5xx, which tells Pub/Sub to retry. Returning a 2xx on failure would silently ack and drop the message.

Idempotency with Redis

Pub/Sub guarantees at-least-once delivery. In practice, duplicate delivery is rare but real — network hiccups, consumer restarts, or Pub/Sub's own retry logic can cause the same message to arrive twice. Without idempotency, a user could receive two order confirmation emails for one purchase.

PulseCart uses Redis (Cloud Memorystore) as the deduplication store. The pattern is simple: before processing any event, check if its event_id exists in Redis. If it does, skip. If it doesn't, process and then write the key with a TTL.

# services/idempotency.py
import aioredis
from config import settings

redis = aioredis.from_url(settings.redis_url, decode_responses=True)

IDEMPOTENCY_TTL = 86400  # 24 hours — longer than Pub/Sub's max retry window


async def is_duplicate(event_id: str) -> bool:
    result = await redis.get(f"processed:{event_id}")
    return result is not None


async def mark_processed(event_id: str) -> None:
    await redis.set(
        f"processed:{event_id}",
        "1",
        ex=IDEMPOTENCY_TTL
    )

The TTL matters. You need it to be longer than Pub/Sub's maximum retry window for a given subscription (configurable, but defaults to 7 days with exponential backoff). 24 hours is conservative for most cases — tune it to your subscription's actual retry policy.

One subtlety: mark_processed is called after successful processing, not before. If processing fails halfway through, the event_id stays unset in Redis and Pub/Sub will redeliver. This means your downstream operations (sending a message, writing to Cloud SQL) also need to be idempotent — or wrapped in a transaction that either fully succeeds or fully rolls back. For PulseCart's use case, the messaging service handles this by checking for an existing message_sent record before sending.

Pattern 2: Cloud Tasks for Delayed and Retryable Work

Not everything should happen immediately. PulseCart's cart abandonment reminder shouldn't fire the moment someone leaves the site — that would be jarring and almost certainly wrong (they might just be switching tabs). The right behavior is to wait two hours, check if the cart is still abandoned, and only then send a reminder.

This is exactly what Cloud Tasks is built for: scheduling work to happen at a specific time in the future, with explicit retry control.

Creating a Cloud Tasks Task

# services/tasks.py
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
from config import settings
import json
import datetime

tasks_client = tasks_v2.CloudTasksClient()

QUEUE_PATH = tasks_client.queue_path(
    settings.gcp_project_id,
    settings.gcp_region,
    "pulsecart-cart-reminders"
)

HANDLER_URL = f"https://{settings.cloud_run_consumer_url}/tasks/cart-reminder"


async def schedule_cart_reminder(event: dict, delay_seconds: int = 7200):
    scheduled_time = datetime.datetime.utcnow() + datetime.timedelta(seconds=delay_seconds)

    timestamp = timestamp_pb2.Timestamp()
    timestamp.FromDatetime(scheduled_time)

    task = {
        "http_request": {
            "http_method": tasks_v2.HttpMethod.POST,
            "url": HANDLER_URL,
            "headers": {"Content-Type": "application/json"},
            "body": json.dumps({
                "event_id": event["event_id"],
                "user_id": event["user_id"],
                "cart_id": event["payload"].get("cart_id"),
                "total": event["payload"].get("total"),
            }).encode(),
            "oidc_token": {
                "service_account_email": settings.service_account_email
            }
        },
        "schedule_time": timestamp,
        "name": f"{QUEUE_PATH}/tasks/cart-reminder-{event['event_id']}"
    }

    response = tasks_client.create_task(
        request={"parent": QUEUE_PATH, "task": task}
    )
    return response.name

The task name (cart-reminder-{event_id}) is deterministic. Cloud Tasks will reject a duplicate task creation with the same name within the deduplication window — this gives you free idempotency on task scheduling. If schedule_cart_reminder is called twice for the same cart.abandoned event (because Pub/Sub re-delivered it), only one task gets created.

The Cart Reminder Handler

# routers/tasks.py
from fastapi import APIRouter, HTTPException, status
from services.cart import get_cart_status
from services.messaging import trigger_personalized_message
import logging

logger = logging.getLogger(__name__)
router = APIRouter(prefix="/tasks", tags=["tasks"])


@router.post("/cart-reminder", status_code=status.HTTP_204_NO_CONTENT)
async def handle_cart_reminder(payload: dict):
    cart_id = payload.get("cart_id")
    user_id = payload.get("user_id")
    event_id = payload.get("event_id")

    # Re-check cart status at execution time — it may have been purchased
    cart = await get_cart_status(cart_id)

    if cart is None or cart.status == "purchased":
        logger.info(f"Cart {cart_id} no longer abandoned, skipping reminder")
        return

    await trigger_personalized_message(
        {"user_id": user_id, "event_id": event_id, "payload": payload},
        template="cart_abandonment_reminder"
    )

This is the critical check that justifies using Cloud Tasks over a simple asyncio.sleep. Two hours after the event was published, the handler re-validates that the cart is still abandoned before sending anything. If the user purchased in the meantime, the task exits cleanly. You can't do this reliably in memory — if your Cloud Run instance restarts during the wait, the reminder is lost.

Push vs Cloud Tasks: When to Use Each

The decision is straightforward in practice: if the work needs to happen now, use a push subscription into Cloud Run. If the work needs to happen later, or if you need to re-validate state before acting, use Cloud Tasks.

Wiring It Together in main.py

# main.py
from fastapi import FastAPI
from routers.consumer import router as consumer_router
from routers.tasks import router as tasks_router
import logging

logging.basicConfig(level=logging.INFO)

app = FastAPI(
    title="PulseCart Consumer",
    version="1.0.0"
)

app.include_router(consumer_router)
app.include_router(tasks_router)

@app.get("/health")
async def health():
    return {"status": "ok"}

Cloud Run Configuration Notes

A few Cloud Run settings that matter for consumer reliability:

Concurrency: Cloud Run defaults to 80 concurrent requests per instance. For a consumer doing lightweight work (validate, route, trigger), this is fine. If your handlers do heavy I/O, lower this to prevent one slow request from starving others.

Min instances: Set to at least 1 for the commerce-events consumer. Cold starts on a push subscription mean Pub/Sub retries while your instance is spinning up, which adds unnecessary latency on high-priority events.

Timeout: Default is 300 seconds. For push subscription handlers that should be fast, set this lower (30–60 seconds) so slow requests fail loudly rather than silently consuming your concurrency budget.

What's Next

Day 5 brings in Airflow — specifically Cloud Composer — to handle PulseCart's batch and scheduled workflows: the nightly aggregation jobs, dead-letter reconciliation, and how the orchestrated layer complements rather than duplicates the reactive one we've built here.

This service is the entry point to PulseCart's entire event pipeline. Everything downstream — Cloud Run consumers, Cloud Tasks, Airflow DAGs — depends on what comes out of this producer. Getting the validation, routing, and delivery semantics right here saves pain at every layer below.

What the Producer Needs to Do

The producer has four responsibilities, in order:

1. Receive an event via HTTP POST from internal services (checkout, cart, user signup)

2. Validate the event against the Pydantic schema for that event type

3. Enrich it with a event_id and timestamp if not already present

4. Publish it to the correct Pub/Sub topic with the right message attributes

It deliberately does nothing else. No business logic, no database writes, no downstream calls. Its job is ingestion and publishing — fast, reliable, and stateless.

Project Structure

pulsecart-producer/
├── main.py
├── models/
│   ├── __init__.py
│   ├── base.py          # PulseCartEvent base schema
│   └── commerce.py      # CartItemAdded, OrderPlaced, CartAbandoned, etc.
├── services/
│   ├── __init__.py
│   └── publisher.py     # Pub/Sub publishing logic
├── routers/
│   ├── __init__.py
│   └── events.py        # FastAPI route handlers
├── config.py            # GCP project ID, topic names, env config
└── requirements.txt

The Event Models

We introduced PulseCartEvent in Day 1. Here's the full set of models for PulseCart's commerce events:

# models/base.py
from pydantic import BaseModel, Field
from typing import Any, Dict
from datetime import datetime
import uuid

class PulseCartEvent(BaseModel):
    event_type: str
    event_id: str = Field(default_factory=lambda: f"evt_{uuid.uuid4().hex[:12]}")
    timestamp: datetime = Field(default_factory=datetime.utcnow)
    user_id: str
    payload: Dict[str, Any]

# models/commerce.py
from .base import PulseCartEvent
from typing import List, Optional

class CartItemAddedEvent(PulseCartEvent):
    event_type: str = "cart.item_added"
    session_id: str

class OrderPlacedEvent(PulseCartEvent):
    event_type: str = "order.placed"
    order_id: str

class CartAbandonedEvent(PulseCartEvent):
    event_type: str = "cart.abandoned"
    session_id: str

class PaymentFailedEvent(PulseCartEvent):
    event_type: str = "payment.failed"
    order_id: str

# Registry: maps event_type string to its model class
EVENT_REGISTRY = {
    "cart.item_added": CartItemAddedEvent,
    "order.placed": OrderPlacedEvent,
    "cart.abandoned": CartAbandonedEvent,
    "payment.failed": PaymentFailedEvent,
}

The EVENT_REGISTRY dict is the key pattern here — it lets the router dynamically resolve the correct model for any incoming event type without a chain of if/elif blocks.

The Publisher Service

# services/publisher.py
from google.cloud import pubsub_v1
from google.api_core.exceptions import GoogleAPICallError
from config import settings
import json
import logging

logger = logging.getLogger(__name__)

publisher = pubsub_v1.PublisherClient()

TOPIC_MAP = {
    "user-actions":    ["cart.item_added", "product.viewed", "search.performed"],
    "commerce-events": ["order.placed", "cart.abandoned", "payment.failed"],
    "system-events":   ["message.personalized_send", "inventory.low_stock_alert"],
}

def resolve_topic(event_type: str) -> str:
    for topic_suffix, event_types in TOPIC_MAP.items():
        if event_type in event_types:
            topic_name = f"pulsecart.{topic_suffix}"
            return publisher.topic_path(settings.gcp_project_id, topic_name)
    raise ValueError(f"No topic mapping found for event_type: {event_type}")


async def publish_event(event: dict) -> str:
    topic_path = resolve_topic(event["event_type"])

    message_data = json.dumps(event, default=str).encode("utf-8")

    try:
        future = publisher.publish(
            topic_path,
            data=message_data,
            ordering_key=event["user_id"],
            event_type=event["event_type"],       # message attribute for filtering
            event_id=event["event_id"],           # attribute for deduplication downstream
        )
        message_id = future.result(timeout=10)
        logger.info(f"Published {event['event_type']} | event_id={event['event_id']} | msg_id={message_id}")
        return message_id

    except GoogleAPICallError as e:
        logger.error(f"Pub/Sub publish failed for {event['event_type']}: {e}")
        raise

A few things worth noting here:

ordering_key=event["user_id"] — as established in Day 2, user-scoped ordering ensures events for a given user arrive in sequence at the consumer.

Message attributes — event_type and event_id are published as Pub/Sub message attributes (not inside the payload). This lets consumers filter by event_type at the subscription level without deserializing the message body, which is faster and cleaner.

future.result(timeout=10) — this blocks until Pub/Sub confirms the message was received. For a synchronous HTTP endpoint this is acceptable; the publish call typically resolves in under 100ms. If you need higher throughput, you'd batch publishes or use async publishing with callbacks instead.

The FastAPI Router

# routers/events.py
from fastapi import APIRouter, HTTPException, status
from models.commerce import EVENT_REGISTRY
from models.base import PulseCartEvent
from services.publisher import publish_event
from pydantic import ValidationError
import logging

logger = logging.getLogger(__name__)
router = APIRouter(prefix="/events", tags=["events"])


@router.post("/ingest", status_code=status.HTTP_202_ACCEPTED)
async def ingest_event(raw_event: dict):
    event_type = raw_event.get("event_type")

    if not event_type:
        raise HTTPException(
            status_code=status.HTTP_400_BAD_REQUEST,
            detail="Missing required field: event_type"
        )

    model_class = EVENT_REGISTRY.get(event_type)
    if not model_class:
        raise HTTPException(
            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
            detail=f"Unknown event_type: {event_type}"
        )

    try:
        event = model_class(**raw_event)
    except ValidationError as e:
        logger.warning(f"Validation failed for {event_type}: {e}")
        raise HTTPException(
            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
            detail=e.errors()
        )

    message_id = await publish_event(event.model_dump())

    return {
        "status": "accepted",
        "event_id": event.event_id,
        "message_id": message_id
    }

The endpoint returns 202 Accepted, not 200 OK. This is intentional: the event has been accepted and published, but processing happens downstream asynchronously. A 200 would imply the work is done, which it isn't.

The response includes both event_id (generated by the Pydantic model) and message_id (returned by Pub/Sub). Callers can use event_id to trace the event through the pipeline — it travels with the message as an attribute all the way to the consumer.

The Main App

# main.py
from fastapi import FastAPI
from routers.events import router as events_router
import logging

logging.basicConfig(level=logging.INFO)

app = FastAPI(
    title="PulseCart Event Producer",
    description="Ingestion service for PulseCart commerce events",
    version="1.0.0"
)

app.include_router(events_router)

@app.get("/health")
async def health():
    return {"status": "ok"}

Config and Environment

# config.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    gcp_project_id: str
    environment: str = "dev"

    class Config:
        env_file = ".env"

settings = Settings()

# .env (never commit this)
GCP_PROJECT_ID=your-gcp-project-id
ENVIRONMENT=dev

Pub/Sub authentication uses Application Default Credentials — on Cloud Run, the service account attached to the Cloud Run service handles this automatically. Locally, you run gcloud auth application-default login.

At-Least-Once Delivery and What It Means for Producers

Pub/Sub guarantees at-least-once delivery — every message will be delivered to every subscription at least once, but potentially more than once. This is not a producer concern; it's a consumer concern. The producer's job is simply to publish reliably.

What the producer does own is making sure every event has a stable, unique event_id. This is what consumers use to deduplicate — if the same event_id arrives twice, the consumer can detect and discard the duplicate. We'll cover the consumer-side deduplication pattern with Redis in Day 4.

Batching for High-Throughput Scenarios

For most PulseCart events, single-message publishing is fine. But user action events (product.viewed, search.performed) can arrive in high bursts. For those, batching reduces Pub/Sub API calls and lowers cost:

# Batch settings on the PublisherClient
from google.cloud.pubsub_v1.types import BatchSettings

batch_settings = pubsub_v1.types.BatchSettings(
    max_messages=100,
    max_bytes=1 * 1024 * 1024,  # 1MB
    max_latency=0.05,           # 50ms max wait before flushing
)

publisher = pubsub_v1.PublisherClient(batch_settings=batch_settings)

With these settings, the client accumulates up to 100 messages or 1MB (whichever comes first) and flushes within 50ms. For the pulsecart.user-actions topic, this is a sensible default. For pulsecart.commerce-events, where latency matters more than throughput efficiency, keep single-message publishing.

Testing the Producer Locally

# Run the service
uvicorn main:app --reload --port 8000

# Publish a test cart.abandoned event
curl -X POST http://localhost:8000/events/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "cart.abandoned",
    "user_id": "usr_8821",
    "session_id": "sess_44fa9b",
    "payload": {
      "cart_id": "cart_99012",
      "items": [{"product_id": "prod_991", "quantity": 1}],
      "total": 89.99
    }
  }'

# Expected response
{
  "status": "accepted",
  "event_id": "evt_a3f9b21c4d8e",
  "message_id": "136969346945"
}

For local testing against real Pub/Sub, use a dedicated dev project or the Pub/Sub emulator (gcloud beta emulators pubsub start) to avoid polluting production topics.

What's Next

Day 4 builds the Cloud Run consumer that receives these events via Pub/Sub push subscriptions — including how to handle at-least-once delivery with Redis-based idempotency, and where Cloud Tasks fits in for delayed workflows like cart abandonment reminders.

Before writing any producer code, it's worth spending time on Pub/Sub's design — specifically topics, subscriptions, and the decisions around them. Getting this right early saves you from painful restructuring once consumers are in production.

How Pub/Sub Works

The model is straightforward:

• A topic is a named channel. Producers publish messages to a topic.

• A subscription is an attachment to a topic. Consumers read from subscriptions, not directly from topics.

• One topic can have multiple subscriptions. Each subscription gets its own independent copy of every message published to that topic.

This means you can add a new consumer (a new subscription) without changing the producer or any existing consumers. The producer doesn't know or care who's listening.

Producer (FastAPI)
      │
      ▼
  [Topic: pulsecart.commerce.events]
      │
      ├──▶ [Subscription: sub-realtime-consumer]   → Cloud Run
      ├──▶ [Subscription: sub-cloud-tasks-router]  → Cloud Tasks
      └──▶ [Subscription: sub-airflow-ingestion]   → Cloud Composer DAG

PulseCart's Topic Structure

A common mistake is creating one topic for everything. It feels simpler upfront but creates problems: consumers that only care about order.placed still receive every product.viewed event and have to filter client-side. Routing gets messy. Access control becomes coarse.

PulseCart uses three topics, aligned to the event categories from Day 1:

Separating commerce-events from user-actions is particularly important. Commerce events are higher-stakes — they need tighter retry policies, stricter monitoring, and potentially different IAM permissions. Mixing them with high-volume user action events makes both harder to manage.

Push vs Pull Subscriptions

This is the decision that shapes how your consumers are built. Pub/Sub supports two delivery modes:

Pull Subscriptions

Your consumer service calls Pub/Sub's API to fetch messages. It controls the polling rate and acknowledges each message after processing.

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    "your-project-id", "sub-airflow-ingestion"
)

response = subscriber.pull(
    request={"subscription": subscription_path, "max_messages": 100}
)

for msg in response.received_messages:
    print(f"Received: {msg.message.data.decode()}")
    subscriber.acknowledge(
        request={
            "subscription": subscription_path,
            "ack_ids": [msg.ack_id]
        }
    )

Use pull when: the consumer controls its own processing pace (Airflow DAGs pulling batches on a schedule), or when you're processing large volumes and want to batch messages explicitly.

Push Subscriptions

Pub/Sub delivers messages to an HTTPS endpoint — your Cloud Run service URL. No polling needed; messages arrive as HTTP POST requests.

{
  "subscription": "projects/your-project/subscriptions/sub-realtime-consumer",
  "message": {
    "data": "eyJldmVudF90eXBlIjogIm9yZGVyLnBsYWNlZCIsIC4uLn0=",
    "messageId": "136969346945",
    "publishTime": "2025-06-18T10:31:05Z",
    "attributes": {
      "event_type": "order.placed"
    }
  }
}

Your Cloud Run service receives this, processes it, and returns a 2xx to acknowledge. No ack_id management needed.

Use push when: you have a serverless consumer (Cloud Run) that should react immediately to each message, and you want Pub/Sub to handle the delivery rather than polling.

PulseCart's Subscription Breakdown

Message Ordering

By default, Pub/Sub does not guarantee message ordering across a subscription. For most of PulseCart's events, this is fine — a product.viewed event being delivered slightly out of order has no consequence.

But for commerce events, order can matter. A cart.abandoned event being processed before cart.item_added would trigger an abandonment email for a cart the system doesn't know exists yet.

Pub/Sub handles this with ordering keys. Messages published with the same ordering key are delivered in order to a given subscriber.

from google.cloud import pubsub_v1
import json

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("your-project-id", "pulsecart.commerce-events")

event = {
    "event_type": "cart.abandoned",
    "user_id": "usr_8821",
    "payload": { "cart_id": "cart_99012", "total": 89.99 }
}

future = publisher.publish(
    topic_path,
    data=json.dumps(event).encode("utf-8"),
    ordering_key="usr_8821",       # ordered per user
    event_type="cart.abandoned"    # message attribute for filtering
)

Using user_id as the ordering key ensures all events for a given user are delivered in publish order. This is the right scope for PulseCart — user-level ordering, not global ordering (which would be a performance bottleneck at scale).

Note: ordering keys require the subscription to have message ordering enabled. You can't enable it retroactively on a live subscription without downtime.

Dead-Letter Topics

Even with retries, some messages will never be successfully processed — a malformed payload, a consumer bug, a dependency that's permanently unavailable. Without a dead-letter topic, those messages either retry indefinitely (blocking the subscription) or get dropped silently.

PulseCart uses a dead-letter topic per main topic:

pulsecart.commerce-events       →  pulsecart.commerce-events.dead-letter
pulsecart.user-actions          →  pulsecart.user-actions.dead-letter
pulsecart.system-events         →  pulsecart.system-events.dead-letter

When a message exceeds its maximum delivery attempt count (we use 5), Pub/Sub automatically forwards it to the dead-letter topic. A separate monitoring subscription on each dead-letter topic triggers an alert, and a periodic Airflow DAG (Day 5) reconciles and reprocesses eligible dead-lettered messages.

Configuring this in Terraform looks like this:

resource "google_pubsub_subscription" "realtime_consumer" {
  name  = "sub-realtime-consumer"
  topic = google_pubsub_topic.commerce_events.name

  ack_deadline_seconds    = 30
  enable_message_ordering = true

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.commerce_events_dead_letter.id
    max_delivery_attempts = 5
  }

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "300s"
  }
}

The exponential backoff retry policy (10s to 300s) prevents a struggling consumer from hammering a downstream dependency in a tight retry loop.

Acknowledgement Deadlines

Every Pub/Sub message has an acknowledgement deadline — the window within which your consumer must ack the message before Pub/Sub considers it undelivered and retries. The default is 10 seconds.

For Cloud Run consumers doing lightweight work (validate, route, trigger), 10–30 seconds is fine. For consumers doing heavier processing (calling an external API, writing to Cloud SQL), you'll want to either extend the deadline or design the consumer to ack fast and handle the heavy work asynchronously via Cloud Tasks (exactly the pattern we cover in Day 4).

What's Next

Day 3 builds the FastAPI producer service that validates events against our Pydantic schemas and publishes them to the Pub/Sub topics we've just designed — including how to handle batching, message attributes, and at-least-once delivery in practice.

PulseCart's first version worked exactly like this. A single checkout endpoint did everything: validated the cart, charged the payment, updated inventory, triggered a confirmation email, and queued a personalization signal — all in one request. At low traffic, it was fine. At scale, it became a liability. One slow email provider call could time out the entire checkout. A spike in orders backed up inventory updates. The system was only as fast as its slowest step.

This is the core problem that event-driven architecture solves — and it's the reason PulseCart's pipeline looks nothing like that original design.

The Problem With Request-Response at Scale

Synchronous request-response couples the caller to every downstream operation. When your checkout endpoint calls the email service, it's blocked until the email service responds. If inventory updates are slow, checkout is slow. If the personalization service is down, checkout fails.

This creates three concrete problems:

Latency compounds. Each synchronous step adds to the total response time. Three services averaging 100ms each means your user waits 300ms minimum — before accounting for retries, timeouts, or database contention.

Failures cascade. A transient failure in one downstream service propagates up to the caller. Your checkout endpoint shouldn't fail because an email provider returned a 503.

Scaling is uneven. Checkout traffic spikes at different times than, say, personalization scoring. Coupling them in a synchronous chain means you scale everything together, whether or not every service actually needs it.

The Event-Driven Alternative

Instead of one endpoint doing everything, PulseCart decouples each action into a discrete event. The checkout endpoint does exactly one thing: persist the order and publish an order.placed event. Everything else — email, inventory, personalization — reacts to that event independently and asynchronously.

The caller gets a fast response. Downstream services process at their own pace. A failure in email doesn't affect inventory. Each service scales independently.

This is the shift: from orchestration (one caller tells everyone what to do, in order) to choreography (each service reacts to what happened, when it's ready).

PulseCart's Event Taxonomy

Before writing any code, you need to define what events exist in your system, what they mean, and what triggers them. Getting this wrong early is expensive — a vague event schema leads to consumers that guess at intent and producers that overload single event types with unrelated concerns.

PulseCart organizes events into three categories:

1. User Action Events

Things a user explicitly does. These are the raw inputs to the system.

{
  "event_type": "cart.item_added",
  "event_id": "evt_01HZ9K2MNP3Q",
  "timestamp": "2025-06-18T10:23:41Z",
  "user_id": "usr_8821",
  "session_id": "sess_44fa9b",
  "payload": {
    "product_id": "prod_991",
    "product_name": "Wireless Headphones",
    "quantity": 1,
    "unit_price": 89.99
  }
}

Other user action events: cart.item_removed, product.viewed, search.performed, user.signed_up

2. Commerce Events

System-confirmed state changes. These represent something that actually happened in the platform — not just an intent.

{
  "event_type": "order.placed",
  "event_id": "evt_01HZ9K7TRP2W",
  "timestamp": "2025-06-18T10:31:05Z",
  "user_id": "usr_8821",
  "order_id": "ord_77234",
  "payload": {
    "items": [
      { "product_id": "prod_991", "quantity": 1, "unit_price": 89.99 }
    ],
    "total": 89.99,
    "currency": "USD",
    "payment_status": "confirmed"
  }
}

Other commerce events: order.cancelled, cart.abandoned, payment.failed, refund.issued

3. System-Triggered Events

Events generated by internal services, not by user actions. These drive automation and personalization.

{
  "event_type": "message.personalized_send",
  "event_id": "evt_01HZ9M1KQP9A",
  "timestamp": "2025-06-18T12:31:05Z",
  "user_id": "usr_8821",
  "payload": {
    "trigger": "cart.abandoned",
    "channel": "email",
    "template_id": "tpl_abandon_v3",
    "scheduled_at": "2025-06-18T14:31:05Z"
  }
}

Other system events: recommendation.updated, inventory.low_stock_alert, user.segment_changed

Defining the Schema in Python

In PulseCart, every event is validated against a Pydantic schema before it hits the pipeline. This isn't optional — it's the contract between producers and consumers. A consumer that receives a malformed event and silently ignores it is worse than one that rejects it loudly.

from pydantic import BaseModel, Field
from typing import Any, Dict
from datetime import datetime
import uuid

class PulseCartEvent(BaseModel):
    event_type: str
    event_id: str = Field(default_factory=lambda: f"evt_{uuid.uuid4().hex[:12]}")
    timestamp: datetime = Field(default_factory=datetime.utcnow)
    user_id: str
    payload: Dict[str, Any]

class CartItemAddedEvent(PulseCartEvent):
    event_type: str = "cart.item_added"
    session_id: str

    class Config:
        json_schema_extra = {
            "example": {
                "event_type": "cart.item_added",
                "user_id": "usr_8821",
                "session_id": "sess_44fa9b",
                "payload": {
                    "product_id": "prod_991",
                    "quantity": 1,
                    "unit_price": 89.99
                }
            }
        }

The base PulseCartEvent establishes the envelope — event_type, event_id, timestamp, user_id, payload. Specific event types extend it and add their own fields. Every event goes through this validation before being published to Pub/Sub.

What Makes a Good Event

A few principles that hold in production:

Events describe what happened, not what to do. order.placed is a good event. send_confirmation_email is a command masquerading as an event. Consumers decide what they do with an event — that decision doesn't belong in the event name.

Events are immutable facts. Once an event is published, it happened. You don't update events; you publish new ones. This matters especially for audit trails and replay scenarios.

Include enough context to be useful. A consumer shouldn't need to make 3 more API calls to figure out what to do with an event. If a cart.abandoned event doesn't include the user's email or at least their user_id, the email consumer is stuck.

Keep payloads focused. Don't dump the entire user profile into every event. Include what's relevant to that specific state change, and let consumers fetch additional context if they need it.

PulseCart's Event Flow, End to End

Here's the full picture of what we're building:

User Action
    │
    ▼
FastAPI Ingestion Service
    │  (validates with Pydantic, assigns event_id + timestamp)
    ▼
Pub/Sub Topic (e.g. pulsecart.commerce.events)
    │
    ├──▶ Cloud Run Consumer (real-time reactions: trigger personalized message)
    │
    ├──▶ Cloud Tasks (delayed work: cart abandonment reminder in 2 hours)
    │
    └──▶ Airflow DAG (nightly: aggregate, reconcile, update recommendation scores)

Each layer has a specific job. The ingestion service doesn't know or care what happens downstream. Pub/Sub delivers to whoever is subscribed. Consumers are independent and can fail without affecting each other or the producer.

We'll build each piece in the posts ahead.

What's Next

Day 2 covers Pub/Sub topics and subscriptions in depth — how to structure PulseCart's event backbone, when to use push vs pull subscriptions, and how dead-letter topics save you when things go wrong downstream.