The queue had been growing for six days before anyone noticed. A Cloud Tasks queue processing cart abandonment reminders had started backing up after a handler deployment introduced a subtle bug — an unhandled edge case that caused tasks to fail silently and retry indefinitely. No alert, no dashboard, no on-call ping. Just 140,000 tasks sitting in a queue, retrying every 5 minutes, hammering a downstream email service.
The bug was a one-line fix. The 140,000 retries took two days to drain.
The real problem wasn't the bug. It was that nobody had set up queue depth alerting, because nobody had explicitly decided that Cloud Tasks was the right tool and what "healthy" looked like for it. The choice had been made by default.
Don't choose your async pattern by default.
The Decision Framework
Three questions determine which tool to use:
1. Can you afford to lose this work if the process restarts? If yes → FastAPI background tasks. If no → Pub/Sub or Cloud Tasks.
2. Does the work need to happen at a specific time in the future? If yes → Cloud Tasks. If no → Pub/Sub.
3. Do multiple independent consumers need to react to the same event? If yes → Pub/Sub. If no → Cloud Tasks or background tasks.
|
Background Tasks |
Pub/Sub |
Cloud Tasks |
| Survives process restart |
❌ |
✅ |
✅ |
| Delayed / scheduled execution |
❌ |
❌ |
✅ |
| Multiple consumers |
❌ |
✅ |
❌ |
| Built-in retry |
❌ |
✅ |
✅ |
| Deduplication |
❌ |
Manual (Redis) |
✅ (task name) |
| Observability |
❌ |
Cloud Monitoring |
Cloud Monitoring |
FastAPI Background Tasks — Use Sparingly
FastAPI's BackgroundTasks runs work in the same process as the request, after the response is sent. It's the simplest option and the most dangerous one to overuse.
from fastapi import BackgroundTasks
def send_welcome_email(user_id: str):
# runs after response is returned
email_service.send(user_id, template="welcome")
@app.post("/users", status_code=201)
async def create_user(payload: UserCreate, background_tasks: BackgroundTasks):
user = await db.create_user(payload)
background_tasks.add_task(send_welcome_email, user.id)
return user
What it's good for: fire-and-forget work where loss is acceptable — sending a welcome email, logging an analytics event, invalidating a cache. Low volume, low stakes.
Where it breaks: Cloud Run scales to zero. When an instance shuts down, any in-flight background tasks die with it. There's no queue, no retry, no visibility. At moderate traffic, you won't notice. At scale, you will.
Never use background tasks for anything that must complete — payment confirmations, order receipts, inventory updates.
Pub/Sub — Use for Event-Driven Fan-Out
Pub/Sub is the right choice when an event needs to reach multiple independent consumers, or when you want the producer completely decoupled from what happens next.
# Producer — doesn't know or care who's consuming
async def publish_order_event(order: Order):
await publisher.publish(
topic_path,
data=order.model_dump_json().encode(),
event_type="order.placed",
ordering_key=order.user_id,
)
# Consumer 1 — sends confirmation email
# Consumer 2 — updates inventory
# Consumer 3 — feeds personalization score
# All subscribe independently, all receive the same message
What it's good for: anything event-driven where multiple things need to react — the PulseCart pipeline is built entirely on this. Also the right choice when consumers are owned by different teams and should be independently deployable.
Where it breaks: Pub/Sub is not a scheduler. You can't say "deliver this message in 2 hours." You can't guarantee exactly-once delivery without a Redis deduplication layer (covered in PulseCart Day 4). And dead-letter queues need explicit setup — messages don't automatically surface when they fail.
Cloud Tasks — Use for Delayed and Retryable Work
Cloud Tasks is the right choice when work needs to happen at a specific time, needs explicit retry control, or needs to check state at execution time rather than at event time.
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime
async def schedule_abandonment_reminder(user_id: str, cart_id: str, event_id: str):
scheduled_time = datetime.datetime.utcnow() + datetime.timedelta(hours=2)
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(scheduled_time)
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{HANDLER_URL}/tasks/cart-reminder",
"body": json.dumps({
"user_id": user_id,
"cart_id": cart_id,
"event_id": event_id,
}).encode(),
"oidc_token": {"service_account_email": SERVICE_ACCOUNT},
},
"schedule_time": timestamp,
"name": f"{QUEUE_PATH}/tasks/reminder-{event_id}", # deterministic name = free dedup
}
tasks_client.create_task(request={"parent": QUEUE_PATH, "task": task})
The handler re-validates state at execution time — if the cart was purchased in the two hours since scheduling, the task exits cleanly without sending anything. This is the pattern you can't replicate with Pub/Sub alone.
What it's good for: delayed workflows (cart reminders, payment retry after N minutes), work that needs idempotency without a Redis layer, and any job where you want per-task retry configuration rather than subscription-level policy.
Where it breaks: Cloud Tasks is one consumer per task. If you need fan-out, it's the wrong tool. And as the story at the top shows — queue depth alerting is not optional. Set it up before the queue goes live, not after.
The Alerting You Need for Cloud Tasks
The lesson from the opening story:
# Terraform — Cloud Tasks queue depth alert
resource "google_monitoring_alert_policy" "tasks_queue_depth" {
display_name = "Cart Reminders Queue Depth High"
conditions {
display_name = "Queue depth > 5000 for 10 minutes"
condition_threshold {
filter = "resource.type=\"cloudtasks.googleapis.com/Queue\" AND metric.type=\"cloudtasks.googleapis.com/queue/depth\""
threshold_value = 5000
duration = "600s"
comparison = "COMPARISON_GT"
}
}
notification_channels = [var.alert_channel_id]
}
Add this at provisioning time, not after an incident.
The One-Line Summary
Use background tasks for low-stakes fire-and-forget. Use Pub/Sub when multiple consumers need the same event. Use Cloud Tasks when timing, retries, or state re-validation matter.
When in doubt, Cloud Tasks over background tasks — the observability alone is worth it.
