By Day 5, every layer of PulseCart's event-driven pipeline exists as application code. The FastAPI producer, Cloud Run consumers, Cloud Tasks queues, and Airflow DAGs are all written and tested. What doesn't exist yet is the infrastructure that runs them — and right now, that infrastructure lives only in someone's head or a series of manual GCP Console clicks.
Manual infrastructure is a liability. It can't be reviewed, versioned, or reproduced reliably. When you need to spin up a staging environment or recover from a misconfiguration, you're guessing. Terraform eliminates the guesswork — every resource is declared in code, version-controlled alongside the application, and reproducible across environments.
This post provisions PulseCart's complete infrastructure and wires it to a GitHub Actions pipeline for zero-downtime deploys.
Project Structure
pulsecart-infra/
├── main.tf
├── variables.tf
├── outputs.tf
├── versions.tf
├── modules/
│ ├── pubsub/
│ │ ├── main.tf
│ │ └── variables.tf
│ ├── cloud_run/
│ │ ├── main.tf
│ │ └── variables.tf
│ ├── cloud_tasks/
│ │ ├── main.tf
│ │ └── variables.tf
│ ├── cloud_sql/
│ │ ├── main.tf
│ │ └── variables.tf
│ ├── redis/
│ │ ├── main.tf
│ │ └── variables.tf
│ └── composer/
│ ├── main.tf
│ └── variables.tf
└── environments/
├── dev.tfvars
└── prod.tfvars
Splitting into modules keeps each service's resources self-contained and reusable across environments. The environments/ directory holds variable overrides — dev uses smaller machine types and lower min-instance counts; prod uses production-grade sizing.
versions.tf
terraform {
required_version = ">= 1.5"
required_providers {
google = {
source = "hashicorp/google"
version = "~> 5.0"
}
}
backend "gcs" {
bucket = "pulsecart-terraform-state"
prefix = "terraform/state"
}
}
provider "google" {
project = var.gcp_project_id
region = var.gcp_region
}
Remote state in GCS means the state file is shared across the team and won't get lost when someone's laptop dies. Enable state locking (GCS does this automatically via object versioning) to prevent concurrent applies from corrupting state.
Module: Pub/Sub
# modules/pubsub/main.tf
locals {
topics = ["commerce-events", "user-actions", "system-events"]
}
resource "google_pubsub_topic" "topics" {
for_each = toset(local.topics)
name = "pulsecart.${each.key}"
message_retention_duration = "604800s" # 7 days
}
resource "google_pubsub_topic" "dead_letter_topics" {
for_each = toset(local.topics)
name = "pulsecart.${each.key}.dead-letter"
}
resource "google_pubsub_subscription" "realtime_consumer" {
name = "sub-realtime-consumer"
topic = google_pubsub_topic.topics["commerce-events"].name
ack_deadline_seconds = 30
enable_message_ordering = true
push_config {
push_endpoint = var.cloud_run_consumer_url
oidc_token {
service_account_email = var.service_account_email
}
}
dead_letter_policy {
dead_letter_topic = google_pubsub_topic.dead_letter_topics["commerce-events"].id
max_delivery_attempts = 5
}
retry_policy {
minimum_backoff = "10s"
maximum_backoff = "300s"
}
depends_on = [google_pubsub_topic.topics, google_pubsub_topic.dead_letter_topics]
}
resource "google_pubsub_subscription" "airflow_ingestion" {
name = "sub-airflow-ingestion"
topic = google_pubsub_topic.topics["commerce-events"].name
ack_deadline_seconds = 60
enable_message_ordering = true
dead_letter_policy {
dead_letter_topic = google_pubsub_topic.dead_letter_topics["commerce-events"].id
max_delivery_attempts = 5
}
}
The for_each loop over local.topics creates all three main topics and their dead-letter counterparts without repetition. Adding a new topic means adding one string to the locals block.
Module: Cloud Run
# modules/cloud_run/main.tf
resource "google_cloud_run_v2_service" "producer" {
name = "pulsecart-producer"
location = var.gcp_region
template {
scaling {
min_instance_count = var.producer_min_instances
max_instance_count = 10
}
containers {
image = var.producer_image
resources {
limits = {
cpu = "1"
memory = "512Mi"
}
}
env {
name = "GCP_PROJECT_ID"
value = var.gcp_project_id
}
env {
name = "REDIS_URL"
value_source {
secret_key_ref {
secret = google_secret_manager_secret.redis_url.secret_id
version = "latest"
}
}
}
}
service_account = var.service_account_email
}
traffic {
type = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
percent = 100
}
}
resource "google_cloud_run_v2_service" "consumer" {
name = "pulsecart-consumer"
location = var.gcp_region
template {
scaling {
min_instance_count = var.consumer_min_instances
max_instance_count = 20
}
containers {
image = var.consumer_image
resources {
limits = {
cpu = "2"
memory = "1Gi"
}
}
env {
name = "GCP_PROJECT_ID"
value = var.gcp_project_id
}
env {
name = "REDIS_URL"
value_source {
secret_key_ref {
secret = google_secret_manager_secret.redis_url.secret_id
version = "latest"
}
}
}
}
service_account = var.service_account_email
}
traffic {
type = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
percent = 100
}
}
The traffic block set to TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST at 100% is what makes Cloud Run deployments zero-downtime by default — new revisions receive traffic only after they pass health checks. If a new revision fails its health check, traffic stays on the previous revision automatically.
For the consumer, min_instance_count is set higher in prod than dev (typically 2–3 for the consumer) to avoid cold starts on push subscription deliveries.
Module: Cloud Tasks
# modules/cloud_tasks/main.tf
resource "google_cloud_tasks_queue" "cart_reminders" {
name = "pulsecart-cart-reminders"
location = var.gcp_region
rate_limits {
max_concurrent_dispatches = 100
max_dispatches_per_second = 50
}
retry_config {
max_attempts = 5
max_retry_duration = "3600s"
min_backoff = "10s"
max_backoff = "300s"
max_doublings = 4
}
stackdriver_logging_config {
sampling_ratio = 1.0
}
}
max_concurrent_dispatches prevents the cart reminder handler from being overwhelmed during a backlog catch-up. stackdriver_logging_config at 1.0 logs every task dispatch — useful for debugging, though you'd lower this in prod once things are stable to reduce logging costs.
Module: Cloud SQL
# modules/cloud_sql/main.tf
resource "google_sql_database_instance" "pulsecart" {
name = "pulsecart-postgres"
database_version = "POSTGRES_15"
region = var.gcp_region
settings {
tier = var.db_tier # db-g1-small for dev, db-custom-4-15360 for prod
availability_type = var.db_availability_type # ZONAL for dev, REGIONAL for prod
backup_configuration {
enabled = true
start_time = "03:00"
point_in_time_recovery_enabled = true
transaction_log_retention_days = 7
}
ip_configuration {
ipv4_enabled = false
private_network = var.vpc_network_id
}
database_flags {
name = "max_connections"
value = "200"
}
}
deletion_protection = var.deletion_protection # true in prod, false in dev
}
resource "google_sql_database" "pulsecart" {
name = "pulsecart"
instance = google_sql_database_instance.pulsecart.name
}
REGIONAL availability type in prod means Cloud SQL automatically fails over to a standby instance in another zone if the primary goes down. For PulseCart's commerce-events pipeline, this matters — a database outage that takes down the consumer would cause messages to pile up in Pub/Sub until the subscription's retry window expires.
deletion_protection = true in prod is non-negotiable. It prevents terraform destroy from accidentally dropping your production database.
Module: Redis (Memorystore)
# modules/redis/main.tf
resource "google_redis_instance" "pulsecart" {
name = "pulsecart-redis"
tier = var.redis_tier # BASIC for dev, STANDARD_HA for prod
memory_size_gb = var.redis_memory_gb
region = var.gcp_region
redis_version = "REDIS_7_0"
authorized_network = var.vpc_network_id
redis_configs = {
maxmemory-policy = "allkeys-lru"
}
}
STANDARD_HA in prod gives Redis a read replica and automatic failover. The idempotency layer we built in Day 4 depends on Redis being available — if Redis is down, the is_duplicate check fails and the consumer falls back to processing without deduplication. Whether that's acceptable depends on your tolerance for occasional duplicate sends; for PulseCart's transactional emails, we'd rather fail closed and have the consumer return a 5xx (triggering Pub/Sub retry) than silently process without idempotency.
Module: Cloud Composer
# modules/composer/main.tf
resource "google_composer_environment" "pulsecart" {
name = "pulsecart-composer"
region = var.gcp_region
config {
software_config {
image_version = "composer-2-airflow-2"
pypi_packages = {
"apache-airflow-providers-google" = ">=10.0.0"
"apache-airflow-providers-postgres" = ">=5.0.0"
}
env_variables = {
PULSECART_PROJECT_ID = var.gcp_project_id
PULSECART_ENV = var.environment
}
}
workloads_config {
scheduler {
cpu = 0.5
memory_gb = 1.875
storage_gb = 1
count = 1
}
web_server {
cpu = 0.5
memory_gb = 1.875
}
worker {
cpu = 2
memory_gb = 7.5
storage_gb = 10
min_count = 1
max_count = 4
}
}
node_config {
service_account = var.service_account_email
network = var.vpc_network_id
}
}
}
Worker autoscaling (min_count = 1, max_count = 4) means Composer adds workers during DAG runs and scales back down when idle. For PulseCart's three DAGs running nightly, this keeps costs reasonable without under-provisioning during peak DAG execution.
GitHub Actions CI/CD Pipeline
Two workflows: one for the application (build, push Docker image, deploy to Cloud Run), one for infrastructure (terraform plan on PR, terraform apply on merge).
Application Deploy
# .github/workflows/deploy.yml
name: Deploy to Cloud Run
on:
push:
branches: [main]
paths:
- 'pulsecart-producer/**'
- 'pulsecart-consumer/**'
jobs:
deploy:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # required for Workload Identity Federation
steps:
- uses: actions/checkout@v4
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
service_account: ${{ secrets.SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
- name: Build and push producer image
run: |
docker build -t gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }} \
./pulsecart-producer
docker push gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }}
- name: Deploy producer to Cloud Run
run: |
gcloud run deploy pulsecart-producer \
--image gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }} \
--region ${{ secrets.GCP_REGION }} \
--platform managed \
--no-traffic # deploy revision without routing traffic yet
- name: Run smoke tests against new revision
run: |
NEW_URL=$(gcloud run revisions list \
--service pulsecart-producer \
--region ${{ secrets.GCP_REGION }} \
--format 'value(status.url)' \
--limit 1)
curl --fail "$NEW_URL/health"
- name: Shift traffic to new revision
run: |
gcloud run services update-traffic pulsecart-producer \
--to-latest \
--region ${{ secrets.GCP_REGION }}
The --no-traffic flag deploys the new revision without routing any traffic to it. Smoke tests run against the new revision's URL directly. Only if those pass does the final step shift traffic. If smoke tests fail, the pipeline stops and the previous revision continues serving 100% of traffic — this is the zero-downtime guarantee.
Workload Identity Federation (id-token: write) replaces long-lived service account keys in GitHub secrets. GCP verifies the GitHub Actions OIDC token directly, eliminating a credential rotation concern entirely.
Infrastructure Plan and Apply
# .github/workflows/terraform.yml
name: Terraform
on:
pull_request:
paths: ['pulsecart-infra/**']
push:
branches: [main]
paths: ['pulsecart-infra/**']
jobs:
terraform:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
pull-requests: write
steps:
- uses: actions/checkout@v4
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
service_account: ${{ secrets.SERVICE_ACCOUNT }}
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: "1.5.7"
- name: Terraform Init
working-directory: pulsecart-infra
run: terraform init
- name: Terraform Plan
working-directory: pulsecart-infra
run: |
terraform plan \
-var-file="environments/${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}.tfvars" \
-out=tfplan
- name: Post plan to PR
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const plan = require('fs').readFileSync('pulsecart-infra/tfplan.txt', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '```terraform\n' + plan + '\n```'
});
- name: Terraform Apply
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
working-directory: pulsecart-infra
run: terraform apply -auto-approve tfplan
Terraform plan output is posted as a PR comment — reviewers see exactly what infrastructure changes the PR introduces before it merges. Apply runs only on merge to main. This pattern prevents infrastructure drift and keeps changes reviewable.
Environment Variable Matrix
| Variable |
Dev |
Prod |
producer_min_instances |
0 |
1 |
consumer_min_instances |
0 |
2 |
db_tier |
db-g1-small |
db-custom-4-15360 |
db_availability_type |
ZONAL |
REGIONAL |
redis_tier |
BASIC |
STANDARD_HA |
deletion_protection |
false |
true |
Dev can scale to zero to keep costs low during development. Prod keeps minimum instances alive to eliminate cold starts on critical paths.
What's Next
Day 7 closes the series with observability — monitoring Pub/Sub subscription backlog, Cloud Tasks queue depth, Airflow DAG failures, and what actually breaks when PulseCart processes at 10x its current scale.
