Production Infrastructure: Terraform, Autoscaling, and Zero-Downtime Deploys

By Day 5, every layer of PulseCart's event-driven pipeline exists as application code. The FastAPI producer, Cloud Run consumers, Cloud Tasks queues, and Airflow DAGs are all written and tested. What doesn't exist yet is the infrastructure that runs them — and right now, that infrastructure lives only in someone's head or a series of manual GCP Console clicks.

Manual infrastructure is a liability. It can't be reviewed, versioned, or reproduced reliably. When you need to spin up a staging environment or recover from a misconfiguration, you're guessing. Terraform eliminates the guesswork — every resource is declared in code, version-controlled alongside the application, and reproducible across environments.

This post provisions PulseCart's complete infrastructure and wires it to a GitHub Actions pipeline for zero-downtime deploys.

---

Project Structure

pulsecart-infra/
├── main.tf
├── variables.tf
├── outputs.tf
├── versions.tf
├── modules/
│   ├── pubsub/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── cloud_run/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── cloud_tasks/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── cloud_sql/
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── redis/
│   │   ├── main.tf
│   │   └── variables.tf
│   └── composer/
│       ├── main.tf
│       └── variables.tf
└── environments/
    ├── dev.tfvars
    └── prod.tfvars

Splitting into modules keeps each service's resources self-contained and reusable across environments. The environments/ directory holds variable overrides — dev uses smaller machine types and lower min-instance counts; prod uses production-grade sizing.

---

versions.tf

terraform {
  required_version = ">= 1.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }

  backend "gcs" {
    bucket = "pulsecart-terraform-state"
    prefix = "terraform/state"
  }
}

provider "google" {
  project = var.gcp_project_id
  region  = var.gcp_region
}

Remote state in GCS means the state file is shared across the team and won't get lost when someone's laptop dies. Enable state locking (GCS does this automatically via object versioning) to prevent concurrent applies from corrupting state.

---

Module: Pub/Sub

# modules/pubsub/main.tf

locals {
  topics = ["commerce-events", "user-actions", "system-events"]
}

resource "google_pubsub_topic" "topics" {
  for_each = toset(local.topics)
  name     = "pulsecart.${each.key}"

  message_retention_duration = "604800s"  # 7 days
}

resource "google_pubsub_topic" "dead_letter_topics" {
  for_each = toset(local.topics)
  name     = "pulsecart.${each.key}.dead-letter"
}

resource "google_pubsub_subscription" "realtime_consumer" {
  name  = "sub-realtime-consumer"
  topic = google_pubsub_topic.topics["commerce-events"].name

  ack_deadline_seconds    = 30
  enable_message_ordering = true

  push_config {
    push_endpoint = var.cloud_run_consumer_url

    oidc_token {
      service_account_email = var.service_account_email
    }
  }

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter_topics["commerce-events"].id
    max_delivery_attempts = 5
  }

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "300s"
  }

  depends_on = [google_pubsub_topic.topics, google_pubsub_topic.dead_letter_topics]
}

resource "google_pubsub_subscription" "airflow_ingestion" {
  name  = "sub-airflow-ingestion"
  topic = google_pubsub_topic.topics["commerce-events"].name

  ack_deadline_seconds    = 60
  enable_message_ordering = true

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter_topics["commerce-events"].id
    max_delivery_attempts = 5
  }
}

The for_each loop over local.topics creates all three main topics and their dead-letter counterparts without repetition. Adding a new topic means adding one string to the locals block.

---

Module: Cloud Run

# modules/cloud_run/main.tf

resource "google_cloud_run_v2_service" "producer" {
  name     = "pulsecart-producer"
  location = var.gcp_region

  template {
    scaling {
      min_instance_count = var.producer_min_instances
      max_instance_count = 10
    }

    containers {
      image = var.producer_image

      resources {
        limits = {
          cpu    = "1"
          memory = "512Mi"
        }
      }

      env {
        name  = "GCP_PROJECT_ID"
        value = var.gcp_project_id
      }

      env {
        name = "REDIS_URL"
        value_source {
          secret_key_ref {
            secret  = google_secret_manager_secret.redis_url.secret_id
            version = "latest"
          }
        }
      }
    }

    service_account = var.service_account_email
  }

  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
}

resource "google_cloud_run_v2_service" "consumer" {
  name     = "pulsecart-consumer"
  location = var.gcp_region

  template {
    scaling {
      min_instance_count = var.consumer_min_instances
      max_instance_count = 20
    }

    containers {
      image = var.consumer_image

      resources {
        limits = {
          cpu    = "2"
          memory = "1Gi"
        }
      }

      env {
        name  = "GCP_PROJECT_ID"
        value = var.gcp_project_id
      }

      env {
        name = "REDIS_URL"
        value_source {
          secret_key_ref {
            secret  = google_secret_manager_secret.redis_url.secret_id
            version = "latest"
          }
        }
      }
    }

    service_account = var.service_account_email
  }

  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
}

The traffic block set to TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST at 100% is what makes Cloud Run deployments zero-downtime by default — new revisions receive traffic only after they pass health checks. If a new revision fails its health check, traffic stays on the previous revision automatically.

For the consumer, min_instance_count is set higher in prod than dev (typically 2–3 for the consumer) to avoid cold starts on push subscription deliveries.

---

Module: Cloud Tasks

# modules/cloud_tasks/main.tf

resource "google_cloud_tasks_queue" "cart_reminders" {
  name     = "pulsecart-cart-reminders"
  location = var.gcp_region

  rate_limits {
    max_concurrent_dispatches = 100
    max_dispatches_per_second = 50
  }

  retry_config {
    max_attempts       = 5
    max_retry_duration = "3600s"
    min_backoff        = "10s"
    max_backoff        = "300s"
    max_doublings      = 4
  }

  stackdriver_logging_config {
    sampling_ratio = 1.0
  }
}

max_concurrent_dispatches prevents the cart reminder handler from being overwhelmed during a backlog catch-up. stackdriver_logging_config at 1.0 logs every task dispatch — useful for debugging, though you'd lower this in prod once things are stable to reduce logging costs.

---

Module: Cloud SQL

# modules/cloud_sql/main.tf

resource "google_sql_database_instance" "pulsecart" {
  name             = "pulsecart-postgres"
  database_version = "POSTGRES_15"
  region           = var.gcp_region

  settings {
    tier              = var.db_tier   # db-g1-small for dev, db-custom-4-15360 for prod
    availability_type = var.db_availability_type  # ZONAL for dev, REGIONAL for prod

    backup_configuration {
      enabled                        = true
      start_time                     = "03:00"
      point_in_time_recovery_enabled = true
      transaction_log_retention_days = 7
    }

    ip_configuration {
      ipv4_enabled    = false
      private_network = var.vpc_network_id
    }

    database_flags {
      name  = "max_connections"
      value = "200"
    }
  }

  deletion_protection = var.deletion_protection  # true in prod, false in dev
}

resource "google_sql_database" "pulsecart" {
  name     = "pulsecart"
  instance = google_sql_database_instance.pulsecart.name
}

REGIONAL availability type in prod means Cloud SQL automatically fails over to a standby instance in another zone if the primary goes down. For PulseCart's commerce-events pipeline, this matters — a database outage that takes down the consumer would cause messages to pile up in Pub/Sub until the subscription's retry window expires.

deletion_protection = true in prod is non-negotiable. It prevents terraform destroy from accidentally dropping your production database.

---

Module: Redis (Memorystore)

# modules/redis/main.tf

resource "google_redis_instance" "pulsecart" {
  name           = "pulsecart-redis"
  tier           = var.redis_tier   # BASIC for dev, STANDARD_HA for prod
  memory_size_gb = var.redis_memory_gb
  region         = var.gcp_region

  redis_version      = "REDIS_7_0"
  authorized_network = var.vpc_network_id

  redis_configs = {
    maxmemory-policy = "allkeys-lru"
  }
}

STANDARD_HA in prod gives Redis a read replica and automatic failover. The idempotency layer we built in Day 4 depends on Redis being available — if Redis is down, the is_duplicate check fails and the consumer falls back to processing without deduplication. Whether that's acceptable depends on your tolerance for occasional duplicate sends; for PulseCart's transactional emails, we'd rather fail closed and have the consumer return a 5xx (triggering Pub/Sub retry) than silently process without idempotency.

---

Module: Cloud Composer

# modules/composer/main.tf

resource "google_composer_environment" "pulsecart" {
  name   = "pulsecart-composer"
  region = var.gcp_region

  config {
    software_config {
      image_version = "composer-2-airflow-2"

      pypi_packages = {
        "apache-airflow-providers-google" = ">=10.0.0"
        "apache-airflow-providers-postgres" = ">=5.0.0"
      }

      env_variables = {
        PULSECART_PROJECT_ID = var.gcp_project_id
        PULSECART_ENV        = var.environment
      }
    }

    workloads_config {
      scheduler {
        cpu        = 0.5
        memory_gb  = 1.875
        storage_gb = 1
        count      = 1
      }
      web_server {
        cpu       = 0.5
        memory_gb = 1.875
      }
      worker {
        cpu        = 2
        memory_gb  = 7.5
        storage_gb = 10
        min_count  = 1
        max_count  = 4
      }
    }

    node_config {
      service_account = var.service_account_email
      network         = var.vpc_network_id
    }
  }
}

Worker autoscaling (min_count = 1, max_count = 4) means Composer adds workers during DAG runs and scales back down when idle. For PulseCart's three DAGs running nightly, this keeps costs reasonable without under-provisioning during peak DAG execution.

---

GitHub Actions CI/CD Pipeline

Two workflows: one for the application (build, push Docker image, deploy to Cloud Run), one for infrastructure (terraform plan on PR, terraform apply on merge).

Application Deploy

# .github/workflows/deploy.yml
name: Deploy to Cloud Run

on:
  push:
    branches: [main]
    paths:
      - 'pulsecart-producer/**'
      - 'pulsecart-consumer/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write   # required for Workload Identity Federation

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v2

      - name: Build and push producer image
        run: |
          docker build -t gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }} \
            ./pulsecart-producer
          docker push gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }}

      - name: Deploy producer to Cloud Run
        run: |
          gcloud run deploy pulsecart-producer \
            --image gcr.io/${{ secrets.GCP_PROJECT_ID }}/pulsecart-producer:${{ github.sha }} \
            --region ${{ secrets.GCP_REGION }} \
            --platform managed \
            --no-traffic   # deploy revision without routing traffic yet

      - name: Run smoke tests against new revision
        run: |
          NEW_URL=$(gcloud run revisions list \
            --service pulsecart-producer \
            --region ${{ secrets.GCP_REGION }} \
            --format 'value(status.url)' \
            --limit 1)
          curl --fail "$NEW_URL/health"

      - name: Shift traffic to new revision
        run: |
          gcloud run services update-traffic pulsecart-producer \
            --to-latest \
            --region ${{ secrets.GCP_REGION }}

The --no-traffic flag deploys the new revision without routing any traffic to it. Smoke tests run against the new revision's URL directly. Only if those pass does the final step shift traffic. If smoke tests fail, the pipeline stops and the previous revision continues serving 100% of traffic — this is the zero-downtime guarantee.

Workload Identity Federation (id-token: write) replaces long-lived service account keys in GitHub secrets. GCP verifies the GitHub Actions OIDC token directly, eliminating a credential rotation concern entirely.

Infrastructure Plan and Apply

# .github/workflows/terraform.yml
name: Terraform

on:
  pull_request:
    paths: ['pulsecart-infra/**']
  push:
    branches: [main]
    paths: ['pulsecart-infra/**']

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.5.7"

      - name: Terraform Init
        working-directory: pulsecart-infra
        run: terraform init

      - name: Terraform Plan
        working-directory: pulsecart-infra
        run: |
          terraform plan \
            -var-file="environments/${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}.tfvars" \
            -out=tfplan

      - name: Post plan to PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const plan = require('fs').readFileSync('pulsecart-infra/tfplan.txt', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '```terraform\n' + plan + '\n```'
            });

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        working-directory: pulsecart-infra
        run: terraform apply -auto-approve tfplan

Terraform plan output is posted as a PR comment — reviewers see exactly what infrastructure changes the PR introduces before it merges. Apply runs only on merge to main. This pattern prevents infrastructure drift and keeps changes reviewable.

---

Environment Variable Matrix

Variable	Dev	Prod
producer_min_instances	0	1
consumer_min_instances	0	2
db_tier	db-g1-small	db-custom-4-15360
db_availability_type	ZONAL	REGIONAL
redis_tier	BASIC	STANDARD_HA
deletion_protection	false	true

Dev can scale to zero to keep costs low during development. Prod keeps minimum instances alive to eliminate cold starts on critical paths.

---

What's Next

Day 7 closes the series with observability — monitoring Pub/Sub subscription backlog, Cloud Tasks queue depth, Airflow DAG failures, and what actually breaks when PulseCart processes at 10x its current scale.