Deployment¶

Deploying Scribe is Terraform apply in the selected workspace. Terraform resolves API, frontend, Triplet, and OCR GAR tags to immutable digests. The deployment has no VM, Cloud Compose, Traefik, or cloud Vault dependency.

The OCR wrappers use the repository's terraform/modules/cloud-run-service module with Google providers v8. It vendors the MIT-licensed libops/terraform-cloudrun-v2 source at commit 8718cd663a74fd33f45306321b4f2b11c83814d5 because the upstream provider constraint excludes v8. Module and resource names retain their Terraform state addresses. Review a workspace plan before applying the provider upgrade.

Bootstrap secrets¶

While the existing dev/prod Vault service is still available, authenticate with gcloud and keyless Application Default Credentials, then copy its four application credential maps:

GCLOUD_PROJECT=your-project make secret-manager-secrets WORKSPACE=dev
GCLOUD_PROJECT=your-project make secret-manager-secrets WORKSPACE=prod

The command discovers the corresponding Vault service, uses the same operator login as make vault-secrets, and writes google_oauth, openai, gemini, and database/app to deterministic deployment-scoped Secret Manager names. It verifies every map after writing, preserves opaque values, and prints only the logical secret names. An unchanged value does not create another version. Missing optional provider credentials become empty maps; missing OAuth or database credentials fail. VAULT_ADDR, VAULT_TOKEN, and VAULT_ADMIN_TOKEN may be supplied explicitly for a locally accessible Vault.

Terraform adopts these exact bootstrap secrets with declarative imports. The copied database password initializes the new Cloud SQL user. The schema jobs initialize fresh databases. Preview credentials are independently generated by Terraform; previews cannot read dev/prod credentials or create provider secrets.

Apply¶

The foundation root owns enabled project APIs and the shared Artifact Registry. Apply it before application workspaces. It contains no Cloud Compose module. It enables the Recommender API (recommender.googleapis.com) for cost and instance-sizing recommendations; reading recommendations also requires the appropriate project or billing-account IAM permissions. Foundation also grants jjc223@lehigh.edu Cloud SQL Viewer and Cloud SQL Client on the project for instance inspection and Auth Proxy connectivity. These additive IAM memberships are owned once by foundation, independently of runtime workspaces. They do not grant instance administration, Secret Manager access, or database login privileges. SQL connections still require database credentials and network access to the private PSC endpoint; the instance has no public IP.

terraform -chdir=terraform/foundation init -backend-config=bucket=YOUR_STATE_BUCKET -backend-config=prefix=scribe-foundation
terraform -chdir=terraform/foundation apply -var=project_id=YOUR_PROJECT
GCLOUD_PROJECT=YOUR_PROJECT make tf-dev ACTION=apply
GCLOUD_PROJECT=YOUR_PROJECT make tf-prod ACTION=apply
GCLOUD_PROJECT=YOUR_PROJECT make tf-preview PR=23 ACTION=apply

The protected main workflow runs CI, builds API/frontend/Triplet images in GAR, refreshes OCR images when needed, applies foundation, and invokes make tf-prod. Pull requests run CI only; automatic preview deployments are disabled during active development. Operators can still apply a preview workspace manually. To remove an existing preview, dispatch Terraform Apply on main with action destroy-preview and its PR number, or run make tf-preview PR=N ACTION=destroy.

Terraform creates two finite Cloud Run migration jobs. Each starts a keyless Private Service Connect Cloud SQL proxy, applies its schema, and stops the proxy before exit. A straight Terraform provisioner sequence executes both jobs with --wait before creating API or worker revisions. A failed migration fails the apply.

The API service runs frontend, API, Triplet, PDF, and SQL-proxy containers. The private worker service runs a request-driven worker, an API for source reads, Triplet, and a SQL proxy. All Cloud Run services use a minimum of zero instances, including the production application, worker, and OCR services. Request-based billing allows idle workers to scale to zero. Authenticated Pub/Sub pushes wake workers for transcription; Cloud Scheduler invokes bounded maintenance every thirty minutes for outboxes, retention, and recovery of missed publishes or expired leases. The invocation identity has no data access. Committed application events also push maintenance wake-ups, so outbox delivery does not normally wait for the schedule. The thirty-minute fallback leaves an idle window for scale-down; a lost wake-up can delay recovery until that pass. The Scheduler job runs in us-east4, a supported Scheduler location, and invokes the worker in its configured runtime region; it carries no application payload. Transcription attempts cancel after nine minutes, before Pub/Sub's ten-minute push deadline, and follow the existing fenced, bounded retry policy. Maintenance requests finish within four minutes. No background work runs between requests. Only the frontend and worker request ports are ingress ports; all other listeners stay inside their instance.

When services or images change, Terraform executes backend_readiness_job and ocr_readiness_job. The backend job checks the deployed API and worker image digests, canonical origin, and database readiness over HTTPS. The OCR job sends a real image through the registered private OCR endpoints. These jobs have no database, upload, or secret access. A failed probe fails deployment. The production workflow reports up to five recent OCR probe failure markers after a plan or apply. Diagnostics contain only the failed stage, never tokens, endpoint URLs, or inference responses.

Operator SQL connection using IAM¶

Foundation grants jjc223@lehigh.edu roles/cloudsql.viewer for inspection and roles/cloudsql.client for IAM-authorized Cloud SQL Auth Proxy connections. Apply foundation before trying these commands. Use the pinned Cloud SQL Auth Proxy version recorded in terraform/managed_runtime.tf and a MySQL client.

Run the proxy on an operator host with network access to the deployment's PSC endpoint and resolution of its private Cloud DNS record. A normal laptop or Cloud Shell session without that connectivity cannot reach this private-only instance. IAM grants do not establish a network route; Scribe does not provision an operator VPN or bastion.

Authenticate the CLI and the proxy's Application Default Credentials as the same operator, then start the proxy bound to loopback:

gcloud auth login jjc223@lehigh.edu
gcloud auth application-default login
PROJECT=lehigh-lyrasis-catalyst
INSTANCE=scribe-mysql
CONNECTION_NAME=$(gcloud sql instances describe "$INSTANCE" \
  --project="$PROJECT" --format='value(connectionName)')
cloud-sql-proxy --psc --address=127.0.0.1 --port=13306 "$CONNECTION_NAME"

In another terminal, connect with an operator-provisioned MySQL account. The password prompt keeps the password out of shell history and process arguments:

mysql --protocol=TCP --host=127.0.0.1 --port=13306 \
  --user=YOUR_DATABASE_USER --password scribe

Check the selected database with SELECT DATABASE(), CURRENT_USER(); and stop the proxy with Ctrl-C after use. Use a separately authorized account for the triplet database. The foundation grant does not expose application passwords or create a MySQL account.

This is IAM-authorized proxy connectivity. Passwordless IAM database login additionally requires the cloudsql_iam_authentication flag, an instance IAM database user, roles/cloudsql.instanceUser, and explicit SQL privileges. Those are currently unconfigured: do not use --auto-iam-authn or an email address as the MySQL username with the current deployment.

Configuration and destruction¶

Database sizing and commitment proposal¶

The default SQL tier is db-custom-1-3840 (one dedicated vCPU, 3.75 GiB). Production retains regional HA; dev and previews remain zonal. This replaces the previous two-vCPU, 7.5-GiB default. The reported steady load of under 900 MiB and 10% CPU suggests this candidate has headroom, but verify peak memory, CPU, connections, query latency, and processing concurrency against live monitoring before applying. A tier change can interrupt MySQL for several minutes. Verify backup/PITR health first, then check application readiness and the same metrics after resizing. Restore cloud_sql_tier=db-custom-2-7680 through Terraform if the smaller instance cannot sustain the workload.

Commitment purchases must be restricted to the prod workspace. Dev and preview plans must create no commitments, even when given production purchase inputs. Do not put purchase resources in foundation or shared service modules. Any future Terraform purchase implementation must gate resource creation directly on terraform.workspace == "prod" and verify prod, dev, and preview plans. No commitment purchase automation is currently configured.

Propose a one-year Cloud SQL commitment for the resized production instance's continuous CPU and memory spend only, after confirming its live tier and region, existing commitments, billing currency, and account-specific rates. Do not include dev, previews, storage, backups, or networking. SQL CUDs give a 25% one-year discount on eligible usage. At identical regional unit rates, halving both CPU and memory halves compute cost; adding the CUD reduces that component to 37.5% of the previous on-demand compute cost. Total invoice savings will be smaller because storage and other charges remain.

For illustration only, using the pricing page's default Iowa HA rates of $0.0826/vCPU-hour and $0.014/GiB-hour, with 730 hours/month:

The October 28, 2026 billing report confirms these unit rates in us-east5: 42 regional vCPU-hours cost $3.4692 and 150 regional GiB-hours cost \(2.10. The smaller HA instance therefore has a proposed new-model one-year commitment of **\)0.101325/hour**, or approximately $887.61 over 8,760 hours. This is compute only; storage, backups, and networking remain additional. Confirm existing commitments and the account's purchase model before ordering. The new spend-based model uses discounted hourly spend as the purchase amount; the legacy model uses on-demand equivalent spend. Confirm the account's model before entering an amount. Commitments are billed monthly and cannot be cancelled; they are not a prepaid credit balance.

Cloud Run's configured minimum instance counts are zero. The operator-approved purchase sizing basis is the most recent ten complete calendar days in the supplied report, October 18-27, 2026, rather than an hourly minimum. Exclude GPU and network charges. The current Compute flexible CUD offers 17% for one-year request-based usage and 28% for instance-based usage and jobs. Size the purchase using the observed billing-mode mix:

Divide by 240 hours: the eligible on-demand average is $0.0798574/hour and the new-model discounted commitment is \(0.059832947/hour. Propose a one-year Compute flexible commitment rounded down to **\)0.0598/hour**, approximately $43.65/month or $523.85 over 8,760 hours. The report also contains $0.926129 of GPU charges in this window, which remain on demand. Daily totals cannot prove continuous hourly utilization or distinguish runtime workspaces; this sizing assumes the report was filtered to the intended project and represents expected production use. Unused hourly commitment is still charged, and bursts above coverage remain on demand. The commitment fee is not a ceiling on total Cloud Run spend.

The supplied daily billing report covers September 29 through October 28, 2026, including a partial current day. Excluding October 28 leaves 29 calendar days and $341.17 of net project charges, with recent usage potentially still arriving. Cloud SQL accounts for $137.31, Cloud Run $131.24, Compute Engine $38.19, Gemini API $21.64, Artifact Registry $8.26, and other services $4.53. The empty CUD recommendations response is not an inventory of active orders.

Cloud Run daily charges fell from roughly $24/day on October 9-11 to roughly \(0.45/day on October 15-25, with later bursts. Historical instance-based spend must not determine a new commitment after migration to request-based billing. The operator selected the last ten days as the commitment sizing basis above. GPU charges (\)2.52 across the full report) are excluded from that commitment.

N4 VM CPU and memory account for $20.32 but have no reported charges after October 8; do not buy a VM commitment for this historical footprint. Remaining Compute Engine charges are principally disk and static IP charges. The report does not establish another ongoing eligible service requiring a commitment. Gemini API spend does not by itself prove eligibility for a Vertex AI/Gemini Enterprise savings plan; verify exact eligible SKUs before proposing one.

CUD benefits apply automatically at the billing-account level; SQL is regional and Compute flexible commitments can span projects and regions. A production Terraform gate restricts purchase execution, not benefit allocation. If dev and production share a billing account, production-only benefit attribution needs separate billing-scope verification before purchase. The pinned Google provider does not currently expose a spend-based procurement order resource; do not substitute a Compute Engine regional resource commitment or an apply-time purchase script.

Purchase the production commitments manually¶

The pinned Google and Google Beta providers have no native resource for either Cloud SQL spend-based CUDs or the Compute flexible CUD used by Cloud Run. Purchase both once through Cloud Billing. Terraform applies in prod, dev, and preview perform no commitment purchases and cannot duplicate or renew them.

Use a Billing Account Administrator identity. Find the billing account linked to the production project:

gcloud billing projects describe lehigh-lyrasis-catalyst \
  --format='value(billingAccountName)'

1. Open Billing in Google Cloud console and select that linked account.
2. Open Committed use discounts (CUDs). Inspect existing active and pending commitments first; purchase only coverage not already committed.
3. Confirm production SQL is running db-custom-1-3840 with regional HA in us-east5 before buying the smaller SQL commitment.
4. Select Purchase CUD and enter the SQL purchase below. Verify that the amount field uses the new model's discounted hourly spend, in USD.
5. Review the preview summary, region, one-year term, recurring commitment fee, and service-specific terms, then finalize the purchase.
6. Repeat for Compute flexible to cover Cloud Run. A new Cloud Run-specific regional commitment is not the product to select; Compute flexible covers eligible Cloud Run usage across regions.

If the console identifies the account as using legacy on-demand-equivalent amounts, do not paste discounted amounts into that field. SQL's corresponding on-demand coverage is $0.1351/hour. Recalculate the Compute flexible purchase using the account's offer and displayed pricing model before proceeding. If the console rounds an amount, use its final displayed fee when recording the purchase. These fees exclude storage, backups, GPUs, network charges, and usage beyond hourly coverage. Unused hourly coverage does not carry forward.

After each purchase, verify the order is listed as pending or active, and record its order ID, billing account, amount, currency, scope, start/end dates, and renewal setting in the operator's billing record. The proposals above are not evidence of a completed purchase. Recheck the CUD analysis report after billing data arrives to confirm coverage and utilization. CUDs apply automatically; no Cloud Run revision or SQL deployment setting attaches them. Keep renewal manual so a later year is reviewed against current production usage. An accidental purchase cannot be undone with Terraform; contact Cloud Billing Support.

These are production-budget purchases. Billing-account-wide benefits may also cover eligible dev or other-project usage sharing the account; the manual purchase process does not enforce production-only benefit attribution.

See Google's purchase instructions and Cloud Run Compute flexible CUD guide.

Allowed ingress CIDRs are enforced by the Cloud Run frontend. It accepts only the reviewed direct run.app forwarding topology, establishes external HTTPS, and forwards one validated client address to the loopback API. Custom load balancers require a separate reviewed topology. Production requires nonempty allowed_ips before apply. Runtime quotas are decoded from config.yaml and may be overridden with the bounded Terraform variables.

Production Cloud SQL and serving services have deletion protection. Preview resources have independent names, buckets, queues, credentials, and database instances, and can be removed with make tf-preview PR=N ACTION=destroy. No preview teardown requires a shared Vault namespace.

See configuration, troubleshooting, and backup and restore.

SQL connectivity uses a deployment-owned Private Service Connect endpoint and private Cloud DNS record. It creates no producer VPC peering to retain after a preview is deleted. Terraform removes the endpoint and DNS records before removing its SQL instance and application network.