ci(infra): apply per-environment tfvars via -var-file (fixes prod private topology)#429
Merged
Merged
Conversation
The Terraform deploy step passed only individual -var flags and never referenced prod.tfvars / dev.tfvars, so enable_networking and enable_private_endpoint silently fell back to their defaults (false). As a result the pipeline deployed a PUBLIC topology even though both tfvars declare the intended PRIVATE topology (VNet-integrated Container Apps env + Cosmos private endpoint, public access disabled). This left production in a broken half-state: a manual 'apply -var-file=prod.tfvars' had disabled Cosmos public access, but the Container Apps environment was created without VNet integration (which is immutable post-creation), so the backend egresses over the public internet and Cosmos's firewall rejects it — every /chat returns 500. Fix: select the per-environment var file (production → prod.tfvars, else dev.tfvars) and pass it as -var-file, listed before the explicit -var flags so those (env, subscription, location, ACR, images, iteration) still override the file. Networking/model config now comes from the tfvars as intended. NOTE: applying this to an environment whose Container Apps env was created without a VNet will FORCE-REPLACE that environment (VNet integration is immutable), recreating the contained Container Apps. Plan/redeploy during a maintenance window. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
james-tn
added a commit
that referenced
this pull request
Jun 9, 2026
…aform apply (#430) PRODUCTION RECOVERY. Two bugs combined to take down the production Container Apps environment during the private-topology redeploy (#429): 1. The container_apps subnet had no delegation, so creating the VNet-integrated Container Apps environment failed with: ManagedEnvironmentSubnetDelegationError: The subnet of the environment must be delegated to the service 'Microsoft.App/environments'. Add the required delegation (Microsoft.App/environments + Microsoft.Network/virtualNetworks/subnets/join/action). 2. The deploy step ran 'terraform apply ... | tee' inside an if-condition with no pipefail, so the pipeline reported tee's (success) exit status and MASKED the failed apply. The old environment had already been destroyed, so the job went green while production was left with no environment and no container apps. Capture terraform's real exit code via PIPESTATUS so a failed apply fails the job instead of silently reporting success. With the delegation in place the environment can be (re)created, and the PIPESTATUS fix ensures any future apply failure surfaces instead of corrupting the environment. Co-authored-by: James N. <james.nguyen@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a latent bug in the Terraform deploy step: it passed only individual
-varflags and never referencedprod.tfvars/dev.tfvars, soenable_networkingandenable_private_endpointsilently fell back to their defaults (false). The pipeline therefore deployed a public topology even though both tfvars declare the intended private topology (VNet-integrated Container Apps env + Cosmos private endpoint, public access disabled).Why this matters (root cause of the prod
/chat500s)Production ended up in a broken half-state:
terraform apply -var-file=prod.tfvarshad set Cosmospublic_network_access= Disabled, butResult: every
/chatreturns 500 (Forbidden ... blocked by your Cosmos DB account firewall settings), which is what's failingintegration-testson PRs tomain(e.g. #426) — even though those PRs don't touch the deployed backend.The change
Select the per-environment var file and pass it as
-var-file(listed before the explicit-varflags so env/subscription/location/ACR/images/iteration still override the file):Applying this to an environment whose Container Apps env was created without a VNet will force-replace that environment (VNet integration is immutable), which recreates the contained Container Apps → brief backend/MCP downtime. This also creates the VNet/subnets and the Cosmos private endpoint. Plan and redeploy during a maintenance window.
Both
dev.tfvarsandprod.tfvarsare internally consistent (enable_networking=true+enable_private_endpoint=true), so per-developerintegration-*environments will likewise deploy the private topology on their next run.Related
main.