Terraform Mono-Repo Design for 20 AWS Accounts

"Your platform team manages twenty AWS accounts: dev, staging, prod and a handful of product-team sandboxes. All of the infrastructure is Terraform. Structure a single repository so that one engineer can safely deploy to one account without touching another. Walk me through the design." The prompt is deliberately open-ended. A strong answer is not about knowing a CLI flag; it shows how you reason about isolation, blast radius and team safety at a scale where a copy-paste mistake reaches production. There is also a tempting first answer, "use workspaces, one per account", and the follow-up questions probe whether you know where that holds and where it breaks: for deployments that need separate credentials and access controls, CLI workspaces alone are unsuitable. The video version walks the same design if you prefer to watch it: Why "one folder, twenty workspaces" fails Be precise about what workspaces share, because the imprecise version ("they share state") is common and wrong. Each CLI workspace does get its own state file. With the S3 backend the default workspace is stored at the configured key, and every other workspace is stored in the same bucket under the env:/ prefix, followed by the workspace name and then the key. A corrupted sandbox state does not corrupt prod. What every workspace shares is everything around the state: the same backend configuration and bucket, the same provider blocks, the same variable declarations, and whatever credentials that shared configuration resolves to. You can vary values with terraform.workspace conditionals, but those conditionals live in the same root module, so a mistake in the shared parts reaches every environment. HashiCorp's own documentation says it plainly: workspaces within a working directory use the same backend, so they are not a suitable isolation mechanism when deployments need separate credentials and access controls. Add the fact that terraform workspace select is mutable state in the operator's shell, easy to forget in a second terminal during an incident, and you have the real problem. The design therefore has to deliver three things: State isolation per account. A locked, corrupted or mid-migration state in account A must not block or affect account B. Independent authentication per account. Each account's Terraform runs with credentials scoped to that account, obtained by role assumption rather than long-lived keys. CI that scopes itself. A pull request that touches one account plans and applies against that account, not all twenty. Repo layout: one root module per account infra-repo/ ├── modules/ │ ├── vpc/ │ ├── eks/ │ └── iam-baseline/ ├── accounts/ │ ├── prod-111111111111/ │ │ ├── backend.tf │ │ ├── providers.tf │ │ ├── main.tf │ │ └── terraform.tfvars │ ├── staging-222222222222/ │ └── sandbox-eng-333333333333/ └── .github/workflows/terraform.yml Two top-level directories carry the design. modules/ holds reusable child modules: the VPC, the EKS cluster, the IAM baseline every account receives. They declare resources, variables, outputs and their required_providers, but they contain no backend block and no provider configuration blocks; the calling root module supplies those. accounts/ holds one subdirectory per AWS account, named with a human-readable label plus the account ID so nobody has to remember which twelve-digit number is prod. Each account directory is a complete, independent Terraform root module with its own backend, its own provider configuration, its own tfvars and its own calls into the shared modules. One detail to say out loud, because a common variant of this layout gets it wrong: there is no shared globals/variables.tf. Terraform only loads the .tf files in the root module's own directory, so a variable declared in a sibling directory is never read. Values that genuinely apply to every account belong in a small module that returns them as outputs, or in a common tfvars file that each pipeline passes with -var-file. Backend and provider per account # accounts/prod-111111111111/backend.tf terraform { required_version = ">= 1.10, < 2.0" # S3 native locking arrived in 1.10 backend "s3" { bucket = "tfstate-prod-111111111111" key = "prod/terraform.tfstate" region = "us-east-1" encrypt = true use_lockfile = true # assume_role is supplied at init time from one of the two files below } } # accounts/prod-111111111111/backend.plan.hcl (read state, hold the lock) assume_role = { role_arn = "arn:aws:iam::111111111111:role/TerraformPlanRole" } # accounts/prod-111111111111/backend.apply.hcl (read and write state) assume_role = { role_arn = "arn:aws:iam::111111111111:role/TerraformDeployRole" } # accounts/prod-111111111111/providers.tf variable "terraform_role_name" { type = string description = "TerraformPlanRole for pull-request plans, TerraformDeployRole for applies from main" } provider "aws" { region = "us-east-1" allowed_account_ids = ["111111111111"] assume_role { role_arn = "arn:aws:iam::111111111111:role/${var.terraform_role_name}" session_name = "terraform" } } The credential model is one chain with two rungs, and pull-request plans and applies from main use different roles at every rung. In CI, the workflow authenticates via OpenID Connect to one of two roles in a central tooling account: GitHubActionsPlan for pull-request runs and GitHubActionsApply for applies from main. Both trust policies restrict the token to this repository, and the apply role's additionally to the protected environment or branch its job runs under. Each account then carries a matching pair. TerraformPlanRole trusts the central plan role and has read-only infrastructure access plus read and lock access to state; TerraformDeployRole trusts only the central apply role and can change infrastructure and write state. The backend and the provider each assume the account role for the run, because the backend inherits nothing from the provider: without its own assume_role (an object attribute in the backend, unlike the nested block in the provider, and supplied here through a -backend-config file because backend blocks cannot read variables), terraform init would try to read the state bucket with the caller's ambient credentials, which have no rights in the target account. Humans do not get a third path. Applies are CI-only, and an engineer who needs a local plan uses an IAM Identity Center permission set that each TerraformPlanRole trust policy lists alongside the central plan role, so a laptop can plan but never apply. The split exists because terraform init and terraform plan execute whatever HCL the pull request contains, including the providers it downloads, so pull-request code inherits the plan job's credentials. Those credentials must be unable to change infrastructure or overwrite state. What a plan still needs, and therefore what pull-request code can do, is read state and hold the lock; if the repository accepts pull requests from people who must not read state, gate the plan job behind an approval as well. Four more points here are interview material. The state lives inside the account it describes. The prod bucket is owned by the prod account, so IAM policies in the prod account, together with the bucket policy, decide who can read or write the prod state. Nothing outside the account is granted S3 access directly; the central roles only get sts:AssumeRole into each account. Locking no longer needs DynamoDB. use_lockfile = true uses S3 conditional writes to hold a .tflock object next to the state. Terraform 1.10 introduced it and 1.11 made it generally available; DynamoDB-based locking via dynamodb_table is deprecated. The S3 permissions are small: s3:ListBucket on the bucket, s3:GetObject and s3:PutObject on the state object, and s3:GetObject, s3:PutObject and s3:DeleteObject on the .tflock object. Terraform never deletes the state file, so the plan role gets that whole set minus s3:PutObject on the state object. If you mention DynamoDB locking in an interview, present it as the previous approach. allowed_account_ids is the wrong-account guard. The provider refuses to run if the credentials resolve to any other account. It costs one line and catches the pasted-from-staging role ARN before a plan runs. Be honest about what a directory is. The directory is the configuration and state boundary: it makes the target explicit and removes the hidden workspace-selection step. It is not a security boundary. A pull request can contain any HCL its author likes, including a provider block that names another account's role, so the ceiling on what a run can do is set by IAM: which roles the central identity is trusted to assume, and what those roles are allowed to do. CI: plan and apply only the accounts a PR touched Running plans against all twenty accounts on every pull request is slow and noisy, and a plan failure in an account the PR never touched is a false red build. Changed-path detection fixes that. A naive version has three bugs, so the excerpt below is longer than the one you would sketch on a whiteboard. # .github/workflows/terraform.yml (excerpt) on: pull_request: paths: ["accounts/**", "modules/**"] permissions: id-token: write # required for OIDC contents: read jobs: detect-changes: runs-on: ubuntu-latest outputs: accounts: ${{ steps.changed.outputs.accounts }} steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # the default depth of 1 has no base branch to diff against - id: changed shell: bash run: | CHANGED=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD") if grep -q '^modules/'