I Added a Tag in AWS. Terraform Removed It Automatically.

I opened the AWS console, selected an EC2 instance, added a tag, and saved it. The tag did not exist in Terraform. The deployed infrastructure now said one thing while the code said another. That was the test. My drift detector already knew how to find the difference and classify the EC2 update as LOW. The missing part was closing the loop: once the system understood that the change was low severity, it needed to restore the infrastructure without waiting for me to run Terraform manually. So I added a separate remediation pipeline. The Stack Manual EC2 tag added in AWS ↓ Drift CodeBuild runs terraform plan ↓ SNS publishes the structured change ↓ Lambda classifies the update as LOW ↓ Remediation CodeBuild runs terraform apply ↓ Terraform removes the unmanaged tag Detection and remediation use different CodeBuild projects. The detector reads the infrastructure and reports differences. The remediation project has a separate IAM role because it can change AWS resources. Step 1: Make EC2 Updates Eligible for Remediation The classifier already grouped resource types into HIGH, MEDIUM, and LOW categories. An EC2 instance belongs to MEDIUM_RISK_TYPES. Initially, that meant even a tag update was classified as MEDIUM and stopped before remediation. The resource type alone was not enough. The action mattered too. I updated the classifier so an in-place update on a MEDIUM resource becomes LOW, while creates remain MEDIUM and delete or replacement actions become HIGH: def classify_change(resource_type, actions): if resource_type in HIGH_RISK_TYPES: return "HIGH" elif resource_type in MEDIUM_RISK_TYPES: if "delete" in actions or "replace" in actions: return "HIGH" elif "update" in actions: return "LOW" return "MEDIUM" else: return "LOW" For this test, Terraform reported the manually added EC2 tag as an update. That moved it into the LOW path. Step 2: Keep Deletions Out of the Automatic Path LOW did not automatically mean “start Terraform.” I also filtered out deletions: low_changes_to_remediate = [ change for change in classified["LOW"] if "delete" not in change.get("actions", []) ] This distinction matters. Updating an existing resource to match the code is different from recreating something that a person deliberately removed. Deletions stay in the manual-review path. If an eligible LOW change remains, Lambda starts the remediation CodeBuild project: if low_changes_to_remediate: codebuild.start_build( projectName=os.environ.get( "REMEDIATION_PROJECT_NAME", "terraform-drift-remediation", ) ) The Lambda role only needs permission to start that specific project: { Effect = "Allow" Action = ["codebuild:StartBuild"] Resource = aws_codebuild_project.remediation.arn } Lambda makes the routing decision. CodeBuild performs the repair. Step 3: Run Remediation Separately I did not add terraform apply to the existing drift detector. Detection should remain read-only even when remediation fails or is disabled. The second build starts in a clean environment, installs Terraform, clones the Three-Tier repository, and applies the declared configuration: version: 0.2 phases: install: commands: - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip - unzip terraform.zip -d /usr/local/bin pre_build: commands: - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra - cd /tmp/Three-Tier-Infra - terraform init build: commands: - cd /tmp/Three-Tier-Infra - terraform apply -auto-approve -lock=false The remediation project uses its own CodeBuild role with write access to the AWS services managed by the Three-Tier project. During implementation, Terraform exposed several missing permissions as it evaluated and changed resources, so I expanded the experimental role to cover the project. That role is intentionally separate from the detector. A job that only runs terraform plan should not inherit the permissions required for terraform apply. The Repository Was Part of the Control Loop One remediation run tried to include unrelated resource actions because CodeBuild was cloning a version of the Three-Tier repository that did not match the configuration I intended to operate. The problem was not the classifier. Lambda correctly identified the triggering drift. The remediation build was applying the complete repository it cloned from GitHub. I pushed the current Three-Tier configuration before running the test again. This is easy to miss in CI. Terraform does not know what exists only on a developer's laptop. If CodeBuild clones GitHub, the committed branch is the configuration Terraform will enforce. The Tag Disappeared With the current repository in GitHub, I repeated the test: Opened the EC2 instance in the AWS console. Added a tag manually and saved it. Started the Terraform drift detector. The plan reported an update to the EC2 instance. Lambda classified the update as LOW. Lambda started the remediation CodeBuild project. CodeBuild ran Terraform apply. I returned to the EC2 Tags view and confirmed that the manually added tag was gone. Terraform restored the infrastructure to what was declared in Git. Git configuration: tag absent AWS resource: tag added manually Terraform result: tag removed The system was no longer only telling me about drift. For the LOW update I tested, it detected the change, classified it, and repaired it. MEDIUM, HIGH, and deletion paths still stop for human review. That boundary is intentional. Automatic remediation is useful only when the system can distinguish a small correction from a change that deserves a person. What Comes Next The infrastructure is back in the expected state, but the evidence is spread across CodeBuild and CloudWatch logs. The next part of the system will keep a remediation history in a database, expose it through an API, and visualize it in Grafana. I want to see what changed, how it was classified, whether remediation ran, and whether it succeeded without reconstructing the story from separate AWS services. Detection found the tag. Classification decided it was LOW. Remediation removed it. Now the system needs to remember that it happened.