The Authorization Gap in AI Operations: Building a Policy-Enforced EKS Control Plane with Cedar, Step Functions, and Systems Manager

Originally published in AWSBuilder The hardest problem in AI-driven operations is not getting an agent to diagnose an incident. Modern models can correlate logs, metrics, deployment events, traces, Kubernetes state, and historical incidents well enough to produce plausible remediation proposals. The harder question begins one step later: Who decides whether the proposed action is actually allowed to touch production? That distinction matters because reasoning quality and operational authority are different properties. An agent can be highly accurate and still eventually make a bad decision. If that decision carries unrestricted production authority, model accuracy becomes a weak safety boundary. A better architecture assumes that recommendations can be wrong and constrains what happens when they are. Telemetry + Cluster State | v AI Investigator | v Structured Remediation Proposal | v Deterministic Evidence Collector | v Policy + Risk Evaluation | | | v v v AUTO APPROVAL DENY | | | Human Decision | | +----> Revalidation | v Bounded Runbook | v EKS Executor | v Independent Verification The important architectural decision is not which model sits at the top. It is that the model does not own the bottom half. Start with an executable contract An operations agent should never hand an executor a natural-language instruction such as: Fix the payments service. There is almost nothing meaningful to authorize in that request. Instead, require the agent to produce a typed remediation proposal: { "action": "RollbackDeployment", "cluster": "prod-eks", "namespace": "payments", "workload": "payments-api", "observedRevision": 42, "targetRevision": 41, "reason": "error_rate_regression", "executionBounds": { "timeoutSeconds": 180, "maxUnavailable": 1 } } This document describes intent, not truth. That distinction is critical. The model may propose that revision 41 is the rollback target. It should not be trusted to assert that revision 41 is healthy, that no incompatible database migration occurred, or that revision 42 is still running when execution begins. Those facts need to come from deterministic systems. A separate evidence collector can enrich the request: { "deploymentGeneration": 108, "currentRevision": 42, "previousRevisionHealthy": true, "statefulMigrationDetected": false, "maintenanceFreeze": false, "evidenceTimestamp": "2026-08-19T13:01:14Z" } This gives us the first hard boundary: The agent proposes the action. Trusted infrastructure establishes the state under which that action may be considered. Without that separation, policy enforcement becomes security theater. An agent capable of supplying both the request and the evidence used to authorize that request can effectively authorize itself. Put authorization outside the agent Amazon Verified Permissions is useful here because it externalizes authorization decisions into policies written in Cedar. The application asks whether a principal may perform an action against a resource in a particular context, and receives an authorization decision. Conceptually, a rollback policy could resemble: permit ( principal == RemediationActor::"eks-remediator", action == RemediationAction::"RollbackDeployment", resource is KubernetesWorkload ) when { context.previousRevisionHealthy == true && context.statefulMigrationDetected == false && context.maintenanceFreeze == false && context.evidenceAgeSeconds 0.95 Confidence may be useful for deciding whether the system needs more investigation. It is a poor substitute for operational invariants. The control plane should ask questions such as: Is the target revision known? Is it healthy? Has persistent state changed? Is the request still current? Is this namespace eligible for autonomous remediation? Does the executor have authority for this resource? Is a change freeze active? These are much stronger authorization signals than whether an LLM reports high confidence. Cedar also supports explicit forbid policies and a default-deny evaluation model. A matching forbid overrides permits, and a request without an applicable permit is denied. That makes certain boundaries straightforward: forbid ( principal, action, resource ) when { context.production == true && context.modifiesPersistentData == true }; Some operations should simply never enter an autonomous path. Deleting persistent data, changing cluster-wide authorization, modifying identity infrastructure, or executing arbitrary shell commands are reasonable examples. Do not turn Cedar into an incident workflow engine There is an architectural trap here. It is tempting to make the policy system return increasingly complicated outcomes: ALLOW_AUTO ALLOW_WITH_APPROVAL ALLOW_IF_SRE INVESTIGATE_MORE ESCALATE_SEV1 I would avoid that. Authorization and operational risk classification are related, but they are not the same responsibility. Keep the classifier deterministic and separate: Risk classifier | +--> AUTO | +--> HUMAN_APPROVAL | +--> ESCALATE Then ask the authorization system whether the identified actor is permitted to execute the requested operation in that path. This keeps Cedar focused on authorization rather than turning policy expressions into a hidden orchestration language. Step Functions owns workflow state Authorization tells us whether an operation may happen. It does not solve what happens before or after the decision. This is where AWS Step Functions fits naturally. A remediation state machine could look like: CollectEvidence | v ValidateProposal | v ClassifyRisk / | \ AUTO APPROVAL DENY | | | WaitForHuman | | +------+ | v RefreshEvidence | v Reauthorize | v ExecuteRunbook | v VerifyRecovery / \ SUCCESS FAILED | v ESCALATE For human approval, Step Functions supports the callback-with-task-token pattern. A workflow can pause until an external process returns the task token using SendTaskSuccess or SendTaskFailure. Human approval is a natural use case for this pattern. The critical state in that diagram, however, is not WaitForHuman. It is: RefreshEvidence Human approval expires Suppose the system proposes: Rollback payments-api from revision 42 to revision 41 An SRE reviews the evidence and approves the action. Eight minutes pass before execution. During those eight minutes, the deployment pipeline releases revision 43. The original approval is now describing a production state that no longer exists. Executing the approved rollback blindly would mean modifying revision 43 based on analysis of revision 42. This is a time-of-check versus time-of-use problem. The solution is simple conceptually: Approval authorizes intent. It does not freeze production state. Immediately before execution, recollect the critical evidence and evaluate authorization again. The remediation contract should also carry preconditions: { "expectedGeneration": 108, "expectedRevision": 42, "targetRevision": 41 } The executor reads the Deployment immediately before mutation. If it finds: expected generation: 108 observed generation: 109 execution stops. Do not ask the model whether generation 109 is "probably okay." Return the workflow to investigation. Once the underlying state changes, the authorized operation is no longer necessarily the same operation. Make execution boring The execution layer should be deliberately less intelligent than the investigation layer. AWS Systems Manager Automation provides runbooks consisting of defined parameters and sequential actions. Custom runbooks can execute scripts, invoke AWS APIs, call Lambda functions, and compose other automation actions. That allows us to expose something like: Runbook: EKS-BoundedDeploymentRollback Inputs: Cluster Namespace Deployment ExpectedGeneration ExpectedCurrentRevision TargetRevision TimeoutSeconds Instead of: execute(command) That difference is one of the strongest controls in the architecture. The AI cannot decide to append: kubectl delete namespace payments because there is no arbitrary command parameter. The execution vocabulary should be intentionally small: RollbackDeployment RestartSelectedStatelessPods ScaleDeploymentWithinBounds CordonNode RemoveKnownBadPod Each action gets its own validation, permissions, preconditions, timeout, and verification behavior. Systems Manager Automation can invoke a purpose-built Lambda executor through aws:invokeLambdaFunction. I would prefer this pattern over turning EKS worker nodes into generic administrative hosts using Run Command. Systems Manager Run Command is designed to execute commands on managed nodes. Technically, it could be used to run Kubernetes administration commands from a managed host. Architecturally, that creates a generic command channel precisely where we are trying to remove one. A dedicated executor with a narrow API is easier to reason about. Least privilege must exist below policy Policy alone is not enough. Imagine that a Cedar policy is accidentally changed and begins permitting an operation it should deny. If the executor has cluster-admin privileges, the policy error immediately becomes a production capability. The better model is: Policy boundary + IAM boundary + EKS authentication boundary + Kubernetes RBAC boundary + Runbook parameter boundary Amazon EKS access entries associate IAM principals with Kubernetes access. Permissions can be supplied through EKS access policies or by mapping the principal to Kubernetes groups and implementing permissions with Kubernetes RBAC. For a tightly constrained remediation executor, custom Kubernetes RBAC is often attractive because the exact verbs and resources can be controlled. For example: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: payments name: bounded-remediator rules: apiGroups: ["apps"] resources: deployments deployments/scale replicasets verbs: get list watch patch Then bind the Kubernetes group associated with the executor's EKS access entry. The executor should not receive access to: Secrets ClusterRoles ClusterRoleBindings PersistentVolumes CustomResourceDefinitions Namespaces ServiceAccounts unless a specific remediation capability genuinely requires it. The principle is stronger than least privilege: Make prohibited actions technically impossible from the remediation identity. Do not let the executor declare success There is another subtle boundary after execution. Suppose Kubernetes accepts: PATCH deployment/payments-api and returns success. Did remediation succeed? We know only that the API server accepted the mutation. We do not know whether: Pods became Ready Service endpoints recovered application errors dropped latency normalized dependencies remained healthy the original SLI recovered customer transactions succeeded Execution success and recovery success are different states. So verification should belong to a different component. Executor | | mutation v Kubernetes | v Independent Verifier The verifier should preferably operate with read-only permissions and consume both Kubernetes state and service-level telemetry. For a rollback, it might evaluate: Deployment available replicas ReplicaSet convergence Pod readiness EndpointSlice endpoints HTTP synthetic check 5xx rate p95 latency incident-specific SLI The actual criteria must be workload-specific. A batch processor cannot be validated like an HTTP API. A Kafka consumer may require lag and processing-rate checks. A stateful service may require consistency or replication signals. This is why "AI remediation platform" is often too broad a product boundary. Safe remediation is easier when capabilities are designed around known workload classes and explicit recovery contracts. What happens when verification fails? The obvious answer is: Undo the remediation. That answer is dangerous. The remediation may have changed state. A second rollback may compound the failure. The system may no longer satisfy the preconditions under which the first action was approved. Automatic reversal should therefore be treated as another remediation request, with its own policy and preconditions. A safer default is often: Verification failed | v Freeze further automation | v Capture current state | v Escalate to human Autonomous systems need a reliable way to stop being autonomous. Design failure behavior before success behavior The control plane itself will fail. That needs explicit semantics. If Verified Permissions cannot return a decision: DENY not: continue because this is an emergency If the evidence collector cannot determine whether a stateful migration occurred: UNKNOWN must not silently become: FALSE If human approval times out, block or escalate. If the runbook receives an unexpected parameter: ABORT If production state changes between authorization and execution: REINVESTIGATE If verification is inconclusive: ESCALATE These decisions may reduce the percentage of incidents handled autonomously. That is acceptable. Automation coverage is not the primary reliability metric for an autonomous remediation system. The more meaningful question is: How much production authority can we safely delegate while keeping the worst credible failure bounded? The architectural lesson The interesting part of AI operations is no longer whether an LLM can call kubectl. It can. The harder engineering problem is building a system where a bad recommendation cannot automatically become an unbounded production change. That requires separating responsibilities: AI Reason about the incident Propose an action Evidence collector Establish current operational truth Risk classifier Determine the execution path Amazon Verified Permissions + Cedar Authorize the exact operation AWS Step Functions Own workflow state and approval boundaries AWS Systems Manager Automation Expose bounded operational runbooks EKS access entries + Kubernetes RBAC Constrain technical capability Independent verifier Determine whether the system recovered None of those mechanisms makes the AI correct. That is precisely why they matter. A mature production architecture assumes that eventually the model will be wrong, the evidence will become stale, a dependency will behave unexpectedly, or a policy will contain a defect. The architecture should make those failures survivable. The goal is not an AI agent powerful enough to operate production without humans. The goal is an operations control plane disciplined enough that increasingly capable agents can participate without inheriting unrestricted production authority. That is a much more useful definition of autonomous operations.