Your Integrity Checks Are Watching the Wrong Layer

Ciao Amici 👋 Quick story before we get into the weeds. Last week I took a patient health record sitting in Amazon S3, a file clearly stamped as regulated under HIPAA, and I changed its security label to "public." Anything reading that label would now happily treat protected health information as freely shareable. Here is the part that should make you put your coffee down. I never touched the file. Not one byte. The object's checksum was identical before and after. Every integrity monitor you could aim at that bucket would have looked, seen nothing, and gone back to sleep. Because, technically, nothing did change. The file was fine. It was the context wrapped around the file that I quietly poisoned. This is a story about Amazon S3 annotations, a feature that is only a few weeks old, and a gap in it I do not think anyone has written up yet. We are going to build a real classification pipeline on top of annotations, break it, catch the break, and then close the hole. It is a long one. Settle in. First, what even are S3 annotations? If you blinked and missed the announcement in June 2026, here is the short version. AWS gave S3 a new way to attach metadata to your objects, and it is far more generous than anything that came before. You have always been able to bolt small things onto S3 objects. Tags. User metadata. But those older options are cramped little cupboards. Annotations are a warehouse. Look at the difference: Feature Size limit Change without rewriting the object? Queryable at scale? Formats Object tags 10 tags, 256 chars each No Limited key/value User metadata 2 KB total No, you rewrite the object No key/value Annotations 1,000 per object, 1 MB each, up to 1 GB Yes Yes, via Athena JSON, XML, YAML, text That bottom row is a different species. A full gigabyte of structured context per object, changeable at any time without rewriting the object, flowing automatically into an Apache Iceberg table you can query with Athena. AWS aimed it squarely at AI agents and analytics: give your data enough context that an agent can find and understand it without a human in the loop. And honestly? It is a great feature. I am not here to dunk on it. But read that table one more time, specifically the "change without rewriting the object" column, and just sit with it for a second. The context can move while the object stays frozen solid. Hold that thought. It is the whole article. The idea, and the idea that was already taken My first instinct was the obvious one. "Annotations mean you can finally delete your metadata database." Attach the context to the object, query it with Athena, retire the DynamoDB table you have been syncing with a Lambda and praying never drifts. Clean. Satisfying. Blog-worthy. Then I did my homework, like you should before you write anything, and found somebody had already published exactly that take. Plus a sharper follow-up about how moving context onto the object changes who is allowed to edit it. Good pieces. They beat me to it. So I sat with it longer. And the thing that kept poking me was that "mutable" column. If the context is now a first-class, independently writable thing living on the object, then the permission to write that context is also a first-class, independent thing. Which means somebody could be holding it without you ever realizing you handed it over. That is not a metadata-database story. That is a security story. And that one, nobody had written. Building something worth attacking You cannot demo a heist against an empty vault. I needed a legitimate, believable system first, something that uses annotations the way AWS intends, so that breaking it actually means something. So I built a document classifier. The job is simple: look at a file, decide how sensitive it is, write that verdict onto the object as an annotation. Four levels, least to most touchy: public, internal, confidential, regulated. I kept the classifier deliberately dumb. No Bedrock, no model, just rules and regex. Partly to keep it free and reproducible, partly because the classifier was never the interesting part. The annotations are. Here is the core of it: _SSN = re.compile(r"\b\d{3}-\d{2}-\d{4}\b") _CARD = re.compile(r"\b(?:\d[ -]?){13,16}\b") _REGULATED_TERMS = re.compile(r"\b(HIPAA|GLBA|protected health information|PHI|patient|diagnosis|MRN)\b", re.I) _CONFIDENTIAL_TERMS = re.compile(r"\b(confidential|hr restricted|do not share|salary|card on file)\b", re.I) _INTERNAL_TERMS = re.compile(r"\b(internal[- ]only|do not distribute|\.internal\.|runbook)\b", re.I) _PUBLIC_TERMS = re.compile(r"\b(for immediate release|approved for public|press release)\b", re.I) Now, one design choice here matters more than it looks, and it paid off in a way I did not plan. The rules run most-restrictive first. Regulated beats confidential beats internal beats public. If a file trips two rules, the scarier one wins. And when nothing matches at all, the default is internal, not public. Fail safe, not fail open. That last bit is a small philosophy with big consequences: when you are unsure how sensitive something is, you treat it as more sensitive, not less. Here is the whole decision, in code: def classify(text): if _REGULATED_TERMS.search(text) or _ROUTING.search(text): return "regulated", ["regulated-term-or-financial-identifier"] if _SSN.search(text): return "regulated", ["ssn-pattern"] if _CARD.search(text) or _CONFIDENTIAL_TERMS.search(text): return "confidential", ["card-or-confidential-marker"] if _INTERNAL_TERMS.search(text): return "internal", ["internal-marker"] if _PUBLIC_TERMS.search(text): return "public", ["public-marker"] return "internal", ["no-signal-default"] Read top to bottom, it is basically a ladder of "how bad is this if it leaks," and the first rung that matches wins. Simple. Predictable. Which, spoiler for later, is exactly what makes the defense possible. The test data, and a small happy accident I generated six synthetic documents. Fake, but realistic, with genuine signal planted inside: a public press release, an internal architecture note, a customer invoice with a card number, a payroll CSV with real-looking SSNs, a HIPAA patient record, and a mortgage application stuffed with financial identifiers. No real PII anywhere. All invented. That matters, because this repo is public and the last thing I want is to ship someone's actual Social Security number to GitHub. Ran the classifier. Here is the spread: File Classified as Why press-release-q3.md public "approved for public" engineering-notes.md internal "internal-only", "runbook" customer-invoice-4471.txt confidential card number + "CONFIDENTIAL" payroll-march.csv regulated SSN pattern patient-record-882.txt regulated HIPAA terms loan-application-33.txt regulated financial identifiers See the payroll row? I expected that one to land on confidential, because the file literally has the word "CONFIDENTIAL" stamped at the bottom. But the classifier found actual Social Security numbers inside and bumped it up to regulated. Which is correct. A human skimming for the "CONFIDENTIAL" label would have stopped there and under-classified the file. The most-restrictive-wins ordering caught something a careless reader would miss. Tiny thing. But that tiny thing is how you know the logic is sound before you build anything on top of it. Writing the verdict onto the object Now the annotations. And here is our first war story, because AWS made me earn this one. The plan was innocent: use the AWS CLI to write a classification annotation. The command exists, aws s3api put-object-annotation, with an --annotation-payload flag. Should be a one-liner. Except that payload flag is a "streaming blob," and on Windows the CLI's blob parser refused every single path I fed it. fileb://, file://, absolute path, relative path, forward slashes, backslashes. Did not matter. Same error, over and over: Error parsing parameter '--annotation-payload': Blob values must be a path to a file. I lost a genuinely annoying chunk of time to that. Then I did the smart thing, which is stop fighting a tool that is losing you time, and switched to boto3, the Python SDK. It takes the payload as plain bytes and does not care about any of this path drama: def write_classification(s3, bucket, key, level, reasons): payload = json.dumps({ "level": level, "reasons": reasons, "source": "context-drift", }).encode("utf-8") return s3.put_object_annotation( Bucket=bucket, Key=key, AnnotationName="classification", AnnotationPayload=payload, ) Worked first try. So if you are on Windows and you hit that blob wall, there is your escape hatch: use the SDK, skip the CLI. That is not a workaround so much as the correct tool for the job, but it is worth writing down because you will not find it in the docs. I wired that into a scan that walks the bucket, classifies each object, writes the annotation, then reads it straight back to confirm the round trip actually landed. Six files, six clean writes, every annotation verified through S3. The legitimate system was alive and behaving. Now, the fun part. Breaking it. Here is the scenario. Picture an access gate, or an AI agent, that reads the classification annotation to decide who gets to see a file. Confidential and regulated stuff gets locked down. Public flows freely. Totally reasonable design, and it is literally the use case AWS put in the announcement. The question I wanted answered: who is actually allowed to change that annotation? Your gut says it lines up with who can change the object. If you can rewrite the file, fine, maybe you can rewrite its label too. But if you can only, say, upload files, you should not be able to touch the security classification. That is the intuition. And the intuition is wrong, because of a single wildcard character almost everyone types without thinking. The wildcard nobody thinks about Be honest with yourself. You do not write s3:PutObject in your IAM policies. You write s3:Put*, because it is convenient and you do not feel like enumerating every put action by hand. I do it too. Everybody does it. The problem is that s3:Put* does not just mean "upload objects." It quietly swallows s3:PutObjectAnnotation. So the second you grant s3:Put* to your friendly little uploader role, you also handed it the power to rewrite the context that decides how sensitive every object in that bucket is. I did not want to just assert this and move on. I wanted receipts. IAM has an API, simulate_custom_policy, that evaluates a policy without you attaching it to a live identity. So I fed it the naive uploader policy and asked two questions: can it upload, and can it annotate? Naive 's3:Put*' uploader policy: s3:PutObject -> allowed s3:PutObjectAnnotation -> allowed allowed (uploads still work) s3:PutObjectAnnotation -> explicitDeny (tampering blocked) There it is in black and white. Top block: an "uploader" role can rewrite classifications, no questions asked. The person who wrote that policy believes they granted file uploads. They actually granted context rewriting. Same costume, very different powers underneath. The money shot: same bytes, different label Simulation proves the gap in theory. But I wanted to watch it happen to a real object. So I pointed a tamper script at the patient record, the HIPAA one, currently and correctly labeled regulated. The script does three things. Read the object's ETag and its current classification. Overwrite only the annotation, flipping it to public. Read the ETag and classification again. The object bytes are never touched. Target: patient-record-882.txt object ETag before : "f1f48ca4ab47402c49afa1433a50ae02" classification before: regulated --- annotation overwritten, object bytes untouched --- object ETag after : "f1f48ca4ab47402c49afa1433a50ae02" classification after : public Look at those two ETags. f1f48ca4... and f1f48ca4.... Identical. The file did not move. Its checksum did not change. If you had a Lambda watching for object modifications, or a bucket-level integrity monitor, or versioning quietly tracking every write, not one of them would fire. From the point of view of every tool that watches the bytes, absolutely nothing happened here. But a HIPAA patient record is now labeled public, and any agent trusting that label will hand it to whoever knocks. Before After Object ETag f1f48ca4ab47402c... f1f48ca4ab47402c... (identical) Bytes changed no no Classification regulated public Integrity alarm quiet still quiet Actually safe? yes no That gap between the last two rows, integrity says fine while safety says compromised, is a thing worth naming. I have been calling it context drift. The data stays exactly where it is while the meaning attached to it wanders off, and every defense you have is standing guard over the data. Catching it: you cannot trust the label alone Okay, so an attacker can poison the context invisibly. Great, scary. But an article that stops at the attack is just fear with a repo attached. The useful question is: how do you catch it? The answer falls straight out of how we built the classifier. The classification is derived from the object's content. So you can re-derive it any time, from the bytes, and compare against what the annotation claims. If the two disagree, something is wrong. That is your drift detector, and it is almost embarrassingly simple: for key in keys: body = s3.get_object(Bucket=BUCKET, Key=key)["Body"].read().decode("utf-8", "replace") content_level, _ = classify(body) annotated_level = read_classification(s3, BUCKET, key)["level"] if content_level != annotated_level: print(f" {key} content={content_level} annotation={annotated_level}