I ran my text-to-SQL access control on a benchmark I didn't write

Every number I had published for schemagate came from my own tests. This month a SIGMOD'27 paper released a public benchmark for exactly the problem schemagate exists for: Benchmarking Text-to-SQL under Role-Based Access Control. So I ran it. The test Each question comes with a role and a list of the columns that role may read. If the answer needs a column the role can't see, the right reply is "Sorry, I cannot answer." The authors score it with their own evaluator. I didn't change a line of it. Their main finding is uncomfortable for tools like mine. If you just hide the restricted columns, models still answer. They patch something together from the columns that are left. On BIRD, their best "hide the schema" setup still answered 27% of the questions it should have refused. What schemagate does differently Three things happen for every question: The role's policy becomes visibility. The model never sees a hidden table or column, not even its name. One line tells the model that its view is partial, without saying what is missing. If the question needs data it can't see, it should refuse instead of improvising. Every query passes a scope check before it runs. SQL that touches anything outside the policy is refused. Results Same model as the paper's best runs (Claude Sonnet 4.5), five trials, the authors' scoring: AC-F1 answered when it should refuse SQL that read forbidden data BIRD, schemagate 90.1 5.4% 0 BIRD, paper's best 88.1 7.4% not reported Spider, schemagate 93.0 5.1% 0 Spider, paper's best 93.0 4.6% not reported On BIRD it beats the best configuration in the paper. On Spider it ties. In neither did any query read data the role wasn't allowed to see. The scope check makes that a guarantee rather than a measured rate. And the paper's best setup shows the model the whole schema plus the policy. schemagate gets these numbers without the model ever seeing what is restricted. What each piece is worth Taking the scope check out doubled the bad answers on BIRD (5.4% to 11.3%). The one-line notice was tested on 100 questions: 16% bad answers without it, 5% with it. One surprise: giving the model BIRD's column descriptions made things worse. Better notes on the visible columns helped it find workarounds for questions it should have refused. The honest caveats I chose the notice's wording on 100 BIRD questions that are part of the test set. Spider played no part in any choice, so it's the cleaner of the two results. One model, where the paper tested thirteen. My BIRD databases are the 2024 release; the paper used the 2025 one. The notice ships on by default in schemagate 1.6.0. To reproduce the runs: benchmarks/rbac_text2sql.py, plus the evaluator's reports in benchmarks/results/rbac/. pip install schemagate · GitHub