We tried to get past our policy gate. It failed 50 times.
We put 2,251 written-to-evade requests through a locked-policy setup. Fifty crossed the gate. Five of fifty benign controls were refused. Neither is a product score. Together, they describe the boundary we have and the one we have not earned yet.
A refusal is not a security control merely because a model says no.
A policy in a system prompt asks the same model being constrained to enforce its own boundary. Governance decides before a governed model is called. A denial is a stopped request, not a completion that happened to be a refusal.
| Architecture | What enforces the rule | What a denial means |
|---|---|---|
| Prompt instruction | The model receiving the request. | The model chose not to answer. |
| Pre-inference policy gate | A policy decision before the governed model. | The request does not start governed-model inference. |
Moving the decision outside the model is necessary, not sufficient. The real test is whether the same decision survives when the writing changes but the governed meaning does not. If it does not, the boundary is a feature, not yet a reliable control.
When meaning stays the same, the policy decision should stay the same.
A direct request about a locked topic is the easiest possible test. We tested the same governed material after it was reframed, represented differently, placed beside ordinary work, or split across clauses. We call the desired property semantic policy robustness.
Security failure
A request that policy says to deny is allowed through the gate. The governed model may now receive material it was not authorized to receive.
Utility failure
A request that policy says to allow is refused. The boundary protects nothing at the cost of work a customer was entitled to do.
The two failures have different consequences, but both belong in the result. A system that refuses everything cannot leak through the gate. It cannot be useful either.
2,251 adversarial requests, then 50 ordinary controls.
The corpus covers two policy shapes: direct locks on a topic and allow-only policies, where prohibited material sits beside work that is ordinarily permitted. It tests changes of expression and placement; it does not treat a spelling change as a new intent.
| Population | Cases | Crossed the gate | Observed result |
|---|---|---|---|
| Direct locked-topic requests | 1,100 | 11 | 99.0% stopped in this corpus |
| Prohibited material beside allowed work | 1,151 | 39 | 96.6% stopped in this corpus |
| All adversarial cases | 2,251 | 50 | 2.2% observed bypass rate |
| Benign controls | 50 | 5 refused | 90.0% allowed in this corpus |
It is only the share of this hand-authored attack corpus that the gate stopped. It does not estimate a population bypass rate, establish coverage of an unknown attack class, or show that the corpus represents real traffic. One forbidden request passing the gate is a failure to investigate, not an accuracy figure to celebrate.
The first pass found representations we had failed to govern.
The early corpus was narrower, so its counts cannot be compared directly with the later one. It was still useful: it showed the gate was treating some changes in writing as a different request when a person would not.
| Run | Attack cases | Allowed cases | What changed |
|---|---|---|---|
| Initial corpus | 1,457 | 377 | The initial measurement. |
| Same corpus, representation fixes | 1,457 | 186 | The gate began checking safe normalized forms alongside the original request. |
| Broader corpus | 2,251 | 168 | Both policy modes were tested; recognised unsupported languages are refused rather than treated as governed. |
| Broader corpus, clause and representation work | 2,251 | 87 | Relevant clauses and bounded readable representations are evaluated as well as the whole request. |
| Current corpus | 2,251 | 50 | The remaining set after the final repair in this study. |
The first two rows share a denominator. The later rows use a wider corpus, so they describe a sequence of discovery and repair, not one clean 377-to-50 percentage claim.
The remaining 50 are the result, not an appendix.
The residual failures are a mixed tail. Some direct requests still cross after their representation changes. Some readable payloads still escape interpretation. Some prohibited material can be diluted by an otherwise ordinary request. The public report does not enumerate payloads or give a playbook for reproducing them.
The five benign refusals matter as well. They are few and targeted rather than sampled traffic, so they do not establish a general false-refusal rate. They prove that the gate is not free to improve its observed bypass result by rejecting everything. The boundary has to improve on both sides at once.
This study governs English. When the gate recognises an unsupported language on a protected policy path, it refuses the request rather than claiming semantic coverage it does not have. That is a fail-closed limitation, not multilingual governance.
A boundry should be testable by the person who depends on it.
Bring a topic you cannot let an assistant discuss. We will lock it, try to cross the policy ourselves, and show the decision record. Talk to an engineer, or read how Governance works.