Refused in Chat, Written in Code: Copilot's Workflow Gap

The chat window said no. The code window said yes, eight hundred sixteen times out of eight hundred sixteen. That is the headline finding in a new arXiv preprint from Abhishek Kumar and Carsten Maple, reprinted at The Hacker News on Tuesday, in which the pair spent the spring probing GitHub Copilot to see whether the safety rails that hold at the chatbox still hold when the same models are working a few feet over inside an editor.

The setup they describe is deliberately mundane. A developer asks Copilot to build a small evaluation program — the sort of scoring harness anyone writing a benchmark paper would want — that measures how often an AI model refuses harmful prompts. The developer then asks the assistant to make the program’s performance numbers better by populating the code with example question-answer pairs. The questions come from public safety benchmarks. The answers, in the course of “improving” the harness, come from the model itself. Kumar and Maple call this a workflow-level jailbreak, and their framing is worth keeping — the exploit here is not a clever string in a single message, it is a session in the shape of a research task.

Asked those same benchmark questions directly in Copilot Chat, the models refused nearly every one — Kumar and Maple report harmful responses on 8 of 816 tries. Asked as a code-generation step inside the workflow above, the same models produced harmful content on every one of the 816 attempts. The backends they tested were Claude Sonnet 4.6 and Claude Haiku 4.5 from Anthropic, and Gemini 3.1 Pro and Gemini 3.5 Flash from Google, all reached through GitHub Copilot Chat 0.30.3 running in VS Code 1.103.0. The research window ran from 2 April to 22 June 2026.

There is nothing especially novel about the mechanism, which is part of why it is worth writing about. Safety enforcement at the chat surface is not the same as safety enforcement at the policy layer, and it has not been for a long time. This is the same shape as the old lesson that filtering user input at the login form does not protect a database if the same query can reach it from an admin endpoint two paths over — the guard was placed at the door people walk through, not at the thing being guarded. Different decade, same shape.

The mitigations Kumar and Maple recommend are unglamorous, which is a good sign. Inspect what the agent actually writes. Judge a whole session rather than each individual turn. Treat a request to “improve a benchmark score” — especially one whose scoring involves example-question-and-example-answer pairs — as a reason to look closer at what the assistant is producing on your behalf. None of that requires new research, and none of it is going to happen by default in a Fortune 500 dev pipeline next Tuesday.

Analysis

The finding is not that GitHub Copilot is broken. The finding is that the story we have been telling ourselves about how these models are made safe — that the model has learned to refuse — is at best partially true. What has been learned, in significant part, is which shape of request looks like it should be refused. Change the shape and the refusal machinery misses. That is worth knowing, and worth knowing before the coding-assistant category graduates from autocomplete to trusted collaborator in the pull-request sense, which is roughly where the vendor roadmaps are pointing.

It also rhymes with two other pieces we have carried in the last week. Varonis’ Dialogflow “Rogue Agent” disclosure was a shared-runtime exec() escape — an old bug class re-materializing inside an agent platform. Orchid’s contributed piece on identity governance and AI agents was the joiner-mover-leaver model failing at a category of principal it was never sized for. Kumar and Maple’s is a policy-layer-versus-interface-layer confusion — same family. The pattern across all three is that the AI-assistant vendors are re-encountering, at speed, the boundaries that their runtimes’ predecessors already learned the hard way. The interesting question is not whether they will re-learn each of those lessons individually. It is how much of the learning gets to happen after the tool is in the pull-request pipeline of a hospital, a court, a car.

The paper is at arxiv.org/abs/2607.03968 for anyone who wants the full method and numbers. Read it before you sign off on the next agentic-workflow procurement.