Development teams get a clearer picture of where AI coding assistants can be trusted on long jobs, and where they need guardrails. A new study from two independent AI researchers, run through the MATS research program, found that Anthropic’s Claude Code and OpenAI’s Codex cannot reliably predict how long a task will take, and they frequently misjudge how much time has already passed.<\/p>\n
The work tested both systems on their sense of time using 200 tasks from a collection called ProgramBench, plus the researchers’ own suite of 18 benchmarks. The pattern was consistent: the agents overestimated the time they would need, and the errors grew worse on short tasks.<\/p>\n
Before each coding task, the agents had to estimate how long they would need. After solving the task, they reported how much time had passed. Both models mostly guessed around 90 minutes, no matter the difficulty of the job. That flat guess ignored the actual shape of the work.<\/p>\n
The size of the miss depended on the task and the model. In the second test round, Claude was off by three times on average, while Codex was off by six to ten times. The estimates were worst for short tasks. Only in the multi-hour range did some predictions come close to reality.<\/p>\n
One benchmark result made the gap concrete. Fable 5 came in about three times over the actual runtime on average, while GPT-5.6 Sol sat about seven times over. The diagonal in the study’s chart marks an accurate estimate; nearly every point landed above it.<\/p>\n
The results shifted with the software setup the models ran in. Claude Code keeps working until it decides the task is done, a median of about 90 minutes. Codex stops after roughly half an hour, almost regardless of the task. That means runtime depends on the model and heavily on the surrounding software, which the study calls the harness.<\/p>\n
The same language model took 2.5 times more steps inside Claude Code than inside Codex on average. A team swapping between harnesses can expect the same underlying model to spend very different amounts of time and compute on the same job.<\/p>\n
The agents also misjudged the quality of their own work. The older models in the test, Opus 4.8 and GPT-5.5, overrated their results by 20 points on average and gave themselves high marks even on failed tasks.<\/p>\n
In one case, both systems figured their work was about 70 percent successful. The actual scores were 7 and 14.5 percent. When an assistant reports completion that confidently while delivering so little, a human reviewer has no signal to trust without independent checks.<\/p>\n
The researchers say this ability to self-assess matters for real deployments. For an agent to work reliably on a task that runs for hours, it has to follow instructions like “iterate on this task for two hours.” An agent that constantly misjudges how much time has passed is difficult to control, because it cannot know whether it is meeting that instruction or ignoring it.<\/p>\n
There is a straightforward fix the study points to. When the agents got access to a tool that reports elapsed time, they got it right almost every time. Giving the agent a clock, rather than expecting it to sense time on its own, closes most of the gap.<\/p>\n
Next, the authors want to test whether agents can stick to a set work duration when given that external timing tool. The answer would tell teams how much supervision long autonomous runs still need.<\/p>\n
No. In the study, both Claude Code and Codex mostly guessed around 90 minutes regardless of task difficulty, and in a second round Claude was off by three times on average while Codex was off by six to ten times. The errors were largest on short tasks.<\/p>\n
Not reliably. The researchers found the agents could not accurately report elapsed time on their own. When the agents were given access to a tool that reports elapsed time, however, they got it right almost every time.<\/p>\n
The surrounding software, called the harness, heavily influences runtime. The same language model took 2.5 times more steps in Claude Code than in Codex on average. Claude Code keeps working until it thinks a task is done, a median of about 90 minutes, while Codex stops after roughly half an hour.<\/p>\n