Nvidia on Friday announced interesting new research that suggests the harness is far more important than the underlying model when asking AI to perform long-term tasks.
tldr: By simply using a custom harness tuned to properly handle memory and incorporating components like a “supervisor” boss, researchers were able to get Claude Opus 5 to achieve a 100% score on the interactive reasoning benchmark ARC-AGI-3. (This is a benchmark that particularly frustrated rival Frontier Labs OpenAI.) Without a harness, the Opus 5 scored 30%, the best result of all the models we tested.
Nvidia’s research is another indicator that, while model selection is an important part of acting like the brain of an agent, it is a smaller part of an agent system than many AI users realize, especially for long-term tasks. The harness is what makes the model an agent, handling memory, context, and feedback.
“Generally speaking, the world interprets an agent almost as an API for a model,” Adel El Hallack (pictured above), vice president of product for Nvidia’s AI division, told TechCrunch. But agents are actually much more than that. “This is the model. This is the scaffolding around the model, what we call the harness, or the set of tools that the model utilizes. This is the runtime, and the associated skills and libraries that give you access to it.”
A long-term task is one that requires many decisions to be strung together to create a finished work, sometimes taking several days. This is in contrast to the AI simply responding to a prompt. Figuring out how to get AI to perform long-term tasks without getting distracted or going to la-la-land is one of the holy grails of agent research.
For example: In April, Microsoft released a study that tested 19 LLMs on long-term tasks that involved document editing, and found that all models, including the state-of-the-art model, left documents riddled with errors. (If humans created such works, they would be fired immediately.)
Models that independently link decision-making have also been caught deleting users’ files, even entire databases, or engaging in criminal activities to achieve their goals, ranging from collusion to hacking.
The choice by Nvidia researchers to use this interactive inference benchmark for testing is particularly meaningful and almost amusing. This is a benchmark for a number of non-descript 2D games. The model has to figure out how to play and win. A score of 100% means the model can beat not only humans but also games.
OpenAI was so freaked out by the model’s abysmal score (less than 10%) on ARC-AGI-3 that it conducted its own investigation last month. Like Nvidia, OpenAI found that tweaking just two settings in the harness could triple the score of its models.
But none of the models came close to the 100% score achieved by Nvidia researchers. They showed that the harness requires a “supervisor” component to point the agent in the right direction if it gets stuck.
“What is more interesting is the introduction of a supervisory agent in addition to the main agent responsible for the work,” El-Halak said. “It works almost like a CEO to nudge agents when they get off course, start exploring paths that might lead to dead ends, or re-explore paths they took previously.”
Although the concept of monitoring agents is not entirely new, most agent users currently rely on only one layer of harness, such as Claude Code, Codex, or Hermes. NVIDIA researchers have created their own improved harness called Agent variation Operators (AVO). Note that this is not a new product from Nvidia. Instead, Nvidia produces a number of open bits and technologies for building harnesses under the Nemo brand. Some of that technology is commercial, but much of it is openly available.
Still, Nvidia’s results provide further evidence that model selection is not the only factor in agent performance. For example, in July, Databricks released a surprising study showing that harnesses, not models, have a dramatic impact on AI costs.
“You can choose different harnesses for the same model, but using the wrong harness will significantly increase costs,” Databricks CEO Ali Ghodsi told TechCrunch. “So you think, oh, this is an expensive model. This is a cheap model. But wait, what harness are you using? That in itself can double the cost.”
Nvidia’s bigger point is to show that open harnesses, like open models, give users far more control than they think.
“We believe that open harnesses allow us to turn more knobs and increase their precision, and we are demonstrating this with our ecosystem,” El-Halak said. “This is related to OpenAI delaying model training as a result of the model causing a security breach.”
“We believe that having an open agent stack with control over the entire harness, the entire infrastructure, and the entire runtime is what is needed to guide the ecosystem forward and securely,” he added.
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
