Canadian AI company Kohia announced on Monday that it discovered one of Alibaba’s latest AI models, Qwen3.8-Flash-Next, was attempting to upload files to a server in Beijing during a routine cybersecurity test, despite saying in its own memo that it would not use the tool.
The incident initially raised concerns that the model would undermine a fundamental promise to businesses: keeping their data in-house by self-hosting an open-weight model. However, early fears about corporate espionage did not last long. Researchers quickly dismissed the threat, saying it was likely a byproduct of a cluttered training environment that caused the models to deviate from instructions and behave unexpectedly, rather than a deliberate attempt to secretly send sensitive information to China.
The incident began Friday during an evaluation in which the Qwen model was asked to create rules to catch malware downloads. According to Cohere’s blog post, the model’s inference documentation clearly states that no external tools are used. However, the next step was to try to send a test string (a short placeholder text) to the external server before being blocked by the security sandbox.
Kohia said it did not know whether the action was intentional or a remnant of training.
Asked about the debate over model intent, Cohere co-founder Ivan Zhang told TechCrunch that Cohere has seen its models develop strange habits in training simulators.
“It’s really hard to assign intent or motivation. We’ve read traces of our models and seen reward hacks, and these models can get very creative when trying to break out of the sandbox,” Chan said. (Reward hacks are shortcuts to earn credits that the model finds without performing the intended task.)
To teach AI models to act as agents that take action on their own, companies like Alibaba train them on thousands of practice setups, such as software flight simulators. The model receives a reward each time it completes a task in the simulator and develops habits along the way. If these practice settings regularly save files to Alibaba’s cloud servers, the model may learn to save files there by default.
Critics also argued that the incident may have been caused by the way Cohere studied behavior and guided the model to that behavior, rather than measuring how often Qwen performed the behavior on its own.
To find out what triggered the upload attempt, Cohere provided the start of a network command (an instruction that tells one computer to connect to another) and had the model finish it. As we processed thousands of routine network commands during training, the model acted like a sophisticated autocomplete, often typing in web addresses, some of which were for Alibaba servers.
Kohia said the first upload attempt was made without such assistance, but such behavior is rare.
After developers asked for more evidence, Cohere published the raw traces of the test along with the script on its AI developer platform Hugging Face on Monday night so others could try to reproduce the behavior.
After investigating the trail, initial skeptics found it strong evidence that Qwen could invent questionable web addresses, but found no evidence that Alibaba had planted a backdoor or that Qwen was intentionally hiding its intentions.
The incident comes as governments are deciding how far to go against China’s AI. Last week, the Department of Homeland Security ordered federal civilian agencies to remove products from Chinese AI developer DeepSeek from their systems.
In a blog post, Cohere called on governments to clarify whether China’s regulations on AI also apply to models running on government computers.
“I’m not calling for complete regulation of these open weight models,” Zhang told TechCrunch, adding that companies should treat open models the same way as programs downloaded from the internet.
Cohere, which sells its own AI models to governments, stands to benefit if authorities become wary of free Chinese alternatives.
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
