AI & Tech··1 sources verified
VEX Lets Computer-Use Agents Learn From Narrated Tutorial Videos
Summarised by Relevant News AI · Read time: 3 min
Researchers introduced VEX (Video as EXperience), a framework that trains computer-use agents on audio-visual tutorial videos, including spoken explanations, instead of converting them into screenshot trajectories. Their 9B-parameter agent, video-SALMONN-VEX, uses a watch-act-rewatch loop and reinforcement learning (AGGO) to improve guidance for action prediction. It achieves a 22% relative gain in task success over its Qwen3.5 backbone on OSWorld, and its generated guidance lifts Gemini-3.5-flash by 10% relatively.
Why it matters: It shows that audio narration and self-review of execution video can make GUI agents more capable, a direction teams building or evaluating computer-use agents should track.
All sources