How does a vision agent work?
A vision-language model interprets the current frame, then the agent plans and acts through tools. The loop must observe the result after every action, because the screen or environment may have changed. Computer use is one implementation where screenshots guide mouse and keyboard actions.
Where do vision agents fail?
Small text, occlusion, similar controls, spatial ambiguity, stale frames, and hidden state can all produce a plausible but wrong observation. Require confirmation from the target system for important outcomes rather than trusting the image alone. An AI agent should expose uncertainty and stop before irreversible actions. A multimodal LLM supplies perception, but permissions and verification belong to the surrounding agent. The common production failure is not perception but format: the model sees correctly and still returns output downstream code cannot parse, so the agent loop breaks on its own plumbing.
How did we keep a vision model’s output usable in production?
With a schema, not a better camera. In the campaign localization pipeline we built with Freedman, a vision model checks campaign imagery for cultural fit across markets, and its early structured output was mostly invalid. Fixing the schema and the prompt at the source took it to consistently valid, which mattered more to throughput than any gain in visual judgment, because every invalid response stalled the pipeline behind it. The other production lever was concurrency: cultural assessment originally ran market by market and too slowly for the largest markets, and running assessments in parallel brought even the largest market inside its time limit.