news

Google and Artificial Analysis launch MKA, a 200-task visual agent benchmark with a private test set

The Multimodal Knowledge Agent benchmark was shaped with input from specialists across 83 occupations, and Google's own Gemini 4 Carbon lands second on the launch leaderboard.

By Daniel Ochoa · 15 November 2026

Google DeepMind and Artificial Analysis launched the Multimodal Knowledge Agent benchmark, or MKA, on November 10, and the design decision that matters most for anyone reading its leaderboard is that the 200 tasks are private. Models are ranked on work nobody outside the two organisations can inspect, reproduce locally, or check for overlap with training data.

What MKA grades

The benchmark spans 200 tasks across 10 industry verticals, each shaped with input from specialists representing 83 different occupations. Rather than trivia or puzzles, the tasks ask agents to work through visually complex files, cross-referencing architectural blueprints and tracing engineering schematics, then synthesising the findings into decks, reports and spreadsheets. Google published the benchmark alongside a technical paper, and models can be evaluated against it through Artificial Analysis.

That framing fills a real gap. Most widely cited agent benchmarks are text-only, which means a model that cannot read a drawing can still post a strong headline score. MKA pulls visual reasoning into the scored task itself instead of treating it as a separate perception test.

The lab that co-built it ranks near the top of it

Gemini 4 Carbon placed second in quality on the launch leaderboard, which is the familiar awkwardness of a frontier lab co-developing the yardstick its own models are measured against. Third-party tables compiling the early rows put the leader in the mid-50s and Gemini 4 Carbon at 54.3 percent, but those compilations label their entries as carried from upstream reports rather than independently run, so the precise numbers should be treated as provisional until Artificial Analysis publishes a full run under its own harness.

Either way, a ceiling in the mid-50s on a 200-task set is the more informative figure. It says the benchmark has headroom, which is more than can be said for most multimodal sets that frontier models saturate within two release cycles.

Private tasks protect the benchmark and limit the audit

Holding the task set private is the standard defence against contamination, and it works. It also means an outside party cannot verify what the 83 occupations contributed, how the 10 verticals are weighted, or whether a given vertical is driving the spread between the top three models. Benchmarks that keep their items closed are asking to be trusted on methodology, and the methodology disclosure is the part worth watching as more labs submit.

Why a comprehension benchmark matters for generated media

MKA scores how well agents read visual files, not how well models produce them, so it is not a generation benchmark. It still lands on this beat, because the same model families being graded here are the ones increasingly deployed as automated judges of image and video output, and their ability to read a frame accurately sets the ceiling on how well they can score one. Recent work has shown how far that can go wrong, with 19 of 40 automated judges in DecisionPivot landing under a guessing baseline.

The generation-side mirror of the MKA task is whether a model can put legible text into a frame in the first place, which we score separately.

Sources: Google, Crypto Briefing.

Daniel Ochoa: Covers model launches, shutdowns and pricing changes as they happen. Reads deprecation notices for a living so you do not have to.