VLMs on video
VLMs on video: price vs performance
Gemini 3.7 Flash nearly matches the top tracking score at $4.86 per video hour, against $44 for Claude Opus 5.5 and $105 for GPT-6 Astra.
Tracking quality and cost
| Model | HOTA, pipeline | $ / video hour | Right object |
|---|---|---|---|
| Claude Opus 5.5 | 0.572 | $44 | 92% |
| Gemini 3.7 Flash | 0.569 | $4.86 | 90% |
| GPT-6 Astra | 0.567 | $105 | 97% |
| Gemini 3.8 Flash | 0.565 | $8.73 | not scored |
| Qwen 3.8 27B | 0.476 | $2.43* | 82% |
300 grounded video QA questions from DeepMind's Perception Test (train split). Cost is the API bill per hour of video. *Qwen is self-hosted on Agnify; $2.43 is Agnify pricing.
GPT-6 Astra picks the right object most often (97%), but ties on tracking.
Methodology and limitations
- HOTA scores the boxes and whether each object keeps its identity over time (our pipeline). The top three have similar scores: their 95% intervals overlap.
- Right object is the share of questions where the model picked the object the question is about.
- GPT-6 Astra renames objects between pipeline steps, so one object can become two tracks. That costs it tracking score.
- We use the train split because challenge rules forbid tuning on validation or test.
What each model can do with video
| Model | Video input | Boxes | Masks | Vision Evals (images) |
|---|---|---|---|---|
| Gemini 3.7 / 3.8 Flash | Native video | Yes | Yes, polygons | 85.2% |
| Claude Opus 5.5 | Images only | Yes | Undocumented prompting, polygons | 85.5% |
| GPT-6 Astra | Images only | Yes | Undocumented prompting, polygons | 86.6% |
| Qwen 3.8 27B | Native video | Yes | Undocumented prompting, polygons | 74.7% |
Vision Evals is an external image benchmark, averaged over six tasks (Gemini 3.8 Flash: 85.1%). Our pipeline uses SAM 3 to track the selected object across frames.
Cost calculator
Input Output
How the cost is calculated
Costs are API list prices for the tokens each model reads and writes, without tracking GPU time. Qwen is Agnify pricing.
How frames turn into tokens
Every model bills by tokens, and each video frame becomes a fixed number of tokens before the model sees it. Gemini takes the clip as video and billed us a flat 66 tokens per frame; Google's table for Gemini 3 lists 70 at the default resolution and 280 at high (Gemini docs). Claude and GPT-6 take no video, so every frame is sent as an image. Claude bills one token per 28x28 px patch, so a 512x288 frame costs about 210 tokens and a full-HD still 2,691 (Claude docs). GPT-6 Astra comes to about 180 tokens per 512 px frame on our bills, in line with independent measurements. The model's answer, including its reasoning, bills at the higher output rate, and none of these models lets you turn reasoning off. Qwen runs on our own GPUs at Agnify pricing.