VLMs on video
VLMs on video: price vs performance
GPT-6.1 Terra tracks as well as GPT-6.1 Sol (0.617 against 0.616) at less than half the price: $8.20 per video hour against $19. GPT-6 Astra leads at 0.622 for $105. Gemini 3.7 Flash scores 0.570 at $3.70.
HOTA, pipeline vs $ / video hour
Tracking quality and cost
GPT-6 Astra and GPT-6.1 Sol pick the right object most often (97%). Nemotron loses most of its score before tracking starts, by picking the wrong object or none.
| Model | HOTA, pipeline | $ / video hour | Right object |
|---|---|---|---|
| GPT-6 Astra | $105 | ||
| GPT-6.1 Terra | $8.20 | ||
| GPT-6.1 Sol | $19 | ||
| Claude Opus 5.5 | $43 | ||
| Gemini 4 Argon | $16‡ | ||
| Gemini 3.7 Flash | $3.70 | ||
| Gemini 3.8 Flash | $8.73† | not scored | |
| Qwen 3.8 27B | $1.85* | ||
| Nemotron 3 Nano Omni | $0.60** |
300 grounded video QA questions from DeepMind's Perception Test (train split). Cost is the API bill per hour of video. *Qwen is self-hosted on Agnify; $1.85 is Agnify pricing. **Nemotron is self-hosted too; $0.60 is an estimate (3B active parameters, FP8 on one GPU). †Gemini 3.8 Flash is from the earlier pipeline and not re-run yet. ‡Gemini 4 Argon is at its introductory price, which doubles on 1 Dec 2026.
Methodology and limitations
- HOTA scores the boxes and whether each object keeps its identity over time (our pipeline). GPT-6 Astra and GPT-6.1 Sol are both ahead of Gemini 3.7 Flash (paired bootstrap, 95% intervals above zero); Claude Opus 5.5 only narrowly. Sol and Astra are within noise of each other: Sol trails by 0.006, 95% CI [-0.016, +0.002]. GPT-6.1 Terra is level with Sol (+0.001, 95% CI [-0.015, +0.017]) and ahead of Gemini 3.7 Flash and Claude Opus 5.5. Gemini 4 Argon ties Opus (-0.001) and trails Sol by 0.023.
- Right object is the share of questions where the model picked the object the question is about.
- Since 26 Sep the pipeline refers to each object by number between steps. Before, a model that reworded an object's name split it into two tracks. Sol and Astra lost the most to this: Sol 0.552 before, 0.616 now; Astra 0.567 before, 0.622 now.
- Nemotron 3 Nano Omni runs FP8 on one GPU. It left the answer empty on 24 of 300 questions, often with the answer in its own reasoning; those count as zero.
- Cup games with a hidden object remain the hardest questions for every model. Most of Gemini 4 Argon's gap to Sol comes from them: it often tracks the cup instead of the hidden object.
- GPT-6.1 Terra gave no keyframe on 18 questions, against 12 for Sol and 0 for Opus and Argon. It reads the visibility rule in our keyframe step literally.
- We use the train split because challenge rules forbid tuning on validation or test.
What each model can do with video
| Model | Video input | Boxes | Masks | Vision Evals (images) |
|---|---|---|---|---|
| Gemini 3.7 / 3.8 Flash | Native video | Yes | Yes, polygons | 85.2% |
| Gemini 4 Argon | Native video | Yes | Not tested | - |
| Claude Opus 5.5 | Images only | Yes | Undocumented prompting, polygons | 85.5% |
| GPT-6 Astra | Images only | Yes | Undocumented prompting, polygons | 86.6% |
| GPT-6.1 Sol | Images only | Yes | Not tested | - |
| GPT-6.1 Terra | Images only | Yes | Not tested | - |
| Qwen 3.8 27B | Native video | Yes | Undocumented prompting, polygons | 74.7% |
| Nemotron 3 Nano Omni | Native video | Undocumented, 0-1000 grid | Not tested | - |
Vision Evals is an external image benchmark, averaged over six tasks (Gemini 3.8 Flash: 85.1%). Our pipeline uses SAM 3 to track the selected object across frames.
How the cost is calculated
Costs are API list prices for the tokens each model reads and writes, without tracking GPU time. Qwen is Agnify pricing.
Tokens per frame, billed flat with the clip sent as video
Docs: 70 default, 280 high (Gemini docs)How frames turn into tokens
Every model bills by tokens, and each video frame becomes a fixed number of tokens before the model sees it. Gemini takes the clip as video and billed us a flat 66 tokens per frame; Google's table for Gemini 3 lists 70 at the default resolution and 280 at high (Gemini docs). The calculator uses the same 66 for Gemini 4 Argon, which we did not measure separately. Claude and the GPT models take no video, so every frame is sent as an image. Claude bills one token per 28x28 px patch, so a 512x288 frame costs about 210 tokens and a full-HD still 2,691 (Claude docs). GPT-6 Astra comes to about 180 tokens per 512 px frame on our bills, in line with independent measurements. The model's answer, including its reasoning, bills at the higher output rate, and none of these models lets you turn reasoning off. Qwen runs on our own GPUs at Agnify pricing.
Cost calculator
Input Output