VLMs on video
VLMs on video: price vs performance
GPT-6 Astra now leads on tracking at $105 per video hour. Gemini 3.7 Flash scores 0.570 against Astra's 0.622 at $3.70, a twenty-eighth of the cost.
Tracking quality and cost
| Model | HOTA, pipeline | $ / video hour | Right object |
|---|---|---|---|
| GPT-6 Astra | 0.622 | $105 | 97% |
| Claude Opus 5.5 | 0.594 | $43 | 93% |
| Gemini 3.7 Flash | 0.570 | $3.70 | 90% |
| Gemini 3.8 Flash | 0.565† | $8.73† | not scored |
| Qwen 3.8 27B | 0.481 | $1.85* | 85% |
300 grounded video QA questions from DeepMind's Perception Test (train split). Cost is the API bill per hour of video. *Qwen is self-hosted on Agnify; $1.85 is Agnify pricing. †Gemini 3.8 Flash is from the earlier pipeline and not re-run yet.
GPT-6 Astra picks the right object most often (97%), and now keeps it as one track.
Methodology and limitations
- HOTA scores the boxes and whether each object keeps its identity over time (our pipeline). Only GPT-6 Astra is clearly ahead of Gemini 3.7 Flash (paired test, p = 0.005). Claude Opus 5.5 wins and loses against Gemini almost equally often.
- Right object is the share of questions where the model picked the object the question is about.
- Since 26 Sep the pipeline refers to each object by number between steps. Before, a model that reworded an object's name split it into two tracks. GPT-6 Astra lost the most to this (0.567 before, 0.622 now).
- Cup games with a hidden object remain the hardest questions for every model.
- We use the train split because challenge rules forbid tuning on validation or test.
What each model can do with video
| Model | Video input | Boxes | Masks | Vision Evals (images) |
|---|---|---|---|---|
| Gemini 3.7 / 3.8 Flash | Native video | Yes | Yes, polygons | 85.2% |
| Claude Opus 5.5 | Images only | Yes | Undocumented prompting, polygons | 85.5% |
| GPT-6 Astra | Images only | Yes | Undocumented prompting, polygons | 86.6% |
| Qwen 3.8 27B | Native video | Yes | Undocumented prompting, polygons | 74.7% |
Vision Evals is an external image benchmark, averaged over six tasks (Gemini 3.8 Flash: 85.1%). Our pipeline uses SAM 3 to track the selected object across frames.
Cost calculator
Input Output
How the cost is calculated
Costs are API list prices for the tokens each model reads and writes, without tracking GPU time. Qwen is Agnify pricing.
How frames turn into tokens
Every model bills by tokens, and each video frame becomes a fixed number of tokens before the model sees it. Gemini takes the clip as video and billed us a flat 66 tokens per frame; Google's table for Gemini 3 lists 70 at the default resolution and 280 at high (Gemini docs). Claude and GPT-6 take no video, so every frame is sent as an image. Claude bills one token per 28x28 px patch, so a 512x288 frame costs about 210 tokens and a full-HD still 2,691 (Claude docs). GPT-6 Astra comes to about 180 tokens per 512 px frame on our bills, in line with independent measurements. The model's answer, including its reasoning, bills at the higher output rate, and none of these models lets you turn reasoning off. Qwen runs on our own GPUs at Agnify pricing.