The Map Was Wrong

We audited the ground truth on the two standard video-caption benchmarks. 70% of CaReBench clips and 66% of DREAM-1K entries had factual errors in the GT itself.

Wrong sports, wrong camera angles, misspelled on-screen text, claims of actions that don’t happen on screen. Every model that’s ever reported a DREAM-1K AutoDQ or CaReBench score has been ranked against a reference that is wrong about something in roughly two out of every three clips.

That’s the headline. The rest of this post is the audit, the methodology, and a 2B open-source model that lands #3 on the cleaner TimeLens-Bench leaderboard while the field is still chasing the wrong target.

TL;DR

The two standard dense-caption benchmarks (CaReBench, DREAM-1K) ship with broken ground truth: 70.2% of CaReBench clips and 65.8% of DREAM-1K entries have factual errors that a video-aware judge flags. Their judges are text-only and can’t catch any of it; worse, when a model adds correct dense detail the GT omits, the judge marks it as a hallucination. On TimeLens-Bench, which uses no LLM judge, Marlin-2B (2B params, open source) lands #3, ahead of GPT-5, GPT-4o, and Qwen2.5-VL-7B. Posts 2 and 3 cover the training journey and the audit-judge release.

About Marlin-2B

2B-parameter video-language model · Apache 2.0 · fine-tuned from Qwen3.5-2B-Base · built by NemoStation. Demo · Hugging Face · Part 1 of 3 in our build-journey series.

---

Most benchmarks aren’t ready for real-life evaluation

We started by evaluating Marlin-2B on the standard public dense-caption benches, CaReBench and DREAM-1K. Both use text-only LLM judges that read our model’s caption next to the ground-truth caption and decide which side gets each atomic fact right. The judge never sees the video. That choice produces two problems.

Problem 1: when the model is more accurate than the GT, the model loses

Here’s a real row from our DREAM-1K eval. The clip shows a yellow humanoid creature in a half-kneeling pose. Our model wrote:

“yellow humanoid creature stands in a crouched position”

The DREAM-1K GT said:

“the character half kneels on the ground”

No mention of color. The text-only judge ruled on our predicted event:

“The video description mentions a character half kneeling, which is similar to a crouched position, but does not specify the color of the character.”

The judge marked it neutral, didn’t credit the prediction, and gave the same ruling to four more “yellow…” predictions on the same clip, each time citing that the GT does not specify color. The judge can’t watch the video, so when the candidate adds correct detail the GT happens to omit, that detail is unverifiable and gets dropped. Our model was being penalized for being more accurate than the reference.

Figure: same candidate caption, same GT, same clip. Left, the text-only judge can’t verify “yellow” and defaults to neutral. Right, the video-aware judge sees the creature is yellow and credits the candidate. The text-only protocol is what every public dense-caption benchmark uses today.

Problem 2: the GT is wrong far more often than the field admits

To measure how often this was happening, we built a video-aware audit judge: Gemini-3-Flash with the actual video frames in the prompt, scoring the GT caption directly. Across roughly 2,000 GT entries:

Figure: GT-error rates by benchmark. The field has assumed GT is essentially correct; the audit shows otherwise. 70.2% and 65.8% are the conservative numbers; 2-8% of clips are bad enough that the GT itself scored ≤5/10.

A few representative consensus-flagged errors from CaReBench:

idx=3: states the video is filmed from her perspective, but it is a front-facing/selfie camera view idx=4: typo in banner text, “NEW CIDS” instead of “NEW VIDS” idx=6: claims she finishes applying eyeshadow to her left eye, which is never shown in the video idx=9: describes the background as a blue and yellow gradient when it is a solid teal/mint green idx=12: the athlete is wearing a sleeveless tank top, not a short-sleeve shirt idx=13: describes the javelin as blue, whereas it is primarily yellow/orange with a blue grip section

And from DREAM-1K:

idx=0: GT claims they grabbed two green monsters; the video shows one green and one blue-with-purple-spots idx=1: GT narrates “one character raises their hand and then puts it down. The other character walks away and is followed by the former, both characters go up a staircase together.” None of those events happen in the clip; the characters just walk around the space.

DREAM-1K idx=1. No hand-raising, no staircase. The characters just walk around the space.

idx=7: GT calls a blue figure “cat-like”; the video shows a mechanical praying-mantis-like insect idx=11: GT says “sword fight”; the actual combat is a body tackle with an axe-like weapon idx=14: GT says “large bird”; the creature is a dragon with scales, horns, and a reptilian tail

This is what the standard text-only judges have been grading against for the past two years. They can’t watch the video, so they can’t catch any of it. The leaderboards aren’t a map of model capability; they’re a map of how closely each model’s vocabulary happens to match the GT’s vocabulary, even when the GT is wrong.

---

Why this gap exists, and what’s actually grading what

Here’s how the major dense-caption and grounding benchmarks evaluate models:

On the two benches that try to grade dense caption quality, the field’s default is a text-only LLM judge. That made sense when the benchmarks were designed; Gemini-2.5-Flash didn’t exist yet. The limitation is inertial now, not technological. The benches still use text-only judges, the text-only judges still can’t audit the GT, and five minutes of running a video-aware judge over the GT side exposes the problem at scale.

LLM-judge failure modes aren’t a new observation. Zheng et al, 2023, Wang et al, 2023 on positional bias, Saito et al, 2023 on verbosity bias have all documented them. What’s different here is scale: the majority of every dense-caption benchmark’s GT is wrong, not just a few percent.

---

Why we kept the model small

Iteration is the rate-limiting step in finding judge bugs. We re-ran the full 1000-clip CaReBench audit a dozen-plus times in three weeks across different judge prompts. That cadence is only affordable if a full eval finishes in under two hours. A 2B model trains comfortably on H100s with a frozen ViT and DeepSpeed Zero-2, and a full re-eval costs an evening. The same 2B fits on a single L4 GPU for inference, so the public demo runs cheaply.

The audit judge (Gemini-3-Flash) is what surfaces the GT errors. The small model is what made it affordable to re-run that audit many times across different prompts. Without the iteration count, the errors stay invisible.

The same constraint applied to data. We regenerated roughly 400K grounded training captions (200K dense captions plus 200K temporal-grounding events) every time we tightened the teacher prompt. We tightened it more than you’d think; that story is in Post 2.

---

Why Qwen3.5-2B specifically

Qwen3.5-2B uses a hybrid attention layer stack: 24 layers total, with 18 Gated DeltaNet layers (linear attention, O(n)) and 6 full-attention layers in a strict pattern of 3 linear then 1 full, repeating. Full attention sits at layers 3, 7, 11, 15, 19, and 23. Verified from the model’s config.json layer_types array.

Practical consequences for this project:

• Decode is much faster than an all-full-attention 2B baseline. Each GDN layer is O(n) instead of O(n²), which compounds during GRPO rollouts.
• KV-cache footprint is smaller per sequence, so we batch more rollouts in parallel.
• Token counts are heavy. From our actual training set (226K records across 5 source files), median is roughly 2,100 tokens per example. Longer clips from ActivityNet- and COIN-style sources sit at roughly 5,000 tokens typical and 7,800 at p90. The cap before we drop a clip is around 8,500 tokens. Attention savings hit twice: on prefill, and on every decode step.

We trained on H100s with ms-swift and vLLM-backed rollouts, which gave us 16 rollouts per prompt during GRPO data filtering. The public demo at vlm.nemostation.com runs the trained model on a single L4 GPU.

Figure: Qwen3.5-2B’s 24-layer stack. Full attention at layers 3, 7, 11, 15, 19, 23 (six total). The other 18 use Gated DeltaNet (linear attention). Visual tokens are injected via DeepStack residual into the first 3 layers.

The 6 full-attention layers keep long-range modeling intact for dense captioning. The 18 GDN layers make decode cheap enough to fit GRPO rollouts on one GPU.

---

The TimeLens-Bench leaderboard

TimeLens-Bench is the bench we trust most because there’s no LLM judge in the loop. Scoring is pure span-IoU arithmetic on (start_sec, end_sec) predictions. Whatever a model scores here is roughly what it’s actually doing.

We evaluated Marlin-2B on the refined TimeLens-Bench (Zhang et al, 2025, Dec 2025), which is a manually re-annotated version of Charades-STA, ActivityNet-STA, and QVHighlights. Our sample counts match the paper’s exactly (3363 / 4500 / 1541), so this is apples-to-apples on the same data.

Non-Marlin rows from Zhang et al, 2025, Table 1. Marlin-2B rows from our own evaluation on the same refined TimeLens-Bench data. Full R@1@τ breakdown in Appendix A.

Figure: Charades-TimeLens mIoU across the leaderboard. Marlin-2B GRPO (orange, 2B params) lands at #3, ahead of TimeLens-7B, Gemini-2.5-Flash, and Qwen3-VL-8B.

Sorted by Charades-TimeLens mIoU, our 2B model is #3 overall, behind only TimeLens-8B (8B) and Gemini-2.5-Pro:

• Marlin-2B GRPO beats GPT-5 by +9.55 mIoU on Charades-TimeLens (50.05 vs 40.5) and by +2.98 on ActivityNet-TimeLens. QVH is essentially a tie at -1.3. A 2B open-source model out-grounding GPT-5 by ten points on Charades is the kind of result that travels.
• Marlin-2B GRPO is the only 2B model in the top 10 on every TimeLens-Bench split, and competitive with Qwen3-VL-8B (4× its parameter count): ahead on Charades, slightly behind on the other two.
• Marlin-2B SFT, before any RL, already beats Qwen2.5-VL-7B by +6.4 mIoU on Charades and beats GPT-4o. The +3 to +4 mIoU GRPO adds on top is upside.
• The only models that consistently beat Marlin-2B are proprietary frontier models (Gemini-2.5-Pro, Gemini-2.5-Flash) or open-source 8B models (Qwen3-VL-8B, TimeLens-8B).

How a 2B model ended up in this leaderboard position is Post 2.

---

Where this leaves the field

None of this is a strawman of careless researchers. The text-only AutoDQ protocol predates the era of high-quality multimodal judges. CaReBench and DREAM-1K were both significant contributions in their time. The reason the field’s defaults are broken in 2026 is inertia.

The inertia matters. With roughly 70% of CaReBench GT entries containing at least one consensus-flagged factual error, and 66% of DREAM-1K GT entries similarly flagged, the dense-caption leaderboards aren’t measuring what they claim to. A model that scores well on AutoDQ is mostly vocabulary-matching a wrong reference. A model that adds correct dense detail (color, on-screen text, camera angle) gets penalized for diverging from a noisy GT. The incentive structure points toward writing captions that resemble the reference rather than seeing what’s in the video.

We did three things about it:

• Trained a model that does both dense captioning and temporal grounding jointly. That’s Post 2.
• Rebuilt the audit judge so it grades against the video. Post 3.
• Released val_set, a new 1000-clip benchmark whose GT was generated under that judge. Also Post 3.

One thing worth flagging now: we found a prompt-side fix that consistently reduces Gemini-3-Flash hallucinations. The short version is asking the judge to enumerate visible frame-level evidence before scoring any axis. We use it on both the training-data teacher and the audit judge. Full prompt and ablation in Post 3.

---

Limitations and honest disclosure

• Self-grading risk. The audit numbers (70.2% / 65.8%) come from a Gemini-3-Flash judge. The Marlin-2B training teacher is also Gemini-3-Flash. There’s a non-zero risk the audit favors Gemini’s worldview over the human annotators’. A re-audit with a non-Gemini judge (Claude Opus 4.7 or GPT-5 with video) is planned for Post 3.
• The audit judge itself can be wrong. Spot-checking flagged samples (CaReBench idx=15, DREAM-1K idx=1), we found cases where the audit judge hallucinated its own visual claims rather than catching a real GT error. Some fraction of the 70.2% / 65.8% are judge errors, not GT errors. The non-Gemini re-audit will tighten these numbers.
• What counts as a GT error. We count any clip where the video-aware judge flagged a non-empty errors_gt list. Some flagged items are minor (misspelled watermark, small color disagreement). The more conservative version (GT scored ≤5/10 in all 4 independent runs) is 2.1% on CaReBench and 7.7% on DREAM-1K. The 70% number is the loose definition, the 2% number is the tight one. Both are real.
• Calibration drift across Gemini snapshots. Judge scores aren’t fully stable across gemini-3-flash-preview versions. We pin the model version in every metrics.json and recommend others do the same.
• The yellow humanoid anecdote is one row. At bench scale, our model’s pairwise precision dropped from roughly 0.87 to 0.50 as captions got denser. The bench-wide fraction of “neutral” rulings that are actually correct details only the v2 re-audit can quantify.
• Inference protocol per model. Each row in the TimeLens-Bench leaderboard uses the inference setup the model was trained for. Marlin-2B uses FPS_MAX_FRAMES=240, Qwen2.5-VL / Qwen3-VL / MiMo-VL use 2 FPS sampling from the TimeLens paper, GPT and Gemini use protocols described in the paper’s appendix. The benchmark data is identical for everyone. Forcing every model into one shared frame budget would penalize models trained at higher resolutions.

---

What’s next

In Post 2, we walk through how we built the roughly 400K-clip grounded training set and the SFT → GRPO → SimPO recipe that lifted Marlin-2B from 45.73 to 50.05 mIoU on Charades-TimeLens. One interesting finding was that 4 bugs killed each training stage; we’ll walk through each of them. The teacher-side version of the frame-evidence prompting trick is also documented there.

In Post 3, we release videoeval-v2: the audit judge code, the judge prompts, the 1000-clip GT-error annotations on CaReBench and DREAM-1K, and val_set, a new 1000-clip benchmark with timestamped GT audited against video at generation time. Apache 2.0, full open release.

If you’re shipping a video VLM, your AutoDQ / CaReBench numbers are probably grading your model against a 70%-broken reference. Open an issue on the Marlin-2B HF repo with a sample of your judge outputs and we’ll look at them with you.

• 🐠 Try the demo: vlm.nemostation.com
• ⭐ Star the model: huggingface.co/NemoStation/Marlin-2B
• 📩 Follow for Posts 2 and 3

---

Written by Aryan Jain.

Acknowledgements: the TimeLens team at ARC Lab / Tencent PCG for the refined VTG benchmark we evaluate on; the ms-swift team for the training framework; the Qwen team for the base model; Hazy Research for visual style inspiration; and the maintainers of the open-source video datasets we built on top of (Charades, ActivityNet, QVHighlights, CaReBench, DREAM-1K).

---

Appendix A — Full TimeLens-Bench numbers

Per-split, per-model R@1@{0.3, 0.5, 0.7} and mIoU. Marlin-2B rows are our own evaluation on the refined TimeLens-Bench annotations (n counts match the paper exactly: 3363 / 4500 / 1541). All other rows are from Zhang et al, 2025, Table 1.

Charades-TimeLens (n = 3363)

ActivityNet-TimeLens (n = 4500)

QVHighlights-TimeLens (n = 1541)

---

Appendix B — Methodology and reproducibility

Bench audit. The 70.2% and 65.8% headline numbers come from running an independent video-aware judge (Gemini-3-Flash, frames + GT caption + axis-scoring prompt) over the GT side of each benchmark and counting how many entries received a non-empty errors_gt list. CaReBench used 4 independent runs (consensus across all 4 = 70.2%; any single run = 98.2%; severe ≤5/10 in all 4 = 2.1%). DREAM-1K used a single-side video-aware judge on the GT (any-error = 65.8%; severe = 7.7%).

TimeLens-Bench numbers. Non-Marlin rows from Zhang et al, 2025, Table 1 verbatim. Marlin-2B rows from our own evaluation on the same refined TimeLens-Bench annotations. Scoring is pure span-IoU arithmetic; anyone with the same weights and annotations reproduces the numbers deterministically.

Architecture. Qwen3.5-2B layer_types array directly from the public Qwen/Qwen3.5-2B-Base config.json.

What’s public today vs coming with Post 3. Public now: Marlin-2B model weights, the TimeLens paper baseline numbers, the Qwen3.5-2B architecture details. Releasing with Post 3: the audit judge code, the judge prompts (including the frame_evidence mechanic), the 1000-clip GT-error annotations on CaReBench and DREAM-1K, and the new val_set benchmark. If a number doesn’t seem to add up before then, open an issue on the Marlin-2B HF repo and we’ll share that slice directly.

---

Appendix C — Glossary