Skip to content

Expert creative judgment for visual AI

Working animators, VFX artists and editors judge AI video motion to the frame, and we score AI judges against them.

Book a callGet the benchmark results

Prompt

Photorealistic, static side-on wide shot. A person in a grey hoodie skateboards across a city plaza at sunset, pushes off twice and carves into a turn. Real-time speed.

  1. Take 1

    Best motion
    A skater in a grey hoodie pushes off twice across a sunlit plaza and rolls on, feet steady on the board.

  2. Take 2

    A skater pushes across the plaza, then pops a kickflip the prompt never asked for and lands rolling.

  3. Take 3

    Worst motion
    A skater pushes, does a kickflip the prompt never asked for, and dust puffs out of the wheels on clean concrete.

  4. Take 4

    A skater pushes, flips the board early in a trick the prompt never asked for, then rolls away across the plaza.
Close-up of take 3 at 0.8 s: the pushing foot kicks up dust from clean, dry concrete

Phantom dust

Frame strips of 4 takes. Example task. Calls written by Bresla to show the format, not by our panel. AI-generated, Grok Imagine Video 1.5 Lite (xAI), October 2026
Best AI judge
26.5%. Share of 400 tasks where the best of 20 AI models picked both the best and the worst image
Human experts
68.9%. Share of tasks where human experts picked both the best and the worst image
Visual Aesthetic Benchmark (ICML 2026 workshop, images)

Experts name the failure to the frame.

The same take, frame by frame. Clean, dry concrete throws up no dust, yet this take raises it from the first frame.

Frame 1 of 60 0.0 s

Dust: the pushing foot kicks up a cloud from clean, dry concrete.

AI-judge agreement with experts (kappa), from coarse criteria to fine body-part motion (NextMotionQA, 2026)
0.70 to 0.10
Share of clips where the best video model passes the prompt and physics checks (VideoPhy-2, 2025)
32.6%
Example task. Calls written by Bresla to show the format, not by our panel. AI-generated, Grok Imagine Video 1.5 Lite (xAI), October 2026

One of these four flips is worse than the rest.

Watch the four takes; Bresla's call follows as you scroll.

Prompt

Photorealistic, static side-on medium shot. A cook flips a pancake in a frying pan; it turns once in the air and lands back in the pan. Kitchen light. Real-time speed.

  1. Take 1
  2. Take 2Worst motion
  3. Take 3Best motion
  4. Take 4

Bresla picked take 2 as the worst.

Nothing tosses the pancake: the pan only tilts, and for the first 1.5 s the pancake hangs over the pan.

3.3 s Cause and effect: the pan only tilts, yet the pancake stands up on its edge and flips over.

Take 3 has the best motion.

Every take floats; take 3 is the only one where the pan's toss causes the flip.

Example task. Calls written by Bresla to show the format, not by our panel. AI-generated, Grok Imagine Video 1.5 Lite (xAI), October 2026

Four steps from a lab's brief to scored data.

  1. Brief

    The lab says what its model must get right.

  2. Design

    We write the tasks and a craft rubric.

  3. Judge

    Experts compare outputs, pick the best and the worst, and tag each failure.

  4. Deliver

    Scores, preference pairs, rubrics and training tasks, with a quality report.

Labs receive three kinds of data.

  • Expert evaluations
  • Rubrics and preference data
  • Checkable training tasks

Quality is measured: gold-standard checks, expert agreement, and AI judges tested against the panel.

Each of these clips breaks one rule of real motion.

Seven example calls Bresla wrote on AI clips. Each clip stops on the frame its call names.

  • A woman walks along a hallway towards a tall mirror and meets her reflection.
    Interpenetration4.9 s She walks into the mirror and her body passes through the glass.
  • A tennis ball is dropped onto a wooden floor and bounces.
    Collision / bounce3.2 s The ball climbs higher after its third bounce than after its first.
  • A boy jumps off a low garden wall onto the grass.
    Weight and momentum0.6 s The boy hangs in mid-air beside the wall for over a second before he lands.
  • A street performer juggles red balls in a town square.
    Pop / morph4.5 s A fourth ball has appeared; the juggler started with three.
  • A hand holds a match against a dark background as smoke rises and a flame appears.
    Fluid, cloth, smoke0.0 s Smoke pours from the match before it is lit.
  • A man holds a black umbrella on a city street at dusk.
    Action broken1.5 s The open umbrella folds shut, then opens again.
  • A girl blows out the candles on a birthday cake.
    Cause and effect2.0 s She blows for three seconds and the flames stand upright.
Example task. Calls written by Bresla to show the format, not by our panel. AI-generated, Grok Imagine Video 1.5 Lite (xAI), October 2026

Bresla Motion v1

Designed in full and run once our pre-seed round closes.

7 models from 7 labs

Veo 3.1
Google
Seedance 2.5
ByteDance
Kling 3.0
Kuaishou
MiniMax H3
MiniMax
Wan 3.0
Alibaba
Grok Video 1.5
xAI
FLUX 3 Video
Black Forest Labs

8 motion categories

  • Walking and running
  • Jumps, lifts, getting up
  • Hands and objects
  • Contact and collisions
  • Falling and throwing
  • Liquids and cloth
  • Animals
  • Camera moves

12 failure tags

Contact slide
Planted foot, hand or wheel skates
Interpenetration
Solids pass through each other or the body
Collision / bounce
Wrong rebound or friction; teleports instead of bouncing
Weight and momentum
Floats, falls too slowly or too fast, starts or stops without cause
Timing / speed
Unmotivated slow motion, uneven speed
Pop / morph
Things appear, vanish, duplicate or change shape, size or identity
Fluid, cloth, smoke
Implausible flow, pour, splash or drape
Anatomy
Extra or missing limbs, broken hands, impossible joints
Action broken
Incomplete, repeats, runs backwards, static, or no effect
Cause and effect
Effect missing, wrong or out of order
Camera
Wrong move type (zoom vs dolly), ignored path, unmotivated shake
Glitch
Flicker, local blur, wrong shadow or reflection

4 claims, fixed before judging

  • C1 Experts agree with each other far more than AI judges agree with experts.

    Supported if: The best AI judge is below the expert ceiling, and the 95% CI of the gap excludes 0. Reference from images: AI 26.5%, experts 68.9%.

  • C2 AI judges miss fine-motion failures (contact slide, weight and momentum, timing) more often than gross ones (pop/morph, anatomy, interpenetration).

    Supported if: The hit rate on fine-motion failures is lower, and the CI of the difference excludes 0.

  • C3 AI judges are fooled by motion that looks clean.

    Supported if: AI judges call the frozen clip the worst less often than experts do.

  • C4 The 7 models differ on motion, and the motion ranking differs from the general-preference ranking.

    Supported if: Descriptive only: reported with CIs, no threshold.

Get one email when the results are published.

Bresla uses your email for that one message only. Privacy

Bresla's panel will be working animators, VFX artists and editors.

Everyone on the panel will be:

  • Vetted on real work
  • Credited by name if they want it
  • Paid €40+/h for all work, including the test
  • Told plainly that their judgments become open data that AI companies use to test and improve models
  • Free to withdraw before publication
Register your interest

or email Andrei at andrei@bresla.ai. The panel forms once our pre-seed round closes.

Labs pay for expert judgment,
and the rules now ask where data comes from.

Andrei Balan

Founder

Andrei built Algebo.ai, an AI math tutor, in four months (January to May 2026). Algebo won:

Andrei builds Bresla's judging tools in-house.

Bresla is at the idea stage; the panel and the benchmark start once our pre-seed round closes.

Judgment is the missing data
in visual AI.

Book a callEmail Andrei at andrei@bresla.ai