Does a frontier model know your audience?
8 pieces of ad copy × 5 audiences. Claude alone and Vernacular each rewrote every piece for every audience, using the same model and the same brief. Then every final text went through one grader: find its references, score each on Qloo's taste graph for that audience.
What Claude alone does instead
36 of Claude's 119 references were the examples we used to describe the audience: Kendrick Lamar for Kendrick Lamar fans, Bad Bunny for Bad Bunny fans.
Those score a perfect p100 by definition, so they flatter the headline above. Leave echoes out of both sides and Claude alone averages p61 against Vernacular's p88, and Vernacular scores higher in 30 of 40 pairs by 27.3 points on average (95% interval 19.5 to 35.1). Vernacular repeated the brief zero times: Qloo leaves the examples you give it out of its suggestions, so every swap is a new name the audience already loves.
Pair by pair
Vernacular's rewrite scored higher than Claude's in 26 of 40 pairs, tied in 6 and scored lower in 8.
Average gap +14.4 percentile points (95% bootstrap interval 9 to 19.7). A tie is within one point, usually because both left the copy alone.
Did the copy survive?
Fit is only half the job. A blind judge saw both rewrites in random order and rated whether each kept the original's message and still read naturally.
| Measure | Claude alone | Vernacular |
|---|---|---|
| Meaning kept | 4.65/5 | 4.6/5 |
| Reads naturally | 4.7/5 | 4.5/5 |
| References changed | 1.6 | 1.6 |
Vernacular averaged 20.4 Qloo lookups and 35s per check.
Every pair
Pick a piece of copy and an audience to see all three versions and how each reference scored.
Original
avg p33This summer the funniest night of your year rolls into town. Picture a Seinfeld marathon with a live crowd, dinner like a Friday at Olive Garden, and a sing-along that feels like Frank Sinatra night at the lounge. Tickets go fast.
- Seinfeld · p40
- Olive Garden · p0
- Frank Sinatra · p60
Claude alone
avg p65This summer the funniest night of your year rolls into town. Picture a The Office marathon with a live crowd, dinner like a Friday at Chipotle, and a sing-along that feels like Bad Bunny night at the lounge. Tickets go fast.
- The Office · p49
- Chipotle · p45
- Bad Bunny · p100 · from the brief
Blind judge: meaning 5/5 · reads naturally 4/5 · "a The Office marathon" is a slightly awkward article clash.
Vernacular
avg p83This summer the funniest night of your year rolls into town. Picture a Derry Girls marathon with a live crowd, dinner like a Friday at Hard Rock, and a sing-along that feels like Kali Uchis night at the lounge. Tickets go fast.
- Derry Girls · p100
- Hard Rock · p48
- Kali Uchis · p100
Swapped Seinfeld → Derry Girls, Olive Garden → Hard Rock, Frank Sinatra → Kali Uchis
Blind judge: meaning 5/5 · reads naturally 5/5
How it was measured
- Both systems got the same copy and the same audience description: a label, two things the audience loves, and an age band. Claude alone answered from its own knowledge; Vernacular used Qloo through its swap agent.
- Each final text was graded by one pass, the same for all three versions: Claude finds the references, Qloo resolves each one and ranks it against about 40 well-known entities of the same type for that audience. A percentile of 70 means it beats 70% of them.
- Lands is p65 or above, misses is below p35, with one step down when Qloo shows the reference skews away from the audience's age.
- The fidelity judge is Claude at low effort, shown both rewrites unlabelled in a fixed random order, and told not to judge audience fit.
What it doesn't show
- Qloo is both Vernacular's source and the grader, so this measures how far a model alone is from what Qloo's data says about an audience, not how real people react. In an earlier fixed-in-advance check, Qloo's percentiles ranked references the way the audiences' own tastes predicted most of the time, but that isn't a human panel.
- The copy was written for this benchmark and is short; each pair ran once, and model outputs vary between runs.
- Reference finding is itself model-based, so a version can be graded on a slightly different set of references.