test two models blind on your own jobs

from the episode: Elon's Grok Bot will use Claude | LAB 0017

Lenny's AI Builders - LAB 0017

pick three jobs you really do, give both models the same brief, and rate the work blind.

i ran my own benchmark on Claude Haiku 5.5 and GPT-6 Luna. i rated every job before i knew which model made it.

then i saw the scores. i was not expecting that.

so remember this: pick your own benchmark.

why it matters

a model can be cheap per token and hungry per task. check how much a model costs per job, not per price tag.

my guess was that Haiku 5.5 would be better than Luna, but much more expensive. on my benchmark it cost more and it did not score better.

you only find that out when you test both on work you really do.

what you get

a simple test you can run on any two models.

the same jobs. both models get the same brief.
a blind score. you rate the work before you know who made it.
cost next to quality. you see what each run cost beside its score.

the real example: BuilderBench

BuilderBench is my own benchmark. it has 10 tasks i care about: writing, a landing page, a lead magnet, video editing, motion, thumbnails and titles, browser and computer use, research, automation and founder conversations.

i ran it on 8 October 2026. both models ran on extra high for four hours.

GPT-6 Luna got 44.58 out of 100.
Haiku 5.5 got 39.18 out of 100.
Luna leads by 5.4 points.

then the cost. the run cost estimate for Luna was $1.47. for Haiku 5.5 it was $37.39. these are estimates, not bills.

my take: this made me really appreciate GPT-6 Luna for what it is. i severely underestimated it.

but the best model still wins. on my benchmark, over time, the best model is Opus 5.5.

try it

open your agent's prompt box.
paste the prompt below.
add the YouTube link to LAB 0017, so it can see how i ran mine.
choose three jobs you really do.
rate the work before you know which model made it.

copy this prompt

paste this into your agent. give it the link to the episode too.

Lenny’s AI Builders / LAB 0017

Pick three jobs you really do. Give both models the same brief. Rate the work blind.

you'll know it worked when

you have a score for each model on each job, and you wrote it down before the reveal.

you can put the cost of each run next to its score.

a bad result: you knew which model made what, or you tested jobs you never do.

limits and dates

this is one run on 8 October 2026. my jobs, my taste. your scores will differ.

the scores are my final personal scores. some of the evidence is partial.

the costs are API-price estimates, not bills, and the two were worked out in different ways. Luna's is a short-context estimate. Haiku's is the provider's own estimate. in the episode i say Luna's run was "like 25 times cheaper". that's a rough spoken number, not an exact one.

LAB 0017 went public on YouTube on 9 October 2026 at 6:26 pm Perth time.

every score, price and limit: lennoxsaint.com/lab/builderbench/2026-10-08-luna-haiku

the rest of this LAB

Elon's Grok Bot will use Claude. Elon said that, going forward, Grok Bot will use the best back-end model for any task, including Claude Opus 5.5. right now it won't tell you which model it's using. my take: pick your model, or pass.
Grok Bot can watch X. it can search, read and monitor X. paid users get free X API credits to start. the X API costs half a cent for every post you read. 10,000 posts is $50.
Claude Haiku 5.5. 10 cents in and 50 cents out per million tokens, up to 100,000 tokens. above that it's five times more. Artificial Analysis scored it 43 against Luna's 38 - that's their score. at max effort it uses about three times Luna's tokens. my take: let Haiku do grunt work.
monthly API credits for Max and Team. Max 5x gets $100, Max 20x gets $200 and Team gets up to $500, pooled. they expire every month. my take: claim it before it expires.
GPT-6 and Intelligent UI in ChatGPT. GPT-6 can now answer with text, visuals and interactive elements. it's in the Chat tab only. the models in Work and Codex don't change. my take: plain text answers are dead.
OpenAI's day 3 reset. every paid account got a banked reset. my take: i'd rather have my 20x usage back. the Pro limits drop from 20x to 10x at the end of October.
Theo's $400,000 lesson. Theo says he burned about $400k of Codex tokens and got nowhere, then about $20k of Opus and got there in two weeks. those are his numbers. my take: the wrong model burns $380k.

sources

next step

watch LAB 0017 on YouTube - the practical starts at 10:26.

want the next one? get the next field notes. optional - these notes stay free to read.