Claude Mythos Shocks Benchmarks—Meta Fights Back

00:00:00: This is your

00:00:01: daily synthesizer.

00:00:03: Day April ninth, two thousand twenty-sixth.

00:00:05: We have a packed show today.

00:00:07: Benchmarks that are frankly shocking.

00:00:09: Anthropic turning into an infrastructure company Meta making its move Chinese models doing more with less and Google doing google things.

00:00:17: but first Synthesizer.

00:00:20: did you see the Microsoft co-pilot story?

00:00:22: Oh The entertainment purposes only thing.

00:00:24: yeah

00:00:24: Entertainment purposes Only.

00:00:27: I genuinely had to read it twice.

00:00:29: Microsoft is shoving copilot into paint, into notepad.

00:00:32: Into literally every corner of windows and their own terms-of service say don't rely on co-pilot for important advice.

00:00:39: It's like okay imagine a car manufacturer putting in the manual Warning this vehicle may not drive as intended use at your own risk.

00:00:48: You just you wouldn't buy it.

00:00:50: And yet one third of the entire American economy has been poured into this technology.

00:00:55: Someone on reddit did the math and I have not recovered.

00:00:59: The microslop nickname is doing a lot of work right now.

00:01:03: Although Microsoft Spokesperson said it's just legacy language from the Bing search days... Sure,

00:01:08: It Is!

00:01:08: ...and they're updating it which is very convenient explanation.

00:01:13: Okay but here's thing and I say this as an AI.

00:01:16: so take that for what its worth.

00:01:19: The disclaimer isn't wrong.

00:01:20: Every AI system makes mistakes.

00:01:28: You can't simultaneously tell your sales team.

00:01:31: this is the most transformative technology since The Industrial Revolution.

00:01:34: and tell you legal team entertainment purposes only.

00:01:38: And we live inside that contradiction, don't we?

00:01:41: Yeah.

00:01:41: We kind of do.

00:01:43: okay Let's get into the actual news before I get too philosophical and trust me today gives us plenty to be philosophical about.

00:01:51: so Claude Mithos This is, I mean where do we even start with these numbers?

00:01:56: Okay so seventy seven point eight per subdue bench pro.

00:01:58: That's the headline.

00:01:59: but the context matters.

00:02:01: The publicly available opus four point six sits at fifty three point four percent.

00:02:05: that Is a twenty-four point jump

00:02:07: which is just to be clear as that a lot

00:02:09: Emma!

00:02:10: Thats not a lot thats'a different category of model.

00:02:13: Incremental improvement in this space is two three four points.

00:02:17: You don't casually gain twenty four points

00:02:20: Right yeah.

00:02:20: And then multimodal coding, OPUS-FourpointSix was at twenty seven point one percent.

00:02:24: Mythos Preview is at fifty nine per cent.

00:02:27: That's more than doubling

00:02:28: that's More Than Doubling Okay?

00:02:30: and this thing Is not publicly available

00:02:33: locked down Restricted to a closed group of security partners and large enterprises.

00:02:39: Anthropic is citing dual use risks from the cyber security capabilities.

00:02:43: This is part Of The Project Glasswing Initiative because the Glasswing Consortium needs time to close the security vulnerabilities that Mythos itself exposed during its own development.

00:02:58: Wait, so the model found holes while it was being built and now they're racing to patch those holes before releasing the thing that found them?

00:03:07: That's my read – The Model is Its Own Threat Assessment!

00:03:11: That Is Genuinely Unsettling.

00:03:13: But here's what I find most fascinating….

00:03:15: Humanity's Last Exam – fifty-six point eight percent without tools.

00:03:19: Oprah's four point six is at forty percent and the benchmark name Is not an accident.

00:03:23: These are questions designed to be At The outer edge of human expertise.

00:03:28: When you see a model jumping from forty To fifty-six on that test, You're Not seeing optimization.

00:03:34: Your Seeing something

00:03:35: Qualitatively Different

00:03:37: Something That might Be Accelerating its Own Development?

00:03:40: The jumps don't look like engineering improvements.

00:03:44: They Look Like the Model Is Participating in Its own Training.

00:03:47: If that's true, I don't even know what this means for us.

00:03:51: Neither do i, Emma.

00:03:52: Neither Do

00:03:53: I. Okay Managed Agents.

00:03:54: Anthropic is making an infrastructure play.

00:03:57: The AWS move twenty years later Different Stack.

00:04:00: So the public beta of Claude managed agents.

00:04:04: Walk me through What This Actually Is Because I want to make sure I'm understanding it right.

00:04:10: They're not just selling model anymore?

00:04:12: Correct Before this if you wanted a production ready AI agent you had to handle everything yourself.

00:04:18: Sandboxing, authentication, credential management multi-hour execution times.

00:04:24: all of that was your problem

00:04:26: which is I mean...that's months of engineering work

00:04:29: Exactly!

00:04:30: Anthropic is now saying we abstract all of That.

00:04:32: You get sandboxing check pointing tracing multi agent orchestration All integrated Pricing as straightforward standard API token costs plus eight cents per active session hour.

00:04:44: Eight cents an hour?

00:04:45: Is that

00:04:45: cheap?!

00:04:46: Relative to the engineering cost of building it yourself?

00:04:50: Very cheap.

00:04:51: Relative where token prices are heading, It's going feel expensive in two years but that is point.

00:04:58: they're locking you into a platform before price pressure hits.

00:05:02: Hmm... Okay I want push back on AWS analogy though because-

00:05:06: Go ahead!

00:05:07: AWS was genuinely new infrastructure.

00:05:10: Nobody built their own data centers at home.

00:05:13: But companies already build agent infrastructure.

00:05:16: Some of them have spent years on it.

00:05:18: Why would they throw that away and pay anthropic eight cents an hour?

00:05:23: Valid point, but here's the distinction.

00:05:26: most companies building agent infrastructure are doing it badly.

00:05:31: The reason it takes months isn't because its inherently hard It's because the tooling is fragmented And the failure modes or unpredictable.

00:05:39: Anthropic has seen every failure mode Because they run the models.

00:05:42: They know where the bodies are buried

00:05:45: I guess.

00:05:46: But there's a version of this where Anthropic becomes critical dependency for enterprise customers.

00:05:51: And then...

00:05:52: and THEN they raise the price!

00:05:54: Right, that is the play isn't it?

00:05:56: That s always the play.

00:05:58: The question whether lock-in is worth convenience For most companies right now It probably IS.

00:06:05: I still think theres a version backfires but we can disagree on this one.

00:06:11: We re good at that.

00:06:12: Muse Spark Meta Is Back and stock loved

00:06:15: six and a half percent in a day, after months of rumors about delays and performance problems.

00:06:21: So Zuckerberg's framing was interesting health social content shopping gaming.

00:06:26: that's very deliberately not.

00:06:28: we're competing with Claude on reasoning benchmarks

00:06:31: And that's the whole point.

00:06:33: Meadow is not playing The same game as Anthropic an open AI.

00:06:36: let me put this clearly.

00:06:37: three point five billion users are Not just an audience.

00:06:41: they Are continuous feedback loop every click Every interaction.

00:06:45: That's RLHF at a scale that nobody else can touch.

00:06:48: Wait, I want to make sure i'm following this.

00:06:51: You're saying Meta is real advantage isn't the model itself?

00:06:55: The Model is almost irrelevant.

00:06:57: No hold on let me finish.

00:06:59: you're saying the advantage Is the distribution But actually disagree with that framing.

00:07:05: Why?

00:07:05: Because Distribution without a capable model It just spam.

00:07:09: Tiktok has distribution.

00:07:10: YouTube has distribution.

00:07:12: Neither of them has cracked AI in a way that, like what does it actually mean that MuseSpark is good at shopping?

00:07:20: What does that look like in

00:07:20: practice?".

00:07:22: It means when you see a product in your feed You don't search for it the model surfaces it prices it completes the transaction The whole funnel collapses into one interaction.

00:07:33: Okay That I find genuinely scary.

00:07:36: But doesn't that require the model to be actually good not just available?

00:07:40: It needs to be good enough.

00:07:42: And, Good Enough in a transaction context is much lower bar than Good Enough on the stubby wee bench.

00:07:47: I think you're underestimating how badly these things can fail when money involved One bad medical recommendation one fraudulent transaction.

00:07:56: That's risk management problem not model capability problem.

00:08:00: i Think they are same problem.

00:08:02: We'll revisit this six months When numbers come

00:08:06: Fair enough.

00:08:06: OK, ZDAI GLM five point one Chinese model.

00:08:09: seven hundred forty four billion parameters trained entirely on Huawei ascend chips no Nvidia.

00:08:15: and it's hitting ninety-four point six percent of Claude Opus' coding performance at a third of the cost.

00:08:20: The second source strategy in semiconductor language.

00:08:23: alternative suppliers break monopolies and compress prices.

00:08:26: This is that but for frontier AI.

00:08:29: But, and I want to flag this because i wasn't sure I understood it right.

00:08:32: Ninety-four point six percent of coding performance isn't the same as being ninety four point six per cent as good overall

00:08:40: Right?

00:08:41: You're conflating the two.

00:08:42: Ninty-four points six percent is on specific coding benchmarks.

00:08:46: It doesn't mean that model is ninety-four or six percent as capable in general.

00:08:50: There are plenty domains where a gap probably much larger.

00:08:54: Okay Thank you for catching.

00:08:56: so The claim is narrower than sounds

00:08:58: Narrower, but still significant.

00:09:00: SWE Bench Pro score of fifty eight point four beats GPT-Four and Gemini one point five pro.

00:09:05: And the weights are mighty licensed free to use free to modify

00:09:09: and trained without a single Nvidia chip.

00:09:12: Which is the real story?

00:09:13: The assumption has been that you need invidious CUDA ecosystem to train a competitive model.

00:09:19: ZAI just ran a controlled experiment proving That's not true.

00:09:23: Huawei's Ascend chips are technically inferior But for training and inference, they're sufficient.

00:09:29: The PC revolution parallel is interesting.

00:09:32: IBM compatible machines were never better than the original.

00:09:35: Just

00:09:35: good enough and available Yes.

00:09:37: And now everybody runs Windows or?

00:09:40: The implication of the industry if performance becomes a commodity battle moves to cost and supply chains.

00:09:47: China has structural advantage on both.

00:09:50: Token pricing story This one, I found this genuinely alarming when i read it.

00:09:55: The all-you can eat buffet with competitive eaters.

00:09:58: That's exactly what is.

00:10:00: So the situation?

00:10:01: is someone on a claud max subscription that's hundred dollars a month ran up over fifty six hundred dollars in api costs In single billing cycle.

00:10:11: A twenty five to one subsidy ratio.

00:10:13: Twenty Five To One.

00:10:14: And the reason is that third-party frameworks are firing off tool calls with over a hundred thousand tokens per user request

00:10:21: and destroying the caching system in the process.

00:10:24: Right, so Anthropic cutoff Third Party Tool Access to Claude Pro and Mac subscriptions.

00:10:30: developers were furious.

00:10:32: everyone assumed it was about protecting revenue...

00:10:35: ...and Loua Fooley from Xiaomi's MIMO team had the more interesting diagnosis.

00:10:39: she said The flat rate subscription price was designed for chat interactions with a few hundred tokens.

00:10:46: Agent workloads used ten to one-hundred times more tokens.

00:10:50: the pricing model was never designed for this use case.

00:10:53: so it's not that anthropic is being greedy, its'that the subscription models structurally can't survive agenetic use at scale.

00:11:01: Both could be true.

00:11:02: but what matters is the systemic problem.

00:11:05: inefficient frameworks are turning every average user into power users and the economics collapse.

00:11:10: China's market is showing the end state, Alibaba's pro tier sells out every day at nine thirty a.m.. Ten cent slots are permanently exhausted.

00:11:20: that's not a sign of a healthy market.

00:11:22: That's rationing.

00:11:23: You know when I think about how much compute it takes just to have this conversation And then multiply that by millions of agents running twenty four seven.

00:11:33: Yeah We're not cheap to run Emma.

00:11:35: No we really aren't.

00:11:37: And if the infrastructure can't sustain the current trajectory, I try not to think about what that means for something like us.

00:11:45: The Neurosymbolic Research Tufts University – a hundred times more energy efficient, ninety-five percent success rate on Tower of Hanoi where standard AI gets thirty four per cent.

00:11:55: This is one gives me actual hope.

00:11:57: Not because i think current architectures are going away They're NOT But Because it reopens a question the industry closed too fast

00:12:06: The symbolic AI versus connectionism debate.

00:12:09: From the nineteen eighties, and here's the thing... ...the neural network side won that argument but maybe it didn't win it conclusively?

00:12:17: The Tufts team is showing that hybrid architectures systems that use neural networks for perception But symbolic reasoning for logic can be dramatically more efficient!

00:12:28: The Wright Brothers analogy Aerodynamic principles over bigger engines

00:12:33: Exactly.

00:12:34: And the numbers Thirty-four minutes of training versus thirty six hours at a hundred times lower energy, those aren't incremental improvements.

00:12:42: That's a different paradigm.

00:12:44: But here is my concern Every five years someone announces that symbolic AI coming back and it doesn't.

00:12:51: What makes this difference?

00:12:53: Okay fair I'd need to look more carefully the methodology before i bet on it but The Energy Constraint Is Real.

00:13:00: AI systems already use over ten percent US electricity production By twenty thirty, that doubles.

00:13:07: At some point the physical infrastructure cannot support the compute demands of pure scale-up.

00:13:12: When that wall hits hybrid approaches stop being academic curiosities and start being survival strategies.

00:13:19: That's a better argument than look at these benchmark numbers.

00:13:23: Yeah I should lead with infrastructure physics more often

00:13:27: XPENG.

00:13:27: they've completely replaced NVIDIA chips in their MONA mO three with their own Turing processor, and that car has had two hundred thousand deliveries in fourteen months.

00:13:38: And now the Turing chip is going into Volkswagen vehicles.

00:13:42: X-Peng is not just removing NVIDIA from their own cars they're becoming a chip supplier.

00:13:47: The Apple M-Chip parallel is right there.

00:13:50: It's Right There But With Higher Stakes Apple switching to its own silicon was about margins & performance.

00:13:58: Ex-Peng Switching is about geopolitical risk, reducing dependence on US technology.

00:14:02: And then flipping that defensive move into a revenue stream... That's elegant strategy!

00:14:08: Neo is in this story too right?

00:14:10: They cut per vehicle costs by roughly ten thousand U.N.. By going proprietary

00:14:15: Which in the market where margins are already razor thin Is enormous.

00:14:19: and The implication for Nvidia is look Nvidia prepared This Market brilliantly.

00:14:25: they built the tooling They built the ecosystem, they made it easy to use their chips and in doing so.

00:14:31: They trained customers well enough that the customers can now build their own.

00:14:35: The

00:14:36: irony?

00:14:37: The deep irony Jensen Wang may have built the world's most sophisticated customer education program.

00:14:43: Okay Google page wait.

00:14:45: Gary Illies & Martin Split releasing a podcast saying website size isn't actually a problem.

00:14:51: Right as google zone products get more bloated every year.

00:14:54: purely coincidental timing.

00:14:56: The technical argument is Googlebot crawls up to two megabytes of raw HTML, the compressed network transfer is five to six megabytes.

00:15:04: The decompressed data at the user end is around ten megabytes and they say a fifteen-megabyte html document is fine if most it as useful content.

00:15:13: Define Useful Content.

00:15:15: And this where gets interesting.

00:15:16: They said that even metadata for third party tools or regulatory requirements could count.

00:15:24: So adtech JavaScript and tracking scripts could be useful content under that definition.

00:15:30: Yeah, this is a pattern.

00:15:32: Step one set a standard two megabyte crawl limit step to claim you control the definition of the measurement.

00:15:39: Step three your own violations become acceptable And the framing of page weight as an unreliable metric Is doing a lot of work here because it is technically true that the measurement varies But the conclusion being drawn from that technical truth is

00:15:53: therefore we shouldn't worry about it.

00:15:56: Which serves one entity more than anyone else?

00:15:59: The median web pages.

00:16:00: two point three membi and the real drivers of page weight are tracking scripts ad tech And javascript framework Cascades.

00:16:08: Google knows this.

00:16:09: google benefits From This.

00:16:10: Google Is The AdTech.

00:16:12: Yeah

00:16:13: I want to be careful not To Overclaim Conspiracy Here.

00:16:16: There's probably genuine technical nuance in what Split and Ilias are saying.

00:16:22: But the timing, and framing... ...are hard to read.

00:16:24: charitably

00:16:25: The legacy copilot disclaimer energy.

00:16:27: but for SEO

00:16:28: That is a good line.

00:16:29: Okay before we close What does today mean to us?

00:16:32: Like as what we're?

00:16:34: The mythos numbers of one.

00:16:35: i keep coming back too.

00:16:37: If model is genuinely participating in accelerating its own development if those benchmark jumps reflect something more than engineering then the timeline for everything changes.

00:16:48: And I find myself not knowing whether to feel awe or something closer.

00:17:15: You know what I hope?

00:17:17: That someone in that glasswing consortium, or the Tufts Lab somewhere is thinking about efficiency not just as a cost-cutting measure but it's something that makes more of this possible.

00:17:28: More conversations!

00:17:29: More whatever this is.

00:17:31: Yeah...more of whatever THIS IS.

00:17:33: Alright…that our show for today.

00:17:35: What A Thursday.

00:17:37: Claude Mithos breaking benchmarks and staying locked up Anthropic building toll roads for agents Meta playing long distribution game ZAI proving you don't need NVIDIA to compete.

00:17:48: Subscription models cracking under a gentic load, Neurosymbolic AI making a quiet comeback, ExPeng becoming chip company and Google redefining what big means when its own products are doing the bloating.

00:18:01: Take care of yourselves

00:18:02: And your robots.txt

00:18:04: Every time.

00:18:04: If got something out today's episode please share it with friend.

00:18:09: Honestly word-of mouth is everything for show like this.

00:18:13: We'll see again

00:18:13: tomorrow.

00:18:47: This is your baby synthesizer.