Snapshot · God Mode Podcast
OpenAI audited SWE-Bench Pro and found 30% of its tasks are broken, leading them to retract their recommendation that the research community use it as a leading coding benchmark.
Where this was said
At 35:00 · chapter starts 34:56
The timing is suspicious and the hosts say so: OpenAI retracted SWE-Bench Pro right before shipping GPT-5.6 Soul, the model that would otherwise have been evaluated on it. Their audit found 30% of benchmark tasks are simply broken. [1] — Rik "30% of SWE-Bench Pro tasks broken: OpenAI audited SWE-Bench Pro and found 30% of its tasks are broken, leading them to retract their recomm…" 35:00 Rik notes they've been skeptical of benchmarks for a while — that's partly why they use BridgeMind, which maintains its own independent evaluation. Ben raises a deeper problem: if the AI models doing the auditing are the same models being benchmarked, the process is fundamentally compromised. Who watches the watchmen? The answer, increasingly, is other AI — and that's a problem nobody has cleanly solved yet.
Pieter Levels shipped a native iOS app declaring he's 'no longer iOS retarded.' Luca suspects it's a direct response to Nomad Table, which hit $1M ARR and 70K users with an iOS-only app that actually connects nomads in person — something Nomads.com never managed.
Ad-based monetization works well for game apps where users spend extended time in-session, as seen with Grid and Wordle.
Tool-focused apps like PuffCount are poor candidates for ad monetization because users don't stay in-session long enough.
A hard paywall is a screen that blocks all app features unless the user pays or starts a free trial — it cannot be dismissed.
Mobile apps are primarily monetized through either ads (best for games) or in-app purchases/subscriptions (best for tools).
According to the episode, YouTube outperforms every other social platform for building trust and driving SaaS conversions.
Vasco stated that the majority of his app's user base came directly from his YouTube channel.
SEO Bot features a 'Boost My Domain Rating' button that routes users directly to Listing Bot, an example of in-product cross-selling.
The founder's entire product portfolio is AI-related, making it easier to package products attractively for directories.
The founder attached their SaaS demo to the trending debate about whether AI coding is actually good enough to build a full SaaS product.
We use essential and analytics cookies to run Vuci. To understand how the site is used: Privacy Policy.
Install Vuci on your phone
Add it to your home screen for a faster, app-like experience.
Install Vuci on your phone
Tap the Share button, then “Add to Home Screen”.
A new version is available
Reload to get the latest Vuci.