Podbit · The MongoDB Podcast
Manual evaluation of LLM traces is not scalable, but it's essential early on for establishing a baseline and understanding failure modes. Once that baseline exists, LLM-as-a-judge — where one model scores another's outputs — can automate ongoing evaluation at scale. Langtrace acts as the reporting and visualization layer for both approaches.
Where this was said
At 37:00 · chapter starts 28:17
The demo segment is the most concrete part of the episode. Karthik creates a fresh Langtrace project in real time, generates an API key, and shows that the integration requires a single initialization call in the Python script. The demo script queries MongoDB Atlas Vector Search on the pre-vectorized MFlix dataset, searching for movies related to 'time travel' across 500 candidate embeddings and returning the top 10 results by cosine similarity score. [1] — Karthik Kalyanamaran "With a simple Python script querying MongoDB's Atlas Vector Search on the MFlix movie dataset, Langtrace captures the entire trace: the Ope…" 28:17 When the script runs, Langtrace immediately captures two spans: one for the OpenAI embedding generation call (showing model name, input text, token count) and one for the MongoDB aggregation (showing the vector index name, the plot-embedding path, top-k settings, and matched document metadata). Karthik then demonstrates manual evaluation — creating a custom 'retrieval' metric scored 0-2 and scoring the MongoDB trace — and explains how the human evaluation dashboard plots confidence (proportion of traces evaluated) and score trends over time. He grounds the demo in a practical scenario: in a customer support chatbot, this trace would reveal whether the pricing documentation was actually being matched as the top result for pricing questions. If it isn't, the trace tells you exactly where to intervene.
With a simple Python script querying MongoDB's Atlas Vector Search on the MFlix movie dataset, Langtrace captures the entire trace: the OpenAI embedding call, the MongoDB aggregation pipeline, top-k settings, cosine similarity scores, and matched documents — all visible in a single dashboard without any custom instrumentation code.
Sam had no coding knowledge, so he used ChatGPT voice mode to generate his entire codebase and copy-pasted it into Notepad. A friend later introduced him to Cursor, and he never looked back.
Copy days of Discord chat history, paste it into ChatGPT, and ask it to list recurring pain points. The ones that come up most often are your best product bets.
Sam's top advice: when prompting Cursor, tell it to architect code for 100,000 users from day one. The AI changes its approach, building scalable frameworks instead of brittle one-user code.
With AI coding tools like Cursor, Bhanu replicates an existing free tool for a new keyword in under 5 minutes. What used to be a multi-day build is now a lunch-break task.
Ahrefs, SiteGPT, Cal.com, PostHog, Datafast, Sibyl AI, Bento, Feather, Featurepace, Mintlify, Cloud Code, ChartMogul — Bhanu runs his entire business solo with these 12 tools.
PropGPT runs on React Native with TypeScript and Python for ML, Neon for the database, RevenueCat and Superwall for monetization. LLM costs are $20/month, data APIs $100/month, and after $10K in monthly marketing spend, margins sit at roughly 50%.
Game apps keep users engaged long enough for ads to pay off. Tool apps don't — so if you're building a utility, ads are almost always the wrong call and subscriptions are your only real lever.
Inside SEO Bot, a single button labelled 'Boost My Domain Rating' routes users directly to Listing Bot. That one interaction converts a user of one tool into a user of two — without any marketing cost.
Directory listings are a powerful but underrated growth channel — but only if your product is genuinely interesting enough to earn the click. AI products have a natural advantage here because they're easy to package in a compelling, clickable way.
We use essential and analytics cookies to run Vuci. To understand how the site is used: Privacy Policy.
Install Vuci on your phone
Add it to your home screen for a faster, app-like experience.
Install Vuci on your phone
Tap the Share button, then “Add to Home Screen”.
A new version is available
Reload to get the latest Vuci.