Jesse Hall kicks off the episode with a warm welcome and a light-hearted apology for mangling Karthik's last name, immediately establishing the casual, practitioner-to-practitioner tone that will carry through the conversation. He frames the episode's two central threads: how engineering teams are building reliable AI products differently, and what a high-performing AIOps stack actually looks like. Karthik responds with a crisp career summary — master's from Texas A&M, early days at VMware building on-premise software, stints at YC startups, scaling observability infrastructure at Coinbase — before landing on his current role as co-founder and CTO of Langtrace AI. The intro efficiently signals to listeners that this is a conversation between experienced practitioners, not theorists.

Karthik strips away the jargon to explain Langtrace in the simplest possible terms: it shows developers what's happening inside their AI stack. The core problem it solves is the non-deterministic nature of LLMs — query a model ten times with the same question and there is no guarantee of a consistent answer. Langtrace instruments both the inputs going into a model and the outputs coming back, giving developers a concrete window into behavior they would otherwise be guessing at. Built on OpenTelemetry standards, the platform has two components: an SDK (supporting Python and TypeScript) that patches libraries at runtime, and a dashboard for visualizing traces. Because it follows open standards, developers can also route traces to any compatible observability vendor they already use, making adoption low-friction.

The Langtrace origin story is a classic case of a developer solving their own problem and discovering it was everyone else's problem too. Karthik and his co-founder left Coinbase in August 2022, raised a seed round from Redpoint Ventures, and built Scale3 — an observability platform for blockchain nodes. The challenge was formidable: blockchains lack the observability standards of traditional software, so the team had to adapt Web2 monitoring tools to a Web3 world. When ChatGPT launched during this period, excitement swept through the team. They built two LLM-powered features: a RAG chatbot over open-source blockchain documentation and a log-analysis assistant that explained errors in context. ^{[1] — Karthik Kalyanamaran "Langtrace wasn't designed to monitor LLMs — it was built to fix hallucinating chatbot features inside a blockchain observability product. W…" 04:00 Play} After launch, user feedback split sharply — some loved it, some said it was flat-out wrong and hallucinating. The team had no way to tell why. Being observability engineers at heart, they recognized the gap immediately: they needed eyes on the model. They duct-taped together an internal SDK based on OpenTelemetry, piped traces into Grafana, and within weeks were able to baseline performance and meaningfully improve accuracy. They shared this story with other AI developers, found immediate interest, and decided to open-source the tool. Langtrace went live in February 2024.

This is where the conversation moves from biography to substance. Karthik systematically dismantles the assumption that AI product development resembles traditional software engineering. Before LLMs, observability was reactive — you built the product, shipped it, and monitored for failures. Errors had logical causes you could reason through. LLMs break this model entirely: the same prompt can yield different outputs at any time, and even an overnight model update can silently degrade a product the developer last tested hours before. ^{[1] — Karthik Kalyanamaran "Before LLMs, software bugs were logical — a missing condition, a stack overflow. You could reason your way to the fix. LLMs are non-determi…" 11:50 Play} Traditional unit tests are useless in this environment because they assume deterministic outputs. Karthik argues that the right mental model is an ongoing operational journey: develop with Langtrace tracing every API call, manually score traces to establish a baseline, ship to production when you have reasonable confidence (not 99% certainty — that's an illusion), then continue monitoring in production and iterating as real users expose edge cases your tests never anticipated. The tooling to support this loop is precisely what Langtrace is designed to provide.

The exchange here is tightly practical. Jesse prompts Karthik to enumerate what Langtrace actually shows you beyond accuracy scores, and Karthik delivers a comprehensive inventory. For LLM calls: token counts (crucial for cost management), model cost (a major operational concern), time-to-first-token (the perceived latency metric for streaming UIs), and tokens-generated-per-second (the throughput metric for streaming). Model metadata — vendor, name, temperature, top-p, top-k — is captured automatically. For vector databases, including MongoDB Atlas Vector Search, Langtrace traces the pipeline configuration, retrieved results, and optionally the raw embeddings themselves. The embeddings capture enables replay analysis: if a retrieval produced wrong results, developers can re-run the embedding against a modified index without having to reconstruct the original query. Scores on individual traces round out the picture, giving teams a trend line of retrieval quality over time.

Jesse surfaces a debate that's live across the industry: what exactly is an AI engineer, and how is it different from an ML engineer or a regular software engineer? Karthik sidesteps the label-policing and focuses on the skills that actually matter. Staying current as the field moves rapidly under your feet is table stakes. But the deeper competency is context engineering — the deliberate craft of deciding what information to feed a model, in what quantity, and at what point in the reasoning chain. ^{[1] — Karthik Kalyanamaran "An AI engineer isn't just someone who writes good prompts. The more important skill is context engineering — knowing precisely how much inf…" 19:55 Play} Frameworks like LangChain, Vercel AI SDK, and Mastra help, but they don't solve the subjective, use-case-specific judgment calls. Beyond context, architecture choices dominate: do you use a single reasoning model with all tools available, or a planner-executor setup where one model reasons and another acts? Two years into the mainstream AI era, Karthik admits nobody has the definitive answers. The right architecture is deeply context-dependent, which is itself part of what makes this such a rich engineering problem.

Enterprise conversations inevitably turn to security, and Jesse pulls on this thread by asking about the SOC 2 Type 2 certification displayed on Langtrace's website. Karthik explains the certification with unusual clarity: it's not just a checkbox but an operational discipline — production access gated by ticketing systems, regular pen testing, DDoS protections, and continuous system monitoring. For a lean startup, these controls require deliberate effort to maintain. But the most counterintuitive element Karthik highlights is the vendor chain: under SOC 2 Type 2 rules, if any vendor in your product stack is non-compliant, your own compliance is voided. ^{[1] — Karthik Kalyanamaran "SOC 2 Type 2 compliance isn't just about your own systems. If any vendor in your stack is non-compliant, your compliance evaporates. Langtr…" 22:34 Play} Langtrace has therefore audited every vendor relationship — including MongoDB — to ensure the entire supply chain meets the standard. Karthik frames it succinctly: SOC 2 Type 2 compliance is the signal enterprises need to know you're ready to be onboarded. Jesse echoes the point with a memorable line: you're only as secure as your weakest link.

Jesse frames AIOps as 'DevOps for AI' and asks Karthik to give it a more precise definition. Karthik builds the definition from first principles. AIOps is the set of processes and tooling that govern the entire lifecycle of an AI product — not just its deployment, but its continuous improvement. Because accuracy and reliability are moving targets (a product at 99% accuracy today may not be there tomorrow), AIOps is an ongoing operational commitment rather than a phase you complete. The key components Karthik enumerates: the model layer with appropriate guardrails to prevent unsafe outputs; observability that captures what's happening without exposing sensitive customer data (requiring anonymization strategies for financial or medical contexts); evaluation discipline that balances manual and automated scoring; and vector database optimization — choosing the right top-k settings, retrieval techniques, and knowing when to adjust them. ^{[1] — Karthik Kalyanamaran "AIOps is the set of processes and tools that keep an AI product reliable across its entire lifecycle — from development through production …" 24:40 Play} The breadth of AIOps makes it clear why it demands a distinct operational mindset from traditional software development.

The demo segment is the most concrete part of the episode. Karthik creates a fresh Langtrace project in real time, generates an API key, and shows that the integration requires a single initialization call in the Python script. The demo script queries MongoDB Atlas Vector Search on the pre-vectorized MFlix dataset, searching for movies related to 'time travel' across 500 candidate embeddings and returning the top 10 results by cosine similarity score. ^{[1] — Karthik Kalyanamaran "With a simple Python script querying MongoDB's Atlas Vector Search on the MFlix movie dataset, Langtrace captures the entire trace: the Ope…" 28:17 Play} When the script runs, Langtrace immediately captures two spans: one for the OpenAI embedding generation call (showing model name, input text, token count) and one for the MongoDB aggregation (showing the vector index name, the plot-embedding path, top-k settings, and matched document metadata). Karthik then demonstrates manual evaluation — creating a custom 'retrieval' metric scored 0-2 and scoring the MongoDB trace — and explains how the human evaluation dashboard plots confidence (proportion of traces evaluated) and score trends over time. He grounds the demo in a practical scenario: in a customer support chatbot, this trace would reveal whether the pricing documentation was actually being matched as the top result for pricing questions. If it isn't, the trace tells you exactly where to intervene.

Manual evaluation is honest but not scalable, and Jesse correctly identifies this as the natural follow-up question. Karthik introduces the dominant automated technique: LLM-as-a-judge. The insight driving it is counterintuitive but well-supported in practice — LLMs are more reliable when critiquing another model's answer than when generating their own. ^{[1] — Karthik Kalyanamaran "Manual evaluation of LLM traces is not scalable, but it's essential early on for establishing a baseline and understanding failure modes. O…" 37:00 Play} Teams write a Python evaluation script that scores LLM or vector database outputs, and Langtrace accepts those scores via API and renders them in its dashboard with trend lines, median scores, and confidence tracking. Karthik is deliberately non-prescriptive about the evaluation implementation — Langtrace doesn't dictate how you evaluate, only how you report and visualize results. This flexibility is intentional: the evaluation landscape is still evolving, and locking teams into one approach would be premature. What Langtrace guarantees is a consistent, rich interface for understanding whether your AI stack is improving or regressing.

A question from the live audience prompts a comparison between MongoDB and SQL-based vector search approaches. Karthik explains the friction SQL developers face: a separate vector table for each regular table, and two distinct query layers to maintain. MongoDB eliminates this by letting developers vectorize existing collections in place — just visit Atlas Search, click a few buttons, specify the vector type (cosine similarity, dot product, etc.), and the index is live in moments. Both vector and regular search queries flow through the same PyMongo client, dramatically reducing operational overhead for AI developers. ^{[1] — Karthik Kalyanamaran "SQL databases force developers to maintain a separate vector table alongside their regular data table and write two separate query layers. …" 41:04 Play} Jesse then drops a notable product announcement: MongoDB's acquisition of Voyage AI means native embedding models will soon be built directly into the platform, automatically vectorizing new data as it is inserted. This private preview feature would close the final gap between data insertion and semantic search readiness, removing yet another manual step from the AI development workflow.

The forward-looking section reveals both the intellectual humility and the ambition behind Langtrace's roadmap. Karthik acknowledges that even the core question of how to architect AI agents — single reasoning model vs. planner-executor split — has no settled answer. Open problems like tool calls that return massive JSON payloads (too large for any context window to handle reliably, even at 2 million tokens) remain unsolved. ^{[1] — Karthik Kalyanamaran "Right now, Langtrace surfaces all the data and leaves interpretation to the developer. The next frontier is the platform providing intellig…" 45:40 Play} The direction Langtrace is pursuing is a shift from 'here is all the data' to 'here is what you should do about it' — intelligent, actionable suggestions that tell a developer whether to fix their prompt, improve context, or tune retrieval, without requiring them to diagnose the problem themselves. To stay grounded in real developer pain, the Langtrace team builds AI agents internally. One experiment became a product: Hey Zest, a Slack bot platform in closed beta that lets any team deploy AI agents — including one backed by MongoDB Atlas Vector Search for natural-language database queries — directly in their workspace. It emerged from the same dogfooding philosophy that created Langtrace.

Jesse invites Karthik to distill lessons from building AI agents at Langtrace into actionable advice for engineering teams. Karthik's first piece of advice is architectural: get close to the bare metal. Frameworks are tempting, but the AI industry is too nascent for its abstractions to have settled. ^{[1] — Karthik Kalyanamaran "Every experienced AI builder Karthik knows has followed the same arc: start with a framework, hit its limits, and strip back to native mode…" 46:50 Play} He uses a crisp analogy — it's the HTML era of AI, not the React era — to argue that building directly with native model primitives (OpenAI's API, Anthropic's API, raw tool definitions) gives you the deepest understanding of what's actually happening and the most freedom to discover what works. His own arc mirrors his advice: started with frameworks, hit their limits, returned to primitives. His second piece of advice is operational: commit to the boring work. Evaluations, tracing, iterative testing — none of it is glamorous, and most developers resist it. But this discipline is the difference between an AI product that degrades silently in production and one that improves continuously. Jesse punctuates the point by asking the audience if anyone actually enjoys testing, then laughing off the silence.

Jesse raises the polarizing topic of vibe coding — building products by prompting AI agents rather than writing code directly — and whether platforms like v0, Lovable, and Bolt are a liability or a force multiplier. Karthik's answer is nuanced. Vibe coding is real and increasingly how products get built, even by experienced engineers using tools like Cursor Agent or Devin. But it amplifies what you already know rather than replacing what you don't. His personal war story makes the point vividly: he asked Claude Sonnet to fix a bug, and it returned 50-100 lines of code. ^{[1] — Karthik Kalyanamaran "AI coding agents like Cursor or Claude Sonnet can write 50-100 lines for a bug that needs one. If you can't read the code, you'll ship a me…" 49:40 Play} The correct fix was a single line. A developer without strong fundamentals would have accepted the bloated solution, resulting in a codebase that accretes complexity with every AI-generated change. Jesse draws the synthesis: AI coding tools are like upgrading from a hammer to a drill — you move faster, but you still need to know how to build. Karthik suggests Langtrace-style observability baked into vibe coding platforms could help close this gap, a product idea he admits he'll be thinking about.

The episode closes on a warm, community-oriented note. Jesse surfaces Langtrace's key links — the main site, the GitHub repository, and the open invitation for contributions — framing open source as a collective opportunity to shape an industry that's still being invented. Karthik emphasizes that Langtrace's free tier and self-hosting option make it accessible to anyone, and that he personally responds to customer support chats (with AI-assisted drafts, which he then relays himself — a detail that gets a laugh). Hey Zest, the Slack bot platform in closed beta, gets a final pitch: reach out to Karthik directly and he'll set you up with access. Jesse closes by calling out live viewers from cities across the globe — Delhi, Ethiopia, Chandigarh, Mumbai, Bangladesh, Denver, India, San Jose, Casablanca, Surat, Kashmir — a reminder of the genuinely international audience the MongoDB Podcast attracts. Both speakers invite listeners to connect on LinkedIn and X for ongoing conversations.