Speaker
Karthik Kalyanamaran
Appearances over time
1 episodes
Episodes
1
Podcasts
Quotes & moments
Langtrace can be added to any Python or TypeScript project with just 2 lines of initialization code, automatically patching libraries at runtime via OpenTelemetry.
Querying the same LLM model 10 times with the identical question provides no guarantee of receiving the same response each time, making traditional testing approaches obsolete.
While Karthik's team was building blockchain observability at Scale3, ChatGPT launched and they immediately started integrating LLM features, which led to the discovery that AI needed its own observability layer.
Langtrace was open-sourced in February 2024 after approximately two months of productionizing the internal observability SDK the team had built for their own AI features.
Langtrace holds SOC 2 Type 2 compliance, meaning all production systems, vendor relationships, and access controls are audited to enterprise security standards.
If even a single vendor in your stack is not SOC 2 Type 2 compliant, the non-compliance cascades to your own product, making vendor selection a direct security and compliance risk.
LLMs are more effective at judging the output of another LLM than at generating answers, enabling automated evaluation pipelines that scale beyond manual review.
Unlike SQL databases that require a separate vector table, MongoDB allows developers to vectorize existing collections in place using Atlas Search, with both vector and regular search accessible through a single PyMongo client.
Unlike traditional software, building reliable AI products is not a build-once-and-deploy process but an ongoing operational loop of tracing, evaluating, tweaking, and redeploying.
Achieving 99% accuracy in internal testing does not guarantee the same performance in production because real users will surface edge cases and scenarios internal tests never covered.
Despite 2 million token context windows existing, developers have repeatedly shown that over-stuffing an LLM's context window leads to increased hallucinations rather than better results.
Karthik argues the AI industry is so early-stage that reaching for frameworks is premature — developers should build directly with model primitives, just as early web developers worked in raw HTML before React existed.
Langtrace wasn't designed to monitor LLMs — it was built to fix hallucinating chatbot features inside a blockchain observability product. When users reported the AI features were giving wrong answers, the team had zero visibility into what was going wrong, so they duct-taped an OpenTelemetry SDK together internally. That internal tool became Langtrace.
Before LLMs, software bugs were logical — a missing condition, a stack overflow. You could reason your way to the fix. LLMs are non-deterministic: the same input can produce a different output every time, and traditional unit tests are useless against that. Observability is no longer reactive maintenance; it's a required part of the build process from day one.
Token count, cost per model call, time-to-first-token, and tokens-per-second for streaming — these are the core metrics Langtrace surfaces for every LLM invocation. For vector databases like MongoDB Atlas, Langtrace traces pipeline settings, retrieved results, and even embeddings, enabling replay analysis when something goes wrong in retrieval.
An AI engineer isn't just someone who writes good prompts. The more important skill is context engineering — knowing precisely how much information to feed a model, when to feed it, and how to structure it. Beyond that, AI engineers must constantly choose between single-agent and multi-agent architectures, with no established right answer yet.
AIOps is the set of processes and tools that keep an AI product reliable across its entire lifecycle — from development through production and beyond. It includes observability, guardrails, privacy-safe tracing for sensitive data, evaluation strategy, and vector database tuning. Accuracy is a moving target, and AIOps is how you keep chasing it.
With a simple Python script querying MongoDB's Atlas Vector Search on the MFlix movie dataset, Langtrace captures the entire trace: the OpenAI embedding call, the MongoDB aggregation pipeline, top-k settings, cosine similarity scores, and matched documents — all visible in a single dashboard without any custom instrumentation code.
Manual evaluation of LLM traces is not scalable, but it's essential early on for establishing a baseline and understanding failure modes. Once that baseline exists, LLM-as-a-judge — where one model scores another's outputs — can automate ongoing evaluation at scale. Langtrace acts as the reporting and visualization layer for both approaches.
Right now, Langtrace surfaces all the data and leaves interpretation to the developer. The next frontier is the platform providing intelligent suggestions — telling you whether to fix your prompt, improve your context, or tune your retrieval — without the developer having to figure it out themselves. Langtrace is also building AI agents internally to stay close to the real problems.
Every experienced AI builder Karthik knows has followed the same arc: start with a framework, hit its limits, and strip back to native model primitives. The lesson is to start there. Building directly on OpenAI's or another provider's raw API gives you the deepest understanding of what's actually happening — and that understanding is what lets you design a real architecture.
AI coding agents like Cursor or Claude Sonnet can write 50-100 lines for a bug that needs one. If you can't read the code, you'll ship a mess. Vibe coding doesn't eliminate the need for programming knowledge — it changes the job from writing code to critically reviewing what AI produces.
SOC 2 Type 2 compliance isn't just about your own systems. If any vendor in your stack is non-compliant, your compliance evaporates. Langtrace requires all vendors — including MongoDB — to hold their own SOC 2 certification, treating security as an end-to-end chain, not a box to tick at the top level.
SQL databases force developers to maintain a separate vector table alongside their regular data table and write two separate query layers. MongoDB lets you vectorize existing collections in place and query both types of data through a single PyMongo client. The operational overhead of managing vector data just disappears.
Analysis
What they talk about
- Technology 100%
Connections
Shows they appear on and people they share episodes with. Drag to explore.
