Quote · The MongoDB Podcast
Don't Build Your Own AI (Unless You Have To)
Where this was said
Evaluating AI Models: Benchmarks vs. Task Fit
At 48:54 · chapter starts 47:20
Shane asks whether Zendesk pays attention to public model evaluation benchmarks when choosing which models to build products on. Akshaya's answer is nuanced: benchmarks serve a purpose — they signal general capability — but they are not a reliable proxy for enterprise task fit. The fact that a model can ace LSAT questions or solve complex mathematical problems says little about whether it can resolve a customer support query accurately and empathetically. His analogy is sharp: the top-scoring student in the class is not necessarily the brightest or most capable person in the room. At Zendesk, the practical response has been to evaluate both custom and commercial options in parallel and release both when appropriate — a custom triage model and a GPT-4o-powered feature launched on the same day. The lesson for enterprises is clear: use benchmark scores as a starting filter, not a final verdict. Test models against your own representative data and tasks, and maintain optionality by not over-committing to a single vendor or architecture. MongoDB draws a parallel: for 18 months the team has been working with code assistant providers to build MongoDB-specific evaluation sets, precisely because general benchmark performance doesn't predict performance on MongoDB-specific coding tasks.
Don't launch an AI initiative because AI is the thing to do. Start with a problem statement, validate that AI is part of the answer, then work backwards to data, model selection, talent, and cost. Every AI project that skips this step is setting itself up to fail.
