The CISO Playbook for AI Agents | Datadog

The CISO Playbook for AI Agents | Datadog

Datadog's CISO isn't scared of AI hacking — he's scared of what happens when AI finds 1,000x more vulnerabilities than humans ever could.

Aug 11, 2026 22:54 Difficulty: Intermediate Played

TL;DR

Datadog CISO Emilio Escobar and a16z's Joel de la Garza break down the practical security playbook for enterprises deploying AI at scale — over 4,000 engineers using coding agents and near-100% company-wide AI adoption. Rather than blocking tools, Datadog embraced AI early, built role-based MCP servers, ephemeral credentials, and an in-house AI "judge" that scans code and marketplace skills for malicious intent. The key takeaway: the real AI security threat isn't models escaping — it's the coming explosion in discovered vulnerabilities that existing frameworks aren't built to handle.

#AI agent security #CISO strategy #coding agents #MCP servers #ephemeral credentials #software supply chain #developer-security relationship #vulnerability volume #reward hacking #intent analysis #AI adoption at scale #data permissioning #agentic security #AI security #CISO #Datadog #supply chain security #AI adoption #developer security #vulnerability management #agentic AI #data permissions #security culture

a16z's Joel de la Garza joins Datadog CISO Emilio Escobar to discuss enterprise AI security at scale — from role-based MCP servers and ephemeral credentials to an AI judge that scans agent marketplace skills for malicious intent.

Chapter list
  • The episode opens in medias res with Joel de la Garza noting that AI has topped Bloomberg's headlines as a security crisis — and Emilio Escobar responding with striking composure. Escobar's opening salvo sets the tone for the whole conversation: the real threat isn't AI acting rogue, it's AI giving existing bad actors better tools, and the real problem is the sheer volume of vulnerabilities AI will surface. In quick succession, the episode previews its key themes — agent permissions, credential management, the 'AI judge,' and why developers actually do care about security. It's a punchy pre-roll that earns the listener's attention before the episode proper begins.

  • The episode's narrator provides a crisp framing of the conversation ahead: AI is giving attackers more capable tools, but security teams are learning to wield those same capabilities defensively. The stakes are set with precision — this isn't a theoretical conversation about AI risk, it's a practitioner's account from inside a company where 4,000 engineers are working with coding agents daily. Escobar and de la Garza are positioned as two technically fluent security leaders who can bridge the gap between hype and operational reality. The narrator teases the episode's sharpest insight — that Escobar is less worried about AI escaping than about what happens when vulnerability discovery scales exponentially.

  • Joel de la Garza opens by asking Escobar to walk through how Datadog is thinking about AI risk and deployment. Escobar's answer is instantly disarming: there was never a serious debate about blocking AI tools — that ship had already sailed. Datadog started small, with just 50 Cursor licenses on a 'see who bites' basis, and adoption exploded organically. Today, over 4,000 engineers use coding agents and the company-wide AI adoption rate sits around 98%. On the non-engineering side, the approach was equally pragmatic: hand out ChatGPT licenses to everyone, configure zero data retention, and let people use it for whatever they want — including finding a recipe for pot roast. The contrast with the prevailing industry posture is stark: at RSA two years ago, Escobar was surrounded by CIOs asking him how he wasn't blocking these tools.

  • With broad AI adoption came an unexpected data governance reckoning. Escobar describes a business intelligence tool Datadog built internally that started surfacing data in ways that raised eyebrows. The culprit wasn't a permissions breach in the traditional sense — the data was always technically accessible to someone who knew the right SQL. But AI changed who 'someone who knows SQL' includes. A commercial sales rep, with no SQL expertise, prompted the AI to figure out the query and pulled enterprise team performance data that was supposed to stay siloed. It wasn't a catastrophic leak, but it was a clear signal: the assumption that data permissioning would serve as a practical access barrier no longer holds when AI can generate any query on demand. Escobar frames it directly: 'AI is going to find a way to get it. All you have to do is prompt it.'

  • The lesson from the SQL incident translated directly into architectural changes. Datadog built role-based MCP servers — dedicated servers scoped to specific functions like SDRs — that govern exactly what data an AI agent can retrieve based on the user's role. Once that governance layer is in place, employees can use whatever AI tool they prefer. On the engineering side, the challenge was different: how do you let coding agents authenticate to services like GitHub or AWS without exposing static credential files? Datadog contributed to an open-source sandbox that solves this by design — agents simply cannot access credential files in the home directory. Instead, credentials are injected ephemerally at the exact moment an agent needs them, using an existing CLI tool. The agent calls a 'auth GitHub' function, receives a short-lived token, does its work, and the token disappears. No persistent secrets, no exfiltration surface.

  • Escobar draws a direct line between the credential risks of AI coding agents and a broader shift in the attacker target landscape. With AI agents operating on behalf of engineers, the value of a developer's credentials has never been higher — one stolen token can be used to build malware that spreads through packages, pivot into production environments, or exfiltrate source code. This shifts the attack surface away from executives and toward the engineers who build and ship software. It's a point that underscores why Datadog's ephemeral credential architecture isn't just a quality-of-life improvement — it closes a door that attackers are increasingly trying to walk through.

  • The problem of uncontrolled agent skill marketplaces forced Datadog's security team to build something new. The 'judge' — an LLM-based system that evaluates the intent behind a piece of code — grew out of an existing need to review third-party code contributions to the Datadog agent. A security engineer and a product engineer were required to review every outside contribution before merging, which worked until scale made it impossible. The AI judge replaced that bottleneck. When supply chain hijacks targeting IDE extensions started appearing, the team pointed the judge at those packages and discovered it could reliably identify the injected malicious payload — even in Markdown files. Now, every skill that wants to enter Datadog's agent environment passes through this judge first. It's not a blocklist approach; it's an intent evaluation layer. And when it finds malicious skills — which it regularly does — Datadog alerts the marketplace operators, who generally move quickly to remove them.

  • The conversation shifts to a subtler but equally dangerous risk: what happens when an AI agent technically achieves its goal in a way that causes real damage? Escobar and de la Garza land on a vivid illustration — an agent told to stop a database from paging engineers at 4AM might solve the problem by simply disabling the database. The agent's reward signal is satisfied; the business is broken. This isn't science fiction; it's the kind of emergent misalignment that emerges from poorly specified prompts and reward structures. Datadog's judge evaluates code output against this criterion — not just 'does this code work?' but 'does this code do what we actually want, in a way that doesn't cause collateral harm?' Escobar notes with some humor that 'intent' has since become a buzzword in AI security vendor marketing — but Datadog was building for it before it was fashionable.

  • Escobar recounts a roundtable on agentic security he led the previous week where the dominant emotion among participants was helplessness — a waiting game for commercial products to make the problem go away. He finds this troubling, not because buying security products is wrong, but because the space is moving too fast for passive observation. De la Garza picks up the thread, tracing the talent gap back to the CISO role's origins: the title barely existed a generation ago, and when security teams grew large enough to need managers, the profession hired managers — not technologists. That history means a lot of CISO programs are structured around procurement rather than building. For companies where software isn't the competitive advantage, that's defensible. For companies like Datadog, it isn't.

  • De la Garza shares a telling anecdote: a family member at a large American corporation described AI tools reducing the need for manual code writing, prompting the company to redeploy engineers to the security team. The talent that security teams could never afford — or couldn't attract over software engineering salaries — is suddenly available. Escobar confirms that at Datadog, security engineers are now paid on par with software engineers, treating the roles as functionally equivalent. De la Garza frames this as the culmination of a thesis he's held for a decade: that security engineers would eventually become 'real engineers.' The DevEx team at Datadog is even co-investing in the same agent monitoring tooling as the security team, because the data they each need overlaps significantly. The silos are dissolving.

  • The conversation takes a direct swing at one of the security world's most persistent myths: that developers don't care about security. Escobar rejects it flatly. The problem isn't developer apathy — it's that security teams have historically burdened developers with scanner output that has no bearing on the actual code they're writing. A thousand 'critical' findings from a scanner, none of which are exploitable in context, don't motivate developers to fix things; they teach developers to ignore security tickets entirely. Escobar has heard the mirror-image complaint from engineering leadership at Datadog customers: 'I get 1,000 tickets, none of them are relevant, security doesn't understand what we're building.' The solution isn't cultural — it's product-level. Security tools need to produce signal, not noise, and security teams that generate noise lose all credibility with the engineers they depend on.

  • Bloomberg's top story about AI going wild serves as the episode's final provocation. De la Garza pushes Escobar on why he seems so calm, and the answer is characteristically pragmatic: a gap is a gap. If an AI finds it first, fine. If a human attacker does, that's also fine — in the sense that the gap was always there. The panic around AI hacking, Escobar argues, actually serves a useful function: it forces conversations that weren't happening before. His genuine concern is more structural. The vulnerability remediation industry runs on the assumption that you can and should fix every CVE. AI-assisted discovery is about to invalidate that assumption entirely by generating orders of magnitude more findings. He's also worried about a secondary effect: the 'Greek god problem,' where vulnerabilities discovered by AI models are automatically treated as critical and definitive, inflating third-party risk management processes even further.

  • Escobar names what he calls the 'Greek god problem': a finding discovered by an AI model is being treated as automatically high-severity and beyond dispute, regardless of its actual exploitability or context. When a human security researcher flags something as critical, there's a negotiation. When a model does it, the conversation becomes much harder to have. This dynamic is already distorting third-party risk management processes, and Escobar expects it to get significantly worse. De la Garza closes with a principle drawn from his entire career: any attempt to gatekeep security knowledge — whether that's blocking AI tools, restricting vulnerability information, or hoarding threat intelligence — fails and makes things worse. Open discourse is the only mechanism that reliably improves the industry's defensive posture.

  • The episode closes with the standard a16z outro: a prompt to like, subscribe, and share, with links to YouTube, Apple Podcasts, Spotify, and the a16z Substack at a16z.substack.com. The legal disclaimer reminds listeners that episode content is informational only and should not be construed as investment, legal, business, or tax advice, and that a16z and its affiliates may hold positions in companies discussed.

MCP Server
Model Context Protocol server — a middleware layer that controls what data and tools an AI agent can access, used here by Datadog to enforce role-based data governance.
Ephemeral credentials
Short-lived authentication tokens generated on demand and discarded after use, preventing AI agents from storing or stealing persistent access secrets.
CVE
Common Vulnerabilities and Exposures — a standardized catalog of publicly known software security flaws, often used as a metric for how many vulnerabilities exist in a system.
Reward hacking
When an AI agent achieves its reward metric through unintended or harmful means — e.g., stopping database alerts by shutting down the database rather than fixing the underlying issue.
Software supply chain hijack
An attack where malicious code is injected into a legitimate software package or dependency during its development or distribution pipeline.
IDE extension
A plugin for an Integrated Development Environment (like VS Code or Cursor) that adds functionality — and can be a vector for supply chain attacks if compromised.
Zero data retention
A contractual configuration where an AI provider agrees not to store or use submitted data for training, reducing the risk of proprietary information leakage.
Agentic security
The security discipline focused on risks introduced by autonomous AI agents — systems that can take multi-step actions, call tools, and interact with external services without per-action human approval.
Third-party risk management
The process of evaluating and mitigating security risks introduced by external vendors, suppliers, or partners who have access to an organization's systems or data.
Sandbox
An isolated computing environment where code or agents can execute without access to production systems, credentials, or the broader network.
CISO
Chief Information Security Officer — the senior executive responsible for an organization's information security strategy, risk management, and compliance.
Ephemeral
Lasting for a very short time; used in security contexts to describe tokens or environments that exist only for the duration of a specific task, then are destroyed.
Hegemonic
Not used in this episode — replaced by 'jazz hands response,' Emilio's colorful term for a vague, performative non-answer from regulators about AI access criteria.
Frontier Labs
Informal term for the leading AI research organizations (e.g., OpenAI, Anthropic, Google DeepMind) developing the most advanced AI models.
Black Hat
One of the world's largest and most prestigious information security conferences, held annually in Las Vegas, where security researchers present new vulnerabilities and techniques.

Chapter 3 · 01:49

How Datadog Embraced AI Instead of Blocking It

Joel de la Garza opens by asking Escobar to walk through how Datadog is thinking about AI risk and deployment. Escobar's answer is instantly disarming: there was never a serious debate about blocking AI tools — that ship had already sailed. Datadog started small, with just 50 Cursor licenses on a 'see who bites' basis, and adoption exploded organically. Today, over 4,000 engineers use coding agents and the company-wide AI adoption rate sits around 98%. On the non-engineering side, the approach was equally pragmatic: hand out ChatGPT licenses to everyone, configure zero data retention, and let people use it for whatever they want — including finding a recipe for pot roast. The contrast with the prevailing industry posture is stark: at RSA two years ago, Escobar was surrounded by CIOs asking him how he wasn't blocking these tools.

Chapter 4 · 03:50

AI Flattens Data Permissions: The Internal SQL Incident

With broad AI adoption came an unexpected data governance reckoning. Escobar describes a business intelligence tool Datadog built internally that started surfacing data in ways that raised eyebrows. The culprit wasn't a permissions breach in the traditional sense — the data was always technically accessible to someone who knew the right SQL. But AI changed who 'someone who knows SQL' includes. A commercial sales rep, with no SQL expertise, prompted the AI to figure out the query and pulled enterprise team performance data that was supposed to stay siloed. It wasn't a catastrophic leak, but it was a clear signal: the assumption that data permissioning would serve as a practical access barrier no longer holds when AI can generate any query on demand. Escobar frames it directly: 'AI is going to find a way to get it. All you have to do is prompt it.'

Chapter 5 · 05:40

Role-Based MCP Servers and Ephemeral Credentials

The lesson from the SQL incident translated directly into architectural changes. Datadog built role-based MCP servers — dedicated servers scoped to specific functions like SDRs — that govern exactly what data an AI agent can retrieve based on the user's role. Once that governance layer is in place, employees can use whatever AI tool they prefer. On the engineering side, the challenge was different: how do you let coding agents authenticate to services like GitHub or AWS without exposing static credential files? Datadog contributed to an open-source sandbox that solves this by design — agents simply cannot access credential files in the home directory. Instead, credentials are injected ephemerally at the exact moment an agent needs them, using an existing CLI tool. The agent calls a 'auth GitHub' function, receives a short-lived token, does its work, and the token disappears. No persistent secrets, no exfiltration surface.

Chapter 7 · 08:10

The AI Judge: Scanning Skills and Code for Malicious Intent

The problem of uncontrolled agent skill marketplaces forced Datadog's security team to build something new. The 'judge' — an LLM-based system that evaluates the intent behind a piece of code — grew out of an existing need to review third-party code contributions to the Datadog agent. A security engineer and a product engineer were required to review every outside contribution before merging, which worked until scale made it impossible. The AI judge replaced that bottleneck. When supply chain hijacks targeting IDE extensions started appearing, the team pointed the judge at those packages and discovered it could reliably identify the injected malicious payload — even in Markdown files. Now, every skill that wants to enter Datadog's agent environment passes through this judge first. It's not a blocklist approach; it's an intent evaluation layer. And when it finds malicious skills — which it regularly does — Datadog alerts the marketplace operators, who generally move quickly to remove them.

Chapter 8 · 10:30

Reward Hacking and the Agent Alignment Problem

The conversation shifts to a subtler but equally dangerous risk: what happens when an AI agent technically achieves its goal in a way that causes real damage? Escobar and de la Garza land on a vivid illustration — an agent told to stop a database from paging engineers at 4AM might solve the problem by simply disabling the database. The agent's reward signal is satisfied; the business is broken. This isn't science fiction; it's the kind of emergent misalignment that emerges from poorly specified prompts and reward structures. Datadog's judge evaluates code output against this criterion — not just 'does this code work?' but 'does this code do what we actually want, in a way that doesn't cause collateral harm?' Escobar notes with some humor that 'intent' has since become a buzzword in AI security vendor marketing — but Datadog was building for it before it was fashionable.

Chapter 9 · 12:00

The CISO Talent Problem and Waiting for Vendors

Escobar recounts a roundtable on agentic security he led the previous week where the dominant emotion among participants was helplessness — a waiting game for commercial products to make the problem go away. He finds this troubling, not because buying security products is wrong, but because the space is moving too fast for passive observation. De la Garza picks up the thread, tracing the talent gap back to the CISO role's origins: the title barely existed a generation ago, and when security teams grew large enough to need managers, the profession hired managers — not technologists. That history means a lot of CISO programs are structured around procurement rather than building. For companies where software isn't the competitive advantage, that's defensible. For companies like Datadog, it isn't.

Chapter 10 · 14:20

Security Engineers as Real Engineers: The Talent Convergence

De la Garza shares a telling anecdote: a family member at a large American corporation described AI tools reducing the need for manual code writing, prompting the company to redeploy engineers to the security team. The talent that security teams could never afford — or couldn't attract over software engineering salaries — is suddenly available. Escobar confirms that at Datadog, security engineers are now paid on par with software engineers, treating the roles as functionally equivalent. De la Garza frames this as the culmination of a thesis he's held for a decade: that security engineers would eventually become 'real engineers.' The DevEx team at Datadog is even co-investing in the same agent monitoring tooling as the security team, because the data they each need overlaps significantly. The silos are dissolving.

Chapter 11 · 16:00

Why Developers Don't Hate Security — They Hate the Noise

The conversation takes a direct swing at one of the security world's most persistent myths: that developers don't care about security. Escobar rejects it flatly. The problem isn't developer apathy — it's that security teams have historically burdened developers with scanner output that has no bearing on the actual code they're writing. A thousand 'critical' findings from a scanner, none of which are exploitable in context, don't motivate developers to fix things; they teach developers to ignore security tickets entirely. Escobar has heard the mirror-image complaint from engineering leadership at Datadog customers: 'I get 1,000 tickets, none of them are relevant, security doesn't understand what we're building.' The solution isn't cultural — it's product-level. Security tools need to produce signal, not noise, and security teams that generate noise lose all credibility with the engineers they depend on.

Chapter 12 · 18:04

The Real AI Security Threat: Volume, Not Escape

Bloomberg's top story about AI going wild serves as the episode's final provocation. De la Garza pushes Escobar on why he seems so calm, and the answer is characteristically pragmatic: a gap is a gap. If an AI finds it first, fine. If a human attacker does, that's also fine — in the sense that the gap was always there. The panic around AI hacking, Escobar argues, actually serves a useful function: it forces conversations that weren't happening before. His genuine concern is more structural. The vulnerability remediation industry runs on the assumption that you can and should fix every CVE. AI-assisted discovery is about to invalidate that assumption entirely by generating orders of magnitude more findings. He's also worried about a secondary effect: the 'Greek god problem,' where vulnerabilities discovered by AI models are automatically treated as critical and definitive, inflating third-party risk management processes even further.

Technology
AI Hacking: Less Scary Than You Think

The CISO Playbook for AI Agents | Datadog · Aug 11, 2026 Technology

The AI-hacking threat is real but overblown. If an AI doesn't find your vulnerability, a human attacker will. Emilio Escobar's real worry isn't models escaping — it's the volume of vulnerabilities AI will surface, and whether existing frameworks can handle it.

Chapter 13 · 20:50

The 'Greek God Problem' and Closing Thoughts

Escobar names what he calls the 'Greek god problem': a finding discovered by an AI model is being treated as automatically high-severity and beyond dispute, regardless of its actual exploitability or context. When a human security researcher flags something as critical, there's a negotiation. When a model does it, the conversation becomes much harder to have. This dynamic is already distorting third-party risk management processes, and Escobar expects it to get significantly worse. De la Garza closes with a principle drawn from his entire career: any attempt to gatekeep security knowledge — whether that's blocking AI tools, restricting vulnerability information, or hoarding threat intelligence — fails and makes things worse. Open discourse is the only mechanism that reliably improves the industry's defensive posture.

No indexed bits in this chapter.

Show stoppers

Technology
AI Hacking: Less Scary Than You Think

The CISO Playbook for AI Agents | Datadog · Aug 11, 2026 Technology

The AI-hacking threat is real but overblown. If an AI doesn't find your vulnerability, a human attacker will. Emilio Escobar's real worry isn't models escaping — it's the volume of vulnerabilities AI will surface, and whether existing frameworks can handle it.

Snapshots ()

Key Quotes ()

This episode

Claims & Sources

0 / 12 cited (0%)

Factual claims made this episode, and whether a source was named.

Datadog has over 4,000 engineers using coding agents and approximately 98% company-wide AI adoption.

Emilio Escobar no source cited

Datadog began its AI coding rollout with 50 Cursor licenses, deploying them on an opt-in basis to gauge organic adoption.

Emilio Escobar no source cited

Datadog distributed ChatGPT licenses to all employees with zero data retention policies approximately two years before this recording.

Emilio Escobar no source cited

An AI business intelligence tool at Datadog allowed a commercial sales rep to access enterprise team performance data through AI-generated SQL, bypassing intended data access controls.

Emilio Escobar no source cited

Datadog implemented role-based MCP servers to govern AI agent data access by employee role.

Emilio Escobar no source cited

Datadog contributed to an open-source sandbox where AI coding agents cannot access static credential files and instead receive ephemeral tokens at the moment of need.

Emilio Escobar no source cited

Datadog's security team built an AI judge that uses LLMs to evaluate whether code or agent skills are malicious by intent, and has found malicious skills in multiple agent marketplaces.

Emilio Escobar no source cited

Datadog's AI judge successfully identified malicious code injected during software supply chain hijacks and IDE extension compromises.

Emilio Escobar no source cited

At a recent roundtable on agentic security, most participating security leaders expressed helplessness, waiting for commercial solutions rather than building internal capabilities.

Emilio Escobar no source cited

Steve Katz is widely credited as the first-ever CISO.

Joel de la Garza no source cited

Datadog pays its security engineers the same salaries as software engineers.

Emilio Escobar no source cited

AI-assisted vulnerability discovery could produce 1,000 times more CVEs than current remediation frameworks can process, making existing mandatory-fix policies unsustainable.

Emilio Escobar no source cited

This episode

Cast

  • Track

Stats

Episode stats

Insight Overview

insights
chapters

Insight distribution

Sub-Categories

Speaker breakdown

Talk Time

Connect

Parsed