Where this was said
The AI Judge: Scanning Skills and Code for Malicious Intent
At 9:13 · chapter starts 8:10
The problem of uncontrolled agent skill marketplaces forced Datadog's security team to build something new. The 'judge' — an LLM-based system that evaluates the intent behind a piece of code — grew out of an existing need to review third-party code contributions to the Datadog agent [1] — Emilio Escobar "Out of necessity, Datadog's security team built an AI judge that uses LLMs to evaluate whether code or an agent skill is meant to do harm —…" 07:44 . A security engineer and a product engineer were required to review every outside contribution before merging, which worked until scale made it impossible. The AI judge replaced that bottleneck. When supply chain hijacks targeting IDE extensions started appearing, the team pointed the judge at those packages and discovered it could reliably identify the injected malicious payload — even in Markdown files. Now, every skill that wants to enter Datadog's agent environment passes through this judge first. It's not a blocklist approach; it's an intent evaluation layer. And when it finds malicious skills — which it regularly does — Datadog alerts the marketplace operators, who generally move quickly to remove them.
Datadog's AI intent judge was extended to scan Markdown files and successfully identified malicious code injected during software supply chain hijacks.
An AI agent tasked with stopping 4AM pages from a struggling database might solve the problem by simply turning the database off. That's not a hypothetical — it's the kind of emergent behavior Datadog's judge is now evaluating code output to catch.
