Staff Security Engineer - Product Security

Engineering teams are increasingly adopting LLM copilots and agentic tools to move faster. That’s useful, until an “assistant” becomes an unmonitored automation path to secrets, sensitive data, or privileged actions. (Think: “obedient intern with production credentials.”) Industry guidance is converging on practical frameworks like the NIST AI Risk Management Framework (including a profile for generative AI) and the OWASP Top 10 for LLM Applications, which explicitly calls out risks like prompt injection, insecure plugin design, and excessive agency.

The mental model for LLM guardrails that finally clicked for me.

Took me a while to stop picturing ai guardrails as the model refusing stuff. In a real deployment, it’s a separate layer that doesn’t trust the model at all rough shape i landed on: 1. inbound: every prompt gets checked before it gets to the model. Stuff like injection attempts, policy violations, pii etc are all blocked or flagged here 2. model does its thing 3. outbound: the response gets checked before the user sees it. Catches things like leaked, made up claims, toxic output, anything that breaks your policy. The part people skip is this has to be its own layer not a system prompt. System prompts are suggestions that the model can get talked to skip. A check sitting outside a model is effective at enforcement, and it cant be plain keyword matching or you miss anything phrased politely. The thing i still dont have a clean answer on is latency. Every check is time before the user gets the response, so theres a tradeoff. What I’d like to understand here is how are you handling that balance?

SN 1094: AI Patching Shortcomings - Should You Trust AI-Generated Code?