AI Security & Safety

I’ve spent the last year watching AI security go from a niche concern to the thing that keeps engineering leaders up at night. This isn’t the conference-talk version of AI security and safety. Not the vendor pitch. I’m covering the real threat landscape, the vulnerabilities that keep showing up in production, the regulatory frameworks you’ll actually have to comply with, and the practical defenses that work. If you’re building anything that touches a large language model, this is the stuff you need to know.

The AI Security Threat Landscape Has Changed Completely

For most of software engineering history, security meant protecting deterministic systems. You wrote code, it did exactly what you told it, and attackers exploited the gaps between what you intended and what you actually implemented. Buffer overflows, SQL injection, cross-site scripting. The playbook was well-understood.

LLMs broke that model. We’re now deploying systems that are probabilistic, that interpret natural language instructions, and that can be manipulated through the very interface they’re designed to serve. I’ve been building software for over 14 years, and I’ve never seen a class of vulnerabilities quite like this. The attack surface isn’t a bug in the code. It’s the feature itself.

The numbers tell the story. The AI security market is projected to exceed $60 billion by 2028, growing at over 20% CAGR. But spending doesn’t equal readiness. In a 2024 survey by IBM, 82% of organizations reported they had adopted or were exploring AI, yet fewer than 24% had secured their AI deployments. That gap is where the real risk lives.

What makes right now particularly dangerous is the collision of two trends: AI systems are getting more capable (and therefore more useful as attack targets), and they’re being deployed faster than security practices can keep up. Dario Amodei, CEO of Anthropic, has repeatedly said it plainly: the window between capability and safety is narrowing. His company’s Responsible Scaling Policy represents one of the few serious attempts to formalize safety commitments at the frontier. Whether the rest of the industry follows is an open question.

For a deeper dive into how this landscape evolved and where it’s heading, I wrote a complete guide to AI security that covers the full picture.

Prompt Injection Remains the Unsolved Problem

If there’s one vulnerability that defines this era of AI security, it’s prompt injection. OWASP’s Top 10 for LLM Applications 2025 lists it as LLM01 — the number one risk. It held that position since the list was first published in 2023. That should tell you something.

Simon Willison, the developer and researcher who originally coined the term “prompt injection” by analogy to SQL injection, has been adamant about a distinction most people miss: prompt injection and jailbreaking are not the same thing. Jailbreaking targets the model’s safety filters. Prompt injection targets applications built on top of those models. The stakes are completely different.

As Willison puts it, jailbreaking’s worst case is a “screenshot attack” — someone tricks a model into saying something embarrassing. Prompt injection’s worst case is an attacker gaining access to your users’ private data or triggering privileged actions through your application’s tool integrations. If your LLM-powered app can read emails, execute code, or call APIs, prompt injection isn’t a theoretical concern. It’s the most serious vulnerability in your stack.

I’ve written about why prompt injection remains OWASP’s number one LLM vulnerability in 2026 and what that means for anyone shipping AI agents to production. The short version: there is no complete fix. Every defense — input filtering, output validation, instruction hierarchy, sandboxing — is a mitigation, not a solution. The fundamental problem is that LLMs can’t reliably distinguish between instructions and data when both arrive as natural language.

Why Traditional Security Tools Don’t Work Here

WAFs can’t parse prompt injections. SAST tools don’t understand natural language attack vectors. The entire traditional application security toolchain was built for deterministic systems. When your “code” is a prompt and your “input” is also a prompt, the boundary between the two dissolves.

This is why a new category of AI-native security tooling is emerging — tools that monitor LLM inputs and outputs in real-time, detect anomalous behavior patterns, and enforce guardrails at the application layer rather than the model layer. Still early. But the direction is clear.

The OWASP Top 10 for LLMs: A Practical Framework

The OWASP GenAI Security Project has become the closest thing we have to a shared vocabulary for LLM security risks. Their 2025 update, developed by over 600 contributing experts from 18+ countries, covers the vulnerabilities that matter most in production.

Here’s the current top 10, with my take on real-world severity:

The ones I’d flag as most underestimated: LLM06 (Excessive Agency) and LLM08 (Vector and Embedding Weaknesses).

Excessive Agency is what happens when you give an AI agent tool access without proper scoping. The agent takes actions far beyond what you intended. I’ve seen this in production — a customer support agent had write access to a database it should have only been reading from. That’s not a theoretical risk. That’s a Tuesday.

Vector and Embedding Weaknesses matter if you’re building retrieval-augmented generation systems. Poisoning the vector database that feeds your LLM’s context is a subtle, powerful attack vector that most teams aren’t even monitoring for.

AI as an Offensive Weapon: The Other Side of the Equation

AI security isn’t just about defending AI systems. It’s also about defending against AI-powered attacks. This dual-use problem is accelerating fast.

Attackers are using LLMs to generate convincing phishing emails at scale, to discover vulnerabilities in codebases, and to automate reconnaissance. The economics are brutal: it costs an attacker almost nothing to generate thousands of customized social engineering messages. Defenders have to catch every single one.

But the offensive use of AI goes deeper than phishing. DARPA has funded research into AI pentesting agents — autonomous systems that can discover and exploit vulnerabilities without human guidance. Projects like Mythos AI are teaching LLMs to hack, with government backing. The rationale: understanding AI-powered offense is necessary to build effective defense. I think that’s right. But it also means the tools are being built.

On the military side, the picture gets darker. Autonomous targeting systems powered by AI algorithms are making life-and-death decisions with decreasing human oversight. I explored the AI kill chain and military targeting in depth. The ethical questions are as hard as the technical ones. When an algorithm chooses who to target on a battlefield, accountability becomes almost impossible to assign.

Bruce Schneier, the security researcher and author, has argued that AI will change the nature of cybersecurity by shifting the advantage to attackers in the short term. His reasoning is simple: AI dramatically lowers the skill barrier for attacks while defensive AI is harder to deploy correctly. I think he’s right about the short term. The equilibrium will shift as defensive tooling matures. But we’re not there yet.

The Regulatory Landscape: EU AI Act, NIST, and What’s Coming

Regulation is coming for AI. Security teams need to prepare now.

The EU AI Act, which began enforcement in stages starting in 2024, is the most comprehensive AI regulation in the world. It classifies AI systems by risk level and imposes specific requirements on high-risk applications, including mandatory security testing, documentation, and human oversight.

For engineering teams, the practical implications:

1. Risk classification: You need to determine whether your AI system falls into the “unacceptable,” “high,” “limited,” or “minimal” risk category. Most enterprise applications with decision-making authority will land in “high.”
2. Conformity assessments: High-risk systems require documented testing procedures, including adversarial testing for security and safety.
3. Transparency obligations: Users must be informed when they’re interacting with an AI system. System prompts and decision logic need documentation.
4. Data governance: Training data must meet quality standards, and you need to demonstrate compliance.
5. Human oversight mechanisms: High-risk systems must have meaningful human-in-the-loop controls. Not rubber-stamp approvals.
6. Incident reporting: Security incidents involving AI systems must be reported to authorities, similar to GDPR breach notifications.

In the US, NIST’s AI Risk Management Framework (AI RMF) provides voluntary guidance that’s increasingly becoming the de facto standard. It’s structured around four core functions: Govern, Map, Measure, and Manage. If you’re building production AI systems in North America, aligning with NIST AI RMF is the pragmatic move even if it’s not legally required yet.

Yoshua Bengio, Turing Award winner and one of the pioneers of deep learning, has been one of the most vocal advocates for AI regulation. He argues that frontier AI systems pose risks serious enough to warrant government oversight comparable to what we apply to nuclear technology. Whether you agree with that level of urgency or not, the direction is clear: building AI without a security and compliance strategy is becoming professionally irresponsible.

Practical AI Security: What Actually Works in Production

I’ve shipped enough systems to know that security advice is only useful if it’s implementable. Here’s what holds up when you’re running LLMs in production.

Defense in Depth for LLM Applications

The principle hasn’t changed from traditional security — layer your defenses. But the layers themselves look different:

• Input validation and sanitization: Filter known prompt injection patterns before they reach the model. This catches the low-hanging fruit but won’t stop sophisticated attacks.
• Output validation: Never trust LLM output. Validate, sanitize, and constrain outputs before they reach users or trigger actions. Treat the LLM as an untrusted data source. Because it is.
• Least privilege for function calling: If your agent can call tools, scope those tools as tightly as possible. Read-only where possible. Rate-limited always. No tool should have more permission than the specific task requires.
• Monitoring and anomaly detection: Log all inputs and outputs. Watch for patterns that indicate injection attempts, data exfiltration, or unexpected behavior. This is where you catch what your filters missed.
• Sandboxing and isolation: Run LLM workloads in isolated environments. If an agent is compromised, the blast radius should be contained.

The Real-World Security Stack

Having worked through several production AI deployments, the security stack that actually holds up looks something like this: pre-processing guardrails (input filters, context validation), runtime monitoring (real-time output analysis, behavioral anomaly detection), and post-processing controls (output sanitization, action approval workflows). No single layer is sufficient. The system works because each layer catches what the others miss.

The piece most teams overlook is supply chain security. When you pull a model from Hugging Face or use a third-party fine-tuning service, you’re trusting their entire pipeline. Model poisoning is a real threat, and most teams have zero visibility into the provenance of the models they deploy. At minimum, you should be checksumming model weights and validating them against known-good sources.

Hardware and Physical Security: The Forgotten Layer

Most AI security discussions focus on software. But there’s a physical dimension that matters, especially as AI capabilities push into edge devices, IoT, and tactical hardware.

I reviewed the HaleHound CYD for WiFi pentesting recently, and it drove home something I think about a lot: security tools are getting cheaper and more accessible. A $15 device can do what used to require thousands of dollars of equipment. That’s great for defenders who want to test their networks. It also means attackers have the same capabilities.

The same democratization is happening with AI. Local LLM deployments let you run models on consumer hardware, which is fantastic for privacy and LLM cost optimization. But it also means you need to think about physical access to the hardware running your models, the security of model weights stored locally, and the integrity of the inference pipeline end-to-end.

The Palo Alto Paradox: When Security Vendors Become the Vulnerability

One of the most telling stories in AI security is what happened with CVE-2024-3400 and Palo Alto Networks. Here’s a company whose CEO was giving keynotes about AI security threats while their own firewalls had a critical zero-day being actively exploited in the wild. You can’t make this stuff up.

This isn’t just a gotcha. It illustrates a real problem: the organizations selling AI security solutions are themselves running complex software stacks vulnerable to the same classes of attacks they claim to protect against. Andrej Karpathy, former Tesla AI director and OpenAI co-founder, has pointed out that the security surface area of modern AI systems is enormous — spanning the model, the inference infrastructure, the training pipeline, the data storage, and every integration point in between.

The lesson: don’t outsource your security thinking to any single vendor. Understand your threat model. Know where your data flows. Audit your dependencies. The boring fundamentals of security engineering haven’t changed just because we added “AI” to the product name.

Building a Security-First AI Culture

The hardest part of AI security isn’t technical. It’s cultural.

Engineering teams are under intense pressure to ship AI features fast. The incentive structure rewards speed over safety. I’ve seen teams deploy LLM-powered features with no security review, no adversarial testing, and no monitoring. Not because they don’t care about security, but because there was no process that required them to stop and think about it.

Building a security-first AI culture means:

• Making security review a gate in your AI deployment pipeline, not an afterthought
• Training your engineers on LLM-specific vulnerabilities (most AppSec training doesn’t cover prompt injection or model poisoning yet)
• Running regular red team exercises against your AI systems
• Establishing clear ownership for AI security — it can’t be split across ML engineering and traditional security teams without coordination
• Treating AI safety incidents with the same severity as traditional security incidents

The organizations that will be most secure aren’t the ones with the best tools. They’re the ones where every engineer building AI systems understands the threat model.

This is one of those things where the boring answer is actually the right one. Process, training, accountability. Not sexy. But it’s what separates teams that get breached from teams that don’t.

What Comes Next for AI Security

We’re at an inflection point. AI systems are becoming more autonomous, more capable, and more deeply embedded in critical infrastructure. The attack surface is growing faster than the defense ecosystem can cover it. Regulation is tightening but hasn’t caught up. And the fundamental unsolved problems — prompt injection, model interpretability, alignment verification — remain open.

Here’s what I think the next 18 months look like. First, the rise of agent orchestration frameworks will create new categories of vulnerabilities we haven’t even named yet. When you chain multiple AI agents together with tool access across systems, the security implications compound in ways nobody’s fully mapped out. Second, AI-powered offensive security tools will go mainstream, forcing a defensive arms race that most organizations aren’t staffed for. Third, regulatory enforcement will begin in earnest, and organizations that haven’t built compliance into their AI pipelines will scramble.

The engineers who invest in understanding AI security now — not just the tools, but the threat models, the attack patterns, the regulatory landscape — will be the ones who build systems that actually hold up. Everything else is just hoping for the best.

Frequently Asked Questions About AI Security & Safety

What is the biggest security risk with AI systems today?

Prompt injection remains the single biggest security risk for LLM-based applications. It’s OWASP’s number one ranked vulnerability for LLMs in both 2023/24 and 2025. Unlike traditional injection attacks, there’s no complete fix — only layered mitigations. The risk is especially acute for applications that give AI agents access to tools, APIs, or sensitive data.

How is AI security different from traditional cybersecurity?

Traditional cybersecurity protects deterministic systems with well-defined inputs and outputs. AI security must account for probabilistic behavior, natural language attack vectors, and the dual-use nature of AI as both a target and a weapon. The attack surface includes training data, model weights, inference pipelines, and prompt interfaces — none of which exist in traditional application security.

What is the OWASP Top 10 for LLMs?

The OWASP Top 10 for LLM Applications is a framework identifying the ten most critical security vulnerabilities in LLM-based systems. Developed by 600+ security experts from 18+ countries, it covers risks from prompt injection and data poisoning to excessive agency and unbounded consumption. It’s the closest thing we have to an industry-standard checklist for LLM security.

Can AI be used to hack systems?

Yes. AI is already being used offensively for automated vulnerability discovery, social engineering at scale, and autonomous penetration testing. DARPA-funded projects like Mythos AI are building AI pentesting agents that can discover and exploit vulnerabilities without human guidance. The barrier to entry for AI-powered attacks is dropping rapidly.

What regulations apply to AI security?

The EU AI Act is the most comprehensive AI regulation, classifying systems by risk level and requiring security testing, documentation, and human oversight for high-risk applications. In the US, NIST’s AI Risk Management Framework provides voluntary but increasingly standard guidance. Additional sector-specific regulations are emerging in healthcare, finance, and defense.

How do you prevent prompt injection attacks?

You can’t fully prevent prompt injection, but you can mitigate it through layered defenses: input sanitization and filtering, strict output validation, least-privilege tool access, instruction hierarchy enforcement, and continuous monitoring. Treating LLM output as untrusted data — the same way you’d treat user input in a web application — is the foundational principle.

What is AI red teaming?

AI red teaming is the practice of adversarially testing AI systems to discover vulnerabilities, biases, and failure modes before they’re exploited in the wild. It includes testing for prompt injection, jailbreaking, data extraction, and unexpected behaviors. Major AI companies and government agencies now run regular red team exercises, and the practice is becoming a regulatory expectation for high-risk AI deployments.

Should I run AI models locally for better security?

Running local LLM deployments can improve security by keeping data on-premise and eliminating API call exposure. However, it introduces different risks: physical access to model weights, supply chain concerns around model provenance, and the operational burden of securing inference infrastructure. The right choice depends on your threat model, data sensitivity, and operational maturity.