AI Model Security: Risks, Controls and a Defensive Framework

Access control is still the weak point. In IBM’s 2026 Cost of a Data Breach research, conducted with the Ponemon Institute across 602 organizations hit by breaches, roughly one in five reported an AI-related breach. Of those, 92% lacked proper AI access controls, and only 40% of all organizations reported using access controls on AI models and data [1].

AI model security is not one control or one scanner. It spans the model’s origin, the data used to train, tune and ground it, the access path, the deployment environment, connected tools and the way outputs are handled. This guide organizes defensive controls across that lifecycle, maps them to recognized frameworks and helps you decide what to secure first, before a model touches sensitive data or production workflows.

Key takeaways

• Treat the model as one asset inside a system: the artifact, its data, the serving runtime, the application and connected tools each need controls.
• Close the access gap first: per-application identity, scoped keys and authenticated endpoints.
• Verify provenance before deployment with approved sources, pinned versions and signed artifacts where publishers provide them.
• NIST AI RMF, the OWASP Top 10 for LLM Applications and MITRE ATLAS do different jobs, and none of them is a certification.
• Re-review controls whenever the model, provider, tools, data or incident picture changes.

What does AI model security cover?

AI model security protects a model and the system around it from tampering, misuse and data exposure. Draw the boundary around five layers:

• Model artifact: weights, configuration and the files that load them.
• Data: training, fine-tuning, evaluation and retrieval data, plus prompts and outputs in transit.
• Serving runtime: infrastructure, endpoints and keys.
• Application: the code that assembles prompts, calls the model and uses its outputs.
• Connected services: tools, APIs and other agents the model can call.

Model security overlaps with two neighbors without replacing them. Application security still covers authentication, input validation and dependencies. AI governance decides who may approve a use case, what risk is acceptable and who is accountable. Governance platforms, like those in our review of best AI governance tools, help document those decisions, but they do not enforce technical controls at runtime.

The overlap is visible in NIST’s Generative AI Profile, which lists information security and value chain and component integration among 12 generative AI risks. The second covers untraceable third-party components and improper supplier vetting across the AI lifecycle [2].

Threat areas across the model lifecycle

Organize risk by where it enters the lifecycle. The categories below are conceptual and defensive. The OWASP Top 10 for LLM Applications, currently the 2025 edition, gives matching application-level names [3].

Provenance and tampering

A model or dataset can be altered between its source and your environment, or arrive with unclear lineage. OWASP lists supply chain and data and model poisoning as separate risks [3]. NIST notes that generative AI value chains involve many third-party components, such as procured datasets, pre-trained models and software libraries, which may be improperly obtained or not properly vetted [2].

Sensitive data exposure

Models can reveal data from training, retrieval or system instructions. OWASP covers this as sensitive information disclosure and system prompt leakage [3].

Access misuse

Unprotected endpoints, shared keys and broad service accounts let anyone with network reach use the model or its data. Keys copied into notebooks, scripts or prompts spread the problem further, because nobody can tell which application made a call. IBM’s breach figures show how often this gap appears [1].

Unsafe integrations

When a model can call tools, the risk shifts from what it says to what it can do. OWASP calls this excessive agency [3]. In multi-agent systems, the AI agents orchestration layer defines which calls are possible. It does not make those calls safe by itself.

Output handling

Systems that trust model output without validation can pass errors or injected content into databases, browsers or workflows. OWASP lists improper output handling and misinformation [3].

Operational weaknesses

Missing rate limits, budgets, logging or rollback. Without them, a fault or misuse can run up cost, degrade service and leave no record to investigate. OWASP’s unbounded consumption entry covers resource abuse [3].

The joint CISA, NSA and FBI guidance on AI data security adds a data lens. It names the data supply chain, maliciously modified (“poisoned”) data and data drift as three significant risk areas [4].

Build a layered AI model security control framework

Layer controls so one failure does not expose the whole system. Map them to seven areas, each with a named owner.

Procurement and provenance

Source models from known publishers, pin exact versions and record lineage. Where publishers sign artifacts, verify signatures before loading. The model signing project built with OpenSSF, NVIDIA and HiddenLayer reached version 1.0 in April 2025. It binds a model to the identity that produced it and records signatures in a transparency log so users can verify integrity [5].

Development

Keep training and fine-tuning data in controlled stores. Track data provenance and use digital signatures for trusted data revisions, as the CISA-led guidance recommends [4]. Put prompts, system instructions and tool definitions under the same change approval as code.

Deployment

Serve models from hardened, patched infrastructure. Keep keys in a secrets manager, never in code or prompts. Separate test and production environments. Load new third-party model files in an isolated environment first, check them against the hash or signature you recorded, and promote them to production only through the same release process as code.

Identity and network boundaries

Give every application and agent its own identity with least privilege. CISA and partner agencies advise organizations adopting agentic AI to avoid broad or unrestricted access, especially to sensitive data or critical systems, and to start with low-risk use cases [6]. Put model endpoints behind authenticated gateways and limit outbound calls to approved tools.

Data handling

Classify data, apply access controls and encryption by class, and classify model output at the same level as its input [4].

Monitoring

Log access, tool calls and policy blocks. Alert on unusual callers, volume, cost and behavior drift. Connect these alerts to your existing incident response process, with playbooks to revoke a key, roll back a model version and disable a tool without taking the whole application down.

Ownership

Assign an accountable owner to every model endpoint and integration, and document it in your inventory.

Use NIST, OWASP and MITRE ATLAS responsibly

Each framework answers a different question. Use them together and know their limits.

NIST AI RMF is a voluntary risk-management framework, released on January 26, 2023 [7]. Its core functions are govern, map, measure and manage, with govern described as cross-cutting [8]. NIST says version 1.0 is being revised as part of the White House AI Action Plan [7]. The Generative AI Profile (NIST AI 600-1), released July 26, 2024, adds generative AI risks and suggested actions [2].

The OWASP Top 10 for LLM Applications is a community list of application risks, currently in its 2025 edition [3]. Use it to check coverage in design reviews and tests.

MITRE ATLAS describes itself as a living knowledge base of adversary tactics and techniques involving AI, based on real-world attacks and demonstrations by AI red teams and security groups [9]. Its site lists 16 tactics, 208 techniques, 40 mitigations and 73 case studies [9]. Use it for threat modeling and to check detections against known adversary behavior.

None of these is a certification. Mapping controls to a framework shows coverage on paper; it does not prove the controls work. Test them only in owned or explicitly authorized environments, and record the residual risk you accept.

Assess vendors and model supply chains

Most teams run models they did not train. Ask each provider or model source for evidence, not assurances:

• Model lineage: who built the model, from which base model, which data sources at a summary level, and which version you will run.
• Change notices: how you learn about updates, deprecations and behavior changes, and how much notice you get.
• Data handling terms: whether prompts and outputs are retained, for how long, where, and whether they train future models.
• Access controls: SSO, scoped API keys, project isolation and exportable audit logs.
• Incident response: notification timelines, contacts and what the provider shares after an incident.
• Deployment evidence: security documentation for your hosting model, such as dedicated or regional options.

NIST suggests recording data provenance, versioning and known issues from incident databases in generative AI inventory entries [2]. Open-weight models from public hubs need the same discipline. Confirm the publisher’s identity, verify signatures where they exist, record the file hashes you tested and pin them, so a later upload under the same name cannot replace what you approved.

Separate verified, generally available capabilities from roadmap claims, and record the product version and date of every answer. Re-ask at renewal and whenever the model changes. For wider context on where agent platforms are heading, see our agentic AI trends 2027 overview.

A practical implementation sequence

Work in this order, so each step gives the next one the facts it needs:

1. Inventory models and endpoints. Include shadow use, third-party APIs and embedded features, with an owner for each.
2. Classify the data each one touches. Training, retrieval, prompts and outputs all count.
3. Set a control baseline per risk tier. Use the matrix above; higher tiers get approval gates and tighter monitoring.
4. Pilot monitoring on one endpoint. Prove that logs, alerts and ownership work before you scale them.
5. Test approved defensive scenarios. Run them in owned or explicitly authorized environments, mapped to OWASP and ATLAS categories, with written scope.
6. Review residual risk. Record what you accept, who accepted it and when it expires.

Then set re-review triggers: a new model or provider version, new tools or permissions, changed data sources or classification, and any material incident. Each trigger sends you back to step 1 for the affected system.

Summary

AI model security protects the model and everything around it: provenance, data, runtime, application and connected tools. Close access gaps first, verify what you load, layer preventive and detective controls with named owners, and use NIST, OWASP and MITRE ATLAS for the jobs each does best.

Defense in depth beats any single scanner. Your next step: inventory every model endpoint this quarter, assign an owner to each and schedule measured, authorized testing against your baseline.

FAQs

What are the top 3 AI security risks?

There is no universal top three; the ranking depends on your use case. For many deployments, three categories stand out: sensitive data exposure, controlled with classification and retrieval filters; access misuse, controlled with per-application identity and scoped keys; and unsafe integrations, controlled with least-privilege tools and approvals. OWASP’s 2025 list gives a broader application view [3].

How is AI model security different from AI governance?

AI governance assigns decision rights: who approves a use case, what risk is acceptable and who is accountable. AI model security covers the technical and operational protections that enforce those decisions, such as provenance checks, access control, data protection and monitoring. They overlap in inventory and risk review, but a policy alone does not secure a misconfigured endpoint.

Which frameworks can help organize AI security?

Use the NIST AI RMF to run the risk-management process, the OWASP Top 10 for LLM Applications to check application-level risk coverage, and MITRE ATLAS to understand adversary tactics against AI systems. Each serves a different purpose. None is a certification, and mapping controls to them does not prove the controls work without authorized testing.

References

1. AI-powered adversaries and the enterprise risk challenge: Preparing for the new reality. IBM X-Force (Limor Kessem), on the 2026 Cost of a Data Breach report with Ponemon Institute, 29 Jul 2026. Accessed 28 Oct 2026.
2. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). NIST, 26 Jul 2024. Accessed 28 Oct 2026.
3. OWASP Top 10 for LLM Applications 2025. OWASP GenAI Security Project, 2025 edition. Accessed 28 Oct 2026.
4. AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems. CISA, NSA, FBI and international partners, 22 May 2025. Accessed 28 Oct 2026.
5. Taming the Wild West of ML: Practical Model Signing with Sigstore. Sigstore blog (Mihai Maruseac, Google Open Source Security), 4 Apr 2025. Accessed 28 Oct 2026.
6. CISA, US and International Partners Release Guide to Secure Adoption of Agentic AI. CISA with ASD’s ACSC and partners, 1 May 2026. Accessed 28 Oct 2026.
7. AI Risk Management Framework. NIST, AI RMF 1.0 released 26 Jan 2023, page notes revision in progress. Accessed 28 Oct 2026.
8. AI RMF Core. NIST Trustworthy and Responsible AI Resource Center, excerpt of AI RMF 1.0. Accessed 28 Oct 2026.
9. MITRE ATLAS. MITRE, living knowledge base (data release v2026.09, Sep 2026). Accessed 28 Oct 2026.