Back in January 2024, a customer was following up on a missing package with DPD. An average Joe with little to no complicated tech knowledge decided to mess around with DPD’s customer service chatbot. A few cleverly disguised prompts later, he had manipulated the chatbot into swearing, criticizing its own employer’s services, and calling out delays in internal operations. In short, it abandoned every rule it was supposed to follow. The incident went viral, with screenshots shared online, and as you’d expect, DPD pulled its AI system offline within hours. But what started as a joke on social media is a vital case study into why enterprise AI security exists in the first place: to stop an AI system from doing whatever the user pushes it to do, instead of only performing the functions the business intended it to do.
The aforementioned guardrail failure is more common than most executives would assume. IBM’s 2025 Cost of a Data Breach report found that nearly 97% of organizations that suffered an AI-related security incident lacked proper access controls for these systems. Moreover, any breach tied to unsanctioned AI use means an additional cost of $670,000 per incident.
The pattern is strikingly similar, regardless of company size and industry: AI put into production quicker than it can be secured only leads to disaster.
This guide breaks down what enterprise AI security actually means, where the real risks are, and, critically, what measures to put in place if an organization doesn’t want its online chatbot writing colorful poems about it to customers.
Read on to learn more.
Five Key Risks in Enterprise AI Security
The five critical risks to look out for when dealing with enterprise AI security are as follows:
1. Data Leakage Through Prompts, Outputs, or Fine-Tuning
The most dangerous kind of breach is the one that looks more like a coincidence rather than a malicious attempt. It could be something as trivial as an employee noticing an odd response from a model where the response matches client details they never entered themselves. While most external breaches set off every form of mitigation measures in place, internal leaks, such as sensitive information being shown to someone not supposed to see it, rarely do because there is nothing about the request that indicates an attack.
Moreover, the devil is in the details. More specifically, in the contractual nuances most companies use. Company A may give all forms of promises and commitments about not using any of Company B’s data for training, but these promises do not cover data retention. This is on top of various retention practices adopted by companies as a means to monitor for abuse and safety reviews that can last for as long as months, during which the entire time the data remains human-readable.
2. Model Manipulation: Prompt Injection & Jailbreaking
Companies are spending millions to try and “solve” the issue of prompt injections like SQL injections, with little to no success, because SQL injections were fixed once parameterized queries gave databases a hard, enforced separation between the code and data. This meant the string could never be silently reinterpreted as an instruction. Language models do not have that separation yet.
For now, instructions and content arrive in the same context window, with the model being left without any built-in mechanism guaranteeing it can tell them apart. Moreover, those using prompt injections are incredibly patient, waiting turn after turn, to strike once they’ve established innocuous context.
Because shadow AI can remain so effortlessly separated from corporate accounts is what makes it so difficult to detect. An employee can be logged into their personal AI account and generate no SSO events or corporate login that would tie the session back to the organization’s identity provider. All security tools built around monitoring managed accounts have a structural blind spot for activity that never touched the managed account in the first place.
Moreover, it can depend entirely on the employees themselves. Those who are likely to work around the approval process are usually the most technologically capable ones who know which routes to take if they wish to keep a particular use case off the record. Hence, shadow AI is as much a problem tied to employee skill as carelessness, making awareness or training gaps inefficient.
4. Over-Permissioned AI Agents
Over-permissioned AI agents aren’t born (or created). It’s a slow and gradual process where an agent expands without the necessary reviews. This usually means an agent expected to perform tasks is given the relevant permissions to finish those tasks. This is done “for efficiency” in most cases, with individual permission expansions being approved without a full review of the agent’s whole set of privileges.
While the aforementioned scenario leads to problems of its own, it usually leads to another problem: when an agent with such permissions makes an error, there is no chain of accountability. “Who is to blame?” as the bosses at DPD must’ve thought when they found out what their chatbot had been up to that day.
5. Third-Party & Supply Chain Exposure
It has been a traditional practice where a business seeks to diversify its risk portfolio by spreading that risk across as many vendors as possible. However, in the case of AI use, this usually leads to concentration of risk rather than diversification because a large share of AI tools usually sit on top of the same handful of foundational models underneath. This means any outage, safety incident, or policy change at that one foundational provider expands to everyone connected to that model.
This ripple occurs simultaneously through every downstream tool built on that same layer, even when the tools have different purposes, permissions, and use cases, with little to no connection to each other, either functionally or contractually.
How AI Security Helps Protect Enterprise Models
Here’s how leveraging enterprise AI security helps organizations appropriately protect their models:
a. Securing the Data Layer
Data-layer security is usually framed entirely around the security aspect, and for good reason. However, this comes at the cost of this being done regardless of whether the data in question is confidential enough to be kept away from the model. Provenance is another area that usually gets overlooked, owing less to what the data says and more to where it comes from, along with the permissions attached to it.
An example of this would be the marketing team using an AI tool trained on a particular freelancer’s sample under an ambiguous licensing arrangement won’t necessarily discover that problem through a sensitivity review, because nothing in that content will look confidential. It will become a problem later on, when the product/service in question has already shipped.
This should illustrate the importance of data lineage, not just classification alone. Knowing where an important aspect of context originated can help track down a downstream problem to its source. In short, a sensitivity label alone answers whether something should have been used. Data lineage tells you where to go once you know you’ve used data that you shouldn’t have.
b. Securing the Model Layer
Every instance where the model is fine-tuned, updated, or swapped for a newer version, the security tests run on the previous version stop being useful in describing the model in use. However, there’s a flip side to it, as it means any malicious techniques that may have worked against the older version won’t necessarily work with this one, just as it may lead to vulnerabilities that did not exist in the older version.
Model-layer security only works if the continuous validation aspect of it is tied to each version change, rather than a one-time approval for the model at the initial time of deployment.
A more nuanced version of this problem usually comes up in organizations where they are under the false assumption that they’re running a single internal model when they’re running several unlabeled snapshot variations of it. This is because different teams may have integrated it against a different model at different times, leading to different behaviors and inconsistent security posture across the business units, even when everyone thinks they’re all using the “same version”.
c. Securing the Output Layer
When it comes to the output layer, it seems it’s taken as a synonym for content. In simple terms, stop the model from saying something it shouldn’t. However, this framing misses an increasingly troublesome category of output that humans don’t usually read directly, i.e., structured data a model hands off to another automated system, a JSON object consumed by a downstream pipeline, or a field written straight into a database.
In such instances, a manipulated or malformed output isn’t just a bad answer that an alert human may question; it’s a piece of data that another model or system treats as trustworthy input.
Validating the output format and internal consistency, not just the content itself, matters most when humans may not be in the loop to notice if something is wrong. A bad response from a chatbot can easily be caught; a malformed field is more likely to be silently accepted by an automated finance or inventory system that is fully automated with minimal human oversight.
d. Governance & Policy as a Security Control
Every AI use policy eventually runs into a business use case that presents a compelling reason to break it. In such cases, the exceptions built into that use policy are usually a better test of how effective that policy’s governance is rather than the policy document itself.
A program with a fast, effective, and documented process of granting and tracking exceptions can bend, when necessary, without breaking. However, programs that lack such an exception path don’t only avoid exceptions; they accumulate informal ones until the sum of the undocumented workarounds leads to it being indistinguishable from having no policy at all.
This critical difference is what separates governance maturity from governance paperwork. A thick policy book may tell you what the rules are; a working exception policy tells what’s likely to happen when someone with real business pressure asks for it to be broken, and whether the following decision is made while being documented, or without it.
Five Mitigation Strategies That Work Against AI Security Risks
The most effective mitigation measures an organization can adopt as part of their enterprise security plan are as follows:
a. Classify Data Before It Reaches a Model
Arguably, the best way to know which data shouldn’t be fed into a model for training or fine-tuning purposes is still classifying it as such. Most classification programs operate at the file level, with a label covering an entire document. In most cases, this is a lot more generic than what the AI pipeline usually needs. Think of a 10-page proposal, with just one paragraph that has the sensitive personal information in it buried under mountains of marketing copy. A binary choice here would label the whole file as sensitive and block it from being used. The result? A whole document that could’ve been used is lost.
Hence, passage-level or field-level classification addresses that problem and provides an organization real value from its classification processes by being granular enough to separate the sensitive information hidden inside coarse documents.
b. Monitor For Prompt Injection & Anomalous Behavior at Runtime
The hardest aspect of runtime monitoring isn’t detection; it’s having the right degree of sensitivity appropriate to the deployment. Consider this: a research team has an AI tool that sees all exploratory queries as routine. However, a monitoring system that is tuned as tightly as a customer-facing bot’s would immediately flag all these queries, filling the team’s security queue with false positives. At the same time, a team using the customer-facing bot would need the opposite: zero tolerance, with little to no room for unusual usage patterns.
No single enterprise-wide setting exists that would cater to both effectively. Hence, the right threshold is something that needs to be discussed at the departmental level and depends entirely on that department’s specific deployment needs, rather than a company-wide policy decision.
c. Vet Vendors & Models Before Adoption
Though they’re still of great use, it would be unwise to place blind trust in performance benchmarks and published safety standards. This is because the model providers know exactly which benchmarks are likely to be cited in procurement conversations, and all performance can be fine-tuned to deliver on those. Hence, a model that does well in a public safety benchmark demonstrates it can handle the benchmark’s specific scenarios well but says little about how it handles the specific workflow scenarios in each organization.
As a result, the more reliable form of vetting would be to run an individually tailored adversarial evaluation against the organization’s own use cases before any adoption. The results can be matched against the vendor’s published scorecard measures. This will give more context to such numbers while also providing necessary information in the final procurement decision.
d. Set Clear Policies for Employee AI Use
It would surprise organizations how quickly policy documents can become obsolete, even within a few days. Consider an organization's devices: a comprehensive policy for one model, and within 24 hours of that policy becoming effective, a new iteration of that model is released to the public. Annual or even quarterly policy review cycles are far too reactive and built for a category of tech that didn’t change substantially in such short periods.
The fix isn’t a shorter and sharper policy with weekly policy reviews. This situation demands building a new review cadence where the reviews match the frequency of how these tools evolve. This can be different for each organization but can help organizations avoid a critical gap between policy and reality.
e. Align Practices with Emerging Regulation
At the end of the day, an organization’s entire operation can hit a roadblock if it isn’t compliant with regulatory expectations. And as recent trends have shown, countries have become strict in both their legislation and in establishing financial consequences for breaching these laws. The AI Act is the perfect example of this, as it places obligations upon not only deployers but also developers. Meaning an organization simply buying a third-party AI tool can inherit the high-risk obligations based on how the tool gets used, regardless of who built or designed the underlying model.
This means the entire organizational workflow needs a re-evaluation, from procurement to employee training, as such operations affect an organization’s overall ability to be and remain compliant with mounting regulatory expectations globally.
How Securiti Can Help
Securiti, a Veeam company, is a global leader in providing DataAI privacy, security, governance, and compliance solutions that are relied on by some of the world's leading enterprises.
Its plethora of modules can enables organizations to discover, classify and label all sensitive data elements in multi-cloud and self-managed structured and unstructured data systems at a petabyte scale. Data lineage capabilities help teams gain granular insights into the changes and transformations the data goes through during its entire lifecycle, while data access intelligence and governance provides real-time details into user access settings around data systems and the data within these systems.
Each of these capabilities can prove instrumnetal in the identification of potential liabilities and the enforcement of high-level policies that prevent unauthorized access.
Moreover, its Data Security Posture Management (DSPM) is a comprehensive solution that finds every AI asset running across the enterprise environment and classifies the sensitive data being fed into it.
Request a demo today and learn more about how Securiti, a Veeam company, can help you implement effective enterprise AI security across your organization.
Frequently Asked Questions About Enterprise AI Security
Some of the most asked questions about enterprise AI security are as follows:
![[Knowledge Center] Enterprise Readiness Checklist](https://media.securiti.ai/wp-content/uploads/2026/06/08002712/Thumbnail-Enterprise-Readiness-Checklist.webp)
