LLM security for Enterprises: Risks, Framework & Best Practices

In a now-notorious case of consequences awaiting organizations that ignore appropriate LLM security and guardrails, some engineers at Samsung turned to ChatGPT to solve what they thought was a straightforward problem. However, in doing so, they pasted snippets of proprietary source code. Within weeks, Samsung had banned the use of GenAI tools entirely across the organization.

To this day, it remains a widely cited example of what happens when enterprise data meets a public AI tool without the appropriate safeguards. More importantly, it is also a perfect example of why LLM security needs to be treated as a distinct discipline, rather than an afterthought being bolted onto the existing IT policy.

This is because the Samsung example hasn’t been an isolated incident. A 2025 report indicated more than three-quarters of employees have copied and pasted some form of enterprise data into a chatbot query. Some of that data has included personal and sensitive information that was supposed to stay private and encrypted on the organization’s servers.

Enterprises hoping to build their own applications on top of existing LLMs must understand the extent of exposure they expose themselves to when their employees interact with these models. Every prompt, dataset, and connected data source can contribute to the large attack surface. This is further compounded by the fact that LLMs do not fail as traditional software does; they can be manipulated by malicious actors to reveal information and perform unintended actions long before anyone notices any anomalies.

This is precisely the gap LLM security for enterprises must address. Read on to learn more about what this means in practice, what risks enterprises are most likely to run into when leveraging LLMs, and the best practices they need to adopt in pursuit of LLM security.

Why LLM Security Matters

In what seems like ancient history at this point, security risks before GenAI had a fairly predictable shape. A typical scenario usually played out like this. There would be a firewall breach in the form of a phishing email or a misconfigured server. The mitigation measures to counter such threats were similarly simple. However, LLM security means dealing with an entirely new and different kind of problem: a system that fails because it operates exactly as it is supposed to work, i.e., generating an answer that is wrong, harmful, and exposes something it shouldn’t. This is precisely why LLM security necessitates moving from a simple technical footnote to a board-level conversation.

Consider the case of Air Canada. Its online chatbot gave a customer inaccurate information about bereavement fares. Air Canada argued the chatbot needs to be considered a separate entity responsible for its own statements. This was rejected by the tribunal hearing this case, which held the airline responsible for all information provided by the chatbot, similar to how it would be held accountable for any information on its own website. The customer was ultimately awarded a little over $800 in damages, but the real damage wasn’t financial but reputational. Moreover, it set a legal precedent. Enterprises deploying LLMs in any customer-facing role must learn to reframe their question from “is our AI helpful” to “can we stand by what it tells people” when evaluating its deployment.

IBM’s 2025 Cost of a Data Breach Report found that organizations with high levels of unsupervised AI use saw a $670,000 increase in their overall cost of a breach. This is further compounded by the fact that organizations that do experience an AI-related security incident usually lack proper access controls around AI systems. This isn’t the cost of facing sophisticated or novel attack techniques; it’s the price lethargic organizations are left to pay for deploying LLMs faster than the relevant controls necessary to govern them.

LLM Enterprise Application Risks

Most of the risks when it comes to LLM applications aren’t unique to a single vendor or use case. These are widespread issues across industries, further highlighted by their alignment with the OWASP Top 10 Security Risks for Applications.

Prompt Injection

Prompt injection refers to when an attacker successfully crafts an input designed to override a model’s internal instructions. Doing so leads to the model ignoring its built-in guardrails and acting on hidden commands. These can happen either directly through the text a user types straight into the model, or indirectly, through instructions buried inside a document, webpage, email, or any other source being fed to the model.

Training Data Poisoning

Training data poisoning occurs when an attacker successfully manages to introduce manipulated or malicious data into the model’s training or fine-tuning pipeline. This is because models learn patterns through the data they’re fed. In such instances, even the smallest amount of poisoned data can transform how a model responds under specific conditions, unless triggered by a particular command.

Model Theft

Model theft refers to attempts at stealing or reconstructing the model’s internal underlying weights, architecture, or behavior. This can be done either directly by extracting the model file or querying it repeatedly and using the responses as an approximation. The latter technique is also known as model extraction and does not require a direct breach of the system, just enough queries and patience.

Insecure Output

Insecure output refers to when a model’s responses are passed along to the user, application, or system without appropriate checks in place. This occurs when the model’s outputs are treated as trustworthy by default. However, since the models can generate code, commands, or structured data, there is always the chance of an unfiltered output ending up being executed in a way that leads to a vulnerability being exploited.

Adversarial Attacks

Adversarial attacks involve the crafting of inputs that are deliberately worded or designed to confuse a model, bypass its safeguards, and push it towards undocumented behavior. These inputs are subtle and small variations in wording or formatting that are difficult for a human to notice but, once executed, can shift how a model interprets the request.

Compliance Violations

Compliance violations occur when an LLM processes, stores, or generates data in a way that runs against regulatory requirements, such as mishandling personal data, retaining information longer than permitted, or producing output that a regulator treats as improperly generated. Unlike most other risks on this list, this one does not need a model to be compromised for a violation to occur, as even the smallest governance mistake can lead to it.

Supply Chain Vulnerabilities

Supply chain vulnerabilities arise when using components that the enterprise didn’t build itself. This means pretrained models, open-source libraries, and third-party APIs that all get integrated into an LLM application. If these components are compromised or poorly maintained, the risks can be transferred directly onto the organization’s own system.

6 Best Practices For Securing LLMs

At the end of the day, knowing the risks is still only half the work. Once all the relevant risks are identified, processes and practices need to be put in place to mitigate them. To that end, the following six practices ensure that all the aforementioned risks are appropriately addressed.

Input Validation

Any prompt being fed into the LLM remains unproven and untrusted until it can be demonstrated otherwise, regardless of where it comes from. Input validation refers to checking whether the prompts fulfill certain expected parameters and do not contain any suspicious patterns. This allows for user input to be separated from system instructions, ensuring there can be no confusion between the two.

When it comes to dealing with prompt injection, this is arguably the most effective way to do so, especially for the indirect kind of attack where instructions are hidden inside mountains of regular webpages, email attachments, or other kinds of files being fed into the model.

Output Filtering & Monitoring

A similar degree of scrutiny must be given to what the model produces. Hence, output filtering checks all responses before they can reach users or downstream systems. In the meantime, all outputs are screened for unintended data exposure, harmful instructions, or content that violates internal and regulatory policies before leaving the model boundaries.

This addresses the issue of sensitive information disclosure directly, as there can be instances where the model generates a problematic response even when the original prompt contained nothing sinister. Moreover, it also helps organizations identify patterns, with multiple flagged responses usually being a sign of some deeper underlying issue.

Data Governance For Training & Fine-Tuning Data

The data a model is trained on determines everything about the model, i.e., what it knows, how it behaves, and most importantly, what biases or vulnerabilities it exhibits. Governance in this context means tracking exactly where all this training data comes from, verifying its integrity, and restricting how new data is introduced into the training dataset and subsequent pipeline.

Without such a guardrail, poisoning the data a model is to be trained and then fine-tuned on would be easier, with the attacker only needing to influence one dataset.

Access Controls & Rate Limiting On Model Endpoints

A model that is in reach by anyone, with no controls related to how often it can be queried, can be exploited by malicious actors for both theft and abuse. Strong access controls not only limit who and what can reach a model but also exactly what actions they can perform once they do have access, with limits related to the volume of requests a single user or system can send.

This not only drastically reduces the chances of model theft, where repeated queries are used to map out a model’s behavior and its underlying weights, but also adversarial attacks that use large volumes of input queries being sent until one succeeds.

Third-Party & Model Provenance Review

Most enterprises do not build every part of their LLM stack. They rely on pre-trained models, open-source components, plugins, and external APIs. In such instances, third-party or provenance reviews allow for verification of the origins of each component, who maintains them, and whether there was any tampering before they were integrated into the model.

This helps in addressing all supply chain vulnerabilities directly that would’ve been overlooked owing to the origins of the component not being in the enterprise’s control.

Continuous Red-Teaming & Adversarial Testing

Red-teaming refers to the deliberate attempts at breaking the model, by the enterprise itself, through prompt injection, jailbreak attempts, and edge-case inputs. All this helps in the identification of weaknesses before an actual attacker can exploit them. Such activities are run consistently, with a model required to pass each assessment to ensure all identified and non-identified vulnerabilities can be addressed before new tools or real-world use cases can be introduced.

Such tests allow for practical tests of an organization’s protective measures and frameworks and allow for a real-time view of the effectiveness of these measures and the scope for further improvements.

Enterprise LLM Security Framework

The aforementioned processes and steps work best when they’re built into every stage of the model’s life. This means from the moment training data is introduced to the point where the model is eventually phased out and retired. Every stage has its own problems and risks, with selective controls rarely compensating for the gaps left in others. Hence, the following five stages enable an organization to have a practical way of developing a framework that covers the entire lifecycle.

Design & Data Stage

Enterprises must decide where the data needed to train and fine-tune their models will come from. This means decisions related to data sourcing, licensing, and provenance will shape the entire data processing lifecycle of an organization. This elevates the importance of this stage, where the organization must filter out all sensitive or unauthorized content, document data lineage, and have documentation related to which specific datasets were part of its training phase.

Skipping this step means data poisoning is not only possible, but its chances are also significantly heightened as the initial guardrail against it does not exist. Any model trained on data an enterprise cannot fully account for is not only going to be difficult to remediate later on but can lead to severe repercussions down the line with customers, regulators, and partners alike.

Development & Testing Stage

After a model is built, an enterprise must test it against all the ways it can plausibly be misused. Structured red teaming and adversarial testing help in this particular aspect through prompt injections, jailbreak attempts, and edge-case inputs to expose any weaknesses, while they’re both inexpensive and easy to fix.

However, this shouldn’t be a static and one-time exercise, as it needs to be repeated every time the model is updated, fine-tuned, or connected to new tools. This is because each instance creates a potentially new vulnerability that earlier tests did not test against.

Deployment Stage

Once a model is live, organizations must build strict boundaries around how it can be reached and what its task-related permissions are. This means authentication on model endpoints, rate limitations, scoped access based on RBAC, and filtering on both inputs and outputs.

This particular phase is when model theft and adversarial attacks are the biggest concerns, as they turn from theoretical challenges to live ones with the deployed model being exposed to real traffic, attackers, and third-party integrations. All of these expand the attack surface beyond what was tested during the development and testing stage.

Monitoring & Response Stage

A model must consistently be monitored, especially during its deployment. This should cover logging prompts and outputs, watching for unusual usage patterns, and setting up alerts for signs of manipulation or misuse that went undetected during the testing phase.

Additionally, organizations must have a response plan in place for what to do when something is flagged. This allows for detection to lead to instant action rather than simply becoming a log file, and prevents an immediate threat from causing too much damage in the interim.

Governance & Compliance Stage

This runs concurrently with all the aforementioned four stages as it covers the various policies, roles, audit trails, and governance mechanisms an organization adopts related to its model usage. This includes documentation for all model decisions, maintenance of records of tests and other activities, as well as relevant controls put into place as part of regulatory requirements or internal considerations.

Moreover, this helps in addressing any and all compliance-related violations directly by treating LLMs as part of the organization’s overall governance structure rather than a standalone project. Without this stage, the technical controls necessary would still exist, but the organization would not have an effective or efficient way to prove they do.

How Securiti Can Help

While adopting and implementing an LLM is now a strategic necessity for businesses, properly securing these LLMs has become a critical challenge for organizations in itself, not because of its complications but because of a lack of clear understanding about where to begin.

AI-related regulations have slowly begun coming into effect, and in a few years, LLM security will be both a strategic and regulatory requirement for businesses. Securiti, a Veeam company, has the perfect solution for organizations that find themselves in this predicament.

Its Data Security Posture Management (DSPM) solution provides organizations with intelligent discovery, classification, and risk assessment of all their data. Such intelligent discovery and classification capabilities are vital in ensuring LLMs only receive data that they are supposed to receive and nothing that is strictly sensitive or IP.

Additionally, it enables real-time documentation of various assessments to ensure organizations can continuously record proofs of their compliance without compromising on productivity or continuity of operations.

Request a demo today and learn more about how Securiti, a Veeam company, can help you organize your organizational LLM usage in a way that complies with both regulatory and operational standards.

FAQs About LLM Security For Enterprises

Here are some of the most commonly asked questions you may have related to LLM security for enterprises.