Introduction
It’s difficult to overstate the transformative impact that AI has had on businesses over the past couple of years. With the advent of Generative AI (GenAI), specifically large language models (LLMs), there has been a monumental escalation in both productivity and innovation. These aren’t isolated cases of one-off usages, either. A 2024 Microsoft survey indicated that more than 75% of knowledge workers have adopted GenAI tools as part of their daily operations.
This paints a portrait of a modern enterprise in which these LLMs have effectively become ubiquitous business co-pilots rather than experimental forays into new technology. However, using these LLMs introduces a similarly unprecedented degree and scale of cybersecurity and governance risks.
Unlike most traditional software, LLMs are still relatively black boxes, with their input processes both unpredictable and data-hungry. This unpredictable nature can often lead to catastrophic consequences for organizations. Google learned this exact lesson the hard way when, upon its release of Bard in 2023, a single incorrect answer in a demo led to an almost $100 billion decrease in Alphabet’s market value.
Read on to learn more about the unique AI challenges enterprises face regarding LLM deployment and usage, as well as a practical 13-step security and governance checklist that can help address the aforementioned challenges.
Unique Enterprise AI Challenges
1. Non-Deterministic Outputs
LLMs generate probabilistic responses. Consequently, the same input entered will likely produce multiple outputs at different times, all of them valid. Such models are non-deterministic, in other words, unpredictable in what outputs they generate. This unpredictability has been engrained within these models by design to ensure they are able to produce a variety of different outputs, both creative and customized, to accommodate any other specifications or instructions by users.
At the same time, this unpredictability can be a significant problem within an enterprise context, as businesses rely on consistent outputs for both reliability and accountability. For instance, an LLM that has been leveraged as a customer support bot must keep its answers to policy questions consistent. Variance in this context would erode both reliability and customer confidence.
Most GenAI models rely on immense neural networks that do not follow the traditional “if-then” frameworks. Their knowledge related to creating and presenting outputs is derived from the statistical patterns in the training datasets. Randomness is incorporated into these outputs to minimize repetitiveness. Furthermore, most LLMs operate based on probabilistic token predictions rather than hard-coded logic, meaning even the developers of the LLMs themselves cannot present exact output for any given prompt.
In business applications, such non-deterministic behavior can complicate governance matters. Regardless of how many quality checks are run, a single unexpected output can undo everything. Think of an email completion tool that can consistently write friendly emails, only for one day to send an inadvertently offensive email. Governance issues aside, it can cause a PR nightmare. Most importantly, there is the issue of customer trust. Recent research has indicated how users permanently lose confidence in a model if they detect an early error or inconsistency. A phenomenon that has become so recurrent that it has its own name, “algorithm aversion”.
2. AI Hallucinations
Uncertain outputs are one thing; hallucinations exacerbate all their issues tenfold. It is also a widely reported “quirk” of LLMs to hallucinate, i.e., produce statements that sound both objective and factual but are actually false or completely fabricated. These hallucinations can be in the form of a fictitious product name, an invented citation, or, in some cases, a completely bogus answer.
But why do hallucinations occur? This is due to the fact that LLMs don’t “know” facts; they rely on pattern matching done through the training datasets. If these datasets are flawed or even insufficient, chances of hallucinations increase, with the model filling in the gaps in its knowledge with false information.
Naturally, hallucinations are a direct and significant threat to both the reliability and trust of the LLMs. The use cases where a hallucination may occur are far too many. It can range from something as trivial as false weather information to something as serious as denying documented historical events. A New York attorney received this unfortunate lesson the hard way when he fed case files into an AI tool and the model invented fictional case citations, nearly derailing his entire court filings and the case itself, on top of a $5,000 fine for filing those fake citations - real-world consequences meet AI hallucinations.
Business professionals consider such hallucinations to be, by far, their top concern about GenAI. Minor errors and glitches in betas are one thing; highly sophisticated models designed to assist decision-making resulting in such errors are quite another. As recent studies have shown, once a tool loses its trust among users, it will go unused, regardless of how expensive it is or how many improvements it purports to have made. Yet another phenomenon with its own name, “shelfware.”
3. Expanded Cybersecurity Attack Surface
While integrating LLMs into enterprise workflows, products, and services elevates productivity and efficiency, it does leave them more susceptible to new vulnerabilities, particularly AI-specific ones.
Organizations face multiple AI-specific threats, and leveraging AI capabilities means confronting and mitigating those threats. Since most traditional apps do not carry such threats, this usually means creating an AI-threat mitigation strategy from the ground up, which is expensive, complicated, and can often be a logistical nightmare.
Some of the AI-specific threats are deviously simple and yet can render the entire model, and in some cases, every connected app, compromised. Malicious actors have been able to use prompt injection attacks to extract confidential information from the model itself, data poisoning to disrupt the model’s learning processes and plant the seed for future malicious outputs, and model theft and extraction where the actors may steal the entire model itself through API abuse or by obtaining the model file from an insecure storage.
Combined with some of the more traditional data and organizational threats, it becomes hauntingly clear how the exposed “organizational digital real estate” is greater than ever before, requiring unprecedented attention and resources to ensure all such threats are appropriately identified, addressed, and continuously monitored to avoid any unforeseen surprises.
4. Advanced Attacks Leveraging AI
Yes, AI usage has helped organizations raise their productivity, revenues, and overall operational efficiency. So have cybercriminals. Malicious actors not only target AI systems but weaponize them as well. If that wasn’t a big enough problem on its own, there are Ransomware-as-a-Service (RaaS) rackets where cybercriminals can effectively outsource cyberattacks. Leveraging AI capabilities, not only are modern cybersecurity threats more frequent, but they’re also more potent.
Using AI capabilities, traditional attack methods such as phishing and DDoS attacks can be carried out at near-infinite levels of frequency and scale. LLMs can be used to author more effective malware code continuously, there are even GPT services catering specifically to the needs of cybercriminals. Using a combination of these, it is easy to see how malicious actors can create highly believable and compelling social engineering attacks, such as spear-phishing emails that are immune to some of the traditional telltale signs of fraudulent or phishing attacks, such as poor spelling or odd phrasing.
Add in deepfakes, where an individual’s voice or even entire videos can be fabricated to create an elaborate scam, and the dystopian nature of it becomes increasingly clear.
A Practical 13-Step Cybersecurity & Governance Checklist For LLMs
1. Adversarial Risk Management
The most immediate step an organization must take as part of its AI risk mitigation strategy is to integrate AI-specific risks into its risk management frameworks. This involves putting yourself in a potential cybercriminal’s shoes and thinking from their perspective while also deploying effective countermeasures to all identified risks. NIST’s AI Risk Management Framework is a highly reliable reference to use in this instance, as it emphasizes implementing documented processes to “map” and “manage” all perceived AI risks in a continuous manner.
Additionally, MITRE’s ATLAS (Adversarial Threat Landscape for AI) or OWASP’s emerging Top 10 for LLMs catalog, known attack vectors, are also dependable knowledge resources that can be leveraged as checklists to determine an organization's exact threat profile. Regardless of whether an identified threat is high-risk or low-risk, each scenario must be appropriately considered, and a countermeasure process must be put in place to thwart the threat.
This is where a strong data and AI governance strategy becomes important, as it dictates the internal practices that will be crucial in the entire AI risk management exercise.
2. Threat Modelling
Threat modeling involves the systematic identification of potential attacks and attackers, their goals, and most importantly, the weaknesses they may exploit in the AI system’s design. This process should cover the entire AI pipeline, i.e., the data ingestion, model training, model deployment, user interaction, and output integration. The analysis should comprehensively cover each stage, assessing all possible weaknesses and vulnerabilities. Every possibility related to manipulation or misuse must be addressed. Moreover, this should not be a static exercise but rather a consistent process.
In this process, a cross-functional team must be involved in the constant evaluation of threats. There should be a specific method for their operations. For instance, they can leverage the standard threat modeling methodologies, such as STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege), and adopt them in an AI context. Once identified, mitigation strategies can be developed and put into place, with these strategies subject to similar assessments concurrently.
3. AI Asset Inventory
An AI asset inventory can be extremely helpful in the overall governance of all AI resources. Shadow AI assets are a significant problem for organizations. It isn’t uncommon for an enterprise to discover undocumented or “rogue” ML models or datasets within its infrastructure that aren’t supposed to be there. These can include models or datasets that a team may have built for an isolated experiment, deployed for a beta test, and then moved on to using the insights gained for other projects. Hence, a method to “track” and keep all AI-related assets properly accounted for is a major part of any effective AI governance policy. The NIST AI RMF explicitly calls for the need to “inventory AI systems” as part of an organization’s overall governance strategy.
Details in such an inventory must include but do not have to be limited to, key details such as the purpose of the model, the dataset it was trained on, the current version, where it’s deployed, the owner or responsible team, and what criticality or risk tier it falls under based on the regulatory obligations or any other internal metrics being adopted. It can also include all external AI tools being used by employees, which can be tracked via an “AI component CMDB (Configuration Management Database)” as well as integrations. Several LLMs are available embedded within apps. An inventory that documents such context allows for a better assessment of the threat profile.
4. AI Security & Privacy Training
Arguably, the most straightforward step organizations can take in countering AI or really any other threat. Yet, it remains the most overlooked one, as apparent in the fact that insider threats and negligence continue to be a major reason for data security breaches within organizations.
Technology alone is not sufficient to secure AI properly. Employees who develop and interact with LLMs must be made aware of the risks associated with these LLMs and their role in mitigating these risks. Many employees do not understand the scale of what could go wrong, leading to an often lax attitude and organizational culture regarding AI usage.
Effective, regular, and comprehensive training ensures that everyone, from developers and data scientists to marketers and salespeople, thoroughly understands the dos and don’ts of AI use. Any such training program can include resources on good digital and AI practices, limitations of the AI resources they’re working with, common threats they may face, specific threats they may face based on their roles or level of access, and their responsibilities within a regulatory compliance context. Various departments can collaborate on the development and execution of such programs, allowing for tailored modules and resources for the different personnel.
Simulations of phishing attacks and other forms of possible attacks can also be run to assess the overall readiness and awareness of the employees
5. Establishing Clear Business Cases
It is almost becoming a cliche on its own, but not every task requires AI. While AI’s benefits are unquestionable, there must always be a clear business justification and value assessment for each LLM deployment. Herd mentality is becoming an increasingly cumbersome issue in terms of AI adoption, leading to an elevated risk of AI implementation just for the sake of it, rather than for a defined purpose. In such cases, the true implications of AI use are not understood, leading to future issues of governance and a litany of other issues. AI initiatives must be treated like any other initiative within an organization, with questions being raised related to their objectives, success criteria, and risk-benefit analysis. In other words, the Right of Use (RoU) assessment ensures that any AI use appropriately aligns with the strategic goals of the organization.
Furthermore, organizations may consider tying LLM projects with individual business cases. Not only does this allow for resource prioritization, but it also helps in avoiding the aforementioned herd mentality. There are countless examples of organizations that started their AI pilots simply because they felt they had to, rather than as a means to solve a pressing need where AI allowed for a more efficient resolution. By tying LLM projects with individual use cases, organizations can ensure objective assessment criteria in later evaluations on whether they delivered on the expectations.
If yes, what improvements can be made, and if no, why did it fail? These insights are critical in understanding the context and the major stakeholders involved in enterprise AI use. Identification of the relevant stakeholders allows for individual requirements that can also be used to ensure AI use is purposeful.
6. Governance for AI Systems
Enterprises must define the structures, roles, and processes that oversee their AI deployment across the board. Additionally, the traditional IT governance structures do not cover AI concerns such as ethical usage, model bias, or continuous model updates. This exacerbates the need for an AI-specific governance framework that adequately addresses these issues while also ensuring accountability and alignment with the organization’s internal policies.
Some key elements of the AI governance framework can include:
- A governance body/committee that sets policies, reviews major AI initiatives, and handles all identified issues and escalations. This body can include all the relevant stakeholders from all the organizational departments and should be answerable to the CAIO or other C-level executive responsible for overseeing AI governance;
- Organizational principles on AI usage within an organization that both comply with and align with external frameworks such as the OECD AI Principles. These policies must cover critical aspects of AI use, such as human oversight over AI decisions;
- Documentation of all AI systems, with appropriate model cards and risk assessments, that create an extensive record of the entire decision-making process involved in the development, testing, and deployment of the AI systems. Such documentation is invaluable in internal audits as well as regulatory compliance.
- Review procedures behind approvals, testing, and validation of AI systems before they are deployed, including post-deployment assessments. Additionally, there must be procedures for retraining and updates of AI systems, with governance checks to ensure their continuous compliance.
7. Legal Considerations
AI usage raises a litany of legal questions, issues, and challenges for an organization. These range from intellectual property rights to data protection. This highlights the importance of the legal department being involved from the outset in an organization’s AI deployment. Embedding legal considerations and review into the AI lifecycle can not only save an organization from legal troubles down the line but also help establish clear guidelines to ensure both developers and users are aware of the legal boundaries within which to operate.
Some of the more important considerations include IP rights, accountability of usage, data privacy & protection, and vendor contracts & licensing. Each of these poses a unique set of challenges for the organization. However, these considerations must be appropriately considered and assessed by the AI governance body within an organization, with legal counsel in an advisory role to identify any legal deficiencies in the organization’s AI deployment and usage and help plug any such gaps. Furthermore, they can also aid in the development of usage guidelines, ensuring legal compliance is woven deeply within the AI projects and guaranteeing the longevity of each initiative from a compliance perspective.
8. Navigating Regulatory Compliance
This may very well be the most important consideration for enterprises when assessing LLM deployments. AI regulations are proliferating across the world, and if that weren’t a formidable compliance challenge on its own, various industries have their domain-specific frameworks, standards, and regulations related to AI use. Non-compliance not only results in monetary fines, but the reputational damage on its own can irreparably cripple an organization’s brand image and trust in the eyes of the customers.
At the same time, avoiding such a grim scenario only requires regulatory compliance by an organization. Doing so is complicated but not impossible. Whether it's data privacy laws, industry-specific guidelines, or upcoming regulations, the organization must have its legal team and the AI governance body consistently on top of the requirements these place on their AI usage. Additionally, extensive close collaboration between all the teams using and developing AI tools is required to conduct compliance assessments for all the identified AI use cases. If necessary, external audits and certifications can also be considered to assess an organization’s overall compliance status.
9. Secure Deployment of LLMs
An LLM becomes accessible to users and other systems at the deployment stage. In other words, this is the exact stage where the security of the AI system is tested since it is open to attacks. A secure deployment process ensures that the overall infrastructure running the LLM and the embedded integrations are all locked down as much as possible and shielded from any potential attacks.
There are various aspects and considerations organizations may take into account and deploy as part of their LLM deployment, such as strict access controls to individually determine which users and systems have access to the LLMs and to what degree, secure infrastructure to leverage all the appropriate security features such as VPC isolation, customer-managed encryption keys, etc., input/output validation to catch injections, banned content, and other loopholes that may be exploited, continuous monitoring and logging of the LLM’s interactions that can be useful in future auditing, debugging, and forensic analysis if necessary, and lastly, a fail-safe mechanism to have as a bulwark in case the LLM is compromised and continuity of service is a priority, even if it is in the form of a non-AI functionality.
10. Continuous Testing, Evaluation, Verification, & Validation (TEVV)
Implementing a continuous TEVV methodology for LLMs ensures their safety, effectiveness, and reliability over a period. LLMs are a unique technological phenomenon as they evolve and change over time, sometimes within hours, with updates and new datasets being incorporated. Hence, there needs to be a “continuous” process in place that can adequately test, evaluate, and verify the model and, subsequently, validate its performance against its stated purpose.
Key aspects of a TEVV methodology include automated testing pipelines that continuously test a model whenever it's updated through a combination of regression tests and other scenario tests, evaluation metrics that define assessment criteria for an LLM’s performance and track them continuously, human feedback loops where domain experts evaluate a sample of the outputs produced by a model at random for its quality, and validation of the model’s ability to continuously deliver high-quality outputs over a sustained period to eliminate possible degradation in performance.
11. Model & Risk Cards for Enhanced Transparency
Documenting an LLM via model and risk cards, also known as risk registers, allows for both transparency and accountability. Model cards are akin to a nutrition label where important details such as the model’s intended use, architecture, training data description, performance metrics on various benchmarks, ethical considerations, and limitations are described. Similarly, risk cards list down all the identified risks along with the mitigation measures taken to alleviate them.
For an enterprise, these cards can be invaluable as they offer an important overview of the model’s capabilities and limitations for those who may not be well-versed in its technical aspects. Additionally, they form a vital foundation of the organization’s AI risk management process by documenting the risks and mitigation measures while also informing the important stakeholders of what to watch out for. Furthermore, these cards can be updated each time a new model is deployed or an update is made for continuous documentation.
12. Leveraging Retrieval-Augmented Generation (RAG)
RAG is a reliable method to improve the factual accuracy and compliance of an LLM’s outputs. It involves supplementing an LLM with a knowledge retrieval system that allows the model to fetch relevant documents/data from an external source (like a database or search index) rather than relying solely on its pretrained parameters. It then generates a response leveraging both the prompt and the retrieved data. In an enterprise setting, RAG highly mitigates instances of hallucination while also providing data freshness that elevates the overall performance of the LLM in general.
Moreover, it allows for domain specificity without requiring the LLM to be retrained from the ground up. Enterprises have troves of documents such as manuals, reports, intranet pages, and knowledge bases that RAG can effortlessly leverage to empower even a general model to become an expert on an enterprise’s internal knowledge, as opposed to having an LLM trained from scratch on such proprietary data. Such a process is traceable and is effective in building users’ trust, as they’re more likely to trust an AI’s answer when it cites a credible source while also providing a link to it.
13. AI Red Teaming - A Standard Practice
In any case, AI red teaming and stress-testing should be a routine part of an organization’s AI security and quality assurance process. Such tests go beyond the traditional forms of testing as they’re highly aggressive and ensure they behave similarly to malicious actors in pushing the AI’s defensive mechanisms and processes to their limits, thus elaborating its true capabilities and effectiveness. Done properly, it can lay out any model’s vulnerabilities and allow for their mediation proactively.
Such tests can be designed across various contexts and different settings to assess the model’s performance across various vectors. Moreover, frequent tests provide a consistent overview of the model’s performance. They can also be scheduled if something significantly changes, such as a new model adoption, updates, or a model’s integration into a new workflow.
How Securiti Can Help
AI security will become an increasingly important consideration for businesses as their AI usage and subsequent applications surge. These AI-enabled capabilities can only be effectively leveraged if all bases are covered, which includes, perhaps most importantly, data+AI security. However, ensuring the appropriate protection for AI and data resources can be a significant challenge in its own right.
This is where Securiti can help.
Securiti’s Gencore AI is a holistic solution for building safe, enterprise-grade generative AI systems. This enterprise solution consists of several components that can be used collectively to build end-to-end safe enterprise AI systems or, in various other contexts, to address various AI use cases.
This enables an incredibly effective yet simplified enterprise AI system through comprehensive data controls and governance mechanisms that mitigate all identifiable risks proactively. Furthermore, Securiti’s DSPM can be leveraged to ensure appropriate data protection, thereby ensuring compliance with regulatory requirements and industry standards.
Request a demo today to learn more about how Securiti can help you safely build enterprise-grade AI systems.
