The Dangers of Uncontrolled AI: Shadow AI and Ethical Risks

We're in the midst of a generative AI revolution that's reshaping how we approach business. It's imperative to recognize the potential hazards of uncontrolled AI as we seek to harness its benefits while maintaining safety and trust. To do so, businesses must prioritize transparency, safety, and security in their AI models to ensure ethical and legal compliance, fostering trust with customers.

Why Uncontrolled AI is a Recipe for Trouble

Integrating AI services into enterprise data models requires careful control and oversight over the entire AI lifecycle, spanning from creation to deployment. This is essential to reduce risk around security breaches, compromised data privacy, legal violations, and damaged brand trust. Yet an alarming gap exists between adoption and governance. A September 2023 survey from The Conference Board shows that over half (56%) of US workers are using generative AI technologies on the job, and a survey by ISACA indicates that only 10% of organizations have a formal generative AI policy in place.

And so we also enter the era of uncontrolled AI, in which AI governance becomes an increasingly vital priority for businesses that want to integrate AI models safely and transparently while driving positive business impact and meeting legal and ethical requirements. Without the right controls and oversight in place, enterprises encounter a series of risks that can turn their quest for innovation and efficiency into a compliance and security calamity. Here are just a few of those dangers.

Shadow AI is Already Here

In this rapidly changing environment, the race to innovation is more competitive than ever — and privacy and security risks are more relevant than ever. As companies strive to achieve business goals via the expeditious incorporation of AI, those very same organizations are still figuring out what their AI posture will be.

Without complete visibility into all AI systems, deployed internally or through SaaS, hidden models operate with unknown risks that can lead to astronomical costs down the line. To intensify the problem, shadow AI shows signs of proliferating at a faster rate than the parallel challenge of shadow IT that has beset security and governance teams for decades — and continues to.

Unidentified Risks Pave the Way for Unwanted Consequences

A lot of questions still exist around the use of AI — and many of them involve challenges that enterprise security teams have never encountered before.

Because AI models are not just a function of the model’s code or the data that flows through it, but also the logic that “learns” from that data, the output is more of a moving target than many orgs are used to. Blindness to potential risks like bias, discrimination, and "hallucinatory" responses can cause serious setbacks for security, data, and compliance teams, increasing the likelihood of ethical violations and reputational damage. For example, Navy Federal Credit Union was recently subjected to a lawsuit over allegations of racial bias in their mortgage lending practices.

Data Opacity Fuels Privacy Concerns

Organizations have long grappled with data transparency challenges in the world of privacy and security — and the same issues arise in the use of artificial intelligence.

Data generated by AI models is often cloaked in obscurity, raising questions about its origin, use, and accuracy. This unclear data usage lurking in AI models and pipelines raises doubt around entitlements and exposes sensitive information to potential leaks, derailing compliance efforts and exposing enterprises to a world of uncharted vulnerabilities. For example, a leading consumer electronics company banned ChatGPT among its employees after a sensitive code leak happened.

Unsecured Models Create Vulnerabilities

As the use of AI expands, the need to implement data controls on model inputs and outputs also increases.

Sensitive information that is both put into and generated from AI models must meet compliant data protection and privacy standards. Lack of security controls leaves AI models open to manipulation, data leakage, and malicious attacks. Organizations that want to avoid data breach incidents do not have the luxury of making AI security an afterthought; doing so poses a threat to the integrity of the enterprise and the reliability of the brand.

Uncontrolled Interactions Invite Abuse

Unguarded prompts, agents, and assistants open the door to harmful interactions, threatening user safety and ethical principles.

It's crucial to understand how the data generated by these models is being utilized — whether it's being shared in a Slack channel, integrated into a website as a chatbot, disseminated through an API, or embedded in an app. Moreover, these agents, while serving as channels for legitimate queries, also become potential pathways for new types of attacks on AI systems.

Regulatory Violations Risk Ethical and Legal Minefields

Globally, policymakers are perking up, paying attention, and taking action on the safe, secure, and trustworthy use of AI. The EU was the first to put a comprehensive AI law on the books with the aptly named AI Act, and several other nations promptly followed suit, with China, the UK, and Canada proposing or enacting AI legislation. Even the Biden-Harris administration issued an executive order on the matter in late 2023.

Failure to keep pace with global regulations like the EU AI Act and the NIST RMF (Risk Management Framework) puts organizations at odds with responsible and ethical AI development — and exposes them to the substantial financial penalties and damaged brand reputation that can come from non-compliance.

5 Steps to AI Governance

Fortunately, there are ways that enterprises looking to enable the safe use of AI can integrate AI models into their data landscape while meeting legal requirements, upholding ethical standards, and driving positive business outcomes. Here’s how incorporating AI governance into a central Data Command Center enables the safe use of AI:

1. Discover AI Models

The first step is to discover and catalog AI models in use across public clouds, private clouds, and SaaS applications.

2. Assess Risks and Classify AI Models

Evaluate risks related to data and AI models and classify AI models as per global regulatory requirements.

3. Map and Monitor Data + AI Flows

Connect models to data sources, data processing paths, vendors, potential risks, and compliance obligations — and continuously monitor data flow.

4. Implement Data + AI Controls for Privacy, Security, and Compliance

Establish data controls on model inputs and outputs, securing AI systems from unauthorized access or manipulation.

5. Comply with Regulations

Conduct assessments to comply with standards such as the NIST AI RMF and generate AI ROPA reports and AI system event logs.

Beyond merely “controlling” data, forward-thinking businesses that get ahead of the risk posed by uncontrolled AI will not only enable the safe use of AI through better governance that upholds ethical and legal standards, but will unlock untold value in business performance, insight, innovation, and brand reputation.

Read the whitepaper “5 Steps to AI Governance” to learn more about each of the actions above that you can take to start ensuring the safe, secure, trustworthy, and compliant use of AI.