The Unstructured Data Challenge
LLMs has created an opportunity for organizations to extract tremendous value from their unstructured data. However, CDAOs are all too aware of the challenges involved in incorporating unstructured data into large-scale data transformations. In an ideal world, it would be just as easy to use unstructured data as it is to use structured data. Organizations need to know that data can be trusted, that it has been thoroughly sanitized at an element level with granular access entitlements that protect all the data in the data estate. Today, organizations struggle to apply the same degree of governance typically afforded to their business-critical structured data as they do to their ever-expanding reservoir of unstructured data. Meanwhile, eagerly awaited AI initiatives stall.
Organizations that leverage Databricks for analytics and AI face specific technical challenges when working with unstructured data, which comprises approximately 90% of enterprise information. While Databricks excels at handling structured data and has made progress on unstructured sources, teams in complex, hybrid cloud data environments may encounter several critical pain points when attempting to incorporate unstructured sources into their data pipelines:
1. Complex and Manual Preprocessing Requirements
Ingesting unstructured data (including zipped folders, mixed file types, and inconsistent CSV formats) requires preprocessing before it can be loaded into Databricks. Teams typically need to build custom Python scripts or use external tools to parse, clean, and convert data into Delta Lake format, which creates scalability challenges and maintenance overhead.
2. Granular Permission Management is Cumbersome
To build permission-aware AI applications that safeguard confidential and proprietary data, firms must ensure that only authorized users can access sensitive unstructured data. Today, that often requires meticulous configuration. Unity Catalog provides centralized access control, but setting up granular permissions—especially for external locations in cloud storage—is a manual and error-prone process. Why is that? The answer is technical and organizational. Locking down unstructured data in general requires the organization to have comprehensive fine grained permissions established - unfortunately, due to constantly changing data sources, even the best run companies tend to be over provisioned with far too many people having access. For AI use cases, the matter is even more complicated as the AI workflow includes a process called vectorization that turns all the info into an indexable representation LLMs can read and in the process, breaks the access controls you thought you had in the first place.
3. Security and Compliance Risks in Data Sharing and Rapid Deployment
Databricks' collaborative environment, like all modern cloud data platforms, accelerates the speed at which data can be shared, which in turn increases the risk of accidental or intentional data exposure. Unstructured data often contains sensitive information, and, if not thoroughly scanned, it is impossible to ensure that sensitive data is fully accounted for. Rapid data ingestion and sharing often result in partial scans and misconfigured access controls, making it difficult to maintain compliance with regulations such as GDPR, HIPAA, or PCI-DSS.
4. Feature Extraction and Structuring Overhead
It is not enough to find sensitive data in complex multi-user scenarios. Tools must be in place to minimize, redact, and sanitize sensitive data before it is loaded or considered a gold copy. Before unstructured data can be used for analytics or AI, it must undergo complex feature extraction and transformation. Today, this requires additional pipelines and specialized tooling that engineering teams must build and maintain.
Querying unstructured data can be slow and resource-intensive. Transformations such as flattening nested data degrade performance at scale, while unstructured data quickly balloons storage costs and complicates governance. Without the tools to curate and trust the precise unstructured data you absolutely need- no more no less- organizations may get unpleasant surprise bills.
How Securiti Expands Solutions to Unstructured Data Challenges
Securiti has partnered with Databricks to deliver end-to-end, trusted unstructured data management with full context through Securiti’s Gencore AI solution newly directly integrated into Delta Tables and Unity Catalog. This new partnership enables organizations to more easily and quickly build safe, enterprise-grade generative AI (GenAI) systems and AI agents, using high-value, proprietary enterprise data.
Securiti AI enhances Databricks in five powerful ways:
1. Simplified Unstructured Data Ingestion
Gencore AI safely ingests unstructured and structured data from SaaS apps and on-prem systems into Databricks Delta tables. It eliminates the need for custom preprocessing scripts by providing hundreds of native connectors to quickly and securely ingest data at scale from anywhere, including public, private, SaaS, and data clouds.
Data engineers benefit: Instead of building and maintaining custom scripts, teams can leverage Securiti's extensive connector library to streamline the ingestion process, reducing data preparation time by up to 60% as reported by shared Securiti and Databricks customers.
2. Automated Data Sanitization and Protection
Gencore AI helps sanitize (redact, mask, or anonymize) sensitive information before bringing it into Databricks. The solution automatically classifies and redacts sensitive data on-the-fly, ensuring privacy and compliance before data is exposed to AI models or transformed into vectors that can be later retrieved.
Security teams benefit: Before data enters AI pipelines and LLMs, comprehensive checks ensure alignment with AI governance, privacy, security, compliance, and sovereignty requirements - dramatically reducing security and compliance risks.
3. Advanced Data Security & Governance
Built-in data protection, alignment with OWASP Top 10 for LLMs, and a graph-based full provenance view of AI and data enable safe AI systems at scale. Gencore AI implements advanced LLM firewalls to understand the context of all AI interactions, including prompts, responses, and data retrievals, to offer end-to-end protection of enterprise data far beyond easily circumvented model guardrails.
Compliance teams benefit: Custom and pre-configured policies block malicious attacks, prevent sensitive data leaks, and ensure enterprise AI systems align with corporate policies. These context-aware firewalls also preserve access entitlements to documents and files throughout the AI pipeline.
4. Enhanced Unity Catalog Intelligence
Unity Catalog gains enriched context through Securiti's DataAI Command Graph, thus increasing data utilization. Securiti's DataAI Command Graph contains rich context about relationships between files, tables, columns, AI objects, users, permissions, and regulations that can be seamlessly registered within Unity Catalog.
Data administrators benefit: The comprehensive context increases Unity Catalog's utility and enables safer data usage across the platform.
5. The Securiti DataAI Command Graph: A Game-Changer for Databricks
At the heart of Securiti's solution is the DataAI Command Graph—a knowledge graph that provides contextual intelligence about enterprise data. This graph enables:
- Precise selection of relevant files and datasets based on labels, entitlements, regulations, and quality
- Comprehensive visibility into data lineage and relationships
- Preservation of user entitlements at the prompt level, enhancing security and compliance
"Contextual intelligence for both unstructured and structured data is at the heart of GenAI use cases," said Jocelyn Houle, Sr. Director of Product Management, “The DataAI Command Graph automatically builds knowledge about your data that provides insights to the GenAI pipeline at every step for its safe use.”
The graph provides in-depth contextual insights into data objects, such as files, folders, buckets, tables, or columns, including related context, such as sensitive information, entitlements, location, applicable policies and processes, and regulations.
Conclusion: Unlocking the AI adoption with Securiti and Databricks
The partnership between Securiti and Databricks represents a significant advancement in enterprise AI and permission-aware solution-building capabilities. By addressing the critical challenges of unstructured data management, organizations can now unlock the full potential of their data assets while maintaining rigorous security, governance, and compliance standards.
As organizations continue to invest in AI initiatives, solutions like Gencore AI will become essential for scaling enterprise AI responsibly and efficiently. The integration enables teams to focus on innovation rather than wrestling with the complexities of unstructured data management, ultimately accelerating the path to AI-driven business transformation.
To learn more about how Securiti and Databricks can help your organization, visit Securiti's Gencore AI website.
