Understanding the customer is a non-negotiable requirement for businesses. The formula has remained the same for centuries: understand what your customers want and give it to them. The same holds true in the modern age. While customers have an unprecedented degree of options, organizations have also developed various tools to help them gain a better understanding of what their customers want or are likely to want in the future. Web scraping is one such tool.
If done properly, web scraping can create a comprehensive portrait of customers’ purchase patterns, giving organizations a precise indication of which products or services a customer is likely to buy, when, at what price, and via which medium. However, this precision comes at the expense of ethical concerns, especially regarding customers’ personal data.
The Autoriteit Persoonsgegevens (AP), or the Dutch Data Protection Authority, recently issued guidance on web scraping, highlighting the complex legal and privacy issues associated with the practice.
One of the guidance’s most important assertions is that several standard practices related to web scraping may violate the GDPR.
Read on to learn the specifics of the DPA’s guidance and critical insights vital for organizations that wish to continue leveraging this powerful tool while remaining compliant with any relevant regulatory requirements.
What is Web Scraping
Once connected to the Internet, data subjects can request information from online servers, including websites. A GET request (a way to grab data from a data source) is generated from the data subject’s side. The requested server then provides the information, sometimes subject to certain conditions being met, such as a username and password.
The received information can be a webpage, a document, or any other form of data. This entire process can be automated, and per the guidance’s definition, this automated collection and recording of information from web pages is called scraping.
Organizations use it to perform the following tasks:
- Collect information to train algorithms;
- Collect questions and complaints from (potential) customers of an organization via online channels such as social media and review sites;
- Monitor digital messages about an organization for reputation management, sales, or marketing.
The guidance uses two distinct terms, scraping and web crawling, interchangeably, while acknowledging their subtle differences.
Web scraping is a method used by organizations to collect large amounts of data online from a predetermined list of URLs, such as the websites of specific national newspapers.
However, when this list of URLs can be dynamically adjusted, it is called web crawling. This is done through a web crawler (crawler for short) that can automatically update the list of URLs to be visited. A web crawler relies on spiders (small programs) that receive specific instructions in advance, such as “add all URLs you encounter while crawling the list of URLs previously provided”. Additionally, other instructions can be used to limit the list of URLs to visit or follow all links provided, as long as they stay within a specific domain.
A single crawler can manage multiple spiders, allowing crawlers to execute multiple commands simultaneously. Depending on the instructions provided to spiders, they may search a large part of the internet with just a few “start URLs” since the list of URLs to be visited is dynamically adjusted during the process. This dynamic nature of scraping means that organizations often do not know beforehand what data will be collected and processed.
When training artificial intelligence (AI) algorithms, organizations often rely on scraping and crawling to train large language models (LLMs), such as ChatGPT.
Privacy Risks of Scraping
The AP reiterates that the development of scraping techniques itself is not a negative practice. However, the various uses of scraping techniques, such as collecting and recording personal data of data subjects on a large scale in a short time, can pose a significant privacy challenge.
In these instances, data subjects have little control over preventing the scraping of their data since they may not be aware of the scraping in the first place. Additionally, once a piece of information has become available on the internet, it becomes increasingly difficult to delete or protect it from modern scraping techniques.
Algorithms pose a similar problem as they are based on scraped data, which can lead to discrimination and threats to other fundamental rights.
A combination of these privacy risks and threats can make it highly challenging to comply with the GDPR requirements. Hence, organizations must be diligent in organizing their scraping activity, storing and managing scraped data, planning how they will use personal data scrapped, and deploying safeguards to protect the interests of data subjects whose data has been scrapped.
Scraping & GDPR
Under GDPR, organizations must have a legal basis for processing personal data. While this may not always be the case, depending on the type of data processing an organization engages in, the onus is on the organization itself to confirm this.
The main topics related to the applicability of the GDPR to scraping involve the following:
Domestic Exception
These include instances where the scraped data will be used privately and not for professional or commercial purposes from open sources such as the Internet. Additionally, such scraped data may only be used personally and shared with a limited number of people.
The individual may continue using a scraper to search for and store information online since the GDPR will not be applicable in this case.
If the individual has a "hobby project" that they've developed privately and shared with only a few friends, they may proceed with the scraping provided that they do not have a commercial purpose and any data collected through such activity is not published online, even if the scraping was done from a public repository such as GitHub.
Territorial Scope
The GDPR does not apply only to European organizations. According to the GDPR provisions, under certain circumstances, organizations outside the European Union (EU) will also need to comply with the GDPR when processing personal data. These include the following situations:
- The organization offers goods and services to data subjects within the EU;
- The organization monitors the behavior of data subjects within the EU.
Offering Goods and Services Within the EU
An organization must comply with the GDPR if it offers goods or services within the EU, regardless of whether these goods or services are paid for. The mere fact that the website of an organization established outside the EU, which is accessible within the EU, is insufficient to establish the applicability of the GDPR. In such cases, consideration of all circumstances is necessary, including whether it is possible to use an address within the EU to purchase from the organization and whether payment can be made in euros. These can be valid indicators of an organization offering its goods and services in the EU.
Monitoring The Behavior of Data Subjects
The second category of processing that requires organizations outside the EU to comply with the GDPR involves processing personal data when the organization monitors the behavior of individuals within the EU. The guidance provides comprehensive insights into whether scraping can be considered a form of monitoring the behavior of data subjects.
The first factor to consider must be the purpose of the scraping. If scraping is used to collect information about people's behavior to offer them personalized services or advertisements subsequently, then the scrapping activity and the data processing must comply with the GDPR. If the purpose is to train an algorithm that allows data subjects outside the EU to generate images or code, then the GDPR may not be applicable, as this does not come under the definition of monitoring as provided under Article 3 of the GDPR.
In cases where an organization unknowingly collects data subjects’' personal data from within the EU by not applying a geographical filter to the spiders when collecting data, they will still be subject to GDPR provisions. Moreover, applying a geographical filter is not a foolproof method to avoid processing the personal data of EU citizens, as such data is likely to be available on international websites.
GDPR Principles
If an organization scraps information from the Internet, inevitably, it will also scrap personal data. Hence, unless it operates under an exception, it must comply with the GDPR provisions when carrying out the scrapping activity and adopt mechanisms to ensure compliance with the GDPR at the developmental phase.
Under Article 5(1) of the GDPR, processing of personal data must comply with the following principles:
- Legality, Fairness, and Transparency;
- Purpose Limitation;
- Accuracy;
- Data Minimization;
- Storage Limitation;
- Integrity & Confidentiality.
All organizations processing personal data must adhere to these principles and must be able to demonstrate their adherence. This extends to questions such as whether an organization processes personal data when scraping and whether it meets the GDPR's requirements when processing such personal data.
The guidance acknowledges that the multifaceted characteristics of scraping can make it challenging for an organization to comply with all the principles of the GDPR. Some examples it provides include the following:
- Transparency: It can be difficult to inform the data subjects of the scraping activity effectively, hence conflicting with the principle of transparency;
- Minimal Data Processing: A major risk of scraping is the processing of extensive data that may not always be necessary for the purpose of processing, hence conflicting with the principle of minimal data processing;
- Accuracy: An organization may collect extensive data from multiple sources, making it difficult and, in some cases, impossible to determine the accuracy of the data. This can be further exacerbated if the scraped personal data is stored for an extensive period and is no longer up to date.
Principle of Legality
Legal Basis for Processing
As soon as the GDPR becomes applicable to an organization's processing activities, it must have a legal basis for processing. Additionally, an organization must determine how the provisions related to data processing apply to its scraping activities. Some of the scenarios that guidance cites as examples include the following:
- The scraper itself is responsible for the processing;
- The client is the controller, and the scraper is the processor for the entire data processing activity;
- The scraper serves as the controller for part of the processing and the client for another part, making it both a controller and a client, which leads to a joint processing responsibility.
Exactly who the controller is will depend on the circumstances of the case and how the data processing is organized. Hence, the AP cannot give an exact opinion on who is the controller and processor, as such assessments must be made by the organizations involved, depending on the actual situation.
For a private organization or individual, "legitimate interest" will be the only basis to be considered in almost all cases, as other bases, such as the execution of an agreement, legal obligations, vital interests, and duties of general interest/public authority, will not apply in this case.
In principle, consent, i.e., obtaining the data subjects' consent, may also serve as a legal basis. However, as elaborated earlier, owing to the nature of scraping and the scale of data collection involved, it may not be reasonable or possible to identify each data subject and gain their consent.
In such a context, obtaining valid consent can be challenging, and it would be extremely difficult, if not nearly impossible, for organizations to appropriately identify and obtain permission from each individual. Even if such individuals willingly make their data available on the Internet for everyone to view, it cannot be scraped or processed.
Hence, only ‘legitimate interest’ remains as the possible basis.
Legitimate Interests
An organization can rely on legitimate interests as a legal basis for processing personal data. However, the GDPR places strict criteria for its application. Organizations must demonstrate that their interests are not superseded by individuals' fundamental rights when engaging in web scraping.
For an organization to successfully rely on legitimate interests as a legal basis, it must meet the following conditions:
- There is a legitimate interest on the part of the organization or a third party;
- The processing is necessary for the pursuit of this legitimate interest;
- The interests or fundamental rights and freedoms of the data subject(s) do not outweigh the organization’s legitimate interest or that of the third party.
The aforementioned conditions are cumulative. Hence, an organization must meet all three of them.
Condition 1
The AP argues within the guidance that only legally protected interests qualify as legitimate interests. Hence, any interest an organization relies on must be recognized and protected in the form of a legal rule or principle, such as anti-fraud activities, cybersecurity measures, or the right to freedom of information per the Charter of Fundamental Rights of the EU.
The organization must inform all those involved about the aforementioned interests. Only after the organization can prove that it has taken appropriate measures to inform the data subjects can it rely on legitimate interest. If the interest is purely commercial, an organization cannot successfully rely on the legitimate interest basis.
Condition 2
To successfully meet the second condition, an organization must be able to demonstrate that the processing of personal data is necessary to pursue the legitimate interest it relies on as a legal basis. In such an instance, it must comply with the principles of subsidiarity and proportionality to its needs.
Condition 3
Ultimately, an organization must strike a balance between its own or third-party interests and the interests and fundamental rights of its data subjects.
Assessment of Basis For Legitimate Interest
Scraping can be done for various purposes in various ways. Hence, an organization must conduct extensive assessments to determine whether it meets the aforementioned conditions. Furthermore, it must have appropriate safeguards in place to protect the interests of the data subjects and limit any potential infringement during the processing.
Some of the elements that an assessment must consider, depending on the kind of processing being done, include the following:
Scope & Nature of the Data Processing
An organization may collect extensive data while scraping, including information from multiple individuals as well as detailed data about each one. Such data may be collected from various sources and accumulated over an extended period. In other words, the broader the scraper searches, the greater the likelihood of potential infringement on the privacy of data subjects whose data is being collected.
Suppose the database where the scraped data is stored is searchable. In that case, it increases the likelihood of a potential infringement significantly by allowing the creation of potentially detailed profiles of data subjects whose data has been collected.
Hence, it is essential to consider all potential consequences for data subjects before an organization proceeds with scraping and to determine whether it can meet the conditions for legitimate interest.
Sensitive Data
When scraping data from the Internet, sensitive personal data, such as geolocation and financial data, may be collected. The more sensitive data an organization scraps, the greater the risk of privacy breaches. Similarly, the more sensitive data an organization collects about data subjects, the less likely it is to meet the second and third conditions on a legal basis.
Additionally, an organization may intentionally or unintentionally process special personal data or criminal records, further complicating its ability to meet the conditions.
Expectations of Data Subjects
Scraping collects all the data that is made public and visible to everyone by the data subjects. For such data, data subjects may have a reasonably low expectation of not being processed by others. Consequently, the impact of others using this data is typically smaller compared to situations where the data was not made public by the individuals themselves but by a third party, such as a sports club or employer posting an individual’s name on their website.
However, even when individuals themselves make their personal data publicly accessible online, it does not necessarily mean they reasonably expect their data to be processed for a different purpose.
Hence, when scrapping personal data from the internet, it may be difficult or impossible to identify each person whose data an organization wants to scrap and inform them about the data processing they wish to carry out. In such instances, an organization may not be required to inform the data subject individually, as it would involve disproportionate effort.
However, the organization must take appropriate measures to protect the rights, freedoms, and legitimate interests of the data subjects. This includes disclosing all information it would usually provide to the data subject in the form of a privacy policy page on its website.
The policy should detail the type of data being processed, the purpose and legal basis for processing, the organization's contact details, and the rights of the data subject under the GDPR, along with instructions on how to exercise those rights.
In cases where it is possible for an organization to directly inform the data subjects who are subject to scrapping, then it must do so. This enables individuals to understand what happens to their data, assess any potential risks, and have reasonable expectations of an organization's data processing activities.
Consequences for the Data Subjects
The exact consequences for the data subject will depend mainly on the purpose of the scraping. The organization will be responsible for determining consequences based on its scraping plans. For example, if the data is used for statistical analysis or sentiment analysis, where individuals can’t be identified, the impact is minimal. However, if scraping is used to create personal profiles or make hiring decisions based on online activity, the consequences for individuals can be significant. These potential consequences must be carefully considered when determining if the processing meets the third condition (balance of interest) for legitimate interest.
Weak Position Of The Data Subjects
Scraping is not automatically visible to those whose data is scraped. Hence, the data subjects have a lower chance of exercising their GDPR rights and may not be able to oppose the scraping of their data.
Even if the data subjects are aware of their personal data being scrapped, it is not easy to have this personal data removed once it has been published on the Internet or to make it inaccessible for scraping. An organization assessing whether it meets the conditions for legitimate interest must take into account the lack of control that data subjects have over their personal data.
Role of Safeguards
Organizations can mitigate the consequences for data subjects by implementing additional safeguards. These safeguards, tailored to the specific circumstances of the processing activity, can play a crucial role in meeting the third condition for the legitimate interest basis (balancing of interests). However, these safeguards must be voluntary and not merely those already required by the GDPR.
Examples of additional safeguards include:
- Implement measures to enhance the transparency;
- Delete, anonymize, and pseudonymize collected personal data as quickly as possible;
- Approve the right to erasure requests more frequently and broadly than required by Article 17 of the GDPR;
- Comply with internet standards, such as the robots exclusion protocol (robots.txt), which allows website administrators to specify which parts of their site may be accessed by crawlers.
Additional safeguards can strengthen the reliance on the legitimate interest basis for processing. However, their effectiveness depends on the specific processing activities. Implementing such safeguards does not always guarantee a successful claim on the legitimate interest basis.
Special Personal Data
Some personal data is referred to as special personal data. This includes personal data that is considered extra-privacy-sensitive due to the significant impact it can have on the data subject if such data is processed. Hence, this data must receive an extra degree of protection. This includes information about a person's race, political affiliation, religion, beliefs, and sexual orientation. Since processing such data is prohibited under the GDPR unless an exception is applicable, scraping such data is also prohibited.
Conditions for Processing of Special Personal Data
Scraping, and in particular, scraping of social media, involves processing special personal data. If an organization wishes to use scraping, it must carefully check in advance whether its scraping activities will involve processing special personal data.
Furthermore, organizations need to know that the combination of various “normal” personal data can result in special personal data. In that case, the organization will still be subject to GDPR requirements for the processing of special personal data, regardless of whether the collected data was accurate or if the organization had any intention of processing such data.
Similarly, the protection requirements for special personal data may also apply to the entire processing activity. This is the case not only when you process only special personal data but also when both 'normal' and special personal data are collected.
Finally, if an organization cannot be certain whether it has processed special personal data, it must ensure that it complies with the protection requirements for such data. This would mean that processing such data would be prohibited unless the organization can invoke one of the exceptions, i.e., explicit consent from the data subject or public disclosure of such data by the data subject.
Explicit Consent of the Data Subject
The prohibition on processing special personal data will not be applicable if the data subject has provided explicit consent for processing the personal data for one or more specific purposes.
Such cases of exception will be extremely rare because it is difficult to identify each individual and ask for their consent. However, if an organization is able to organize its scraping activities in such a way that it can ask the data subjects for permission before processing their data, it must keep the following considerations in mind:
- All consent must be freely given;
- The data subject must be given a clear and explicit declaration to agree to;
- Before requesting consent, the data subjects must be informed about:
- The identity of the organization;
- The specific purpose(s) of the processing;
- The types of personal data being collected and processed; and
- The right to withdraw consent at any time.
- Any provided consent will only apply to a specific processing purpose;
- The organization must properly document the consent obtained, including details of the information provided to the individual at the time consent was given.
Public Disclosure By Data Subject
The prohibition on processing special personal data will also not apply if the data subject has manifestly made the data public. This includes information being made public in the following manner.
Self-Published Data
The special personal data must have been made public by the data subjects themselves. For instance, a person might openly share information about their health, sexual orientation, or religious beliefs in a publicly accessible blog. However, when scraping data from the internet, organizations must be aware that any special personal data they have collected and recorded intentionally or unintentionally may not always have been made publicly available by the individuals themselves, such as when a family member shares information on a social media site about someone’s medical situation. In such cases, the exception to the prohibition on processing special categories of personal data does not apply.
Obvious Disclosure of Personal Data
The special personal data must have been “manifestly” made public by the data subject. This means that the data subject’s intention to make such information publicly accessible to a wider audience must be unambiguous and made through an active act.
However, even in such cases, certain conditions will still apply. For example, if an individual uploads their picture on a publicly accessible social media platform, they may not intend to make any biometric data that can be extracted from such a picture public. Biometric data will only be considered to have been made public if the data subject concerned has deliberately made it available in a specific biometric template, not just a facial image.
If we consider the exceptions to the prohibition on processing special personal data discussed above, it becomes clear that when scraping information from the internet, it is often difficult or impossible to distinguish between normal personal data and special personal data. As a result, there is a high likelihood that special personal data will be scraped unintentionally, which is prohibited except in exceptional cases. This may lead to processing being prohibited due to the inclusion of special personal data.
Even if an organization does not scrape the data itself but uses data scraped by someone else, its processing may still be prohibited if it involves special personal data and it cannot successfully invoke one of the exceptions to the prohibition on processing such data.
Personal Data of Criminal Nature
In addition to special personal data, scraping may involve processing personal data of a criminal nature, also known as criminal personal data. Such data is also provided with extra protection under Article 10 of the GDPR.
Criminal personal data can only be processed:
- Under government supervision;
- When processing is permitted under European or Dutch law, with appropriate safeguards to guarantee the protection of the rights and freedoms of data subjects.
The general prohibition on processing this type of data can be lifted for scraping such data if the data subject has manifestly made it public or given explicit consent for processing such data for one or more specific purposes.
Summary of Legality
Based on the AP’s position, a purely commercial interest that is not legally protected cannot be considered a legitimate interest within its meaning under Article 6(1)(f) of the GDPR. For organizations that wish to develop their tools and software with scraping, such as GenAI, simple commercial interest will not qualify as a legitimate interest to collect and use personal data to train such a tool or software. If there is also a non-commercial interest involved, only then will it be considered a legitimate interest, provided it meets the second and third conditions of the legitimate interest.
If such processing meets all three conditions, the organization must also verify whether it is processing special personal data or criminal personal data, with all relevant exceptions and requirements applicable.
All things considered, only a very targeted form of scraping seems to be lawful for private organizations, as it is difficult to judge what use of scraping or scraped data is permitted in general terms. Hence, to make such an assessment, it is necessary to know the details of each specific processing activity. However, some processing operations have to be more GDPR-compliant than others, such as:
- Public news websites that portray current events;
- An organization’s own online web pages, such as with customer reviews and other forms of customer communications;
- Public online forums on information security.
Similarly, there are certain cases of scraping that can probably never be set up in a GDPR-compliant manner, such as:
- Internet profiles of data subjects being set up and resold;
- Private social media accounts or private forums;
- Social media accounts of data subjects, even if they’re public, to collect information to determine whether a data subject will receive a requested insurance policy.
However, these are just illustrative examples and as mentioned earlier, each assessment will have to be assessed on a case-by-case basis to meet the requirements of the GDPR.
Mandatory DPIAs
Per Article 5 of the GDPR, an organization must demonstrate that it processes personal data lawfully, fairly, and transparently. Hence, organizations need to assess the lawfulness of their scraping activities before initiating them. A suitable way for this is to conduct a data protection impact assessment (DPIA).
By conducting regular DPIAs, an organization can proactively identify and mitigate potential privacy risks. Scraping activities can be monitored with precision in real-time, allowing adjustments to be made before they cause significant harm.
How Securiti Can Help
The primary issue with web scraping, as highlighted by the Dutch DPA’s guidance, is the improbability of collecting consent for this practice. Furthermore, there are issues with algorithmic bias and discrimination that may arise from GenAI models being trained on extensive scraped datasets, which will inevitably contain biases.
Attempting to resolve these issues manually will not only be an exercise in futility but also put significant strain on an organization’s resources. Hence, automation is the way to go.
That is where Securiti can help. It is the pioneer of the Data Command Center, a centralized platform that enables the safe use of data and GenAI. Additionally, it provides unified data intelligence, controls, and orchestration across hybrid multi-cloud environments. Globally renowned and reputable enterprises rely on Securiti's Data Command Center for data security, privacy, governance, and compliance, as well as the safe use of data and GenAI capabilities.
Furthermore, organizations can gain access to vital modules such as data mapping and assessment automation that collect and maintain an inventory of data assets and data processing activities in a Sensitive Data Catalog and generate records of processing (RoPA) reports, privacy impact assessments, and data protection impact assessment aligning with global privacy regulations such as the GDPR.
Request a demo today and learn more about how Securiti can help you comply with the GDPR as well as other major data protection regulations from across the world.
