The Underlying Ethics Of Data Scraping & Mining

Article Contibuted by SAIM

Data scraping is an inevitable part of the way the internet works. Companies and individuals are interested in various bits of data that would take a lot of time to collect manually. It can take some technical knowledge to scrape efficiently, but it can be a very useful skill. However, some site owners have voiced their disapproval of the practice. And they have various legitimate reasons for that.

As usual, the truth lies somewhere in the middle. On the one hand, site owners should not fight general (non-interfering) scraping and should accept it as a fact. On the other, those interested in collecting data this way should abide by certain ethical rules.

Why do scrapers use rotating proxies? 

It’s not uncommon for scrapers to wish to stay under the radar when doing their work. This can often be for legitimate purposes. For example, certain sites may only be accessible through a specific geographic location – in this case, using a rotating proxy can be a good solution. These proxies allow the scraper to extract data for various regions seamlessly. Click here if you’re not familiar with the concept of rotating proxies. 

But in any case, anyone doing this for legitimate reasons should give site owners the opportunity to contact them if they need to. Leaving as many contact details as possible is crucial for establishing a good relationship, especially if you’re planning to scrape there a lot of data.

When Is It Okay to Scrape the Web in the First Place?

Web scraping can be used for many reasons. An individual may want to download a list of descriptions of their favourite TV show from its fan wiki. A company might be interested in getting a list of all products’ prices that their competitors offer for the price monitoring. The reasons are practically endless, but they are not all equal. 

Scraping is generally acceptable when you’re doing it to extract some additional value out of existing data. The example with the TV show fan is a good one in this regard. But copying data for the sake of copying it is generally frowned upon. Some might launch a new service pre-populated with data obtained through their competitors. This kind of web scraping use is simply an unethical one.

Scraping Is Sometimes the Only Way

There are cases where scraping is the only way to obtain certain data. For example, a site that doesn’t offer any API for the data you’re interested in. In that case, it’s a good idea for you to identify yourself, leave contact information, and what you’ll do with this scraped data. In this case, the site’s owners can contact you if they have any concerns.

Respecting settings like robots.txt is also important. No, nobody will stop you from scraping a page listed as restricted by the website – but think about why you’re doing it in the first place.

Extra Load on Hosts

Aggressive scraping can also be outright harmful to some sites. This is especially true when it’s done simultaneously from multiple hosts to obtain as much data as possible. If the site’s resources are weak enough, you might accidentally DoS it and prevent legitimate users from accessing it. 

This is one of the main reasons site owners are against the idea of scraping, and it’s definitely a legitimate concern. Scraping should always be done with reasonable limitations, such as a delay between every request and an overall cap on the bandwidth during some period of time.

Accidentally Seeing Things that You Shouldn’t See

It’s also possible to accidentally access parts of a site that you normally shouldn’t be seeing. This often happens with poorly developed sites built from scratch and major platforms that have been misconfigured. Depending on how your scraper works, you might eventually run into other users’ private data, or even things like credentials of the site itself. 

Obviously, an ethical scraper should never take advantage of such discoveries. They should make it a point to notify the site’s owners whenever they run across something like that. Needless to say, not everyone out there respects these unwritten rules.

Scraping Is Inevitable – and Site Owners Must Adjust to That

Some site owners will do everything in their power to limit scraping. But in the end, there’s no way to avoid it when there’s someone determined enough. 

The best course of action is to provide an API that gives as much information as possible to those who may need it for legitimate purposes. This will also reduce activities of   unethical scrapers who don’t have to find workarounds to the site’s security, potentially causing unnecessary load as described above. 

The more we move forward with the internet, the more of a concern this is going to be. Scrapers and site owners need to work together to minimize the friction in their relationships because this will benefit the internet as a whole. 

You Might Also Read:

Why You Should Never Use A Free Proxy:

 

« The History Of The Internet And Its Future
Managing A Remote Team To Protect Against Cyber Attacks »

CyberSecurity Jobsite
Perimeter 81

Directory of Suppliers

Clayden Law

Clayden Law

Clayden Law advise global businesses that buy and sell technology products and services. We are experts in information technology, data privacy and cybersecurity law.

Authentic8

Authentic8

Authentic8 transforms how organizations secure and control the use of the web with Silo, its patented cloud browser.

North Infosec Testing (North IT)

North Infosec Testing (North IT)

North IT (North Infosec Testing) are an award-winning provider of web, software, and application penetration testing.

DigitalStakeout

DigitalStakeout

DigitalStakeout enables cyber security professionals to reduce cyber risk to their organization with proactive security solutions, providing immediate improvement in security posture and ROI.

CYRIN

CYRIN

CYRIN® Cyber Range. Real Tools, Real Attacks, Real Scenarios. See why leading educational institutions and companies in the U.S. have begun to adopt the CYRIN® system.

BackBox Software

BackBox Software

BackBox is a leading provider of solutions for automated backup and recovery software for security and network devices.

BTWorks

BTWorks

BTWorks provides identity management and anti-phishing / smishing solutions for web and mobile apps.

QuintessenceLabs

QuintessenceLabs

QuintessenceLabs offers a suite of Data Security technology, products and solutions to secure digital information in-transit, at-rest or in-use.

Celare

Celare

Celare delivers DPI based network perimeter monitoring solutions with integrated Big Data security analytics and threat detection.

RiskCentric

RiskCentric

RiskCentric is a consultancy specializing in risk management and compliance.

Fraud.com

Fraud.com

Fraud.com ensures trust at every step of the customer's digital journey; this complete end-to-end protection delivers unified identity, authentication and fraud detection and prevention.

Deceptive Bytes

Deceptive Bytes

Deceptive Bytes provides an Active Endpoint Deception platform that dynamically responds to attacks as they evolve and changes their outcome.

Polyrize

Polyrize

The Polyrize continuous authorization platform for SaaS and IaaS stops tomorrow's public cloud cyber threats, today.

Document Security Systems (DSS)

Document Security Systems (DSS)

DSS anti-counterfeit, authentication, and brand protection solutions are deployed to prevent attacks which threaten products, digital presence, financial instruments, and identification.

ByteSnipers

ByteSnipers

ByteSnipers specialize in penetration testings and secure development services. Our focus is on your security.

InfoSystems Inc

InfoSystems Inc

InfoSystems provides reliable IT solutions to build and maintain strong and secure systems for both SMB and enterprise organizations.

Integrity

Integrity

Integrity is a PCI QSA and ISO 27001 certified company specialized in Information Security and IT Consulting.

X Technologies

X Technologies

X Technologies provide world-class engineering, information technology, information security, program management and repair services to Federal, State and commercial customers.

Cufflink

Cufflink

Cufflink makes your business more secure, compliant and trusted. We limit the likelihood and impact of a data breach by controlling exactly what can and can't be done with personal data.

iSTORM

iSTORM

iStorm specialise in supporting organisations who require a range of Privacy, Security and Penetration testing related services.

Quantum eMotion (QeM)

Quantum eMotion (QeM)

Quantum eMotion is a Montreal-based advanced developer leading the way towards a new generation of quantum-safe encryption for the quantum computing age.