The Underlying Ethics Of Data Scraping & Mining

Article Contibuted by SAIM

Data scraping is an inevitable part of the way the internet works. Companies and individuals are interested in various bits of data that would take a lot of time to collect manually. It can take some technical knowledge to scrape efficiently, but it can be a very useful skill. However, some site owners have voiced their disapproval of the practice. And they have various legitimate reasons for that.

As usual, the truth lies somewhere in the middle. On the one hand, site owners should not fight general (non-interfering) scraping and should accept it as a fact. On the other, those interested in collecting data this way should abide by certain ethical rules.

Why do scrapers use rotating proxies? 

It’s not uncommon for scrapers to wish to stay under the radar when doing their work. This can often be for legitimate purposes. For example, certain sites may only be accessible through a specific geographic location – in this case, using a rotating proxy can be a good solution. These proxies allow the scraper to extract data for various regions seamlessly. Click here if you’re not familiar with the concept of rotating proxies. 

But in any case, anyone doing this for legitimate reasons should give site owners the opportunity to contact them if they need to. Leaving as many contact details as possible is crucial for establishing a good relationship, especially if you’re planning to scrape there a lot of data.

When Is It Okay to Scrape the Web in the First Place?

Web scraping can be used for many reasons. An individual may want to download a list of descriptions of their favourite TV show from its fan wiki. A company might be interested in getting a list of all products’ prices that their competitors offer for the price monitoring. The reasons are practically endless, but they are not all equal. 

Scraping is generally acceptable when you’re doing it to extract some additional value out of existing data. The example with the TV show fan is a good one in this regard. But copying data for the sake of copying it is generally frowned upon. Some might launch a new service pre-populated with data obtained through their competitors. This kind of web scraping use is simply an unethical one.

Scraping Is Sometimes the Only Way

There are cases where scraping is the only way to obtain certain data. For example, a site that doesn’t offer any API for the data you’re interested in. In that case, it’s a good idea for you to identify yourself, leave contact information, and what you’ll do with this scraped data. In this case, the site’s owners can contact you if they have any concerns.

Respecting settings like robots.txt is also important. No, nobody will stop you from scraping a page listed as restricted by the website – but think about why you’re doing it in the first place.

Extra Load on Hosts

Aggressive scraping can also be outright harmful to some sites. This is especially true when it’s done simultaneously from multiple hosts to obtain as much data as possible. If the site’s resources are weak enough, you might accidentally DoS it and prevent legitimate users from accessing it. 

This is one of the main reasons site owners are against the idea of scraping, and it’s definitely a legitimate concern. Scraping should always be done with reasonable limitations, such as a delay between every request and an overall cap on the bandwidth during some period of time.

Accidentally Seeing Things that You Shouldn’t See

It’s also possible to accidentally access parts of a site that you normally shouldn’t be seeing. This often happens with poorly developed sites built from scratch and major platforms that have been misconfigured. Depending on how your scraper works, you might eventually run into other users’ private data, or even things like credentials of the site itself. 

Obviously, an ethical scraper should never take advantage of such discoveries. They should make it a point to notify the site’s owners whenever they run across something like that. Needless to say, not everyone out there respects these unwritten rules.

Scraping Is Inevitable – and Site Owners Must Adjust to That

Some site owners will do everything in their power to limit scraping. But in the end, there’s no way to avoid it when there’s someone determined enough. 

The best course of action is to provide an API that gives as much information as possible to those who may need it for legitimate purposes. This will also reduce activities of   unethical scrapers who don’t have to find workarounds to the site’s security, potentially causing unnecessary load as described above. 

The more we move forward with the internet, the more of a concern this is going to be. Scrapers and site owners need to work together to minimize the friction in their relationships because this will benefit the internet as a whole. 

You Might Also Read:

Why You Should Never Use A Free Proxy:

 

« The History Of The Internet And Its Future
Managing A Remote Team To Protect Against Cyber Attacks »

CyberSecurity Jobsite
Perimeter 81

Directory of Suppliers

Cylance Smart Antivirus

Cylance Smart Antivirus

An antivirus that works smarter, not harder, from BlackBerry. Lightweight, non-intrusive protection powered by artificial intelligence. BUY NOW - LIMITED DISCOUNT OFFER.

Jooble

Jooble

Jooble is a job search aggregator operating in 71 countries worldwide. We simplify the job search process by displaying active job ads from major job boards and career sites across the internet.

FREE eBook: Practical Guide To Optimizing Your Cloud Deployments

FREE eBook: Practical Guide To Optimizing Your Cloud Deployments

AWS Marketplace eBook: Optimizing your cloud deployments to accelerate cloud activities, reduce costs, and improve customer experience.

MIRACL

MIRACL

MIRACL provides the world’s only single step Multi-Factor Authentication (MFA) which can replace passwords on 100% of mobiles, desktops or even Smart TVs.

IT Governance

IT Governance

IT Governance is a leading global provider of information security solutions. Download our free guide and find out how ISO 27001 can help protect your organisation's information.

BackupVault

BackupVault

BackupVault is a leading provider of completely automatic, fully encrypted online, cloud backup.

Cyber Security Supplier Directory

Cyber Security Supplier Directory

Free Access: Cyber Security Supplier Directory listing 5,000+ specialist service providers.

XYPRO Technology

XYPRO Technology

XYPRO is the market leader in HPE Non-Stop Security, Risk Management and Compliance.

Authentic8

Authentic8

Authentic8 transforms how organizations secure and control the use of the web with Silo, its patented cloud browser.

CSI Consulting Services

CSI Consulting Services

Get Advice From The Experts: * Training * Penetration Testing * Data Governance * GDPR Compliance. Connecting you to the best in the business.

RISA

RISA

RISA solutions help to secure networks, improve overall network security, and achieve government regulatory compliance.

European Council on Foreign Relations (ECFR)

European Council on Foreign Relations (ECFR)

ECFR is a pan-European think-tank conducting research and promote informed debate on European foreign policy. Cyber security is becoming an intrinsic element of foreign policy debate.

PacCERT

PacCERT

PacCERT is the Computer Emergency Response Team for the Pacific region.

Vintegris

Vintegris

Vintegris are a Certification Authority and manufacturer of innovative systems and applications for the full cycle of digital identity.

GOVCERT.lu

GOVCERT.lu

GOVCERT.lu is responsible for the treatment of all computer related incidents jeopardising the information systems of the government and defined critical infrastructure operators in Luxembourg.

Seconize

Seconize

Seconize empowers enterprises to proactively manage their cyber risks, prioritize remediations, optimize security spending and ensure compliance.

Injazat

Injazat

Injazat Data Systems is an industry recognized market leader in the Gulf region for Information Technology, Data Center and Managed Services.

Intrepid Solutions and Services

Intrepid Solutions and Services

Intrepid Solutions and Services provides technology solutions and professional services to key components of the intelligence and national security communities.