The Underlying Ethics Of Data Scraping & Mining

Article Contibuted by SAIM

Data scraping is an inevitable part of the way the internet works. Companies and individuals are interested in various bits of data that would take a lot of time to collect manually. It can take some technical knowledge to scrape efficiently, but it can be a very useful skill. However, some site owners have voiced their disapproval of the practice. And they have various legitimate reasons for that.

As usual, the truth lies somewhere in the middle. On the one hand, site owners should not fight general (non-interfering) scraping and should accept it as a fact. On the other, those interested in collecting data this way should abide by certain ethical rules.

Why do scrapers use rotating proxies? 

It’s not uncommon for scrapers to wish to stay under the radar when doing their work. This can often be for legitimate purposes. For example, certain sites may only be accessible through a specific geographic location – in this case, using a rotating proxy can be a good solution. These proxies allow the scraper to extract data for various regions seamlessly. Click here if you’re not familiar with the concept of rotating proxies. 

But in any case, anyone doing this for legitimate reasons should give site owners the opportunity to contact them if they need to. Leaving as many contact details as possible is crucial for establishing a good relationship, especially if you’re planning to scrape there a lot of data.

When Is It Okay to Scrape the Web in the First Place?

Web scraping can be used for many reasons. An individual may want to download a list of descriptions of their favourite TV show from its fan wiki. A company might be interested in getting a list of all products’ prices that their competitors offer for the price monitoring. The reasons are practically endless, but they are not all equal. 

Scraping is generally acceptable when you’re doing it to extract some additional value out of existing data. The example with the TV show fan is a good one in this regard. But copying data for the sake of copying it is generally frowned upon. Some might launch a new service pre-populated with data obtained through their competitors. This kind of web scraping use is simply an unethical one.

Scraping Is Sometimes the Only Way

There are cases where scraping is the only way to obtain certain data. For example, a site that doesn’t offer any API for the data you’re interested in. In that case, it’s a good idea for you to identify yourself, leave contact information, and what you’ll do with this scraped data. In this case, the site’s owners can contact you if they have any concerns.

Respecting settings like robots.txt is also important. No, nobody will stop you from scraping a page listed as restricted by the website – but think about why you’re doing it in the first place.

Extra Load on Hosts

Aggressive scraping can also be outright harmful to some sites. This is especially true when it’s done simultaneously from multiple hosts to obtain as much data as possible. If the site’s resources are weak enough, you might accidentally DoS it and prevent legitimate users from accessing it. 

This is one of the main reasons site owners are against the idea of scraping, and it’s definitely a legitimate concern. Scraping should always be done with reasonable limitations, such as a delay between every request and an overall cap on the bandwidth during some period of time.

Accidentally Seeing Things that You Shouldn’t See

It’s also possible to accidentally access parts of a site that you normally shouldn’t be seeing. This often happens with poorly developed sites built from scratch and major platforms that have been misconfigured. Depending on how your scraper works, you might eventually run into other users’ private data, or even things like credentials of the site itself. 

Obviously, an ethical scraper should never take advantage of such discoveries. They should make it a point to notify the site’s owners whenever they run across something like that. Needless to say, not everyone out there respects these unwritten rules.

Scraping Is Inevitable – and Site Owners Must Adjust to That

Some site owners will do everything in their power to limit scraping. But in the end, there’s no way to avoid it when there’s someone determined enough. 

The best course of action is to provide an API that gives as much information as possible to those who may need it for legitimate purposes. This will also reduce activities of   unethical scrapers who don’t have to find workarounds to the site’s security, potentially causing unnecessary load as described above. 

The more we move forward with the internet, the more of a concern this is going to be. Scrapers and site owners need to work together to minimize the friction in their relationships because this will benefit the internet as a whole. 

You Might Also Read:

Why You Should Never Use A Free Proxy: