By Edward Roberts, Distil Networks
Imagine you are responsible for product pricing on an e-Commerce web site. You have a special offer planned for a premium item that you hope will attract customers to your site.
You understand the product’s profit margin and have decided to reduce the price by 7% from the one presently shared with your main competitor. In preparation, you’ve also invested time and money in an online advertising campaign to attract customers to your site.
Advertisement
At the prescribed time, you prepare the web site for the price change. After your price goes live, you check the competitor’s site to confirm that their price is now higher than yours. But to your surprise, they’ve reduced theirs to one cent below your new price. How could this happen?
The answer is price scraping.
Your competitor has used bot technology to continually scrape your company’s site — using that data to maintain its price advantage. Your promotional campaign expense is wasted as you see people clicking your ad but soon abandoning the sale. Your work to gain SEO advantage is also for naught because search engines still list your competitor as having the lowest price. Your site’s brand awareness is diminished, because the competitor offers the same product at a cheaper price. And of course, the planned bump in revenue didn’t occur because the additional sales went to your competitor. The harsh reality is, in e-Commerce, this story is very real and happens every day.
An Industry Under Attack
Web scraping is the act of stealing site content; it’s responsible for millions of dollars in lost annual revenue. On average, Distil Networks finds in its 2016 Economics of Web Scraping Report that 2% of online revenue is lost due to scraping.
Bots comprise almost half (46%) of all web traffic. By extension, half of your site visitors may be irrelevant to your business. Bots perform scraping much faster than humans, at a very low cost, and can exfiltrate publicly available content from any targeted web site.
In fact, e-Commerce is one of the top three web scraping victims. Moreover, such victimization will continue if malicious bots are allowed to scrape web sites unimpeded.
Key findings in the Distil Networks web scraping report reveal that:
- 38% of companies engaged in scraping do so to obtain content: Web scraping is also used for research, contact scraping, price comparison, weather data monitoring and web site update detection.
- Web scraping services cost as little as $3.33 per hour: The average web scraping project costs roughly $135.
- The average scraper earns $58,000 annually: When working for a large company specializing in such activity, some individuals gross up to $128,000 per year.
For many e-Commerce retailers, it comes as no surprise that there is an active economy around building technology to scrape content from web sites. Luckily for retailers, there is also an entire economy dedicated to solutions preventing bots from scraping web sites today.
Or instead you could retaliate against the unscrupulous competitor that is matching your prices using bots. Perhaps you might lower your price from $1,000 down to $1 for a moment, watch the competitor’s bot automation match the price, and then purchase all their inventory at that cheap price?
This scraping retaliation might feel satisfying as a one-off response, but a better long-term solution is one that blocks scrapers from having access to your web site, thereby preventing the fraudulent activity.
Visit here to download a copy of the 2016 Economics of Web Scraping Report.
Edward Roberts leads Product Marketing for Distil Networks and has more than 20 years of experience in technology marketing. Previously he worked for Juniper Networks, heading up Product Marketing for the Counter Security team. Before that he ran marketing for Mykonos Software, a web security company.
