Publications about marketing often invoke the comparison of a digital shelf with a physical one. After all, a digital shelf displays your products for the customer to browse, just like a shelf in a brick-and-mortar store would.
This comparison, however, has its limits. While customers come to browse the physical shelf, your digital shelf goes to the customer. It can constantly and immediately change the way it is arranged to beat other shelves at capturing the customer’s attention.
Constant competition with other shelves for visibility drives the need for digital shelf optimization. Web scraping can help acquire real-time intelligence to keep your shelf moving to the forefront of the virtual marketplace that is forecast to reach $8.1 trillion in sales by 2026.
What is Digital Shelf Optimization?
Digital shelf optimization (DSO) means constantly making changes to your product display on ecommerce platforms in an attempt to keep it as optimal as possible. What you are optimizing are essentially two things:
Advertisement
- How the users find your products
- What they see when looking at them.
The first optimization area breaks down into where your products appear in search engine result pages (SERPs) and your ad placements. The second comprises price, text and visuals that define your product offer. Identifying the performance and optimization opportunities for all these elements requires digital shelf analytics (DSA) to cover a variety of data points.
How can Web Scraping Help?
Many companies can track their own digital shelf data, allowing them to align products across platforms, perform A/B testing and improve inventory management. However, this isn’t enough even if the company is operating in a local market only. Opportunities for digital shelf optimization heavily depend on the changes on competitors’ shelves and how customers react to them.
Thus, the company needs to analyze the competition and general online shopping behavior trends to get actionable insights for DSO, which can only be done by collecting real-time public web data. Fortunately, modern web scraping solutions can fetch public data with nearly unlimited scalability, be it web intelligence from ecommerce marketplaces, competitor websites or public social forums. The fact that you can adapt your digital display very quickly as compared to physical inventory management allows businesses to benefit from multiple types of external data.
Competitors’ out-of-stock data: Using stockouts of other retailers to your advantage requires a constant intake of external data. Competitive stock monitoring provides opportunities to optimize your marketing resources. For example, you can buy ads against competitor OOS offerings, promote the product’s pages on social media in a timely manner and send targeted emails to customers who have the product in an abandoned cart.
Additionally, with external OOS data, you can turn predictive analytics methods used for your own stock management outwards. Forecasting out-of-stocks based on seasonal changes and your competitor’s historic behavior will enable you to plan stock in advance and act at the right moment.
Price data: One of the main advantages of ecommerce versus brick-and-mortar shops is the ease of frequently changing prices for optimal results or, as it is often called, dynamic pricing. Reasonable price flexibility does not damage a brand’s reputation only if it is based on accurate and current data. It may even improve customer experience by providing opportunities to buy items at a lower cost.
Tracking prices across ecommerce platforms will allow for the optimization of dynamic pricing strategies. By integrating automated data gathering with price change automation you can build an efficient infrastructure for winning over customers.
Product titles and descriptions: Search engine optimization is the backbone of making your digital shelf more accessible to the customer. Whether the online shopper is using a general internet search engine like Google or the ecommerce platform’s internal search, you want your product to show up as high in the results as possible.
Scraping product titles and descriptions lets you know which keywords you need to hit. Combined with data on what customers tend to search for when shopping for particular types of products, it will enable you to optimize your product pages.
Customer sentiment data: Gathering customer sentiment data will tell you exactly what people shopping online want to see on your digital shelf. Reviews and comments are where you can find the customer’s perspective on what the market is missing, and adapt your offerings accordingly. Furthermore, they will tell you what your competition lacks and what people tend to complain about when browsing particular products.
With these insights, you will be able to change your product descriptions to highlight information that answers the main questions buyers have and improve your visual presentation. Although customer sentiment data is qualitative, you need large quantities of it that can only be provided by automated scraping in order to have statistically significant results.
Main Challenges in Digital Shelf Optimization
The obstacles to effectively optimizing your digital shelf are related to the quantity and particularity of data needed for the right and timely action.
The need for granular data: The accuracy needed for the aforementioned use cases can only be provided by granular data. In data analysis, granularity refers to the level of detail in the data. Detailed data not only improves the accuracy of information but allows easy integration of datasets from various sources.
Achieving granularity requires collecting data on a very large scale. Companies might be able to gather basic ecommerce data with in-house scraping tools, but rarely on such a scale and in real time. Granular data analysis requires powerful scraping tools for fast collection, storing and processing of the data, which takes time to develop. For many small- to mid-level businesses, looking for opportunities to outsource public data collection is the only resort.
Location-based analytics: Many ecommerce websites have geo-restricted content. One needs to be able to access all the location-based data in real time for comprehensive analytics. Thus, a vast and robust proxy infrastructure is necessary to collect data from diverse regions without interruptions.
Developing such infrastructure in-house would send the costs through the roof. For many companies, the resources needed would defeat the purpose of digital shelf optimization efforts, even if they had the technical knowledge to achieve it. Once again, this challenge calls for a necessity to cooperate with a reliable proxy service provider or a company that serves ready-made scraping solutions and entire datasets.
Summing Up
Digital shelf optimization is a competitive, fast-paced game that firms can benefit a lot from. The analytics needed to play this game right should reflect its pace. Public web data collection provides means to this end but it comes with a host of resource-related challenges that require cooperation with trustful web scraping service providers.
Meanwhile, the online shopper also can benefit from competitiveness in this field. More competition leads to broader options and better deals. The way to empower digital shelves of all-sized businesses to compete on a higher level is to make large-scale scraping tools more accessible.
Tomas Montvilas is a Chief Commercial Officer at Oxylabs, a leading big data infrastructure and proxy solutions provider. He is an expert on organizational growth with over seven years of experience in leadership roles in the areas of sales, marketing, product development and digital transformation. In the last few years he has focused on the tech industry, leading large digital transformation programs and overseeing the development of AI-powered customer data management and analytics solutions. Previously, as a management consultant, Montvilas helped clients around the globe to devise the most effective go-to market strategies. He has supported the launch of countless new ventures aimed at bringing more value to telecommunications, media, and CPG industries.
