Challenges of rateshopping

In this article, WeYield summaries the challenges of collecting competitive pricing data and the factors you should consider when reviewing data suppliers.

Key topics

Why automated rate scraping is an important part of the pricing decision process

In order to determine the best price for your product given your current sales performance, it is important to have an idea of the current market rates and the direction that prices are moving. This will enable you to raise prices (and RPD) confident in the knowledge that it might not affect your overall positioning in the market. Alternatively if you need to generate demand because you are behind schedule, you can lower prices gently to a more appropriate price level and potentially improve your position in results, without requiring a drastic overall drop (1) .

Typically, small rental car businesses have collected this information manually and on adhoc basis. However this is not an effective use of your staff’s time and abilities. By automating the collection of this data, it frees up valuable time to spend analysing the data rather than collecting it. It also ensures that the data can be collected in a consistent timely manner to help you identify key trends in the marketplace. Certainly, as your demand for more competitive information grows, you will soon reach the stage where manual collection is not possible.

Cached vs. Scheduled & Real time rate capture

There are many rateshopping suppliers in the marketplace offering the ability to collect competitor prices. It is important to establish if the supplier is really collecting prices on a real time capture basis, or if they are accessing a cache of prices. At WeYield, we collect prices on an on demand basis. The queries are scheduled to run during the night, so data is available to the clients in the morning of their business day. On top of this, we recently launched Rateshop planner. This tool allows clients to build and schedule their own rate collections, which is advantageous if you need to review destinations / durations / dates not in your standard scheduled data collections.

Cached prices are not the same as real time captures, it means that the data a client sees, is not captured on demand, but periodically by the rateshopping supplier (with no input from the client) and the cache/ temporary storage itself is queried when the client requests their data. This can speed up data retrieval, but the cached data may already be out of date. Often, if there are data collection issues, the gaps are plugged by older cached data. Sometimes suppliers use an API to access the data used to fill the cache, this may not always match the prices displayed on the competitor live websites. There is potential that if using cached prices to make pricing decisions, you might not be making accurate judgements if the data itself does not reflect the price a real customer would see.

Scraping and bot detection

Prices are collected via scripts often referred to as bots. These are pieces of code that are built to visit a website, perform a search (as a customer would do) and then record the results. As the internet has evolved, this has resulted in websites becoming more dynamic (i.e they frequently change layouts, so scripts to scrape data have to be continuously rebuilt) and at the same time, websites have become better at recognising visitors that are not human, and in turn blocking these or worse, misleading the ‘bot visitor’ with incorrect pricing and availability. To counter this data scraping companies use a variety of techniques to mask their identity.

  1. Firstly, they utilise IP address supplied by multiple IP suppliers, so that the target sites see visitors from different locations.
  2. Secondly the script technologies have improved so that the site visit behaviour better resembles human website browsing.
  3. Thirdly each data collection is done as a clean visit, i.e there is no cookie history attached to the IP, it’s as if a brand new computer is used, using a browser for the first time, with no other search history, hence the target site has no cookie information to identify the customer, This is turn limits the visits exposure to the future problem of personalised pricing.

Rise of personalised pricing

As the availability of customer data has increased, travel businesses are becoming increasingly clever in the way they can promote products to customers. In the airline business for example, the New Distribution Capabilities (NDC) technologies, are starting to allow airlines to price a flight ticket differently to every search customer based on a number of factors, including but not limited to past booking history, reward membership status, time to departure, travel purpose, general demand for the flight. This will become the norm in the airline industry in the next few years, and will likely be the same on OTA and large Chain Hotel websites too. By doing a ‘clean’ search automated using technology of WeYileld’s partners, our clients will not see these personalised pricing effects, just the general price intended for the product.

Why prices might differ via manual collection vs. automated means

Our clients often manual check the prices we have collected and it is quite possible there will be some variance. This is likely due to a number of factors:

  1. firstly, prices may have changed since our data collection time due to more automation in the marketplace,
  2. secondly, our price is from a ‘clean shop’, it is quite possible that your browser contains lots of browsing information/cookie history and this might influence the prices the target site displays, especially if it detects that you frequently search but don’t book anything. We have heard instances of major brand car rental companies, not being able to manually shop their competitors sites from their offices as the IP address is recognised and blocked.

Other considerations: Day time throttling / GEO IP Impact

Data vendors have to be conscious of the activity of the sites they scrape, so as not to disrupt their core business (selling products to consumers) as a result we typically see slower data collections during daytime business hours. This is fair so as to not disrupt traffic to the sites and risk being blocked permanently from collecting data from them. Another consideration is the impact of different IP addresses on the prices and availability of products captured during scraping. Clever brokers and brands now utilise technologies to show different prices to different customer markets. It is important that your data vendor is aware of which sites perform this activity and are able to force the correct use of geographical IP addresses when scraping as appropriate to the data request.

Future challenges – Member only rates , Increased appetite for data / automation of price setting.

In the future we are seeing some travel business push priority pricing, for customers who have logged into their site or are part of their rewards scheme. It is important that data vendors monitor this closely as it develops so that you continue to get an accurate picture of pricing in your market. We are also seeing an increased appetite for competitive data as tools are developed to help in the automation of pricing, WeYield is itself looking into this area with great interest.

To summarize:

  • Rateshopping via screenscraping has been around since the early 2000’s and is utilised by key players throughout the Travel Industry.
  • Price monitoring is a key part of the pricing decision process when considered alongside up-to-date data about your own performance.
  • Insist on live shopped rates and not cached ones.
  • Ensure that the rateshopping supplier you choose has a history of high accuracy and the ability to navigate difficult sites / blocking technologies such as captcha pages.
  • Consider your supplier’s ability to scale as your desire for more competitive data increases, do they have a decent sized IT team that is required to ensure all the web bots you use are maintained and accessible at all times.
  • Ensure your data supplier acts ethically when it captures data, you do not want to be blocked from collecting data from your key broker / sales channels during peak season!

Please do not hesitate to get in touch with WeYield if you have any further queries regarding the pricing data collection process.

  1. Refer to WeYield academy about top-down pricing. To be avoided at all times.


Leave a Reply