Comparison shopping website

Comparison shopping agent

In the early development stage from 1995 to 2000, comparison shopping agents included not only price comparison but also rating and review services for online vendors and products. Altogether, there were three broad categories of comparison shopping services.

Later, through mergers and acquisitions, many services were consolidated.

Services

Through 1998 and 1999, various firms developed technology that searched retailers websites for prices and stored them in a central database. Users could then search for a product, and see a list of retailers and prices for that product. Advertisers did not pay to be listed but paid for every click on a price. Streetprices, founded in 1997, was a very early company in this space; it invented price graphs and email alerts in 1998.

Technology

Price comparison sites can collect data directly from merchants. Retailers who want to list their products on the website then supply their own lists of products and prices, and these are matched against the original database. This is done by a mixture of information extraction, fuzzy logic and human labour.

Comparison sites can also collect data through a data feed file. Merchants provide information electronically in a set format. This data is then imported by the comparison website. Some third party businesses are providing consolidation of data feeds so that comparison sites do not have to import from many different merchants. Affiliate networks aggregate data feeds from many merchants and provide them to the price comparison sites. Many of the popular shopping websites provide direct affiliation to the customer who wants to become affiliate partner. They provide their own API to the affiliate partner to show their products with specifications to the affiliate partner's website. This enables price comparison sites to monetize the products contained in the feeds by earning commissions on click through traffic. Other price comparison sites have deals with merchants and aggregate feeds using their own technology.

In recent years, many off the shelf software solutions have been developed that allow website owners to take price comparison websites' inventory data to place retailer prices (context adverts) on their blog or content the only website. In return, the content website owners receive a small share of the revenue earned by the price comparison website. This is often referred to as the revenue share business model.

Another approach is to crawl the web for prices. This means the comparison service scans retail web pages to retrieve the prices, instead of relying on the retailers to supply them. This method is also sometimes called 'scraping' information. Some, mostly smaller, independent sites solely use this method, to get prices directly from the websites that it is using for the comparison.

Yet another approach to collect data is through crowdsourcing. This lets the price comparison engine collect data from almost any source without the complexities of building a crawler or the logistics of setting up data feeds at the expense of lower coverage comprehensiveness. Sites that use this method rely on visitors contributing pricing data. Unlike discussion forums, which also collect visitor input, price comparison sites that use this method combine data with related inputs and add it to the main database though collaborative filtering, artificial intelligence, or human labor. Data contributors may be rewarded for the effort through prizes, cash, or other social incentives.

However, some combination of these two approaches is most frequently used. Some search engines are starting to blend information from standard feeds with information from sites where product stock-keeping units (SKUs) are unavailable.

Empirical projects that assessed the functionality and performance of page-wise SSC engines (AKA bots) exist. These studies demonstrate that no best or parsimonious shopping bot exists with respect to price advantage.

Major websites

Business models

Price comparison sites typically do not charge users anything to use the site. Instead, they are monetized through payments from retailers who are listed on the site. Depending on the particular business model of the comparison shopping site, retailers either pay a flat fee to be included on the site, pay a fee each time a user clicks through to the retailer web site, or pay every time a user completes a specified action - for example, when they buy something or register with their e-mail address. Comparison shopping sites obtain large product data feeds covering many different retailers from affiliate networks such as LinkShare and Commission Junction. There are also companies that specialize in data feed consolidation for the purpose of price comparison and that charge users for accessing this data. When products from these feeds are displayed on their sites they earn money each time a visitor clicks through to the merchant's site and buys something. Search results may be sorted by the amount of payment received from the merchants listed on the website. large price comparison sites.

In addition to comparing tangible goods, some sites compare prices of services, such as insurance, credit cards, phone bills, and money transfer.

Google Panda and price comparison

Like most websites, price comparison websites partly rely on search engines for visitors. The general nature of shopping focused price comparison websites is that, since their content is provided by retail stores, content on price comparison websites is unlikely to be absolutely unique. The table style layout of a comparison website could be considered by Google as "Autogenerated Content and Roundup/Comparison Type of Pages". As of the 2011 updates to its search algorithm, known as Google Panda, Google seems to have started considering these comparison sites to be of low quality.

Niche players, fake test sites and fraud

Due to large affiliate network providers providing easily accessible information on large amounts of similar products from multiple vendors, in recent years small price comparison sites have been able to use technology that was previously only available to large price comparison sites.

This technology includes software and plugins aimed to standardize the typical processes involving price and product comparison. Without much resources it became possible for amateurs to build seemingly professional websites, mostly using the popular Wordpress CMS. These small sites often rely on Google for their visitors, which are monetized using affiliate networks like Amazon.

The low content quality of typical niche sites, often bordering on spam and fraud, is a growing problem from the perspective of consumer protection and the quality of search engines. By playing the algorithm of search engine giant Google, it is possible to place low quality sites prominently in the search results.

Until recently the phenomenon of fake test or comparison websites had escaped public attention. An analysis by testbericht.de discovered that 34,6% of German search traffic related to product tests on the first page of google leads to fake test sites. When a big German newspaper published a report about such a website and consumer protection organization sending out warning letters, observers started to note a sense of panic in the industry, with site owners changing or deleting the content in question. Amazon, being the biggest player in the affiliate market, declined to comment on the matter.

Deceptive comparison sites give the impression of testing a product thoroughly. In reality, the tests are just an aggregation of freely available information, often leading to the most expensive products being recommended. This in turn increases the commission rate the site owners earn for the recommended products.

Google EU Court Case

In 2017, the European Commission fined Google €2.42BN for allegedly monopolising the comparison shopping engine (CSE) market. Google released a statement that the European Commission's assessment will be appealed. Google appealed the decision to the General Court, which largely upheld the fine in November 2021. Google further appealed this ruling to the European Court of Justice. On 10 September 2024, the court dismissed Google's appeal, finalising the €2.42 billion penalty and concluding the legal dispute.

Literature

• Doorenbos, Robert B.; Etzioni, Oren; Weld, Daniel S. (8 February 1997). A scalable comparison-shopping agent for the World-Wide Web. AGENTS '97: Proceedings of the first international conference on Autonomous agents. pp. 39–48. doi:10.1145/267658.267666. Retrieved 30 May 2026.
• Clark, David (March 2000). "Shopbots Become Agents for Business Change". Computer. 33 (2): 18–21. doi:10.1109/MC.2000.820034. Retrieved 30 May 2026.