To click or not to click …
Can the psychology behind recommendation engines and online purchases help us to build better models?
Recommendations while browsing or shopping on the Internet or using a mobile application have become so commonplace these days that many people don’t even give them a second thought.
Despite knowing how recommendation algorithms work, I often find myself asking, ‘What were they thinking?’ And I am sure that many other users are occasionally confused, irritated or even suspicious when some recommendations pop up on their screens.
Is the purpose of recommendation systems to help the user navigate an infinite amount of information, or is it to convert views to purchases, reads or subscriptions?
1. Technically speaking …
Most online and mobile apps and e-commerce websites tend to use a combination of content-based and collaborative filtering systems.
Content-based recommendation engines use a consumer’s purchasing or browsing history to make recommendations for potential future purchases. For example, once you have watched an on-demand movie, the app will recommend other movies that share similar characteristics, like genre or actors.
Collaborative filtering makes recommendations based on preferences of similar users. These could be people who are in your contacts list or users in the same location or other consumers who have made similar purchasing decisions.
Early recommender systems were memory-based and therefore dependent on the size and complexity of an organisation’s database of users. Nowadays we have a proliferation of data providers, many of whom collect real-time crowd-sourced data. With advances in machine learning, modern engines are model-based. Techniques such as regression, clustering and classification are used to create and update models that are increasingly sophisticated in terms of building user profiles and churning out suggestions.
Recommendation engines have become so important that when Netflix introduced its system, it awarded a prize of a million dollars for the best collaborative filtering algorithm to predict user ratings for movies. Amazon uses a similar process which they call item-to-item collaborative filtering.
Well, that’s the technical stuff! But what is it that causes users to either click or ignore recommendations? And, more importantly, what are the factors that lead to more than just clicking?
2. It’s about emotions …
The success of recommendation engines in leading to actual purchases can be explained by various theories and concepts in psychology. Many of these have been used by marketers and salespeople for centuries.
“When dealing with people, let us remember we are not dealing with creatures of logic. We are dealing with creatures of emotion, creatures bristling with prejudices and motivated by pride and vanity.” — Dale Carnegie
- Trust: The user needs to trust, not just the service provider, but also the system itself. Third party users are invaluable in building trust on websites and applications — if a user can see other users’ reviews and choices (both positive and negative), it provides reassurance that the system, organisation or products are continuously being evaluated by independent users. Access to user profiles can provide further comfort, as this enables an evaluation of who is providing reviews or recommendations.
Recent surveys indicate that more that 90% of young adults trust online reviews as much as personal recommendations.
- Cognitive dissonance: Suggestions should be consonant with the user’s previous behaviour and choices, and there should not be contradictory information presented. This is one of the reasons why recommendations often start with similar items that a user has bought or viewed previously, before recommending more obscure items. A good recommendation engine will seek to reduce any inconsistent thoughts or beliefs (i.e. cognitive dissonance) that a user may have about a recommended action. If a user rejects a recommended product or gives it a low rating, the system should not continue to recommend similar products. A site that advertises or implies free shipping before or during the buying process, should not add shipping costs upon checkout.
More than 60% of cart abandonments happen because of extra costs for shipping that consumers were not expecting.
- Flow: A person in a flow state is fully immersed in and enjoying the activity which he or she is performing, and therefore more likely to be open to recommendations. In e-commerce, things like ease of use and speed of processes influence whether a user is in this state during their browsing or shopping experience. Visual or process interruptions during the user’s online experience can also disturb the flow state, and may lead to the user abandoning their purchase.
Some surveys show that almost 30% of shoppers abandon their carts because the checkout process takes too long or is unnecessarily complicated. Other common reasons for cancelling transactions include having to open accounts before purchasing, having to provide shipping addresses multiple times, and having to re-enter credit card information during the transaction process.
- Stereotypes, prejudices and hype: Targeting products or services to certain market segments is often based on stereotypes of what different classes of people spend their money on or are interested in. In the context of misinformation like fake news or clickbait, recommendations on what might interest a user based on previous views or purchases tend to confirm pre-existing beliefs instead of challenging them (known as confirmation bias in psychology). Fake news, for example, tends to spread false narratives about disempowered groups, thereby perpetuating biases and stereotypes. The result is that social, political or economic decision-making occurs based on hype instead of evidence.
Up to 50% of people surveyed in the United Kingdom believed that they encountered fake news daily, and almost 90% of respondents in other surveys felt that fake news was made worse by the internet.
3. Keeping up with the Joneses …
Our online behavior is not just about our individual values. What our friends and neighbors are doing, saying, reading or buying influences how we behave.
“Man is by nature a social animal ….” — Aristotle
Social psychology studies how our behavior, and even our thoughts and feelings, are influenced by other people. An example of this is the phenomenon of ‘groupthink’ where individuals make irrational or dysfunctional decisions in order to fit in with or conform to a group.
The presence of other group members who influence our decisions does not have to be physical or real — it can be implied or even imagined. This has significant implications for online behavior.
- Persuasion: One of the more complex concepts in psychology is persuasion, i.e. the non-coercive process of using a message or other communication to influence someone’s attitudes or behavior. This is linked to the other concepts and processes already discussed, and is the crux of advertising or marketing efforts — converting interest to sales, or other desired outcomes. One of the principles of persuasion is feeling validated by what other people are doing. Whether in the form of ratings (stars, likes or hearts), reviews or recommendations (‘customers who bought this also bought …’), no platform does ‘social proof’ better than web and mobile applications and sites.
Conversion rates (from landing page to completed purchase) are usually low, varying between 1.5% to 5%. Amazon reportedly had a 30% increase in sales after introducing its recommendation system.
4. So it’s about conversion rates?
Understanding the psychology that enables users to enjoy their browsing or shopping experience, and convert their interest into purchases or content consumption, should help developers to create better recommender models.
But what constitutes ‘better’ systems?
- Most studies on the value of recommender systems focus on increase in sales (or reads for content providers).
- The user should receive some quid pro quo (the principle of reciprocity in persuasion or influence theory). For a new user, landing on a company’s website is akin to walking into its business premises for the first time — this is a potential customer who needs help navigating your products or services. A return customer expects personalized and appropriate recommendations.
In addition to enhancing customer experience and profits, any system implemented by a business also exists in the context of society. It is disconcerting that we live in a ‘post-truth society’, one where clickbait and fake news are consumed alongside other material online … sometimes with very little to distinguish between them.
Behaving in an ethical manner is part of every business’ social license to operate … but that is a topic on its own.
