FreshMail created 9 years ago edited 9 years ago

How to use big data in ecommerce

The buzz around big data in the last few years has become louder although attempts to use it to increase internet sales have a 15 year history. And as is often the case with popular trends, attention hasn’t always turned into a better understanding of just what big data is and how it can be used.

Let’s begin with a look at big data itself. The concise technical definition referring to the 3V model (volume, velocity and variety) doesn’t really apply to benefits that a business can gain from utilizing big data in its sales model so I will refer to the traditional sales process using the example of a simple neighborhood store.

Let’s say that you’re thinking about opening a store offering things like fruits and vegetables. You start by determining the overall demand for what you’re selling in terms of the number of residents in the area and similar stores already operating around you. Then you move on to sales statistics - things like the potential sales you might expect in such conditions. This is all the data that you can start with.

But what else can you do? You can ask potential customers about the kinds of things they might like or expect to see in your store. So let’s say you do a bit of door to door market research and you gather as much consumer information as you can. But it turns out that such information (declarative data) isn’t the best solution for your needs. Later I’ll share more about how people rationalize their purchasing decisions regarding products that in fact they don’t need.

So maybe an analysis of actual sales data would be better? You can easily track what goes out the door and what sits on shelves and waits so why not put that information to work? Having transactional data - primarily information about which products sell best - enables you to adjust pricing or create price-based promotions but that’s it. You don’t know why someone bought a particular product and you don’t know what connects particular customers who buy the same product. Most importantly, you don’t have enough data to recognize the correlation between, for example, the sales of a particular product during certain time periods.

This kind of data would most likely help you to put together more effective promotions but there’s no way for you to get it. You are limited to acting on the information you have to confirm or disprove your forecast.

Analyzing customer behavior online is easier than in the real world

Big data brings a qualitative change to the sales process. More than anything else, the internet has made collecting customer data easier than ever before. Every action taken online leaves a footprint. This makes it possible to put them together in a way that was purely abstract until recently. Can something a bank customer posts on Facebook have an influence on his creditworthiness in the eyes of the bank? Does an examination of what he buys in an internet bookstore be the basis for believing that he needs advice on the subject of divorce? Do someone’s phone records have a place in helping an employer evaluate a job candidate? Questions like these were once found only in the realm of science fiction but are making their way into our world.

Answering these questions demands access to certain bits of data as well as the ability to connect and analyze them, something beyond the reach of most small and medium-sized businesses. At the highest levels of international business, however, there are resources available to pursue these challenges and the days of the big data scientist have arrived.

It turns out that big data analysis isn’t just used to help drive sales. The best example of this might be the Google Flu Trends project, in which Google shares information about search results regarding the flu all over the world. It lists the official statistics of cases of the flu and highlights the correlation between the number of cases in a given area and the frequency of search queries associated with it.

Source: Google Flu Trends

The advantage of big data over other data analysis models

How does big data represent such a huge leap forward from traditional models of data analysis? It’s mostly in the fact that we don’t have to search for an answer about why something is happening. You can put together even the strangest combinations of data and you will still gain insight into exactly what is happening.

In professional speak, big data looks for correlation, not causation. A big data analyst doesn’t wonder why someone who declares an interest in something on their Facebook wall is a good target for an ad campaing for a certain brand of clothes or cars. The correlation that he sees after combining everything with the data of others tells him what is happening. The marketing department can then adjust the next step in their advertising on the basis of this information. Without access to big data, this kind of reaction would not be possible.

I’ve already referred to the problems of depending on declarative data about purchasing decisions but let’s return to the subject to examine it in more detail. Research shows that answers about why certain decisions are made can be deceptive. People have a tendency to justify their choices after the fact and assign completely different motives than the ones that actually drove them - usually explaining everything in terms of completely rational motivations for buying a particular product.

Psychologists uncovered this fact by conducting experiments in which subjects were asked to distinguish between two products that were in fact identical but were presented as having some kind of difference. When asked to explain their choice, subjects could “rationally” explain it. This is one of the cognitive errors that we use to explain our perception of reality. It especially helps us in situations where we are not totally satisified with a purchase but we want to create both a positive emotion and a false belief that the purchase was justified (“The car I bought cost more than I thought it would but it’s big and comfortable and will be useful when I have kids…”). Returning to our earlier example of a neighborhood store, declarations from customers about the desire to buy a certain product doesn’t translate into actual purchases of it.

People don’t buy products, they buy better versions of themselves

Everyone in marketing knows Samuel Hulick’s line about why we buy. We create stories around products that are easier to accept rationally, while the actual impact of marketing campaigns moves the realm of emotions. Do people buy a white Kanye West t-shirt for $120 because of the high-quality material it uses or because they need something to wear? Does each new iPhone so much better than the one before that users of the previous model have to upgrade? Maybe consumers can answer these questions but do we need them for our sales strategy? Not necessarily and often not at all.

Source: A.P.C. Store

The sales process based on big data is free from concerns about such explanations - we simply don’t need them. Amazon, one of the giants of ecommerce, used the opinions of editors and critics to create its recommendation system in its early days and depended on this small group to decide if people who read a given book would be interested in another one, which made the editors happy because such decisions were justifiable. Switching to the use of big data in the late 1990’s along with site traffic and transactional data allowed Amazon to create an automatic system for up-selling recommendations that quickly eliminated the “human factor” and currently drives around 30% of Amazon’s sales (according to various sources since, Amazon doens’t publish these stats).

This cast further doubt on the reliability of “rational” decisions and lends credibility to the idea that “If something is stupid but it works, it’s not stupid”.

Big data isn’t just for big players

So how can an average company use the potential of big data? As I mentioned above, even assuming you run an internet store, you still don’t have access to a lot of data that could be useful after some kind of analysis. Remember that one of the advantages that makes big data different is volume great enough to allow us establish correlations.

If we have 1000 customers that have provided us with usable data, analysis can give us totally different results than if we had a database from ten or a hundred thousand customers. Big data is all about the bigger the better - in statistics, further analysis of the data doesn’t always increase our knowledge, which is the exact opposite of big data.

Big data eliminates the issue that faces traditional statistical analysis - generating random samples. In statistics, every distortion of randomness in the process can invalidate the results. In other words, if we want to apply a statistical survey in order to gain knowledge about our customers, we would have a much bigger problem with obtaining useful sales data results than focusing on the potential of big data.

What else can you do?

First of all, don’t look for a reason for everything. Narratives are extremely useful at the stage of creating a brand image, but beginning with the analysis of the data you need to get rid of the need to create stories about why the customer took advantage of a given purchasing decision. If you do not escape the habit to wanting to explain everything, the more data you collect, the more likely that your explanation will be wrong. Why? Because with time you will create more and more complicated hypotheses to explain things and will not be able to cover all the factors that actually influenced the decision. Paradoxically, creating clarification with more data will move you further away from the truth.

So where do you get the data you need? First of all, work with the data that you already have and start getting more of it. You don’t need sophisticated tools - just use what’s collected by the giant of big data, Google. Google Analytics will provide you with a lot of valuable information not only about your own users, but also about online trends. And it's free.

In sales figures analysis, you can consider factors like:

how customers got to your web page
how much time they spent there
what they were interested in ( things like what they looked at before making their final decisions )
the number of bounces.

This kind of information about sales data lets you effectively adjust to the profiles and needs of your customers.

Having data like this enables you to automate your sales and marketing activities. As I mentioned before, the more data you have, the more accurate and useful the results of your analysis will be. Start automating the collection of data as soon as you can and let your system use it to adapt your message - to change the frequency of your newsletters, to send particular offers to particular subscribers, etc. Apart from email communication, use upselling systems based on advanced algorithms like those on Prestashop or Shopify that work similar to the way Amazon’s recommendation system operates.

It’s hard to underestimate the meaning of big data, especially for predicting future trends. Historical data doesn’t and can’t show us what will happen. Big data’s greatest asset is its ability to tell us what’s happening now and help us to react to it in real time.