The Social Data Revolution

We are generating more data than mankind has ever done. How do we intelligently and usefully process this outpouring of knowledge (though not always of wisdom)? Andreas Weigend, a former Chief Scientist at amazon.com discusses the implications for marketing - and for society:

"In 2009, more data will be generated by individuals than in the entire history of mankind through 2008. Information overload is more serious than ever. What are the implications for marketing?

The world has witnessed two revolutions in the way consumer data has been solicited and collected. And consumers have changed the way they use the web to converse, shop and transact. As a result, people have elevated their expectations for good, healthy customer relationships and exchanges. And this has put pressure on marketers to forge astute, coherent strategies for how they engage people, what data they gather, and how they use it.

The first data revolution came about when web commerce got going in earnest. It arose from the dream of collecting data from consumer decision-making. With the advent of the web, firms pondered whether it might be worth saving the vast amounts of data that customers were generating through their clicks and searches. For consumers, there was no hiding: after all, there is no online equivalent of discreetly checking out a magazine while a bookstore employee is looking the other way. Amazon.com has pretty much saved all user data from its beginning.

Back then, customers had no choice but to share their intentions with firms. If a technology enthusiast wanted to find out if a website sold a particular surveillance device, there was no shortcut but to type some keywords into a search box and therefore give the company a valuable intention stream. Companies, therefore, had all the power. Many tried too hard to push products and advertisements. The consumer had no voice.

During the first data revolution, successful companies gained power by collecting, aggregating, and analyzing the customer data they collected. However, most companies did not know what to do and ended up burying their data in tombs.

The second data revolution brought about a new dimension to data creation: users started to actively contribute explicit data such as information about themselves, their friends, or about the items they purchased. These data went far beyond the click-and-search data that characterized the first decade of the web.

An early example of user-generated content was Amazon.com's reviews system. The firm realized that users often trusted recommendations by other users more than promotional material found elsewhere on the web. By enabling users to actively contribute such explicit data, Amazon.com succeeded in leveraging knowledge dormant in its large customer base to help customers with their purchasing decisions.

Later, Wikipedia increased transparency even more by allowing online collaboration. By allowing users to interact and build on top of each other, the site relinquished control over their space. The benefit of allowing such user interaction today is obvious — why spend time on hold with a customer service representative if we can just Google that cryptic error code to see if someone else has already solved the same problem? People learned that by sheer large numbers, an online user community was likely to be more helpful than a representative employed by the company.

Today, the online world has shifted to a model of collaboration and explicit data creation. Successful firms develop systematic ways to encourage and reward users who contribute honest data. A good system does not try to trick customers into revealing demographics or contact information that is useful for the company. Rather, it rewards users with information that is useful to them.

Netflix, for example, allows users to contribute ratings for movies that they have seen. Users have an incentive to contribute accurate data because this will give them better recommendations for new movies. The 1999 "Web 2.0 company" MoodLogic (acquired by All Media Guide, in turn acquired by Macrovision) enabled users to create metadata about their favorite music. Why on earth would they do that? Because they got back playlists, which made it easier for them to discover new music they enjoyed. Such successful companies realized the key feature of a good incentive system: people need to see that they profit from the outcome in some way if they are willing to put in the effort to contribute truthfully.

In the last few years, users have gone a lot further than contributing metadata to movies and music: in fact, they have taken center stage. The center of the universe has shifted from e-business to me-business. Customers are also starting to discover and interact with each other. Knowing that they are not alone has shifted the balance of power from companies back to consumers. And they have begun to demand transparency. Customers are beginning to have a voice. They are realizing that the data they voluntarily contribute can help them and others with making decisions, providing true value. In turn, they want to be treated fairly as individuals by the companies they pay attention and money to.

What are the consequences of this change towards the expectations of consumers?

Successful interactions have become genuine communication with near-instantaneous feedback. For example, PayScale allows users to retrieve real-time salary reports based on their job title, location, education, and experience-but only after they have contributed their own data. As the expectations of users change, firms must spend more time developing incentive systems that will entice more users to participate.

Consequently, the online world is beginning to be ruled by the expectations of the users. No longer is it sufficient for a search engine to cough up some hotels across the world when a weary traveler is looking for a good deal in Bangkok! As these consumer expectations shift, companies that want to stay relevant have no choice but to accept the ideas of the consumer revolution as swiftly as possible. For users, switching costs are cheap — firms can no longer think of "customer relationship management" as providing stickiness for the customer (just like fly paper provides stickiness to the fly). Industries such as real estate and automobiles whose business models are built on information asymmetries will quickly lose their revenues to those who increase transparency using data contributed by consumers.

This leaves us several deep questions to ponder, including what the implications are on customer expectations, and what companies can do to address these expectations