A Blog by Jonathan Low


Feb 8, 2017

The Challenge of Measuring Agile Marketing

Managers are inclined to measure outcomes. But understanding the interrelationship between results and the processes that lead to them is just as important. JL

Kimberly Whitler reports in Forbes:

Agile marketing is a framework for how a marketing team operates. It's important to distinguish between measuring agile methods versus the activities enabled through agile. We're usually more concerned with which worked and what impact they had. Have a framework that aligns activities to outcomes. It guides the linkages between objectives and  activities, spend and outcomes across channels. It encourages the strategic focus on effectiveness and efficiency that helps optimize
The five experts below provide differing perspectives on arguably the most challenging aspect of agile—measuring efficacy.

The team of experts:
Jennifer ZeszutCEO, Beckon
Jim EwelPresident of Peel the Layers and publisher of the Agile Marketing.Net blog
Mark VeroneVP, Global Operations Automation, Gogo
Roland SmartVP of Social & Community Marketing, Oracle & Author of The Agile Marketer: Turning Customer Experience Into Your Competitive Advantage
Scott BrinkerCo-founder & CTO of ion interactive; Editor of chiefmartec.com; Program Chair of MarTech, Author of Hacking Marketing: Agile Practices to Make Marketing Smarter, Faster, and More Innovative 

Kimberly Whitler: How can you measure agile marketing?
Scott Brinker: Since agile marketing is a management framework for how a marketing team operates, it's important to distinguish between measuring agile methods themselves versus measuring the activities enabled through agile marketing. We're usually more concerned with the latter—which experiments worked and what quantitative impact have they had on the business. You can use whichever KPIs are relevant to your business for measuring those outcomes, independent of your adoption of agile management practices.
However, you can measure the success of agile practices by tracking the number and velocity of marketing tasks and stories completed per sprint, the number of experiments run, and aggregate measurements of marketing's net performance over time, including the reported happiness of marketing's stakeholders (including the marketing staff themselves).
Jennifer Zeszut: To be agile, you’re going to have to change the processes and culture of your team to some degree. And so, it’s important to establish upfront with your team what the measures of success will be. Much like using data to optimize marketing, regular review and evaluation of performance against your agility objectives gives you an opportunity to optimize your ways of working. But to get there you need a system that enables agile measurement—one that supports your agile decision making by answering the right questions at the right cadence.
To help ensure marketers can answer the right questions, it’s important to have a framework in place that aligns marketing activities to business outcomes. It guides them to the linkages between their business objectives and their activities, spend and outcomes across all channels. It also helps them identify not only the basic volumetric measures of importance (clicks, conversions, etc.), but also encourages them to explore the important strategic ones that focus on effectiveness (such as paid/earned ratio) and efficiency (such as cost of acquisition), that really help them optimize. With the data sources and metrics in place, the last step is to identify the dimensions required so the data can be sliced and diced in a way that leads to actionable answers. When complete, this process yields an excellent definition of their agile marketing data requirements.
It’s important to consider cadence, too. As mentioned before, cadence is a key factor in unlocking the value of agile—the more frequent your decisions, the more value you’ll drive. Most thoughts turn to digital channels as the most opportunity, which is true – they can keep up with whatever pace your team can sustain. But offline media has opportunities too. You may not be able to change your commitments, but changing the content of your placements can yield great results.
Roland Smart: I think there are two groups of metrics in question here. As Scott points out, there are measurements of the Agile process itself, for example:
• Cycle time (Kanban) and velocity (Scrum) are measurements of productivity or work product. These help teams assess the impact of changes to their process or method.
• Estimating scope, and measuring accuracy in retrospect, helps teams get better at estimating what’s possible for the team within a set timeframe. This helps with optimizing resource allocation.
• Burn down charts measure progress against sprint scope; these can also be applied on the epic level. This measure helps teams stay focused throughout a sprint and anticipate the need to push items into a future release.
• Cumulative flow measures for understanding the consistency of team throughput. Like cycle time and velocity this can help teams understand how their process or method is working.
• Bug/fix measures to help measure the quality of the work being created by the team (i.e. how often are we fixing stuff that was released).
Then there are measures associated with what you’re developing with agile. As Scott points out these can be your standard measures or KPIs. More generally, I think of three major groups:
• Validation measures that help you determine if you’re headed in the right direction. This could be anything from web metrics, to in-store engagement, etc.
• Test-related measures fall into two groups, those to discover and those to optimize. I learned of this approach from Sean Ellis of growthhackers.com—tests to discover determine if there’s even a “there” there. Think about playing Battleship—is there even a ship in this location? Tests to optimize determine if you can improve the experience—how can I sink the ship?
• Prioritization measures help team determine what they should work on next by giving work items a relative score for importance and feasibility.
Whitler: How do you know which of the hundreds of potential projects to pursue—what measures are you looking at?
Smart: This is a prioritization question. There are multiple inputs but one that is quite complementary with agile is a framework that assigns relative values to importance versus feasibility to identify where you’ll get the most bang for your buck.

Whitler: How do you know when or if more resources are warranted? What benchmarks or measurements are you looking for?
Smart: Scope accuracy measures are one way to get a sense of this. It really depends on how quickly you need to get something done. Once you have a cycle time or velocity measure you can predict how long it will take you to reach milestones. If it takes too long than you need to increase throughput (e.g. increase resourcing).
Jim Ewel: Metrics for agile marketing should be business outcome metrics, rather than so-called “vanity metrics.”  Vanity metrics are too often used by conventional marketers to justify the success of big bang campaigns, where because of the cost—both in terms of dollars and time—marketers feel compelled to declare “success” based on whatever metrics make the campaign look good. Examples of vanity metrics include things like page views, attendees at an event, recall of an ad’s message, etc. Agile marketers hold themselves to the same metrics as other portions of the business—revenue, profitability, market share. If a product or service is new to the market, agile marketers measure events that move the sale forward. For example, they measure sales-qualified leads or trials or conversions.
How do agile marketers choose from hundreds of potential projects? They do so based on alignment with the larger goals of the organization. That’s why it’s critical that marketers meet with senior management and sales on a regular basis, and that they show in a transparent way how marketing is impacting the goals of the organization.
Verone: I think the answer to this questions is “it depends” but whatever you do, ALWAYS, ALWAYS measure along the way. Data and reporting are critical to agile marketing. It depends on what are the expectations – what is the strategy? In some cases you might be “testing” a concept and you might conduct some experiments to see which direction is the best path forward. In other cases, you might need to be prepared for plan b, c, or d in the event that something is not working the way it was anticipated. The measurement of success or failure needs to have enough data points to learn from. If something was successful then why was it successful? If something failed, why did it fail? In either case you need data to learn. One method of measurement is A/B testing, split testing or bucket testing. With A/B testing you could launch different versions of a creative or design – Version A and Version B to measure performance against a specific set of variables, clicks, engagement, conversions, etc. between a random and fairly equal sample. Another method is multi-variate testing (MVT) which employs similar methods as A/B testing but with an increased number of variables which could be A+X or A+Y versus B+X or B+Y. Both methods offer real-time feedback and performance metrics to determine whether the experiment is working or not and is a great tool for the agile marketer.


Post a Comment