How we built Asana’s Marketing Measurement Capabilities to power global customer growth
Growing a customer base at scale effectively and efficiently through paid advertising is challenging in the modern digital world. It requires robust and holistic marketing measurement to give marketers visibility into how their campaigns perform at different granularities in a timely manner.
Paid acquisition plays an essential role in helping Asana reach our potential customers worldwide and expand our customer base at scale. We invest in advertising platforms such as social media, display, video, and search, to acquire sign-ups from teams all over the world. Asana’s Marketing Data Science team built a suite of capabilities that measures paid advertising performance in a privacy-conscious way. Here’s how we did it.
Marketing measurement at Asana
Marketing measurement is not a new topic. About 100 years ago, John Wanamaker said, “half of the money I spend on advertising is wasted; the trouble is I don’t know which half”. Our Data Science and Paid Acquisition teams had similar questions:
- How can we know if our advertising spend is effective in driving business outcomes?
- How can we know whether the return of advertising is worth its spend?
- Of the many advertising channels we use, how can we allocate the budget to maximize the return?
Today’s world is a lot different from what John was facing a century ago. First, it’s a digital age with countless options for marketers to spend their advertising dollars. Second, its privacy landscape is evolving quickly which could impact marketers’ ability to measure advertising performance.
To allow marketers to maximize the return of their advertising budget in the digital advertising world in a privacy-conscious way, Asana Marketing Data Scientists have built a suite of marketing measurement workhorses. We led and executed the full lifecycle of developing most of the workhorses, from scoping, researching and prototyping, productionizing, to landing the project among stakeholders, and providing ongoing support to drive the adoption of the data products.
In this post, we’ll focus on Marketing Mix Modeling and User Lifetime Value.
Modern Marketing Mix Modeling
Attribution is often used to keep track of what ads were seen and/or clicked on by a user before signing up. If a specific ad is always present in the customer journey of sign-ups, it probably means it is effective in driving sign-ups. Though this method sounds simple conceptually, it relies on persistent and consistent user identity across all the advertising touchpoints and sign-up.
Modern privacy regulations prevent many ad touches from being tracked.
Marketing Mix Modeling (MMM) allows us to gauge the effectiveness of ads without tracking at the individual level. Instead of using user-level data that associates ad touches with sign-ups, MMM uses aggregated time-series data. The idea of measuring ad performance with the aggregated time-series data is simple: If we usually see a rise in sign-ups after we increase YouTube ads spend, it probably means paid social is effectively driving sign-ups. However, generating accurate measurements to fulfill marketers’ needs is never a simple task. With the advancements in statistical methods, machine learning, and computing power, this task is actually possible to achieve.
Data in this chart is for visualization purpose only and not real
Revolutionize MMM with cutting-edge methods
Our Marketing Data Science team leveraged many cutting-edge methods to push the boundaries of MMM. While many companies can only have MMM insights at channel level every year or twice a year, at Asana, we’re able to obtain granular MMM insights across audiences, tactics, keywords, campaigns for different geographical regions every month.
One major approach that we use is Bayesian statistics. It allows us to share knowledge across periods, regions, and various types of ads. Knowledge sharing helps reduce the data points needed to measure the heterogeneous performance of different types of ads in different regions in driving business outcomes.
Markov chain Monte Carlo (MCMC) is one of the main methods used for true Bayesian computation. It is notoriously expensive computationally and thus, it often gets replaced with other methods in practice. Considering the benefits of a true Bayesian methodology for our use case, we invested in finding a way to speed up MCMC computation. Through research and testing, we are able to build a parallel MCMC process. Our parallelization speeds up the computation by more than 10 times.
The team carried out further research and testing to identify the optimal model parameters. We used domain-specific data transformation methods that try to mimic how advertising affects business outcomes. For example, we leveraged “adstock” transformation to represent the lingering effect of advertising, and “saturation” transformation to represent diminishing returns.
Productionize MMM Pipeline
After the research and methodology development, the team productionized the workflow into an end-to-end pipeline that includes data collection and preparation, data transformation, modeling, reporting and visualization modules. Although we still need a “modeler in the loop” for building MMM, the productionized pipeline encapsulates complex underlying codes and provides a user-friendly interface for Data Scientists to build MMM models with “drag and drop” ease. It also streamlines and speeds up the workflow, and helps us ensure data and model quality.
MMM Calibration
A big question is, how can we know if our MMM is providing accurate estimations of the causal impact of advertising? In causal inference, it is challenging to obtain unbiased estimates of causal impact without conducting hold-out experiments. For this reason, we conduct Geo-Based Experiments and ad platform Conversion Lift Tests to calibrate MMM. We also cross-check MMM results with performance measured by attribution methods, with the acknowledgment that attribution can be very wrong due to the loss-of-tracking data problem.
User Lifetime Value
Measuring the effectiveness of advertising in driving sign-ups is not sufficient for us to understand its profitability. Different sign-ups can bring us drastically different monetization values. What complicates things further is the fact that advertising often acquires individual users, however, Asana’s revenue is derived from the teams and organizations that these individual users are part of. The individual users can have different roles (executive, project manager, individual contributors, etc.) and different levels of adoption of our platform. As a result, their contribution to the revenue Asana collects from their teams/organizations should not be evenly distributed.
Since the monetization value of individual users is difficult to measure using A/B tests, we built causal inference models using observational data, with careful control of confounding factors to disentangle individuals’ impact on the monetization of their teams/organizations.
Another challenge in measuring user lifetime value at Asana is related to our freemium subscription business model. A majority of our paying customers initially adopt our product through free trials, and we later collect revenue from them usually over a very long period of time. However, to make the user lifetime value a lot more useful for paid acquisition, we make predictions of it based on early signals.
Role of Data Scientists in building the data applications
Marketing Data Scientists wear different hats at different stages in the life cycle. We behave like a data product manager when we are planning the roadmap of data products and scoping a project with stakeholders. We are researchers when we are exploring and evaluating different data science solutions to business problems. We show our engineering muscle following engineering workflow and best practices when we productionize machine learning models or build data pipelines.
Everyone on the team plays an important role, and we collaborate with each other and cross-functional teams to seek their expertise, and leverage many tools and documentation that exist within the company (thanks to our Data and Engineering community). Data Scientists at Asana have plenty of opportunities and the flexibility to grow in different dimensions.
Join Asana’s Marketing Data Science team
We’re looking for full-stack Data Scientists who are excited about working in the intersection of Data Science and Marketing to help us uplevel marketing measurement and empower Asana customer growth at all stages in the funnel. You’ll get to work with the amazing Asanas I (Yurong Fan) co-created with on this work, including Data Scientists Dana Ilea, Azoacha Forcheh, Rishabh Meswani, Charles Midwinter and cross-functional partners Sonya Chu, Kevin Ngo, Sam Mazaheri, and Sarah Charlton. You’ll also work with many other Asanas from Growth Marketing, Revenue Marketing, Marketing Analytics, Marketing Operations, etc.”
Check out our open roles on our Careers site and apply today.