Understanding the data behind notifications

Notifications can be a source of timely information, but they can also be disruptive. Here at Asana, we’re looking into how we can make notifications better: How can we only deliver important information, at the right time, with minimal interruptions of productive work? The first step for us was understanding how people are using our notifications right now.

One of our first, and most striking, findings from our new dataset on notifications was that our users seem to like Like notifications– a lot. Not only are Like notifications of all types the most likely to be opened, but they’re the most likely to be re-opened.

But are Like notifications helpful?

One of the most important metrics that we track is collaboration, which we define as any action that creates a notification for another person. Our previous research has shown that collaboration within a team is one of the strongest predictors of whether or not the team is successful in adopting Asana. Helpful notifications, then, are ones that lead to users taking collaborative actions.

Unfortunately, we have no way to establish a causal relationship between a notification and a collaborative action. However, we can approximate this by determining if an action was taken on the object about which the notification was sent, by the person to whom the notification was sent. By this metric, it seems that Like notifications are responsible for more collaborations than the average notification.

But there may be an important confound that we’re not considering in the last analysis. Namely, if a person receives a Like notification about an object, that means they’ve already engaged with the object. It’s possible that the high collaboration rates reflect this previous investment in the object, rather than the utility of the Like notification.

To separate out these two effects, we compared collaboration rates of Like notifications that were opened alone to collaboration rates of Like notifications that were opened in a batched form with other, non-Like notifications about the same object. We hypothesized that if the Likes themselves were driving the collaborations, then the two groups should have similar rates of collaboration, whereas if a more general interest in the object was driving the collaboration, then the Likes batched with non-Like notifications should have a comparatively higher rate of collaboration.

Indeed, we found that the batched notifications were a much stronger driver of collaboration than Likes opened alone. This suggested to us that Likes may not be a strong promoter of product engagement.

Next, we hypothesized that Likes may be playing a social role. Perhaps when people Like others’ contributions regularly, this promotes a reciprocal relationship where others are more likely to open or collaborate on the notifications they send. We examined this hypothesis by performing a Gradient Boosting Regression to identify which user actions are most predictive of people’s sent notifications being collaborated with by others.

However, when we ran our predictive model, we found that Liking things wasn’t even in the top 10 predictors of which actions drive others to collaborate with one’s notifications. Indeed, we identified many factors that influence whose notifications are more likely to be collaborated with, but Likes did not appear to be an important predictor, at least relative to other user actions.

Finally, we examined whether receiving Likes made a user more likely to continue using Asana, particularly if they would otherwise leave the product. In this analysis, we looked at relatively new users who had been absent from Asana for a month, in order to identify which notifications were most highly associated with their return. We determined that notifications played a large role in encouraging users to continue using Asana, but again, Like notifications were not an important factor.

So, what’s next?

In this analysis, we came to the conclusion that Like notifications were occupying a large amount of users’ interest, but without necessarily imparting value. Don’t worry, we have no plans to remove likes from Asana. However, we will use our findings to inform how we treat Likes within the platform: How they should be visualized, delivered, when, and more.

One main takeaway is that there are other actions in Asana that are more important than Likes, which is consistent with how we think about mindfulness and maximizing productivity at work. That said, we know how important validation and praise are in the workplace, and know that our users cherish Likes (and other features that provide delight, like celebrations). As with any decisions we make about our product, we plan to continue balancing judgement with rules and incentives—all while trying to delight our customers every day.