A data-informed approach to community fundraising

Tuesday, October 20, 2015 Nabeel Gillani 2 Comments


A data-informed approach to community fundraising

At Khan Academy, our mission is to provide a free, world-class education for anyone, anywhere. With a staff of fewer than 100 people, the only way we can make good on the “free” part for our 30 million registered learners and millions of others is through the funding we receive from our generous donors. And as the educational needs of billions around the globe continue to evolve, for us to make good on the “world-class” part, we must creatively tap into new sources of funding. Our learners are the reason we come into work every day, and we’re committed to helping them learn regardless of what they can or cannot afford. But what more could we do if those learners (and their families) that are able to chip in actually contributed just a few dollars (or pesos, or rupees, or any other currency) to fuel our mission?

It may sound ambitious -- even crazy! -- to believe we can mobilize our community of users to help us continue to fund our mission. But organizations like Wikipedia and the Obama campaign have shown us how powerful a community of donors can be in the pursuit of an audacious vision. These efforts have relied heavily on experimentation and quantitative analysis to better understand what motivates different kinds of people to donate, and why.

In a similar vein, we’ve started to conduct several different kinds of experiments to better understand what motivates Khan Academy users to donate. Emails, social media, and even carrier pigeons are all channels through which we can reach potential donors (TBD on the carrier pigeons). But given that many of our users find value on the platform itself, how can we encourage giving when they’re on the site, without disrupting their learning experience?


Banner message tests

On-site banners (like the ones below) are one way to encourage users to become donors. Over the past several weeks, we’ve been running experiments to see which banner messages are the most compelling in motivating users to donate. These experiments entail sampling a subset of our users and showing them one of several banners when they’re on the site to investigate which banners lead to the greatest number (and total $ amount) of donations. Users can, of course, always dismiss the banner so they don’t have to see it anymore. Experimenting with different banner messages is one way to learn what users value about their Khan Academy experiences and what motivates them to give. In our latest experiment, we found a particularly interesting trend with two different messages, both of which are shown here:


Message A:

“Khan Academy is a small nonprofit with a big mission: a free, world-class education for anyone, anywhere. With fewer than 100 employees, Khan Academy serves more than 15 million users each month with over 100,000 videos and exercises.

If everyone on the site this week gave $20, it would fund all of our work for the next year. If you’ve gotten something out of our site, please take a second to pitch in and help us get back to teaching!”
 
Message B:

“Khan Academy is a small nonprofit with a big mission: a free, world-class education for anyone, anywhere.

If everyone on the site this week gave $3, it would fund all of our work for the next year. That’s right - the price of a cup of coffee is all we ask. We’re so lucky to be able to serve people like you, and we hope you’ll pitch in to help us continue to do our best work.”
 

Message B only asked for a $3 donation by default, since other (particularly higher) amounts might not make sense (unless you drink really expensive coffee!). Message A, however, varied in its default ask with the following amounts: $1, $2, $3, $19, $201 (with the $20 version of the default ask shown above).

So, which message performed better? Well, it depends on what we mean by “better”. When evaluating message strings, all things equal (except for the string itself), we look at 2 key metrics: the % of banner viewers that actually end up contributing, and the total amount (in $) generated from each message. This is because we care about mobilizing the greatest number of users to give us as much money as they’re able to.

So, which message mobilized a higher fraction of our users to give - Message A, or Message B?

It turns out that users who saw Message B were significantly more likely to donate than those who saw Message A (p < 0.05). But was it really the “coffee”-related text that motivated people to donate, or was it the fact that this message had such a low ask amount of $3? If we look at the percentage of banner viewers who donated when they saw Message A at a $3 default ask, even though this percentage is lower than Message B’s, the difference is not statistically significant (p = 0.33). Moreover, we’ve learned that people tend to donate at lower amounts ($3 or under) significantly more often than higher ones. This suggests that many of our users are inclined to give smaller-sized gifts.

Message B might have brought in the highest number of donors - but did it also raise the most $?

In this case, no - Message A brought in approximately 1.4x more in total gifts than Message B2. If we look at the distributions of gift sizes for each (not including projected revenue from recurring gifts), shown below, we can begin to understand why.


These charts suggest that those users who are asked for only $3 - for example, by equating it to a cup of coffee in Message B - tend not to deviate much from that amount. Therefore, even though many of them donated, they mostly donated at a relatively low amount, yielding lower total $ raised. Those asked for an amount that is not pegged to a specific item (i.e. an amount that fell in the range described earlier) tended to give at those amounts -- but also, at amounts they weren’t asked (e.g., $10). It turns out, Professor Daniel Kahneman wasn’t lying when he said anchoring is real! 


Challenges


There are infinite levels of analyses we can perform to better understand what motivates our users to become donors. There are also several challenges that arise when running experiments like the one described above. Here are a few:

  • Inferring why. While the data can tell us “what” or “which” - e.g., which messages motivate users to donate -- it can’t tell us “why” these users actually donate. This is where supplementing these experimental efforts with qualitative user research is essential (more below). 
  • Sample sizes are often small. Even with the large number of users that visit Khan Academy, because a fraction of users are sampled to be shown banners3, and an even smaller fraction actually ends up donating, the total number of donors is small. This can make it challenging to obtain and evaluate valid results quickly, especially if the goal is to iterate on new message strings to learn as quickly as possible what resonates and what doesn’t. 
  • No perfectly-consistent control group. Running experiments over time opens the door to several confounding factors like the seasonality of site usage or different propensities to give at different times of the year -- all of which could sway results. 
  • Deciding what to test is tough! With so many possible messages, banner styles, amounts, and a host of other variables, picking which variables to test is often half the battle. In these cases, a nice mix of data-informed insights and good ol’ fashioned creativity is crucial.



Looking ahead

A world where every Khan Academy user chips in whatever they can to help us collectively achieve our mission is a world we are all very excited about. In addition to experiments, other research, including qualitative surveys, has taught us even more about the deep commitment many of our users have to Khan Academy. Here’s a powerful quote from a Khan Academy parent:

“...we live right above the poverty line but so appreciate everything khan does for our son and even us grown-ups in learn[ing] for free. We don't have much but if there was a way that we could donate in small amounts ($10 or less) it would make us feel better as a family that we are contributing to our education. We have a 12 year old hispanic son living in the middle of the ghetto but he knows how to write java script [sic] and reads at college level. Thanks for all you do.”

Donate today and help us deliver a free, world-class education for anyone, anywhere.


  • 1. The amounts were launched as a separate, “overlapping” experiment so that the effects of default ask amounts can also be explored independently. 
  • 2. If we account for recurring gifts at an average lifetime of 7 months, this multiplier shrinks to approximately 1.1x. It’s important to note that given the small number of donations, many of these values are highly sensitive to outliers (especially large gifts). 
  • 3. In an effort to incrementally improve our messaging, we’ve chosen to show only a subset of our users banners, with plans to widen the sample size as we learn more about what motivates people to give.  

2 comments:

  1. There is an area of statistics called Small Area Estimation that can help to deal with small sample sizes. Have you tried to apply advanced statistical methods for your research?

    ReplyDelete
  2. Hi, thanks for the blog! "If we look at the percentage of banner viewers who donated when they saw Message A at a $3 default ask".. is there a typo here? I thought Message A was at $20 default ask.

    Also, how did you establish "Moreover, we’ve learned that people tend to donate at lower amounts ($3 or under) significantly more often than higher ones."?

    Just trying to understand this better...thanks!!

    ReplyDelete