It has been two years since the World Bank published a report that stated that over 30 percent of its policy reports have never been downloaded even once and only 13 percent of policy reports were downloaded at least 250 times. The debate among development practitioners that followed made it clear that the World Bank is by far not alone with this phenomenon and that most international organizations, including UNDP, face the exact same challenge.
As UNDP provides support services for implementation of the Sustainable Development Goals (SDGs), we in UNDP’s Knowledge Management Team see the importance of getting insights into the perceived value of our knowledge products and therefore UNDP’s thought leadership in various SDG topics.
In fact, UNDP’s Knowledge Management Strategy 2014-2017 pointed out that UNDP needs to invest in its process of planning, developing and disseminating knowledge products in ways that make them “more relevant to clients’ needs, more flexible and timely in their development and format, and more measurable in their quality and impact.”
During the debate that followed the World Bank’s report, we the Knowledge Management Team at UNDP thought long and hard how to get meaningful data about who is actually reading its publications, to what extent those readers find individual publications useful, and most importantly, for what end those products are actually being used. In order to do this right one would almost need to talk to each individual reader and ask them one by one, which is kind of impossible on an ongoing basis. Or is it?
Well, after several prototypes and tests during the last year, we’ve finally come up with a model to do just that. In March 2016, we tweaked UNDP’s Public Library of Publications so it would present users with a post-download pop-up asking them whether they would be willing to leave their email address so we could contact them later.
In the six month since we introduced this question, over 42,000 users left us their email addresses, and we since followed up with 27,000 of them (through a weekly survey issued a few weeks after a download of a specific publication), asking them how useful they found the specific document they downloaded, what organization they are with, and whether/how the publication made a difference in their work. As of September 2016 we received 1186 survey responses, and the insights we get from our audience goes far and beyond any of the intel we had in the past.
We can now see how useful our publications are to our users, and to what extent specific publications reflect on UNDP’s thought leadership in that topic:
Even with the possibility of a voluntary response bias, the numbers serve as a valuable baseline to track changes in perceived usefulness over time. In addition, we now for the first time get a clear picture who is getting value out of our publications:
And most importantly, we learn from our audience how and for what purpose they use the downloaded publications in their work:
We are also getting great qualitative feedback on how we can improve specific publications in the future, and the individual comments provide great anecdotal evidence at project or community level that demonstrate the impact of UNDP’s work on the ground. Here are some of the impact stories we’ve received:
“The publication was used in the development of our food security and livelihood strategy for the Uganda refugee operation.”
“The publication has been useful as a starting point to persuade managers of Nature Reserves and Forest Reserve to consider ecotourism planning besides conventional forest management planning.”
“Some of the inputs were used in our legislative agenda setting, especially those that are applicable to the Philippines situation.”
“I am working in Rwanda’s Environment Management Authority and the publication is useful for public sensitization.”
“I introduce the paper to PhD students in my development administration class and asked them to prepare a paper on SDG targets.”
“The publication was of fundamental importance for the Pedagogical Political Plan formulation for professional training courses developed within my organization, the Military Police of Mato Grosso, Brazil.”
Going forward, we are making this qualitative feedback available to all our staff, so they will be able to look up their publication and go through all the individual comments the publication received. It is this kind of evidence that shows us where investment in the quality of our publications pays out, and where we need to switch gear, improve our efforts, or shift our focus entirely with regards to specific thematic areas. Most of all, it is these stories that inspire us as staff on a daily basis as they remind us why we are doing what we are doing in our pursuit of sustainable human development.
Of course, this measurement approach is only reaching those who download publications online, and will miss out on all those who receive them through hard copies or through presentations at workshops and conferences.
What did your organization do to get feedback from your offline audience, and do you have any suggestions for how UNDP could fine-tune the above measurement approach? Leave comments below, I’d be glad to hear your suggestions!