US Government strengthens international aid model with qualitative data analysis

USAID uses NVivo to improve learning agenda

Background

The United States Agency of International Development (USAID) is a U.S. government agency that betters the lives of millions across the globe through areas such as improving agricultural productivity, combating maternal and child mortality, providing immediate disaster relief, fostering private sector development, elevating the role of women and girls, and providing humanitarian support.[i] Since its inception during the Kennedy administration, USAID has made an impact in countries across the world.
Funded by the USAID Bureau for Policy, Planning and Learning (PPL), the USAID LEARN contract —implemented by Dexis Consulting Group—is designed to support strategic learning and knowledge management at USAID to improve the performance and effectiveness of its programs around the world.[ii]
USAID LEARN works with multiple teams and departments within USAID through an approach called, “collaborating, learning and adapting (CLA).” Part of this approach includes using Learning Agendas, which are a set of agency-specific questions directly related to its work. When these questions are answered, the Learning Agenda guides staff  through organizational learning in order to create an environment of growth and success.

The Challenge

USAID LEARN conducted 60 interviews and held two focus groups to properly analyze the Learning Agenda’s current landscape within federal agencies. They reviewed brochures and programs already informed by Learning Agendas across agencies, including The Department of Housing and Urban Development (HUD), The Department of Agriculture (USDA), The Department of Labor (DOL), The United Nations Research Institute for Social Development (UNRISD).[iii]
The analysis accumulated textual, video, audio, PDF or image and numerical data. However, traditional quantitative data analysis tools are only able to analyze the numerical data collected. And the traditional, pen/paper/highlighter/sticky note method of analysis would take too long. Individually reviewing each image, interview, and document would waste hours of time and risk the loss of important themes or ideas as time passes. USAID LEARN and Dexis Consulting Group needed a different solution.

Introducing NVivo

USAID LEARN conducted an assessment of the potential software that can analyze all of their qualitative (non-numerical) data in a quick and meaningful way. Because NVivo is often recognized as the industry standard for qualitative data analytics in education, many of the analysts had previous, positive experience working with the solution. NVivo’s functionality, accuracy and speed are exactly what the team was seeking, so it was the unanimous choice.

Results

The USAID LEARN team was impressed to learn from their NVivo research that the Learning Agenda, serving as an integral way to understand both how and why to grow, was in use far more widely than anticipated.

“We discovered that there are over 20 operating units within USAID using Learning Agendas. A variety of people, from HUD to the UN, in all different technical areas are also using them to learn and grow,” said Matt Baker, USAID LEARN’s Monitoring Evaluation, Research and Learning Specialist.

NVivo also helped the USAID LEARN team uncover the themes that make up their Learning Agenda questions.

“Because NVivo can interpret live language, it was able to read over 100 questions and show us how we could distill them down to the best five,” said Baker,

NVivo revealed the top reasons why Learning Agendas are useful by identifying three primary motivations for those who use them:

  1. Expectations of accountability, especially in response to leadership demands
  2. Leadership transitions and structural, strategic, or policy changes
  3. Responses to identified program-related needs

Many other discoveries were made in the landscape analysis that will lead the USAID LEARN contract as the team begins the one-to-two-year process of developing strong Learning Agendas for USAID. Because of this, USAID has the opportunity to improve the effectiveness of its important work, both in Washington and across the globe.

[i] https://www.usaid.gov/what-we-do
[ii] https://usaidlearninglab.org/learn-contract
[iii] https://usaidlearninglab.org/sites/default/files/resource/files/landscape_analysis_report_04_10_17.pdf
 

ABOUT THE AUTHOR

QSR International with USAID and Dexis Consulting Group

The United States Agency of International Development (USAID) is a U.S. government agency that betters the lives of millions across the globe through areas such as improving agricultural productivity, combating maternal and child mortality, providing immediate disaster relief, fostering private sector development, elevating the role of women and girls, and providing humanitarian support. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Research in the field: Shooting Stars and the success of their community program

Australian not-for-profit are utilizing NVivo to evaluate the success of their community program.

Shooting Stars is an initiative of Netball WA and Glass Jar Australia, which uses netball and other incentives, as vehicles to encourage greater engagement and attendance at school of young Aboriginal and Torres Strait Islander girls living in Western Australia’s remote communities and regional towns. Through fostering collaborative relationships with the local community, Shooting Stars works with schools and service providers to implement a regular program of personal development activity and fun for young Aboriginal and Torres Strait Islander girls.

Participation in the program enables girls to have a taste of success to improve their self-image, develop hope and aspirations about their future, and provide motivation to attend school to achieve those aspirations.

Following a successful pilot in 2014 in Halls Creek, the program had its full-scale launch into five locations throughout the State in 2015. Funded by the Department of Prime Minister and Cabinet through to the end of 2017 across six locations, Shooting Stars plans to expand to 14 sites in 2018.

Participants undertake a program that involves netball and physical education activities, such as swimming when the weather is too hot for a game, over two sessions a week. There are also two health and wellbeing sessions covering topics including nutrition and healthy relationships aligned to the Australian Curriculum.

A unique research challenge

Shooting Stars was faced with a unique challenge when wanting to both gather and analyze data to provide feedback on the program. Firstly, staff were dealing with a group of children ranging in age from primary school through to teenagers, and found that the standard method of paper-based surveys was not engaging and did not help in keeping the program as culturally centered as they desired.

To address this issue, a new method of data collection was established through consultation with local cultural leaders. The traditional indigenous yarning circle was presented as an alternative to paper-based surveys, to gain feedback from the girls in the program, in a way that encouraged open discussion.

“The yarning circle is really the idea of sitting around and ‘having a yarn’,” explained Rose Whitau, Research Manager at Shooting Stars – Midwest Gascoyne.

“We’ve found this to be a very participant driven piece of research, through the yarning circles, which are driven by a localised steering committee, made up of our staff, school staff, local cultural leaders, and the remote school attendance strategy provider,” she said.

The next challenge was that yarning circles produced unstructured data, in the form of audio recordings of the sessions, and they required a tool to analyze this data.

“We wanted to turn messy qualitative data into something that was easy to digest, and NVivo has allowed our team to do this,” Whitau explained.

“The purpose of using NVivo is to be able to evaluate the program ongoing, as we want our participants to have an impact on how the program is structured and delivered, which is very empowering for them,” she said.

NVivo is now being used throughout Shooting Stars locations to analyze the results of the yarning circles, of which there have been 22 so far.

“NVivo is the perfect tool for our needs, especially its ability to analyze audio files. Coming from an academic research background, I was quite familiar with the methodology, but I have also been able to easily teach the software to our other staff who come from a range of different professional backgrounds,” Whitau said.

Shooting Stars continues to affect real change, and work to empower indigenous Australian girls to stay in school and engage in their education, as well as maintain a positive attitude toward their health and wellbeing. You can learn more about the great work they’re doing on their website.

ABOUT THE AUTHOR

QSR International with Shooting Stars

Shooting Stars is an initiative of Netball WA and Glass Jar Australia, which uses netball as a vehicle to encourage greater engagement and attendance at school of young Aboriginal girls living in WA’s remote communities and regional towns. Working collaboratively with the local community, schools and service providers to implement a regular program of personal development activity and fun for young Aboriginal girls. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Living Kingston 2035: Creating an informed vision with NVivo

Located in the Melbourne metropolitan area, the Council and management team at the City of Kingston decided to create a shared, long term vision for its diverse community. The project was named Living Kingston 2035 and provided the community with an opportunity to share their needs, interests and aspirations. The Project Team, led by Dr. Robyn Cochrane, used NVivo to progressively analyze and securely manage a significant volume of data which included survey responses, online submissions and forum discussions. By using NVivo, the City of Kingston could efficiently and systematically sort the community’s input to make informed evidence-based decisions and develop plans for the medium to long term.

Benefits of using NVivo:

Introducing Living Kingston 2035

In March 2011, Council resolved to create a long-term community vision for Kingston. The vision was to become a key legacy for the outgoing Council and provide useful input to the decision-making and planning processes of the incoming Council (following elections in October 2012). After considering Officers’ advice, Council resolved to establish a Steering Group and Reference Panel to oversee and guide the community visioning project as well as a Project Team to implement the project.

Planning the consultation

The Project Team recognized the many challenges posed by effectively engaging with the community, particularly about a topic as far-reaching and abstract as a “community vision”. A review of a community visioning exercises undertaken nationally uncovered a range of consultation approaches that had been usefully applied. The Project Team noted the importance of carefully framing key questions that could be used across multiple participation options and decided upon:

Early decisions were made to use several communication channels to inform the community about the project and to offer multiple participation options to enable interested community members to submit their views in the consultation. The consultation was envisaged to comprise traditional paper based surveys, telephone-in options, staff forums, public community forums as well as the use of online, web based and mobile channels. A small number of local government authorities had experimented with web-based technologies and it was quickly agreed that these technologies could provide an additional participation option that might overcome some of traditional consultation challenges. Web-based technologies were regarded as a legitimate and innovative way to meaningfully engage with the community, encourage broader participation and more diverse inputs into the consultation.

Sourcing project funding

The project was initiated and funded by the City of Kingston. However, the Project Team was aware of a community support grants program offered by the Victorian Government for community support initiatives. Contact was made with representatives of the Department of Planning and Community Development to seek advice as to the potential suitability of this project for funding. Following an intensive grant application process and lengthy and competitive external evaluation process, the City of Kingston’s application was successful. A grant of $48,000 was secured to support the consultation and, in particular, the community strengthening and capacity building features of the project.

Turning unstructured data into an informed direction

The Project Team recognized that it was critical to the integrity of the project that the data were efficiently managed and reported to key stakeholders throughout the duration of the project.

NVivo was used at three key milestone stages:

Using NVivo our Project Team had the capacity to produce multiple reports encompassing over 1,000 pages of “raw data”. The reports presented the findings for each broad theme, geographical areas (Council ward) and participation option (survey, online submissions and forum discussions). High-level descriptive reports as well as detailed research reports were produced to suit the needs of specific audiences.

The role of NVivo

Dr. Cochrane was already a proficient user of QSR International’s NVivo having used the product prior to working with the City of Kingston. Thus, the Project Team recognized that NVivo was a relatively inexpensive computer-assisted qualitative data-analysis software tool that could efficiently and effectively manage large volumes of consultation data. Consequently, the Project Team elected to use NVivo 10 because it:

Using NVivo 10 the Project Team were able to:

Although it is difficult to estimate, it may have taken the Project Team around three months to manually analyze the consultation data without NVivo. Managing, analyzing and subsequently reporting the data without NVivo would have been an extremely labor-intensive task. It is highly unlikely that the findings would have been formally reported with the same level of confidence, rigor and visual aids.

The use of best practice computer-assisted qualitative data-analysis software minimizes subjectivity within the sorting and presenting of the data and allows this to occur without researcher bias. It was believed a computer-assisted approach would provide better and clearer outcomes from the consultation.

Living Kingston 2035 Program Leader Dr. Robyn Cochrane stated “We could quickly and confidently report to our management team and Councilors that all of the data had been carefully sorted using NVivo 10. There was little room for researcher bias and loss of data – we could confidently say that every idea was coded, sorted and reported.

Sarah Bishop, Manager – Communications and Public Affairs commented “It was crucial for us to quickly turnaround the consultation data so our Councilors could develop their draft 2013-17 Council Plan. By using NVivo our project team could quickly generate visualizations, frequency tables and customized detailed reports. When these reports were considered alongside the participant profiles for each data set and other information about the municipality of Kingston, Councilors could confidently use the consultation data to make decisions and plan for the future."

Word frequency query for the Arts, Culture and Heritage WISH ideas displayed visually as a tag cloud. The size and density of each word reflects the frequency reported in the WISH ideas. For the purposes of this example, the 100 most frequently reported words (comprising four or more letters) were selected.

Dr. Cochrane added “Sharing in-depth and high-level descriptive versions of the consultation findings with key stakeholders and the community is important to the integrity of the project and is best practice in the public participation sphere. Through this project, the City of Kingston has informed, consulted, involved, collaborated and empowered its community (see iap2.org.au)”.

Related products:

About City of Kingston:

City of Kingston is responsible for ensuring that its 142,000 residents are well governed and serviced. To find out more: kingston.vic.gov.au. The Victorian Government has created a guide which details the roles of local Councils. To find out more: localgovernment.vic.gov.au

About Dr. Robyn Cochrane:
Dr. Robyn Cochrane was employed by the City of Kingston as Project Leader of Living Kingston 2035. Robyn works as a Research/Teaching Associate with Monash University and is the lead researcher with Cochrane Research Solutions. Robyn holds a PhD in Management and has over 25 years of experience in the local government, vocational education and training, higher education and community sectors. Robyn has published articles in academic and practitioner journals and produced a range of research reports for industry partners. Dr. Cochrane’s interests include: qualitative and quantitative research designs and analysis; online and traditional surveying approaches and community strengthening. Contact details: robyncoc@bigpond.net.au