US Government strengthens international aid model with qualitative data analysis

USAID uses NVivo to improve learning agenda

Background

The United States Agency of International Development (USAID) is a U.S. government agency that betters the lives of millions across the globe through areas such as improving agricultural productivity, combating maternal and child mortality, providing immediate disaster relief, fostering private sector development, elevating the role of women and girls, and providing humanitarian support.[i] Since its inception during the Kennedy administration, USAID has made an impact in countries across the world.
Funded by the USAID Bureau for Policy, Planning and Learning (PPL), the USAID LEARN contract —implemented by Dexis Consulting Group—is designed to support strategic learning and knowledge management at USAID to improve the performance and effectiveness of its programs around the world.[ii]
USAID LEARN works with multiple teams and departments within USAID through an approach called, “collaborating, learning and adapting (CLA).” Part of this approach includes using Learning Agendas, which are a set of agency-specific questions directly related to its work. When these questions are answered, the Learning Agenda guides staff  through organizational learning in order to create an environment of growth and success.

The Challenge

USAID LEARN conducted 60 interviews and held two focus groups to properly analyze the Learning Agenda’s current landscape within federal agencies. They reviewed brochures and programs already informed by Learning Agendas across agencies, including The Department of Housing and Urban Development (HUD), The Department of Agriculture (USDA), The Department of Labor (DOL), The United Nations Research Institute for Social Development (UNRISD).[iii]
The analysis accumulated textual, video, audio, PDF or image and numerical data. However, traditional quantitative data analysis tools are only able to analyze the numerical data collected. And the traditional, pen/paper/highlighter/sticky note method of analysis would take too long. Individually reviewing each image, interview, and document would waste hours of time and risk the loss of important themes or ideas as time passes. USAID LEARN and Dexis Consulting Group needed a different solution.

Introducing NVivo

USAID LEARN conducted an assessment of the potential software that can analyze all of their qualitative (non-numerical) data in a quick and meaningful way. Because NVivo is often recognized as the industry standard for qualitative data analytics in education, many of the analysts had previous, positive experience working with the solution. NVivo’s functionality, accuracy and speed are exactly what the team was seeking, so it was the unanimous choice.

Results

The USAID LEARN team was impressed to learn from their NVivo research that the Learning Agenda, serving as an integral way to understand both how and why to grow, was in use far more widely than anticipated.

“We discovered that there are over 20 operating units within USAID using Learning Agendas. A variety of people, from HUD to the UN, in all different technical areas are also using them to learn and grow,” said Matt Baker, USAID LEARN’s Monitoring Evaluation, Research and Learning Specialist.

NVivo also helped the USAID LEARN team uncover the themes that make up their Learning Agenda questions.

“Because NVivo can interpret live language, it was able to read over 100 questions and show us how we could distill them down to the best five,” said Baker,

NVivo revealed the top reasons why Learning Agendas are useful by identifying three primary motivations for those who use them:

  1. Expectations of accountability, especially in response to leadership demands
  2. Leadership transitions and structural, strategic, or policy changes
  3. Responses to identified program-related needs

Many other discoveries were made in the landscape analysis that will lead the USAID LEARN contract as the team begins the one-to-two-year process of developing strong Learning Agendas for USAID. Because of this, USAID has the opportunity to improve the effectiveness of its important work, both in Washington and across the globe.

[i] https://www.usaid.gov/what-we-do
[ii] https://usaidlearninglab.org/learn-contract
[iii] https://usaidlearninglab.org/sites/default/files/resource/files/landscape_analysis_report_04_10_17.pdf
 

ABOUT THE AUTHOR

QSR International with USAID and Dexis Consulting Group

The United States Agency of International Development (USAID) is a U.S. government agency that betters the lives of millions across the globe through areas such as improving agricultural productivity, combating maternal and child mortality, providing immediate disaster relief, fostering private sector development, elevating the role of women and girls, and providing humanitarian support. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Analysing large survey data using automated insights

NVivo Plus enables quick identification of themes and sentiment in University wide student feedback.

An impossible task?

The National Student Survey (NSS) is an annual survey which gathers the opinions, and experiences, of final-year undergraduate students on the course that they have studied. It is widely recognised as an authoritative and highly influential survey in building a picture of the quality of life in higher education in the UK.

At Lancaster University, as with many other institutions, there had to date been no systematic analysis of the qualitative element of the survey performed. The challenge was to develop a better and deeper understanding from the data collected, to improve the student experience of living and studying at Lancaster.

This was taken on as a pilot project by Steve Wright, Ph.D., a Learning Technologist in the Faculty of Health and Medicine at Lancaster University.

Finding the right tool for the task

The NSS is very important to Lancaster, as the score that is achieved has a huge influence on the current standing of the University.

Lancaster was awarded Gold in the Teaching Excellence Framework (TEF) following an outstanding NSS score (84.3% positive score) in 2016. It was also recently named University of the Year by the Times and the Sunday Times Good University Guide 2018.

The aim of the NSS pilot project was to feed into a broad spectrum of institutional activity in preparation for the University’s TEF submission and evaluation.

The first phase looked at a comparison of three possible tools to use for the qualitative analysis:

NVivo was chosen because:

NVivo’s ability to present data visually was also an important factor in the selection process. “In NVivo the visualisations enable you to explore the data which makes it a very powerful tool for both analysis and presentation,” said Steve. Rather than put a bar chart in a report, he presents directly from within NVivo. “The ability to use it as an interactive presentation is what makes it so powerful, for example, to be able to click on a column in the histogram and get the underlying data,” he said.

Testing the automation of sentiment and themes

Lancaster University received 8000 NSS comments, amounting to 25,000 words to be analysed, across the institution.

The NSS asked students to complete three open-ended comments:

Steve was interested in finding out what insights could be gleaned from the data, particularly if they used a systematic approach that could be replicated. The approach would pull out key topics, group those key topics and then explore the sentiment related to those topics. The idea being that as the sentiment is there in the NSS structure (asking for a positive and a negative comment), it is possible to check the accuracy of NVivo’s sentiment analysis (negative/positive) against those and then extrapolate from that, or use it as an example of the confidence you can have in NVivo’s automated sentiment tagging for other datasets.

“We have loads of text but mostly what happens is, we share it with no analysis, and only basic structure with the key people in a department for them to read through. And the easiest thing that happens, when they receive three or four pages of comments, is to read the first few and construct a narrative in their head and immediately get information bias,” said Steve.

“NVivo Plus tools were really good for this. I was able to take this minimally structured data – which only gave the department it related to and the type of a comment and then to extract topics and cross-reference those with the sentiment, as well as provide summaries,” he said.

Benefits of using NVivo

The analysis was well received however it received one critical response, which contended his analysis did not show anything further than the previous statistical analysis. Steve argued that was not the case. “The statistics show that students are broadly happy here. They like certain things but specific areas are shown by the quantitative statistics as being lower, however, what the statistics don’t do, is give any real insights as to the processes, the experiences that inform those lower scores.”

“What the qualitative analysis allows you to do, is to pull out those topics and break them apart to see why some departments had a higher score – to identify good practice, and some of the specific reasons given where there were lower scores to inform interventions and development, for example,” Steve said.
NVivo’s sentiment analysis capabilities played an important part in the data analysis, particularly given the way survey data is collected for the NSS.

“Because of the structure of the NSS, of positive comment and negative comment, we were able to cross-tabulate that with NVivo’s sentiment analysis and get a kind of built-in check of accuracy,” said Steve. “And it was very high. It tends to get things wrong by addition, not omission. i.e. it will classify something as both positive and negative when it’s just positive. The classic instance being ‘I had a load of personal problems and the department was fantastic.’ That is a positive comment, but, because it has the word ‘problems’ in it, it is automatically classified as negative as well.

Overwhelmingly NVivo classifies it correctly, we know that there will be some false matches but they’re a minority and given the volume of data it enables us to work with they can be accounted for,” he said.  “What’s more, this gives us a baseline for being confident in the sentiment analysis when we apply this approach to other student feedback and comment data without this structure.”

Being able to share the project across the University with other staff who are familiar with NVivo is an ambition for the future, as opposed to sharing a static report. Staff can delve straight into the project and discover insights for themselves.

Future Work

The University is planning to repeat the analysis next year, and build upon the framework.

From those who have seen it, there has been some real interest. “I think the real potential is with student or staff surveys. Most organisations have staff surveys, and they ask for extensive qualitative comments and usually, don’t do any sort of systematic analysis with them,” said Steve.

The point of the project was to develop a method, and NVivo assisted with a better analysis of this data. “The questions were:

And I really think the support NVivo provided has a real potential for other sectors with those practical priorities for working with qualitative data rather than the software being part of the somewhat arcane, and highly theoretical, pursuits of qualitative analysis within academia,” said Steve.

He also suggests that there’s a significant opportunity for commercial and public-sector organisations who need to work with unstructured datasets for analysing customer experience, and with a lot of potential for further development of methods and approaches like those introduced here.

About Steve Wright

Earning his Ph.D. in E-Research and Technology Enhanced Learning in 2014, Steve Wright works as a Learning Technologist in the Faculty of Health and Medicine at Lancaster University in the UK. He is also an independent CAQDAS trainer, consultant and certified NVivo expert. As a researcher, he completed five small-scale research projects, in addition to his Ph.D. thesis on sensory learning with a focus on craft beer, with which he took an ethnographic approach.

As an academic-related professional, he’s particularly interested in the e-research area and discovering what is possible for digital tools and how they’ll influence new approaches, which remains his focus. He also has an interest in the development, research, and teaching of methods. Steve’s consultancy and training work is through www.caqdas.co.uk    

About QSR International

Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

NVivo in mixed-methods research

NVivo has a long history of assisting qualitative researchers with managing, and analyzing their data, and being a part of complex research. What you might be interested to know, is that NVivo is also being used by mixed methods researchers to help get a wider picture of the data they’re examining.   

Background

Melody Goodman, PhD is a biostatistician and an Associate Professor in the Department of Biostatistics, at New York University Global Public Health. Her work is focused on racial disparities in urban settings, and she is particularly interested in examining how place is a determinant of disparity, and looking at the concept of how ‘where you live, work, play and pray’ impacts your health outcomes. She works in mixed methods research, and as a biostatistician, has a traditionally quantitative view. Much of Melody’s research has been on health and research literacy and how people understand and use health information.

Speaking about the impact of her research, Melody said “My attempt at working towards addressing disparities has been to increase the knowledge level of the receiver of the information. My work spans a wide spectrum from the generation of new statistical methods all the way out to community engaged research, including a program where we train community members on research methods. I don’t specialize in any particular disease area or any statistical methodological area.”

Introduction to NVivo

Melody is currently using NVivo to validate a quantitative survey tool that assesses the level of community partner engagement in research. When her team were required to evaluate their work, they found that there were no existing validated tools, and they would be required to create their own. “We thought that we could find some existing measure and use that to evaluate our programs. But when we went to the literature there really wasn’t any measures that assess how engaged people felt they were in research processes and research centers. So, we decided to develop our own measure, which was really trying to evaluate a big comprehensive research center that had multiple research projects” Melody said.

As the team were developing a brand new measure, and there was no “gold standard” to validate by, they found the qualitative work they undertook was important for a number of reasons; “We were trying to evaluate from the community health stakeholder’s point of view, instead of the academic’s perspective, and considering what the benefit is for a community member for participating in our research and how engaged they feel in the process” Melody said.

Development of a survey tool requires use of mixed-methods and the team uses NVivo to analyze qualitative data. However, they are also using the software as a project management tool, as Melody noted “We had so many different survey data sets and there’s a lot of rounds, with multiple surveys of experts and participants, we really used it to see where we were, and what data we’d collected. Not only were we using it to analyze the qualitative data, we also used it to keep everything in one place for this project.”

Further work with NVivo

Melody is also using NVivo for another project. Melody read some work by sociologist Elijah Anderson who created the term “white space”, meaning spaces which excluded anyone who wasn’t white, either explicitly or implied. It was particularly timely, given the current social climate, and the history of St Louis (where this project originated), with segregation, white flight, suburbanization, and gentrification.

Melody looked at this existing research from Anderson through the lens of her own work, and her own social experiences. “It’s great work, and he had lots of ethnographical stories, but for me as a biostatistician I really want to measure it. As a black person, I could relate to it, but I didn’t feel like I could convince someone who wasn’t black that this idea really existed. If you didn’t have those life experiences, you may view ethnographic work as anecdotal stories” she said.

Melody was interested in creating a survey measure that would assess if a space was a “white space”. “The first thing we needed to do was talk to residents of St Louis, and it became clear that we needed to speak to both black and white residents, and all up we collected around 50 interviews” Melody explained.

As a biostatistician, Melody was particularly interested in gaining a full picture of the areas participants were mentioning in their interviews, to help inform the research further. “In the interviews, the participants talked about different cities, towns, and places such as shopping malls. NVivo was great because I could link census data which has the actual racial composition, and other factors such as percentage of poverty and median household income, of all the places they mentioned, so we could then code the data not only for the town, but also call upon the quantitative data that goes along with it. This is where NVivo showed what it’s really powerful for in mixed methods work” Melody said.

As a mixed methods researcher, Melody thinks of the quantitative data as the ‘what’, and qualitative data and the ‘why’. Having the ability to merge those two together and compare, for example, if someone reports in an interview that a space is predominately white, with available census data, and triangulate that using NVivo, has been an important part of this project.

Project outcomes

“Ultimately with this project, we want to create a quantitative survey tool that will allow others to assess whether a space is perceived to be a ‘white space’. Currently, we’re a long way off from that, but we had to start with the qualitative, asking people how they think about ‘white space’, how they talk about ‘white space’, how they define it, and if they even know what we’re talking about? It became so timely because of all the things that are going on in our country, and in this community in particular. We’ll probably get more from it than just the survey, because people really gave us so much information and were incredibly honest in their responses” Melody said. 

When it came to selecting software for this project, Melody’s previous experience with NVivo led her to choosing to work with it again. “I had difficulty understanding competitor software. As a traditionally quantitative person, with NVivo I could make sense of what my team were doing, and they were able to generate reports that I could understand” Melody said. 

“When I decided that I was going to learn a qualitative software package, I felt most comfortable attempting to learn NVivo. And it was a challenge to think like NVivo, mainly because I don’t think in qualitative terms, but I found the training to be immensely helpful. Once I did that, I could then go and play and learn more myself” she said.

As for the future of Melody’s mixed methods work, the outlook is positive. “I think I’m in a good space. There’s a need for researchers in mixed methods who can understand the quantitative, and the qualitative and go in and interrogate that data and triangulate the qualitative and quantitative findings” she said.

ABOUT THE AUTHOR

QSR International with Dr Melody Goodman, NYU

Dr. Goodman conducts applied biostatistical and survey research for community-based interventions and health disparities research with a strong focus on measurement. Additionally, through academic-community collaborations, she implements, evaluates, and enhances the infrastructure of community-engaged research, in order to mitigate health disparities. With numerous funders supporting her work, she has published over 70 peer-reviewed journal articles. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

From journalism to research and evaluation: a research journey

Dr. Anupama Shekar shares her story of following her passion for ensuring equitable access to quality education, no matter a child’s economic circumstance, and how it took her from a career in journalism in India, to post-doctoral research in the U.S. Her work has featured a long history of working with qualitative research and evaluation tools, including NVivo.

Introduction

Dr. Anupama Shekar, PhD, is a qualitative researcher and program evaluator with a passion for the field of educational research and evaluation. She is currently an Evaluation Consultant with the Center on Research and Evaluation at the Simmons School of Education and Human Development at the Southern Methodist University (SMU) in Dallas, Texas. Her prior experience includes working as the Director for Evaluation at Teaching Trust in Dallas, Texas, an education leadership non-profit organization. She also worked as an associate researcher, and prior to that a post-doctoral research associate with WIDA which is a national and international prek-12 language development and assessment program housed at the Wisconsin Center for Education Research at the University of Wisconsin-Madison.

Before undertaking her postdoctoral work, she earned her PhD at the Department of Educational Leadership and Policy Analysis at the University of Wisconsin-Madison. She also assisted in the development and evaluation of WIDA’s data literacy program known as LADDER for English language learners. Funded by the US Department of Education's Office of English Language Acquisition, this project essentially helped participating schools make data-driven decisions about English language learners.

Prior to coming to the U.S., Anupama received her Master's degree in journalism at the Symbiosis Institute of Mass Communication and worked as a print journalist for the New Indian Express, a national mainstream newspaper in Tamil Nadu, Southern India. In her role as a print journalist, she focused on public educational leadership and policy issues in South India, where her journey in education began. The school leaders she met served children from low-income families. They greatly impacted her and she was inspired to leave journalism and study educational leadership and policy.

A journey in education leadership and policy research

Anupama recalls why she felt compelled to change careers. “It was the initial encounter that I had with several children from low-income communities,” she said. “They really awakened my interest in studying education leadership and policy formally and improving the public school system in India.”

“Many years later, the first story I wrote for The New Indian Express in 2006, still remains on my desk, “ she said. “It continues to keep me focused on why I began this journey and the importance of working to improve the lives of children from low-income communities anywhere in the world.”

A 14-year-old girl said that she had to work to feed her mother and brothers, and could not go to school. That really stuck with Anupama. Although education is a fundamental right of children under the Indian constitution, thousands of underprivileged children still have no real access to a school or quality education. “At that point I started developing an interest in research and evaluation in education leadership. I wanted to study successful school leadership practices and leaders who advocate for children from low-income groups despite the odds,” Anupama said.

It was when Anupama’s doctorate studies and WIDA work began that NVivo came into the picture. Her professors and other researchers used it, and her own research involved writing up case studies of school leaders in public schools in Tamil Nadu, Southern India. Previous research in the U.S. had examined the contribution of parent involvement in children's educational outcomes, but very little was focused on the role of school principals in fostering parent, family and community involvement practices.

Her analysis of previous research led her to design an exploratory, qualitative, cross-case study and informed her research questions: how do public school leaders in Tamil Nadu foster parent and family involvement? And what are the similarities and differences across schools?

“I used NVivo 9 to explore the initial transcriptions of interviews, contextual observations and field notes. It gave me an initial understanding of all the data and how the school heads initiated and supported parent involvement practices at their schools,” said Anupama.

While NVivo helped gain an initial understanding of the themes in her data, Anupama also used a traditional and manual coding process while interrogating her qualitative data to unpack the complexities in her qualitative case studies.

“Manual coding helped me analyze the story of each headmaster and headmistress and see patterns. I needed to get close to the data to figure out the leaders actions more deeply,” she said. “I also used memos, and documents, and artifacts. I sort of let the curiosities as a researcher take over. I feel moving between manual and software coding really helped me with my dissertation analyses and to triangulate my own thinking and findings,” said Anupama.

She notes how innovative uses of qualitative data helped her accomplish a richer understanding of experiences in the case studies. “The main study findings were that the school heads’ over time created a continuum of overlapping actions that helped foster effective parent involvement. I was really able to get to the core of the school heads’ actions through usage of multiple analyses techniques and constant reflection on the qualitative data. As a qualitative researcher, you really commit to spending extended periods of time to get to the heart of the story” Anupama said.

During her work at WIDA during her doctorate studies, the WIDA’s LADDER project convened many focus groups, as well as individual interviews and mixed methods evaluation. “Each year we produced a program evaluation report and wrote up findings, so NVivo was useful as one of the tools that helped us identify themes and patterns,” said Anupama. “WIDA still offers the LADDER program, and I was there when they were developing the whole program from the ground up,” she said.

When Anupama moved onto her postdoctoral work, WIDA’s Teaching and Learning team were trying to understand best practices in professional learning and professional development. One large project involved multiple qualitative open-ended questions. Anupama found her prior experience helpful. “NVivo was a great tool for me to use then because we were working with a lot of diverse data and it ended up providing great insights,” she said.

Most recently she worked as the Director of Evaluation with Teaching Trust, an educational leadership non-profit in Dallas. Teaching Trust offers high quality training and support for future school leaders, school leadership teams, and teacher leaders to ensure that children in low-income schools across Texas have access to an excellent education.

“Teaching Trust has a strong alumni base and educators who graduated from Teaching Trust programs are out in the field driving positive change for students,” said Anupama. “The Teaching Trust Alumni Network team always gathered and used data effectively to drive their programmatic decisions. In this case, the team was trying to understand through qualitative data, the impact of the Teaching Trust alumni programming from the participant's point of view and how future programming might be improved and changed,” she said.

The Alumni Network team conducted qualitative focus groups of current and former participants. “After every focus group, our team met to extract meaning from the data — the impacts of Teaching Trust programming on participants, personal leadership, student and school outcomes, and what it really meant to be part of the Teaching Trust community,” said Anupama.

The team used both manual and software coding techniques with their qualitative data. “We took a grounded theory approach by listening and gathering data, and bridging perspectives to really unpack the themes and patterns” said Anupama.

“My former colleagues used pen and paper, and I used NVivo to code,” Anupama said. “There is a lot of power in combining multiple qualitative coding techniques because that adds to the validity and reduces researcher isolation. We presented the lessons learned and techniques on the collaborative qualitative approach in a webinar to the American Evaluation Association.” she said.

A passion for qualitative insights

Anupama’s career has evolved through her interest and passion for educational research and evaluation and ensuring people have equitable access to quality education, no matter their background or economic circumstance. Her appreciation for the importance of qualitative research and evaluation has been at the heart of her work.

“Qualitative data tells you something that numbers cannot, and helps you dig deeper to explore the complexities and find powerful insights,” she said. “As a qualitative researcher and evaluator, my challenge has been to find meaning in data, to keep asking why, and to continue digging,” said Anupama.

Anupama also hopes to continue sharing the power of qualitative research and evaluation through her website and blog in the near future. “There is a renewed energy in qualitative research and evaluation that is really exciting. There are people around the world who use qualitative data in very different ways in their work. I think it will be valuable to hear and share their stories as continual learning is the core of qualitative work.”

Next steps in career

Anupama hopes to use her learnings in qualitative research and evaluation at her current work at the Center on Research and Evaluation (CORE) at the Simmons School of Education and Human Development at the Southern Methodist University (SMU) in Dallas, Texas.

“I am excited to be doing projects for CORE and collaborating with their diverse and strong team of researchers and evaluators led by Dr. Annie Wright. They are at the forefront of conducting rigorous research and evaluation that focuses on examining critical issues around children, families and communities.

CORE is constantly striving to push boundaries and was selected as one of the Annie E. Casey Foundation’s expert evaluators nationwide. This shows the focus CORE has on issues around diversity, equity and social justice. I am honored to be learning as a researcher and evaluator with this incredible organization.”

You can follow CORE’s work on Facebook and Twitter.

ABOUT THE AUTHOR

QSR International with Dr Anupama Shekar

Dr. Anupama Shekar, PhD, is a qualitative researcher and program evaluator with a passion for the field of educational research and evaluation. She is currently an Evaluation Consultant with the Center on Research and Evaluation at the Simmons School of Education and Human Development at the Southern Methodist University (SMU) in Dallas, Texas. Her prior experience includes working as the Director for Evaluation at Teaching Trust in Dallas, Texas, an education leadership non-profit organization. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Thematic Analysis Is More Popular Than You Think

The practice of thematic analysis is widely used in qualitative analysis but sometimes not acknowledged or confused with other approaches. Here at QSR International we break down the ambiguities of thematic analysis as an approach, and hope this interpretation can breathe new life into it, as new and emerging forms of content become more integral to the already established research tool.

What is thematic analysis?

Thematic analysis is not a methodology but a tool which can be used across different methods (Boyatzis 1998). It is used to find common themes in content such as:

This practice is dynamic. It can be done manually (by hand), in Excel or through a Computer Assisted Qualitative Data Analysis (CAQDAS) software tool. It traverses traditional qualitative research and quantitative data, allowing researchers to ask more questions of their content.

>> Watch Webinar to Learn how you can use NVivo for Thematic Analysis

When might you choose to do thematic analysis?

Put simply, you may be looking for the right way to explain or express patterns in your content. Consider this example: you are analysing representations of women on social media. You want to collect data from Facebook, Twitter and YouTube as rich datasets so you can access the online conversations and content about your research, organization or topic of interest, but also the valuable data behind the comments, like demographics and locations.

The challenge with importing, managing and analyzing different content types is how do you find the similarities or differences in the media before you? What do you do with it then?

What are the benefits of sifting through content?

Thematic analysis helps you find connections in your content and understanding the underlying themes to help inform decisions. Use queries to ask complex questions and identify new meaning in your data. Test ideas, explore patterns and see connections between themes, topics, people and places in your project. Look for emerging themes, find words and discover concepts using text search and word frequency queries.

Thematic analysis can be used as a technique on its own or it can be used as a first step in a variety of methodological approaches to analysing qualitative data including:

Once you do this, you can search for content based on how it's coded using coding queries. Check for consistency and compare how different users have coded material using coding comparison queries. Cross-tabulate coded content and explore differences in opinions, experiences and behaviour across different groups using matrix coding queries.

How do I visualize my data?

By visualizing your insights, you can explore even further. Get a sense of the larger trends happening and dive in deeper. Discover a new perspective. Identify new and interesting themes. Share results with others.

Visualizations can also provide an easy way to communicate findings with broader audiences.

Why should you do a thematic analysis?

Easily understand how content plays a role in influencing decisions or behaviours.

How do I get started analyzing content and visualizing my insights?

Gain an advantage with NVivo – powerful software for qualitative data and content analysis that helps you make insight-driven decisions.

NVivo has a wide range of visualizations. Below are a few which are particularly useful to thematic analysis:

Editors note: This blog was originally published in March 2017, and was updated in February 2022 for accuracy.

For more information about thematic analysis see these resources:

Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage.

Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, Qualitative Research in Psychology, 3(2), 77–101.

Teaching thematic analysis

Thematic analysis of interview data: 6 ways NVivo can help

On any given day, close to 80% of NVivo users are busy analyzing interviews.

Like you, they're looking for the best way to make sense of their data so they can deliver robust results.

If you’re new to NVivo or just getting started with analyzing interviews (thematic analysis) it can be hard knowing where to start. These tips can help guide you on your way.

>> Watch Webinar to Learn how you can use NVivo for Thematic Analysis

#1. Transcribe the interview recordings

Let's face it, transcribing can be tricky - particularly if you're working with a lot of interviews or have limited time (and budget).

Luckily, NVivo gives you a number of options. For example, you can import the interview recordings into NVivo and:

>> Free Transcription: Try NVivo Transcription free today

#2. Group the responses to each question

If your interviews have already been transcribed, you can import the documents directly into NVivo and get started on analysis.

If you’re working with semi-structured interviews (where participants are asked the same set of questions), you can use heading styles to automatically organize the responses.

For example, you can gather all the responses to Question 1 in one place for easier analysis.

Refer to Automatic coding in document sources for step-by-step instructions.

If your interviews are more free-ranging and conversational, you can use other tools to organize the content by theme.

#3. Find and catalogue themes to make sense of the data

Thematic analysis involves making sense of what your interview participants are saying:

NVivo gives you ways to get a broad feel for what themes are in the data and it also lets you drill down into the material for deeper analysis.

For example, you can run a quick Word Frequency query to see which words your participants are using most often.  The resulting word cloud can give you early insight into emerging themes – and it’s a fun way to ease yourself into analysis:

Taking a more thorough approach to thematic analysis, you can read through each interview and ‘code’ the emerging themes. This involves selecting interesting comments and putting them into containers called ‘codes’. At any time, you can open a code to see all the references you’ve gathered there.

NVivo offers plenty of ways to speed up the coding process – you can use the Automated Insights feature to find themes automatically.

Refer to About Automatic Coding Techniques for the details.

#4. See the connections between themes and move toward analytical insight

As you code your material by theme, you’ll start to develop a list of codes. At regular intervals, you can groom this list – checking whether related themes could be grouped together in a hierarchy.

This is not just ‘good housekeeping’ – it’s a vital step in the analysis process and helps you to see the connections between emerging themes:

You can open any code (by double-clicking on it) to view all the content gathered there but you can also run queries to retrieve your data in revealing ways.

For example, you could see where participants talked about ‘water quality’ in terms of ‘development’ – or where ‘policy’ came up in discussions about ‘water quality’.

NVivo lets you query and visualize your data in all sorts of ways – refer to Move forward with queries and visualizations to find out more.

#5. Make comparisons between participants

If you want to compare what your interview participants say based on attributes like age, gender or location – then you can create a ‘case’ for each person and assign the demographic attributes.

This video gives you a quick overview of how cases work in NVivo:

Creating ‘cases’ for interview participants, paves the way for powerful queries and visualizations. For example, you could create a matrix to see how men and women respond to a selection of themes:

#6. Stay organized and focused on your research design

In the thick of data analysis, it can be easy to lose sight of your research question.

Gathering your material into theme codes and organizing these codes in a ‘sensible’ hierarchy helps you to stay organized and focused.

Mind maps in NVivo are another great way to consolidate your thinking. For example, you could visualize your conceptual framework in a mind map and update it as your thinking evolves:

You should also consider creating a project journal in NVivo.

Keeping an audit trail of your challenges, assumptions, decisions and epiphanies will come in very handy when your supervisor (or client) asks a difficult question.

The journal tells the story of your project, makes your decisions transparent and helps you avoid that terrifying blank page when it comes to writing up.

Editors Note: This blog was originally published in October 2016 and was updated in February 2022 for new information and accuracy.

Find out more

These 6 approaches to thematic analysis are just the tip of the iceberg and we’ll expand on them in upcoming posts.

To find out more – check the online help, watch video tutorials or join the NVivo Users Group on LinkedIn.

How about you? What are your tips for analyzing interviews? Tweet us at @NVivoSoftware using the #NVivotips hashtag.

ABOUT THE AUTHOR

Kath McNiff

Kath McNiff is on a mission to help researchers deliver robust, evidence-based results. If they’re drowning in a sea of data (or floods of tears) she wants to throw them an NVivo-shaped life raft. As an Online Community Manager at QSR, she knows that peers make the best teachers. So, through The NVivo Blog, Twitter and LinkedIn, she shares practical advice and connects researchers so they can help each other. When she’s not busy writing blog posts, swapping stories on social media or training the latest tribe of NVivo users, she can be found wrestling four feisty offspring for control of the remote.

How to make Student Placement More Efficient and Consistent University Wide

We’re delighted to have our new Product and Community Director, and field placement expert, Sally Warlow, presenting a webinar on how she spearheaded an initiative to implement new placement processes and scale the use of Sonia from two to 17 programs, supporting thousands of students  at one university.  


>>Watch the webinar now

You’ll learn how Sally was able to:  

Speed Up Publishing with NVivo and Citavi

I don’t know a researcher on the planet who doesn’t wish they could publish faster. If you work with NVivo, you already know that it can save a substantial amount of time with transcription, coding automation, data analysis, and visualization. But what about all the other steps in the research and writing process? Is there any way to speed them up as well?

To cover the different stages of your research, there are a number of other programs you can use: a citation manager, like EndNote, for keeping your supporting literature organized and for formatting citations and bibliographies when you go to write, a note-taking app like Evernote for keeping track of notes on what you read, an outlining tool, such as Scrivener, for structuring your publication, and a task management app so as not to forget anything. However, having to learn and then switch between a number of tools that don’t interface with each other may not sound appealing when you’re already short on time. This is the problem Citavi was designed to solve.

What is Citavi?

Citavi is an all-in-one referencing and note-taking application designed for academics at all levels and in all disciplines. It supports you with each step of the writing process that involves research literature. First released in 2006, it’s widely used in Europe and is now making its way across the pond to a wider audience by joining the QSR International family.

>>To learn more, watch the free webinar on-demand.

Using NVivo and Citavi together speeds up the publishing process

If you’re working with NVivo to help you analyze the results from your original research or supporting literature, Citavi can help you be more efficient in the other stages of the research process. You can think of NVivo as being for your primary research and analysis, while Citavi is where you take care of any tasks related to your supporting literature.

Below I’ll look at the ways Citavi can speed up your process and how it can be used together with NVivo when taking notes or writing literature reviews.

It’s easier to find and organize sources

Citavi helps you keep better track of the sources you already have and also locate new ones. The Picker browser extensions help you bring in journal articles and web content with just a click, saving you from the hassle of exporting results from databases, although you can still do so for large result sets if you want to. An integrated RSS feed reader lets you add search alerts from databases you frequently consult so that you can directly add the latest publications on your research topic to your project.

Create outlines for a paper as you work

Citavi’s multilevel category system lets you create a detailed outline for your paper. You can then better organize your sources and any notes you take before you start writing your publication draft. As your project grows, your category system can change along with it.

Plan and track tasks in one place

Citavi helps you keep track of important project deadlines and other tasks related to your research project. When you read texts, you can even add tasks to text passages. For example, if you later want to discuss a finding with a colleague or if you want to fact check a particular claim, you can highlight the text in question and create a task for yourself. You can always jump back from the task to the section of the document it was added to, which saves you from having to hunt for it. When collaborating with others, it’s easy to assign tasks to team members and track their progress.

Take and organize notes your way

Both Citavi and NVivo can be used for taking notes. In NVivo you can add annotations and memos about your literature and view them in tabs, and the notes remain tethered to the original source. In Citavi you can annotate in a similar way, but the difference is that your notes can be organized by topic and tagged separate from the original source. Regardless of which program you use for your notes, having them in digital format and all in one place can help you avoid wasting time searching stacks of printed out articles or notebooks when you later want to start writing.

Writing a literature review is even easier

Both Citavi and NVivo can help with writing a literature review, and each program has its own strengths. With Citavi, the notes you take while reading can be tagged and organized as individual elements. In the Knowledge Organizer you can then compare and contrast ideas across the literature and bring them into a preliminary outline. The source information goes along with your notes, so you don’t have to worry about tracking down citations when you go to write.

For a much more detailed picture of the literature without having to read each article first, you can export references from Citavi and then import them into NVivo. NVivo can help you gain insights and track trends and themes across the literature with automated coding and advanced queries, and it can also generate visualizations. You can combine these powerful features with your own coding to get an overview of the relationships between articles, find gaps in the literature, see which authors have collaborated with or cited each other, get an idea of which authors or works have had the biggest impact in your field, etc.

Write up your research, fast

Both approaches to the literature review can be combined in Word. You can export your analysis from NVivo to Word and then use the Citavi Word Add-In to insert any citations you need to add to your analysis.

Beyond the literature review, coding for your original research findings, and maps, charts, and other visualizations created in NVivo can also be exported to Word. From the Citavi Word Add-In you can insert any of the notes from your project that you took while reading. The Word Add-In makes writing more efficient, since you have all of the literature and notes you need at your fingertips and don’t have to switch between documents or apps while writing.

By inserting your category system from Citavi into the document, you can even use a “chapter view” to filter out only the notes and citations you need for a particular section so that you stay focused as you write. As you insert items, the bibliography is automatically generated, and if you find that you need to change citation styles for submission to a different journal, just select the one you want, and the entire document is updated in seconds.

>>To learn more, watch the webinar on-demand<<

ABOUT THE AUTHOR

Jennifer Schultz
Citavi Marketing Manager

Jenny has put her educational training in Library Science and Translation to good use working for Citavi for the past 10 years, where she helped advise users and create training materials. Now Citavi Marketing Manager at QSR International, she looks forward to helping a wider group of researchers make their process better organized and more efficient. In her spare time, you will find Jenny filming and editing videos or gardening. She especially loves growing some of the hottest chile peppers in the world on her balcony.

Extending Your Literature Review With NVivo

Back in 2000, I wrote about how you could you use NVivo for your literature review. The software has changed significantly since then with a different interface and terminology and the possibility to import from a range of bibliographic software but the basics were all in place. For those who still need to learn the basics about using NVivo for literature reviews, I suggest you first have a look at the following link is a more conceptual approach to literature reviews.

However, the latest versions NVivo have added some new possibilities for analyzing your literature. You can use the social network feature to explore relationships between your articles, authors, and books. You may want to map out which researchers write together and analyse the clusters of researchers working in a particular field. Or you may want to code when an author cites, critiques, supports or expands on ideas from other articles.

>> See how NVivo supports literature reviews with a free 14-day trial.

Getting Started

You need to start by creating a case for each author and coding all the author’s work to their case node. [You can see below that there are several articles for some of the authors in the list and for example, there are five articles coded to Kaufmann's case].

You will also need to create a case for each article and code that article in that case node. (see below)


Figure 1: List of case nodes for articles; List of case nodes for authors
NOTE: If you are new to cases in NVivo check the help file –
https://help-nv.qsrinternational.com/20/win/Content/cases/cases.htm

You can create some relationship types that you want to explore between authors and between articles (see below). However, you do not need to create all (or any) of them beforehand. You can create new relationships as you code (see further below).


Figure 2: List of types of relationships when coding literature
Note: If you are new to relationship types and codes in NVivo, check the following help file - https://help-nv.qsrinternational.com/20/win/Content/nodes/relationships.htm

You use relationship codes to code for – yes, you have it – relationships between your articles. To do this, open an article and start reading. When the author is making an interesting comment about another article, code it (using the quick coding tool bar) with the case of the article that you are reading at the left end of the relationship, the relationship type in the middle, and the case of the article they are referring to at the other end. If the case or relationship has not already been created, just type in the name of the new item in the relevant box and it will be created (see below).

Example: Hall (2008) qualifies evidence from Urry (2007).


Figure 3: Coding text from an article for a relationship using the quick coding tool bar

Exploring relationships between articles

You can see and explore in the network sociogram the relationships you have created among articles by clicking on the Explore tab > Social Network Analysis > Network Sociaogram and selecting the case folders for your articles:


Figure 4: Selecting articles to explore in the network sociogram

As you can see from the network sociogram below, there are four relationships between Hall (2009) and Urry (2007) and when I double click on the line, the pop up box opens up to show me what those four relationships are. If I select one, the article with the referenced text will open up in the Detail View.


Figure 5: Network sociogram: detail showing how Hall, Tom (2009) refers to Urry, John (2007)

Exploring the influence of a particular author

In a similar way, I can explore relationships among authors to see who writes with whom. I can also set up relationships to see which articles an author contributed to. See below:


Figure 6: List of relationships between a) authors and articles and b) between authors

Once that is done, I can explore a particular author, using an egocentric sociogram, and see what they wrote and who has referenced their articles.

I just right click over the case node of an author and select, Visualize > Egocentric Sociogram (see below).


Figure 7: Creating an Egocentric Sociogram for the author case node – Urry, John

And a sociogram opens up. The author I selected (John Urry) is represented by a star icon. I can then start to explore what he has written, with whom he has written and how their work has been used by others. The sociogram below shows relationships two steps away from John Urry. It is possible to extend the sociogram to three steps away.


Figure 8: Egocentric sociogram for Urry, John showing a) who he writes with b) literature he has written and c) how that literature has been referred by other articles.

Exploring communities of practice

I can also explore what clusters of researchers write with each other by using the network sociogram with authors.


Figure 9: Network Sociogram showing clusters of authors who write together

Network and egocentric sociograms can be used to visualise relationships within your literature and help identify which works and authors have been key influencers in your field. I have given you a few ideas about how they can be applied to a literature review. Have a play and see what you find.

Editor’s note: This post was originally published in April 2018 and has been updated to reflect information and screenshots from the latest release of NVivo (March 2020).  

ABOUT THE AUTHOR

Silvana di Gregorio, PhD

Silvana is a sociologist and a methodologist specializing in qualitative data analysis. She writes and consults on social science qualitative data analysis research, particularly in the use of software to support the analysis. She is also QSR International's Director of Research.

5 Ways to Make Your Placement Management More Efficient

People managing placements are busy! There are students to manage, site contacts to update, sites to source and university staff to report to. Often, entire programs are managed via complicated and manual systems and processes using Excel or home-grown software that can be out of date or complex to use and integrate.

What if there was a way to remove the administrative burden and empower placement managers to do what they do best: focus on providing the best placement experiences for students?

There is!

Student placement management software can greatly ease administrative burden and let you and your team focus on what’s important: managing placements. By teaming student placement software with these 5 tips, your placement team will be more efficient and less stressed in no time.

>>Learn More About Sonia.  Request a Demo.

 

1. Keep All Placement Data in One Location 

Rather than managing various spreadsheets, documents, databases and handwritten notes, store all placement information in one location. With all data easily accessible, your team is set up to take advantage of time-saving automated workflows and reporting and move through processes faster.

Keeping data together and accessible via permissions also means that you can more easily manage privacy and compliance needs and reduce risk of breaches and potential human error. It also streamlines data management, enabling simple integrations with other tools, such as student information systems.
 

2. Use Automations To Save Time   

Use automated workflows to send emails to students and sites, ensuring no detail is missed and all placement requirements, like immunization and police checks are completed on time and documented appropriately. With automations set up, software like Sonia can take care of previously manual work, while you get on with your day. Learn how to automate placement communications.
Download Effective Communications whitepaper.

3. Empower Your Students When They’re On-The-Go 

Ensure you can easily communicate with students. Take advantage of a mobile app for students to communicate via push notifications, and let students complete timesheets and input data wherever they may be.
 

4. Streamline Reporting and Compliance 

With all data in the one place, you’re well on your way to a more secure and compliant program. With Sonia, you’ll know that you’re compliant with the Family Educational Rights and Privacy Act (FERPA), the Health Insurance Portability and Accountability Act 1996 (HIPAA), the Australian Privacy Act 1988 (Cth) and the General Data Protection Regulations (GDPR).

Use dashboards to track placements and program key performance indicators. Sonia’s dashboard offers one central, automated and digestible source of truth. You can enjoy the freedom of no longer having to spend hours manually creating reports and have immediate insight into the number of students, sites, and placements.

  

5. Centralize Placement Operations For Increased Efficiency 

With all teams at your university using the same systems and processes to manage placements, you’ll streamline work across the organization for placement managers, IT teams and other stakeholders.

All site and student information can be stored and managed in one place, reducing rework, enabling powerful reporting and increasing accuracy. The sites you work with also pass through consistent experiences with your university, regardless of the school they are working with. Overall, one platform reduces time and financial investment required versus running multiple platforms.

Universities around the world use Sonia student placement software to run more efficient placement programs. Reduce the time you spend on administration with the power of Sonia.  
 

Sonia is the leading software solution for student placement management. It’s easy to use, it powers personalized placements, reliable data management and compliance, and provides the flexibility you need to scale across your organization.

Take your placement program to the next level with Sonia.

 

 

How To Reduce Risk In Your Placement Program

It’s never been more necessary to ensure strong risk management in a placement program. Universities and workplaces are managing numerous unknowns, and operational requirements can change on a daily basis.  

Ensure your placement program is set up to successfully manage your duty of care, privacy and compliance responsibilities.  

 

Maintain Policies, Processes and Procedures 

Ensure you have policies, processes and procedures in place that support the safety and wellbeing of your students. Standardized forms and checklists are useful to do this. Make all of your forms and checks paperless, so you can easily store and retrieve data, ensuring procedures are followed and compliance maintained. This includes managing evaluation forms and accreditation requirements.  

 

>>Learn More About Sonia. Request a Personalized Demo Now.

 

Let Technology Reduce Risk of Human Error 

Take advantage of technology that enables automated workflows, so you can set up processes that collect the information you need while you get on with your day. Sonia enables powerful automation that gives you the power to set up, turn on, and collect what you need with the click of a button. This ensures the information needed by your organization and your partner sites is collected and supplied on time, so your reputation of being organized, professional and providing a comprehensive educational experience remains untouched.  

Learn how to automate placement communications. Download Effective Communications whitepaper.
 

Use Tools That Support Compliance and Privacy Requirements 

There are many acts and regulations that placement coordinators need to comply with. To streamline compliance, use a student placement tool that removes any guessing. Sonia is compliant with the Family Educational Rights and Privacy Act (FERPA), the Health Insurance Portability and Accountability Act 1996 (HIPAA), the Australian Privacy Act 1988 (Cth) and the General Data Protection Regulations (GDPR).  
 

Ensure Your IT ‘Just Works’ 

Make sure the software solution you use to support your placement program is easy to integrate with the other systems you use, so you always have access to the most up to date data without any manual intervention. Sonia is IMS certified and complies with LTI (Learning Tools Interoperability),. So, you know it’s easy to plug and play with the rest of your ecosystem in a secure and standard manner, without the need for expensive custom programming. 

If all placement teams in your organization use the same tool, it’s easier for placement managers and your IT team. There’s only one tool to manage, and users share and help each other day to day. You can make it even simpler to manage, by having your student placement software hosted in the cloud by your provider. Ensure any cloud hosting is compliant with relevant regulations, and that the set up provides geo redundancy so your system is always up and running.  
 

Use Reports To Support Quick Decisions and Reactions 

If you needed to rapidly locate and communicate with every student on placement today, could you do it? With a central dashboard that provides a view of up to the minute information on all programs, coupled with a mobile app, Sonia users can locate and contact students in minutes. This is critical in any emergency situation.   

Universities and colleges have a duty of care to students entering cooperative education placements. Liability must be managed, and any chance of negligence removed. Students are the most vulnerable group during placements, and program coordinators have a unique opportunity to influence their safety and wellbeing. Sonia gives you the tools to successfully capture, document, store and recall all the data you need to reduce risk in your placement program.  

 

Sonia is the leading software solution for student placement management. It’s easy to use, it powers personalized placements, reliable data management and compliance, and provides the flexibility you need to scale across your organization.

Take your placement program to the next level with Sonia.

Benefits of Centralized Placement Programs

Why Centralized Placement Management Is the Future 

Higher education institutions are prioritizing student experience, teaching innovation, diversity and workplace readiness. Key programs of work to address these priorities typically involve experiential learning, and more specifically, supporting professional experience via co-op.

Placement programs are often run independently by different schools at a single university. This presents a number of challenges such as lack of central visibility, duplication of effort and unnecessary organizational spending. With universities facing unprecedented pressure on resources and seeking ways to become more effective and efficient, centralizing student placement programs offers significant benefit.  

 

Run More Efficient Experiential Learning Programs  

Streamlining numerous processes will save your institution time and money, at a time where efficiency has never been more critical.  

It’s time to stop spending hours managing student placements in Excel, or trying to create homegrown solutions that are hard to maintain and improve.

 

>> Sonia Student Placement Software Can Help. Request a Demo.

 

Deploying a centralized student placement solution creates economies for placement managers across the university. Not only can they save time with the right technology, but they can leverage each other’s work, and share best practices. Central, yet flexible digital workflows, forms, and reports will have placement staff in every department running more efficiently. Your team is also getting the tools they need to be more successful, supporting higher employee engagement and retention.  

A central solution is more efficient for your IT teams to manage and support. There is only one product to license, and your team should be able to choose between managing your chosen solution on premise, or for even more ease and efficiency, having it hosted by your provider.  

 

Stronger Reporting and Risk Management 

A centralized system offers a single view of all placement programs. In case of emergency, you have a way to locate all placed students immediately and easily contact them. An ‘always on’ mobile app can help your team contact all placed students, which is particularly important in a crisis.  

It’s simpler to create reports with centralized data storage, structures and systems. And you can manage privacy and compliance in one place, removing risk from the organization.

 

Boost Your Reputation With Consistent, High Quality Stakeholder Experiences 

A university or college’s reputation is critical to its ongoing success. Centralizing placement programs offers a unique opportunity to rethink and align on consistent stakeholder experiences throughout the entire organization, supporting a strong community and industry reputation for your brand.  

A consistent process and experience, regardless of which school students and host sites are engaged with, means that all stakeholders have clear expectations and understanding, and programs across all disciplines run more smoothly.  

With a student placement management tool that is customizable for all disciplines at the core of your university’s experiential learning program, your stakeholders will experience the most innovative and intuitive experience possible, ensuring you have the reputation you need to be successful. 

 

If you’re considering a placement solution for multiple schools or for your entire institution, ensure it is flexible and customizable enough to support the terminology and processes that each school requires. Sonia student placement software is the #1 solution for centralized student placement management. 

Sonia is the leading software solution for student placement management. It’s easy to use, it powers personalized placements, reliable data management and compliance, and provides the flexibility you need to scale across your organization.

Take your placement program to the next level with Sonia.