US Government strengthens international aid model with qualitative data analysis

USAID uses NVivo to improve learning agenda

Background

The United States Agency of International Development (USAID) is a U.S. government agency that betters the lives of millions across the globe through areas such as improving agricultural productivity, combating maternal and child mortality, providing immediate disaster relief, fostering private sector development, elevating the role of women and girls, and providing humanitarian support.[i] Since its inception during the Kennedy administration, USAID has made an impact in countries across the world.
Funded by the USAID Bureau for Policy, Planning and Learning (PPL), the USAID LEARN contract —implemented by Dexis Consulting Group—is designed to support strategic learning and knowledge management at USAID to improve the performance and effectiveness of its programs around the world.[ii]
USAID LEARN works with multiple teams and departments within USAID through an approach called, “collaborating, learning and adapting (CLA).” Part of this approach includes using Learning Agendas, which are a set of agency-specific questions directly related to its work. When these questions are answered, the Learning Agenda guides staff  through organizational learning in order to create an environment of growth and success.

The Challenge

USAID LEARN conducted 60 interviews and held two focus groups to properly analyze the Learning Agenda’s current landscape within federal agencies. They reviewed brochures and programs already informed by Learning Agendas across agencies, including The Department of Housing and Urban Development (HUD), The Department of Agriculture (USDA), The Department of Labor (DOL), The United Nations Research Institute for Social Development (UNRISD).[iii]
The analysis accumulated textual, video, audio, PDF or image and numerical data. However, traditional quantitative data analysis tools are only able to analyze the numerical data collected. And the traditional, pen/paper/highlighter/sticky note method of analysis would take too long. Individually reviewing each image, interview, and document would waste hours of time and risk the loss of important themes or ideas as time passes. USAID LEARN and Dexis Consulting Group needed a different solution.

Introducing NVivo

USAID LEARN conducted an assessment of the potential software that can analyze all of their qualitative (non-numerical) data in a quick and meaningful way. Because NVivo is often recognized as the industry standard for qualitative data analytics in education, many of the analysts had previous, positive experience working with the solution. NVivo’s functionality, accuracy and speed are exactly what the team was seeking, so it was the unanimous choice.

Results

The USAID LEARN team was impressed to learn from their NVivo research that the Learning Agenda, serving as an integral way to understand both how and why to grow, was in use far more widely than anticipated.

“We discovered that there are over 20 operating units within USAID using Learning Agendas. A variety of people, from HUD to the UN, in all different technical areas are also using them to learn and grow,” said Matt Baker, USAID LEARN’s Monitoring Evaluation, Research and Learning Specialist.

NVivo also helped the USAID LEARN team uncover the themes that make up their Learning Agenda questions.

“Because NVivo can interpret live language, it was able to read over 100 questions and show us how we could distill them down to the best five,” said Baker,

NVivo revealed the top reasons why Learning Agendas are useful by identifying three primary motivations for those who use them:

  1. Expectations of accountability, especially in response to leadership demands
  2. Leadership transitions and structural, strategic, or policy changes
  3. Responses to identified program-related needs

Many other discoveries were made in the landscape analysis that will lead the USAID LEARN contract as the team begins the one-to-two-year process of developing strong Learning Agendas for USAID. Because of this, USAID has the opportunity to improve the effectiveness of its important work, both in Washington and across the globe.

[i] https://www.usaid.gov/what-we-do
[ii] https://usaidlearninglab.org/learn-contract
[iii] https://usaidlearninglab.org/sites/default/files/resource/files/landscape_analysis_report_04_10_17.pdf
 

ABOUT THE AUTHOR

QSR International with USAID and Dexis Consulting Group

The United States Agency of International Development (USAID) is a U.S. government agency that betters the lives of millions across the globe through areas such as improving agricultural productivity, combating maternal and child mortality, providing immediate disaster relief, fostering private sector development, elevating the role of women and girls, and providing humanitarian support. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Research in the field: Shooting Stars and the success of their community program

Australian not-for-profit are utilizing NVivo to evaluate the success of their community program.

Shooting Stars is an initiative of Netball WA and Glass Jar Australia, which uses netball and other incentives, as vehicles to encourage greater engagement and attendance at school of young Aboriginal and Torres Strait Islander girls living in Western Australia’s remote communities and regional towns. Through fostering collaborative relationships with the local community, Shooting Stars works with schools and service providers to implement a regular program of personal development activity and fun for young Aboriginal and Torres Strait Islander girls.

Participation in the program enables girls to have a taste of success to improve their self-image, develop hope and aspirations about their future, and provide motivation to attend school to achieve those aspirations.

Following a successful pilot in 2014 in Halls Creek, the program had its full-scale launch into five locations throughout the State in 2015. Funded by the Department of Prime Minister and Cabinet through to the end of 2017 across six locations, Shooting Stars plans to expand to 14 sites in 2018.

Participants undertake a program that involves netball and physical education activities, such as swimming when the weather is too hot for a game, over two sessions a week. There are also two health and wellbeing sessions covering topics including nutrition and healthy relationships aligned to the Australian Curriculum.

A unique research challenge

Shooting Stars was faced with a unique challenge when wanting to both gather and analyze data to provide feedback on the program. Firstly, staff were dealing with a group of children ranging in age from primary school through to teenagers, and found that the standard method of paper-based surveys was not engaging and did not help in keeping the program as culturally centered as they desired.

To address this issue, a new method of data collection was established through consultation with local cultural leaders. The traditional indigenous yarning circle was presented as an alternative to paper-based surveys, to gain feedback from the girls in the program, in a way that encouraged open discussion.

“The yarning circle is really the idea of sitting around and ‘having a yarn’,” explained Rose Whitau, Research Manager at Shooting Stars – Midwest Gascoyne.

“We’ve found this to be a very participant driven piece of research, through the yarning circles, which are driven by a localised steering committee, made up of our staff, school staff, local cultural leaders, and the remote school attendance strategy provider,” she said.

The next challenge was that yarning circles produced unstructured data, in the form of audio recordings of the sessions, and they required a tool to analyze this data.

“We wanted to turn messy qualitative data into something that was easy to digest, and NVivo has allowed our team to do this,” Whitau explained.

“The purpose of using NVivo is to be able to evaluate the program ongoing, as we want our participants to have an impact on how the program is structured and delivered, which is very empowering for them,” she said.

NVivo is now being used throughout Shooting Stars locations to analyze the results of the yarning circles, of which there have been 22 so far.

“NVivo is the perfect tool for our needs, especially its ability to analyze audio files. Coming from an academic research background, I was quite familiar with the methodology, but I have also been able to easily teach the software to our other staff who come from a range of different professional backgrounds,” Whitau said.

Shooting Stars continues to affect real change, and work to empower indigenous Australian girls to stay in school and engage in their education, as well as maintain a positive attitude toward their health and wellbeing. You can learn more about the great work they’re doing on their website.

ABOUT THE AUTHOR

QSR International with Shooting Stars

Shooting Stars is an initiative of Netball WA and Glass Jar Australia, which uses netball as a vehicle to encourage greater engagement and attendance at school of young Aboriginal girls living in WA’s remote communities and regional towns. Working collaboratively with the local community, schools and service providers to implement a regular program of personal development activity and fun for young Aboriginal girls. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Identifying the political impact of US talk shows

In the United States it has become popular for presidential candidates to appear on entertainment talk shows. A study was undertaken to explore how viewers accept the humourous information that they watch in late night talk shows as legitimate political information.

Who used NVivo?

David Rhea, Assistant Professor, Communication Studies, Governors State University, Illinois, USA

What was the project?

In recent presidential elections in the US, it has become popular for presidential candidates to appear on entertainment talk shows, as a way of getting their political message across to viewers.

David Rhea, who is Assistant Professor of Communication Studies at Governors State University, undertook a qualitative project to determine how viewers of these talk shows disseminate the information they hear. The purpose of the study was to explore how the viewer comes to accept humorous information that they watch in late-night talk shows as legitimate political information.

“This was my first time using NVivo and I found it to be much easier to organize information than coding manually. With just a little set up time, I was able to code data thematically with just a couple of clicks.”

The value of NVivo and how it was used

David used NVivo to review the interview transcript data collected from talk show viewers, in order to extract themes from the analysis. It gave him the opportunity to analyze the data in a more advanced manner, with a view to running a similar study during the next U.S. Presidential election cycle.

“I loved how the themes were color coded on-screen visually for easy reference of my data and were easily aggregated.”

“I love the new dataset feature. I also do a fair amount of questionnaire research that includes some open-ended questions with qualitative data. When I put that qualitative data into a spreadsheet dataset, it often just gets numerically coded and counted. Results from that data become over-simplified. I look forward to being able to analyze that qualitative data the way it was meant to be analyzed and generate richer results and conclusions than before.”

“I also like how I could send sources from NVivo to my bibliography management program. I definitely think this is a great program for graduate students interested in qualitative research to have experience with when doing their data analysis.”

David recommends users explore the NVivo 9 tutorials.

“Have patience to go through the NVivo 9 tutorials and explore all the great capabilities this program has to offer. Then open your mind to explore all the new ways to analyze data that you haven’t conceived before. Everything from text analytics to visual maps…”

Outcomes from using NVivo

The findings from David’s research were presented at a preconference during the 2008 National Communication Association annual meeting.

The Power of Research and Truth for UX professional

NVivo shows its power in providing defensible findings, in the field of User Experience research.

Background

Zoe Rosen is an anthropologist working in the field of UX (User Experience) Research, where the researcher applies various techniques in order to give context and discover insights. These findings are then used to inform the design process for a product or process. This type of research is needed to discover new information about users, learn the facts, and find any current issues. UX research also helps to understand the users themselves, their needs and is used to identify the requirements of the product.

UX research is incredibly important for anyone designing anything between a product, an interface, content, or a physical interactive service. UX Research provides organizations with insights that empower them to invest capital wisely, by creating something their customers will want, based on tested and researched ideas – not just assumptions.

Zoe works as a UX Researcher contracting to different companies, generally as a part of the strategy and service design teams, and tends to work in one industry on an overarching research piece from 3 to 12 months at a time. She has uncovered important insights for companies across the banking, insurance, telecommunications and medical devices industries, to name a few.

“Because I move between organizations, and industries quite frequently, often I’m not an expert about the specific industry I’m working in. Consequently, I don’t always know what themes I should be looking for, and NVivo is extremely helpful in allowing me to identify those” said Zoe.

Working with NVivo

Being project based work, Zoe often has a limited time in which to complete her research, but also needs to complete it with a high level of depth and rigor, which is where NVivo has been really helpful. “Without NVivo there are so many times where I just would have had too much data and nothing to help me deal with it!” she said.

Zoe’s work has allowed her to uncover unexpected customer insights, that have inspired overall strategy change within the organizations she’s working with. “While working on an insurance project, the company held the misconception that none of their customers read the ‘Product Disclosure Statement’, it was part of their internal story. My work was able to show that not only did more than half of them read it, but they spent time highlighting it, and made notes on it and shared those details” said Zoe.

Using NVivo to analyze and store her data, allows her to easily call on and defend her research when her findings are challenged by her clients. “Often I am reporting findings to staff quite high up the chain of command and it’s great to actually be able to change their perception of the front line of their business. They might have worked in that position ten years ago, but a lot has changed since then, and I have the data to back it up” said Zoe.

Presenting results to clients

NVivo’s visualizations have been key in the way that Zoe’s results are received by her clients. “They’re often quite blown away when the results are presented with NVivo. I can see the change in the business. It changes the way they see things, it changes how they talk, they start changing their goals” she said. Zoe’s work allows her be the voice of the customer, who might otherwise not be being heard or understood by the organization.
Zoe’s reporting stacks up against larger research and design organizations. In one instance, she met with a company who were also consulting at a firm she was contracted at.

Impressed by her reporting and visualizations, they enquired further about her research. Zoe recounts being surprised they weren’t using the same tools as her; “I dug a little deeper and asked them what they used, and how they came to their conclusions in their research work. They told me ‘We don’t do anything like this, we go a little bit more on our intuition’ which really disappointed me, because they’re probably charging a lot more than me, and that’s not research. Intuition is not satisfying, you need data!”

Frequently at a projects conclusion, Zoe will present the results that she has worked through on NVivo, and find the heads of departments are moved to realign their goals. They realize through the uncovered latent opportunities that they could be the first in the industry to fulfil a customer need, and they need to act on it first. “They need to be prepared to take a leap forward. They become of the mind that if they can move on it quickly, they can really be ahead of the game” said Zoe.

There's a lot of power in truth, and at the end of the day, it's what has given the organizations that invest in a UX researcher like Zoe a commercial advantage. 

ABOUT THE AUTHOR

QSR International with Zoe Rosen

Zoe Rosen is a User Experience researcher, with a background in cultural and behavioral anthropology studies. It’s a discipline that she lives and breathes, and loves the research, the ethnographic style, the broadness, the way anthropology is completely dedicated to how people behave and interact with everything around them. Zoe has a wide experience in many industries from banking and insurance, to healthcare. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

NVivo strengthens qualitative research into improving education for Indonesia’s 50 million-plus students

Dr Agustinus Bandur uses NVivo to improve the quality of education in Indonesia.

One of the driving influences behind improving the quality of education across Indonesia’s schools and higher education institutions is Dr Agustinus Bandur. A senior lecturer and strategic research & partnership leader at BINUS University in Jakarta, Dr Bandur has spent more than 15 years researching how to improve leadership and management in education across Indonesia. Dr Bandur also consults to several not-for-profit organizations, including the Florenza Children Resources Center, which he founded in 2008 to help children improve their learning.

Improving education in Indonesia

For Dr Bandur, the issue of improving Indonesia’s education system is compounded by its sheer size and diversity.

With more than 50 million students and 2.6 million teachers in more than 250,000 schools, it is the fourth largest education system in the world, behind only China, India and the United States.

While Indonesia has made major progress in improving its primary and secondary education, serious issues remain around the drop-out rates of students. According to the 2016 National Socio-Economic Survey, around one million children between 7 – 15 years old are not attending primary or junior secondary school. Meanwhile, another 3.6 million adolescents aged 16 to 18 are out of school. There are also concerns around the quality of learning at Indonesian schools. According to UNICEF, only 81% of primary school teachers hold the minimum qualifications required by government.

The role of research

Dr Bandur’s research has been central to discussions around school-based management (SBM) policy reform in Indonesia, which saw education decentralize and shift responsibility to schools in 2005 in an effort to improve management within the education sector. In response to higher education globalization in Indonesian universities, Dr Bandur has also focused his research on higher education internationalization.

While Dr Bandur has plenty of passion for improving Indonesia’s education management and leadership, what he doesn’t have is a lot of time. With so many projects, educational institutions and not-for-profit organizations are relying on his research insights, Dr Bandur realised early on that he needed a faster, more efficient way of collecting, managing and analyzing his research data.

In 2002, Dr Bandur began using NVivo, software designed to support qualitative data analysis. Since then, Dr Bandur relies on it almost every day for his research needs, from conducting literature reviews to content analysis of various sources, thematic and cross-case analysis, as well as mixed-methods research. He also uses NVivo to ensure his articles and books are based on authentic, evidence-based data.

How NVivo Helps

NVivo is the number one software chosen by academics around the world for presenting the most robust, defensible findings from qualitative research. According to Dr Bandur, NVivo saves him significant time and
effort, particularly with queries for content analysis, cluster analysis and visualizing themes, as well as with transcribing data.

NVivo saves time with transcribing

“For any qualitative research, transcribing is tiring,” said Dr Bandur. “Prior to learning about NVivo, I would have to listen to interviews in Windows Media Player and transcribe them into Microsoft Word. It meant having two programs opened at the same time, which took time to use and control. In NVivo, I can hear, see and manage my recorded data much more effectively and efficiently. It is also fascinating with NVivo to capture and transcribe secondary data from YouTube videos prior to conducting primary data collection in the field.”

NVivo identifies themes from hundreds of papers

“Another advantage of NVivo is that I do not have to read all the references to make sense of the data. Instead, with the query system in NVivo, I can search the main theme, word or topic that I am analyzing, saving me from having to skim through hundreds of papers and potentially miss important information. NVivo is also a perfect tool for me as it integrates my papers managed in Mendeley.”

NVivo adds credibility to research through the triangulation technique

“Finally, NVivo adds credibility to my research. It allows me to quickly create project mappings, analyze the attributes and perform cross-case analysis. The triangulation technique – combining data from different sources such as interviews, focus groups and photos – is something I use frequently to improve the strength of my findings. I also find team research helps. With NVivo, my research team can work on the same project, allowing us to conduct inter-rater analysis to measure reliability.”

Best practice research results in greater student achievements

According to Dr Bandur’s research, there is evidence that the implementation of the SBM policy has resulted in improving teaching learning environments and student achievements. He continues to study its effects and train others – along with advocating NVivo as a best practice way to collect, manage and analyze data. “The NGOs I work with, including Wahana Visi Indonesia, SMERU Research Institute and Perkumpulan Prakarsa are all now using NVivo. My Doctoral and Master’s students have also applied NVivo in their Doctoral dissertations and Master’s theses. I strongly recommend NVivo to students, scholars and research because it is a powerful tool for the purposes of conducting content, thematic, and cross-case analyses in qualitative research.”

ABOUT THE AUTHOR

QSR International with Dr Agustinus Bandur

Dr Agustinus Bandur PhD, is a senior lecturer and strategic research & partnership leader at BINUS University in Jakarta. Dr Bandur has spent more than 15 years researching how to improve leadership and management in education across Indonesia. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Analysing large survey data using automated insights

NVivo Plus enables quick identification of themes and sentiment in University wide student feedback.

An impossible task?

The National Student Survey (NSS) is an annual survey which gathers the opinions, and experiences, of final-year undergraduate students on the course that they have studied. It is widely recognised as an authoritative and highly influential survey in building a picture of the quality of life in higher education in the UK.

At Lancaster University, as with many other institutions, there had to date been no systematic analysis of the qualitative element of the survey performed. The challenge was to develop a better and deeper understanding from the data collected, to improve the student experience of living and studying at Lancaster.

This was taken on as a pilot project by Steve Wright, Ph.D., a Learning Technologist in the Faculty of Health and Medicine at Lancaster University.

Finding the right tool for the task

The NSS is very important to Lancaster, as the score that is achieved has a huge influence on the current standing of the University.

Lancaster was awarded Gold in the Teaching Excellence Framework (TEF) following an outstanding NSS score (84.3% positive score) in 2016. It was also recently named University of the Year by the Times and the Sunday Times Good University Guide 2018.

The aim of the NSS pilot project was to feed into a broad spectrum of institutional activity in preparation for the University’s TEF submission and evaluation.

The first phase looked at a comparison of three possible tools to use for the qualitative analysis:

NVivo was chosen because:

NVivo’s ability to present data visually was also an important factor in the selection process. “In NVivo the visualisations enable you to explore the data which makes it a very powerful tool for both analysis and presentation,” said Steve. Rather than put a bar chart in a report, he presents directly from within NVivo. “The ability to use it as an interactive presentation is what makes it so powerful, for example, to be able to click on a column in the histogram and get the underlying data,” he said.

Testing the automation of sentiment and themes

Lancaster University received 8000 NSS comments, amounting to 25,000 words to be analysed, across the institution.

The NSS asked students to complete three open-ended comments:

Steve was interested in finding out what insights could be gleaned from the data, particularly if they used a systematic approach that could be replicated. The approach would pull out key topics, group those key topics and then explore the sentiment related to those topics. The idea being that as the sentiment is there in the NSS structure (asking for a positive and a negative comment), it is possible to check the accuracy of NVivo’s sentiment analysis (negative/positive) against those and then extrapolate from that, or use it as an example of the confidence you can have in NVivo’s automated sentiment tagging for other datasets.

“We have loads of text but mostly what happens is, we share it with no analysis, and only basic structure with the key people in a department for them to read through. And the easiest thing that happens, when they receive three or four pages of comments, is to read the first few and construct a narrative in their head and immediately get information bias,” said Steve.

“NVivo Plus tools were really good for this. I was able to take this minimally structured data – which only gave the department it related to and the type of a comment and then to extract topics and cross-reference those with the sentiment, as well as provide summaries,” he said.

Benefits of using NVivo

The analysis was well received however it received one critical response, which contended his analysis did not show anything further than the previous statistical analysis. Steve argued that was not the case. “The statistics show that students are broadly happy here. They like certain things but specific areas are shown by the quantitative statistics as being lower, however, what the statistics don’t do, is give any real insights as to the processes, the experiences that inform those lower scores.”

“What the qualitative analysis allows you to do, is to pull out those topics and break them apart to see why some departments had a higher score – to identify good practice, and some of the specific reasons given where there were lower scores to inform interventions and development, for example,” Steve said.
NVivo’s sentiment analysis capabilities played an important part in the data analysis, particularly given the way survey data is collected for the NSS.

“Because of the structure of the NSS, of positive comment and negative comment, we were able to cross-tabulate that with NVivo’s sentiment analysis and get a kind of built-in check of accuracy,” said Steve. “And it was very high. It tends to get things wrong by addition, not omission. i.e. it will classify something as both positive and negative when it’s just positive. The classic instance being ‘I had a load of personal problems and the department was fantastic.’ That is a positive comment, but, because it has the word ‘problems’ in it, it is automatically classified as negative as well.

Overwhelmingly NVivo classifies it correctly, we know that there will be some false matches but they’re a minority and given the volume of data it enables us to work with they can be accounted for,” he said.  “What’s more, this gives us a baseline for being confident in the sentiment analysis when we apply this approach to other student feedback and comment data without this structure.”

Being able to share the project across the University with other staff who are familiar with NVivo is an ambition for the future, as opposed to sharing a static report. Staff can delve straight into the project and discover insights for themselves.

Future Work

The University is planning to repeat the analysis next year, and build upon the framework.

From those who have seen it, there has been some real interest. “I think the real potential is with student or staff surveys. Most organisations have staff surveys, and they ask for extensive qualitative comments and usually, don’t do any sort of systematic analysis with them,” said Steve.

The point of the project was to develop a method, and NVivo assisted with a better analysis of this data. “The questions were:

And I really think the support NVivo provided has a real potential for other sectors with those practical priorities for working with qualitative data rather than the software being part of the somewhat arcane, and highly theoretical, pursuits of qualitative analysis within academia,” said Steve.

He also suggests that there’s a significant opportunity for commercial and public-sector organisations who need to work with unstructured datasets for analysing customer experience, and with a lot of potential for further development of methods and approaches like those introduced here.

About Steve Wright

Earning his Ph.D. in E-Research and Technology Enhanced Learning in 2014, Steve Wright works as a Learning Technologist in the Faculty of Health and Medicine at Lancaster University in the UK. He is also an independent CAQDAS trainer, consultant and certified NVivo expert. As a researcher, he completed five small-scale research projects, in addition to his Ph.D. thesis on sensory learning with a focus on craft beer, with which he took an ethnographic approach.

As an academic-related professional, he’s particularly interested in the e-research area and discovering what is possible for digital tools and how they’ll influence new approaches, which remains his focus. He also has an interest in the development, research, and teaching of methods. Steve’s consultancy and training work is through www.caqdas.co.uk    

About QSR International

Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

NVivo in mixed-methods research

NVivo has a long history of assisting qualitative researchers with managing, and analyzing their data, and being a part of complex research. What you might be interested to know, is that NVivo is also being used by mixed methods researchers to help get a wider picture of the data they’re examining.   

Background

Melody Goodman, PhD is a biostatistician and an Associate Professor in the Department of Biostatistics, at New York University Global Public Health. Her work is focused on racial disparities in urban settings, and she is particularly interested in examining how place is a determinant of disparity, and looking at the concept of how ‘where you live, work, play and pray’ impacts your health outcomes. She works in mixed methods research, and as a biostatistician, has a traditionally quantitative view. Much of Melody’s research has been on health and research literacy and how people understand and use health information.

Speaking about the impact of her research, Melody said “My attempt at working towards addressing disparities has been to increase the knowledge level of the receiver of the information. My work spans a wide spectrum from the generation of new statistical methods all the way out to community engaged research, including a program where we train community members on research methods. I don’t specialize in any particular disease area or any statistical methodological area.”

Introduction to NVivo

Melody is currently using NVivo to validate a quantitative survey tool that assesses the level of community partner engagement in research. When her team were required to evaluate their work, they found that there were no existing validated tools, and they would be required to create their own. “We thought that we could find some existing measure and use that to evaluate our programs. But when we went to the literature there really wasn’t any measures that assess how engaged people felt they were in research processes and research centers. So, we decided to develop our own measure, which was really trying to evaluate a big comprehensive research center that had multiple research projects” Melody said.

As the team were developing a brand new measure, and there was no “gold standard” to validate by, they found the qualitative work they undertook was important for a number of reasons; “We were trying to evaluate from the community health stakeholder’s point of view, instead of the academic’s perspective, and considering what the benefit is for a community member for participating in our research and how engaged they feel in the process” Melody said.

Development of a survey tool requires use of mixed-methods and the team uses NVivo to analyze qualitative data. However, they are also using the software as a project management tool, as Melody noted “We had so many different survey data sets and there’s a lot of rounds, with multiple surveys of experts and participants, we really used it to see where we were, and what data we’d collected. Not only were we using it to analyze the qualitative data, we also used it to keep everything in one place for this project.”

Further work with NVivo

Melody is also using NVivo for another project. Melody read some work by sociologist Elijah Anderson who created the term “white space”, meaning spaces which excluded anyone who wasn’t white, either explicitly or implied. It was particularly timely, given the current social climate, and the history of St Louis (where this project originated), with segregation, white flight, suburbanization, and gentrification.

Melody looked at this existing research from Anderson through the lens of her own work, and her own social experiences. “It’s great work, and he had lots of ethnographical stories, but for me as a biostatistician I really want to measure it. As a black person, I could relate to it, but I didn’t feel like I could convince someone who wasn’t black that this idea really existed. If you didn’t have those life experiences, you may view ethnographic work as anecdotal stories” she said.

Melody was interested in creating a survey measure that would assess if a space was a “white space”. “The first thing we needed to do was talk to residents of St Louis, and it became clear that we needed to speak to both black and white residents, and all up we collected around 50 interviews” Melody explained.

As a biostatistician, Melody was particularly interested in gaining a full picture of the areas participants were mentioning in their interviews, to help inform the research further. “In the interviews, the participants talked about different cities, towns, and places such as shopping malls. NVivo was great because I could link census data which has the actual racial composition, and other factors such as percentage of poverty and median household income, of all the places they mentioned, so we could then code the data not only for the town, but also call upon the quantitative data that goes along with it. This is where NVivo showed what it’s really powerful for in mixed methods work” Melody said.

As a mixed methods researcher, Melody thinks of the quantitative data as the ‘what’, and qualitative data and the ‘why’. Having the ability to merge those two together and compare, for example, if someone reports in an interview that a space is predominately white, with available census data, and triangulate that using NVivo, has been an important part of this project.

Project outcomes

“Ultimately with this project, we want to create a quantitative survey tool that will allow others to assess whether a space is perceived to be a ‘white space’. Currently, we’re a long way off from that, but we had to start with the qualitative, asking people how they think about ‘white space’, how they talk about ‘white space’, how they define it, and if they even know what we’re talking about? It became so timely because of all the things that are going on in our country, and in this community in particular. We’ll probably get more from it than just the survey, because people really gave us so much information and were incredibly honest in their responses” Melody said. 

When it came to selecting software for this project, Melody’s previous experience with NVivo led her to choosing to work with it again. “I had difficulty understanding competitor software. As a traditionally quantitative person, with NVivo I could make sense of what my team were doing, and they were able to generate reports that I could understand” Melody said. 

“When I decided that I was going to learn a qualitative software package, I felt most comfortable attempting to learn NVivo. And it was a challenge to think like NVivo, mainly because I don’t think in qualitative terms, but I found the training to be immensely helpful. Once I did that, I could then go and play and learn more myself” she said.

As for the future of Melody’s mixed methods work, the outlook is positive. “I think I’m in a good space. There’s a need for researchers in mixed methods who can understand the quantitative, and the qualitative and go in and interrogate that data and triangulate the qualitative and quantitative findings” she said.

ABOUT THE AUTHOR

QSR International with Dr Melody Goodman, NYU

Dr. Goodman conducts applied biostatistical and survey research for community-based interventions and health disparities research with a strong focus on measurement. Additionally, through academic-community collaborations, she implements, evaluates, and enhances the infrastructure of community-engaged research, in order to mitigate health disparities. With numerous funders supporting her work, she has published over 70 peer-reviewed journal articles. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

From journalism to research and evaluation: a research journey

Dr. Anupama Shekar shares her story of following her passion for ensuring equitable access to quality education, no matter a child’s economic circumstance, and how it took her from a career in journalism in India, to post-doctoral research in the U.S. Her work has featured a long history of working with qualitative research and evaluation tools, including NVivo.

Introduction

Dr. Anupama Shekar, PhD, is a qualitative researcher and program evaluator with a passion for the field of educational research and evaluation. She is currently an Evaluation Consultant with the Center on Research and Evaluation at the Simmons School of Education and Human Development at the Southern Methodist University (SMU) in Dallas, Texas. Her prior experience includes working as the Director for Evaluation at Teaching Trust in Dallas, Texas, an education leadership non-profit organization. She also worked as an associate researcher, and prior to that a post-doctoral research associate with WIDA which is a national and international prek-12 language development and assessment program housed at the Wisconsin Center for Education Research at the University of Wisconsin-Madison.

Before undertaking her postdoctoral work, she earned her PhD at the Department of Educational Leadership and Policy Analysis at the University of Wisconsin-Madison. She also assisted in the development and evaluation of WIDA’s data literacy program known as LADDER for English language learners. Funded by the US Department of Education's Office of English Language Acquisition, this project essentially helped participating schools make data-driven decisions about English language learners.

Prior to coming to the U.S., Anupama received her Master's degree in journalism at the Symbiosis Institute of Mass Communication and worked as a print journalist for the New Indian Express, a national mainstream newspaper in Tamil Nadu, Southern India. In her role as a print journalist, she focused on public educational leadership and policy issues in South India, where her journey in education began. The school leaders she met served children from low-income families. They greatly impacted her and she was inspired to leave journalism and study educational leadership and policy.

A journey in education leadership and policy research

Anupama recalls why she felt compelled to change careers. “It was the initial encounter that I had with several children from low-income communities,” she said. “They really awakened my interest in studying education leadership and policy formally and improving the public school system in India.”

“Many years later, the first story I wrote for The New Indian Express in 2006, still remains on my desk, “ she said. “It continues to keep me focused on why I began this journey and the importance of working to improve the lives of children from low-income communities anywhere in the world.”

A 14-year-old girl said that she had to work to feed her mother and brothers, and could not go to school. That really stuck with Anupama. Although education is a fundamental right of children under the Indian constitution, thousands of underprivileged children still have no real access to a school or quality education. “At that point I started developing an interest in research and evaluation in education leadership. I wanted to study successful school leadership practices and leaders who advocate for children from low-income groups despite the odds,” Anupama said.

It was when Anupama’s doctorate studies and WIDA work began that NVivo came into the picture. Her professors and other researchers used it, and her own research involved writing up case studies of school leaders in public schools in Tamil Nadu, Southern India. Previous research in the U.S. had examined the contribution of parent involvement in children's educational outcomes, but very little was focused on the role of school principals in fostering parent, family and community involvement practices.

Her analysis of previous research led her to design an exploratory, qualitative, cross-case study and informed her research questions: how do public school leaders in Tamil Nadu foster parent and family involvement? And what are the similarities and differences across schools?

“I used NVivo 9 to explore the initial transcriptions of interviews, contextual observations and field notes. It gave me an initial understanding of all the data and how the school heads initiated and supported parent involvement practices at their schools,” said Anupama.

While NVivo helped gain an initial understanding of the themes in her data, Anupama also used a traditional and manual coding process while interrogating her qualitative data to unpack the complexities in her qualitative case studies.

“Manual coding helped me analyze the story of each headmaster and headmistress and see patterns. I needed to get close to the data to figure out the leaders actions more deeply,” she said. “I also used memos, and documents, and artifacts. I sort of let the curiosities as a researcher take over. I feel moving between manual and software coding really helped me with my dissertation analyses and to triangulate my own thinking and findings,” said Anupama.

She notes how innovative uses of qualitative data helped her accomplish a richer understanding of experiences in the case studies. “The main study findings were that the school heads’ over time created a continuum of overlapping actions that helped foster effective parent involvement. I was really able to get to the core of the school heads’ actions through usage of multiple analyses techniques and constant reflection on the qualitative data. As a qualitative researcher, you really commit to spending extended periods of time to get to the heart of the story” Anupama said.

During her work at WIDA during her doctorate studies, the WIDA’s LADDER project convened many focus groups, as well as individual interviews and mixed methods evaluation. “Each year we produced a program evaluation report and wrote up findings, so NVivo was useful as one of the tools that helped us identify themes and patterns,” said Anupama. “WIDA still offers the LADDER program, and I was there when they were developing the whole program from the ground up,” she said.

When Anupama moved onto her postdoctoral work, WIDA’s Teaching and Learning team were trying to understand best practices in professional learning and professional development. One large project involved multiple qualitative open-ended questions. Anupama found her prior experience helpful. “NVivo was a great tool for me to use then because we were working with a lot of diverse data and it ended up providing great insights,” she said.

Most recently she worked as the Director of Evaluation with Teaching Trust, an educational leadership non-profit in Dallas. Teaching Trust offers high quality training and support for future school leaders, school leadership teams, and teacher leaders to ensure that children in low-income schools across Texas have access to an excellent education.

“Teaching Trust has a strong alumni base and educators who graduated from Teaching Trust programs are out in the field driving positive change for students,” said Anupama. “The Teaching Trust Alumni Network team always gathered and used data effectively to drive their programmatic decisions. In this case, the team was trying to understand through qualitative data, the impact of the Teaching Trust alumni programming from the participant's point of view and how future programming might be improved and changed,” she said.

The Alumni Network team conducted qualitative focus groups of current and former participants. “After every focus group, our team met to extract meaning from the data — the impacts of Teaching Trust programming on participants, personal leadership, student and school outcomes, and what it really meant to be part of the Teaching Trust community,” said Anupama.

The team used both manual and software coding techniques with their qualitative data. “We took a grounded theory approach by listening and gathering data, and bridging perspectives to really unpack the themes and patterns” said Anupama.

“My former colleagues used pen and paper, and I used NVivo to code,” Anupama said. “There is a lot of power in combining multiple qualitative coding techniques because that adds to the validity and reduces researcher isolation. We presented the lessons learned and techniques on the collaborative qualitative approach in a webinar to the American Evaluation Association.” she said.

A passion for qualitative insights

Anupama’s career has evolved through her interest and passion for educational research and evaluation and ensuring people have equitable access to quality education, no matter their background or economic circumstance. Her appreciation for the importance of qualitative research and evaluation has been at the heart of her work.

“Qualitative data tells you something that numbers cannot, and helps you dig deeper to explore the complexities and find powerful insights,” she said. “As a qualitative researcher and evaluator, my challenge has been to find meaning in data, to keep asking why, and to continue digging,” said Anupama.

Anupama also hopes to continue sharing the power of qualitative research and evaluation through her website and blog in the near future. “There is a renewed energy in qualitative research and evaluation that is really exciting. There are people around the world who use qualitative data in very different ways in their work. I think it will be valuable to hear and share their stories as continual learning is the core of qualitative work.”

Next steps in career

Anupama hopes to use her learnings in qualitative research and evaluation at her current work at the Center on Research and Evaluation (CORE) at the Simmons School of Education and Human Development at the Southern Methodist University (SMU) in Dallas, Texas.

“I am excited to be doing projects for CORE and collaborating with their diverse and strong team of researchers and evaluators led by Dr. Annie Wright. They are at the forefront of conducting rigorous research and evaluation that focuses on examining critical issues around children, families and communities.

CORE is constantly striving to push boundaries and was selected as one of the Annie E. Casey Foundation’s expert evaluators nationwide. This shows the focus CORE has on issues around diversity, equity and social justice. I am honored to be learning as a researcher and evaluator with this incredible organization.”

You can follow CORE’s work on Facebook and Twitter.

ABOUT THE AUTHOR

QSR International with Dr Anupama Shekar

Dr. Anupama Shekar, PhD, is a qualitative researcher and program evaluator with a passion for the field of educational research and evaluation. She is currently an Evaluation Consultant with the Center on Research and Evaluation at the Simmons School of Education and Human Development at the Southern Methodist University (SMU) in Dallas, Texas. Her prior experience includes working as the Director for Evaluation at Teaching Trust in Dallas, Texas, an education leadership non-profit organization. Every day, QSR International helps 1.5 million researchers, marketers and others to utilize Qualitative Data Analysis (QDA) to uncover deeper insights contained within the “human data” collected via social media, consumer and community feedback and other means. We give people the power to make better decisions by uncovering more insights to advance their area of exploration.

Alberta Health Care Reform: Using NVivo 8 to deliver results, today and tomorrow

The province of Alberta in Canada is home to 3.7 million people, spread across some 661,000 square kilometers. In 2009, Alberta's nine independent health authorities and three provincial entities were merged and a new entity - Alberta Health Services (AHS) - was formed to co-ordinate the delivery of health services across the province. The task was substantial, with the benefits expected to be just as big.

AHS Director of Knowledge Management (KM) - Capacity Building, Dr. Cathie Scott and KM Consultant, Laura Lagendyk are leading a large multidisciplinary mixed methods research project to understand different models of care and how they support collaboration among primary healthcare professionals. The team is using qualitative data analysis software NVivo 8 to deliver immediate outcomes to primary care networks (PCNs) in the field, as well as facilitating the overarching corporate goals of the five-year project.

Primary health care reform and CoMPaIR 

Primary healthcare (PHC) systems are undergoing significant change both across Canada and throughout the world. New models of providing PHC are being explored, but differ according to their context. The one constant is their end goal. Prevention is a key objective of primary health care, and it involves treating short-term health problems, managing long-term health conditions such as diabetes, and encouraging learning to avoid injury and illness.

While there have been many iterations of change within Alberta’s primary healthcare system the most recent reforms began in 2003. Shortly afterward the CoMPaIR project was funded by the Canadian Health Services Research Foundation (CHSRF), the Alberta Heritage Foundation for Medical Research (AHFMR) and the Calgary Health Region. CoMPaIR is an acronym for Contexts and Models in Primary Healthcare and their impact on Interprofessional Relationships. The project's primary brief was to understand the interprofessional collaborative models that healthcare professionals were using to provide care in new and innovative ways. Underlying this program of research is a focus on developing innovative strategies for making better use of evidence to inform practice.

"There has been a substantial shift in the way front-line health care services are delivered in Canada. Over the past six years primary care networks (PCNs) – where groups of family doctors work with the AHS and other health care professionals to coordinate patient care - have come to the fore," Dr. Scott explained.

"While the initiatives put into practice across each province are different, they all share a common feature – practitioners are being asked to work in new ways across professional boundaries. The CoMPaIR project provides a unique opportunity for Alberta's health care professionals to share their own work and learn about the models being used in other sectors too."

Keeping pace with the evolution of the industry 

The research project was designed in three phases: Phase One focused on three primary care networks in Calgary; Phase Two is provincial in scope; and Phase Three adds a national component.

"One of the challenges for us was that the project was already underway when the announcement was made that there would be fundamental changes to Alberta's health care system," Ms Lagendyk said.

"But what that has also meant is that we're actually studying evolving contexts that will influence how the health care industry will move forward in Alberta. That's challenging, but exciting too."

Choosing qualitative data analysis software 

While CoMPaIR involves quantitative research methods in the form of survey work and network analysis, it is largely a qualitative study. The project team's methods have been primarily interview-based, however literature reviews and observation methods have also been used.

Dr. Scott and Ms. Lagendyk have both used qualitative data analysis software in the past, and chose QSR International's NVivo 8 software for use on CoMPaIR.

"With CoMPaIR, we've done some really in-depth data collection around the context, model and outcomes that the different primary care networks have used and why," Dr. Scott said.

"This comes in the form of field notes, individual and focus group interview data, and observation data from different settings such as meetings and clinics. It's all imported into NVivo."

While NVivo 8 can work with many forms of qualitative data, including audio and video, the CoMPaIR team currently transcribes and imports all of their project data in textual formats. Each of the primary care networks is assigned as a 'case' within the software. The data is organized – according to its content and themes – in a hierarchical structure (known as a ‘tree node’ structure).

"We started with a fairly comprehensive tree that’s morphed as we’ve gone along. We've added and combined 'nodes' [or themes] as we get a little more concrete about which areas of the primary care models we’re going to compare and contrast when we put our reports back together," Ms. Lagendyk explained.

"One of the biggest benefits of NVivo is being able to easily classify the data in different ways, including grouping sets of data within NVivo. For instance, we grouped all of the interviews, references and content relating to physicians in one set; all of the nursing materials in another, and so on. Then being able to tease that apart through the matrix structure or any of the queries that you build in is a huge plus."

"NVivo lets us be quite experimental and innovative in our approach. We’re trying to play with the case approach and pull that data together in different ways, while involving different stakeholders in setting the direction for our work," Dr. Scott added.

Delivering immediate benefit in a long term project 

One of the innovative ways that the CoMPaIR research team is using NVivo 8 is to deliver ongoing insights that can be applied immediately, in what is undeniably a long-term project.

"One of the ways we've been working with NVivo is through a team based approach to a number of smaller sub-projects within CoMPaIR. We'll partner with the research members of the team as well as the actual people within the networks to isolate and examine a topic," Dr. Scott said.

"For instance, Laura was working with two primary care networks that took very different approaches to chronic disease management. One had created a stand-alone clinic, and the other was developing a community of practice involving people who were doing hands-on care in diabetes.

"We already had interviews from people within each of those chronic disease clinics and we analyzed them as a mini-project to look at each model and its advantages and disadvantages, and shared those insights."

Dr. Scott and Ms. Lagendyk are also working in a similar way to share immediate insights with health professionals using CoMPaIR's newsletter, ‘The Echo’. The insights shared in the newsletter may be the result of specific sub-projects or based on the research team's observations of knowledge gaps or specific stakeholder wants and needs.

"It's about picking up on those interesting insights or questions that we’re seeing when we’re doing data collection," said Ms. Lagendyk.

"The first one we did was in response to a lot of people within primary care networks asking about how to create policy - they had very little experience with that kind of thing. So the students that were working with us pulled some literature and quotes out of our NVivo database and did a short, two-pager about policy.

"Another was around one of the small sub-projects we completed with social workers. We did targeted interviews with a few social workers and then combined that with the results of our network analysis and released a short paper on what social workers actually do within this primary care network. That was in direct response to the health care professionals in the network, who wanted us to work with them on articulating their roles.

"NVivo 8 has helped us to deliver those immediate results that can be shared across our networks – and that work will still contribute to the larger study. The fact that we can use NVivo 8 to code data and look at it as a stand-alone project and then also seamlessly merge it in with the rest of our data is a huge advantage."

Facilitating team work 

Today the CoMPaIR project team involves no less than 20 researchers and field-based practitioners. Dr. Scott said whereas on other research projects, all team members often code and work within NVivo 8 together, potential conflicts of interest prevent this from happening on their project. However the use of software is still helping to facilitate outcomes across the team.

"Our team isn't made up of just academics, but practice based people too," she said.

"We have specialists across diverse areas including patient family centered care, public engagement, health policy, workforce optimization, economics, knowledge management, as well as physicians and other health professionals who are embedded within primary care teams. Because some people are actually mentioned by others in the interviews, we haven’t, for ethical reasons, opened up the database to the whole team. We’ve had to come up with a strategy that allows each researcher to get what they need out of the data without actually being in the data."

In response, Dr. Scott and Ms. Lagendyk moved away from whole-of-team engagement and instead allowed smaller groups to have ownership over their focus areas through partnering them in NVivo.

"Each of those key members now frames the questions that they want to ask the data, sends those to us and we use NVivo to pull out content that may address their questions," Dr. Scott explained.

"We then have discussions with that team member about what they’re seeing and what’s been pulled out, and how it helps them understand their particular area of interest.

"It has increased our team's level of engagement and facilitates far better outcomes than trying to work across such a large team using manual methods. Especially for this disbursed team, which spans several countries, it's a new way of approaching research, which is quite exciting."

Data sharing and portability 

Dr. Scott also pointed to the value of having a defined analysis structure when introducing new team members to an established project.

"We’re looking at the prospect of bringing in another person to manage the project," she said.

"Even if we didn’t hand them the whole NVivo 8 database straight away, it seems much easier to be able to give them our tree structure and get them starting to think about the ways that we’ve been looking at the data.

"Rather than me doing three or four major brain dumps and having to pick up where I left off, they would have a well established database and documented analysis across several years as their starting point."

Delivering on the project as a whole 

The CoMPaIR project runs until the end of 2010. At its conclusion the research team will share what they've learnt about the models of primary healthcare across Canada. The results will contribute to an in-depth understanding of the influencing factors in the development of primary health care models. Specifically, Dr. Scott and Ms. Lagendyk's research will help health bodies like the AHS understand what's needed to make these systems work, for the good of both the healthcare professionals and the communities they support.

About Alberta Health Services 

Alberta Health Services (AHS) is tasked with co-ordinating the delivery of health supports and services across the Alberta province. The organization brings together 12 formerly separate health entities in the province: nine geographically based health authorities (Chinook Health, Palliser Health Region, Calgary Health Region, David Thompson Health Region, East Central Health, Capital Health, Aspen Regional Health, Peace Country Health and Northern Lights Health Region) and three provincial entities working specifically in the areas of mental health (Alberta Mental Health Board), addictions (Alberta Alcohol and Drug Abuse Commission) and cancer (Alberta Cancer Board). It also includes Alberta's ground ambulance service. The AHS workforce is made up of 90,000 caring and dedicated individuals. Information: www.albertahealthservices.ca

About Dr. Cathie Scott and Ms. Laura Lagendyk

Dr. Cathie Scott, Director, Capacity Building, AHS leads the development and evaluation of processes to build knowledge management capacity within strategic priority areas and across the organization. Her team will support development of capacity for evidence-informed decision-making primarily through project-based learning. Cathie is also the lead for AHS’s commitment to follow through with next steps for SEARCH Canada’s intellectual assets, working in collaboration with other partner organizations. Cathie completed a postdoctoral fellowship focused on knowledge exchange, holds a PhD and MSc in Community Health Sciences, and a BSc in Physical Therapy. Cathie has strong ties with the academic communities.


 Laura Lagendyk has over 10 years experience in academic and applied health research and evaluation and has worked on projects in primary, acute and tertiary care settings. With a Master’s degree in Health Research and a background in science and social work, she continues to further her skills through workshop presentation, group facilitation, and conflict management education. She has a keen interest in working with teams interested in building their capacity to produce and use evidence in innovative projects of direct relevance to their work environment. 

Living Kingston 2035: Creating an informed vision with NVivo

Located in the Melbourne metropolitan area, the Council and management team at the City of Kingston decided to create a shared, long term vision for its diverse community. The project was named Living Kingston 2035 and provided the community with an opportunity to share their needs, interests and aspirations. The Project Team, led by Dr. Robyn Cochrane, used NVivo to progressively analyze and securely manage a significant volume of data which included survey responses, online submissions and forum discussions. By using NVivo, the City of Kingston could efficiently and systematically sort the community’s input to make informed evidence-based decisions and develop plans for the medium to long term.

Benefits of using NVivo:

Introducing Living Kingston 2035

In March 2011, Council resolved to create a long-term community vision for Kingston. The vision was to become a key legacy for the outgoing Council and provide useful input to the decision-making and planning processes of the incoming Council (following elections in October 2012). After considering Officers’ advice, Council resolved to establish a Steering Group and Reference Panel to oversee and guide the community visioning project as well as a Project Team to implement the project.

Planning the consultation

The Project Team recognized the many challenges posed by effectively engaging with the community, particularly about a topic as far-reaching and abstract as a “community vision”. A review of a community visioning exercises undertaken nationally uncovered a range of consultation approaches that had been usefully applied. The Project Team noted the importance of carefully framing key questions that could be used across multiple participation options and decided upon:

Early decisions were made to use several communication channels to inform the community about the project and to offer multiple participation options to enable interested community members to submit their views in the consultation. The consultation was envisaged to comprise traditional paper based surveys, telephone-in options, staff forums, public community forums as well as the use of online, web based and mobile channels. A small number of local government authorities had experimented with web-based technologies and it was quickly agreed that these technologies could provide an additional participation option that might overcome some of traditional consultation challenges. Web-based technologies were regarded as a legitimate and innovative way to meaningfully engage with the community, encourage broader participation and more diverse inputs into the consultation.

Sourcing project funding

The project was initiated and funded by the City of Kingston. However, the Project Team was aware of a community support grants program offered by the Victorian Government for community support initiatives. Contact was made with representatives of the Department of Planning and Community Development to seek advice as to the potential suitability of this project for funding. Following an intensive grant application process and lengthy and competitive external evaluation process, the City of Kingston’s application was successful. A grant of $48,000 was secured to support the consultation and, in particular, the community strengthening and capacity building features of the project.

Turning unstructured data into an informed direction

The Project Team recognized that it was critical to the integrity of the project that the data were efficiently managed and reported to key stakeholders throughout the duration of the project.

NVivo was used at three key milestone stages:

Using NVivo our Project Team had the capacity to produce multiple reports encompassing over 1,000 pages of “raw data”. The reports presented the findings for each broad theme, geographical areas (Council ward) and participation option (survey, online submissions and forum discussions). High-level descriptive reports as well as detailed research reports were produced to suit the needs of specific audiences.

The role of NVivo

Dr. Cochrane was already a proficient user of QSR International’s NVivo having used the product prior to working with the City of Kingston. Thus, the Project Team recognized that NVivo was a relatively inexpensive computer-assisted qualitative data-analysis software tool that could efficiently and effectively manage large volumes of consultation data. Consequently, the Project Team elected to use NVivo 10 because it:

Using NVivo 10 the Project Team were able to:

Although it is difficult to estimate, it may have taken the Project Team around three months to manually analyze the consultation data without NVivo. Managing, analyzing and subsequently reporting the data without NVivo would have been an extremely labor-intensive task. It is highly unlikely that the findings would have been formally reported with the same level of confidence, rigor and visual aids.

The use of best practice computer-assisted qualitative data-analysis software minimizes subjectivity within the sorting and presenting of the data and allows this to occur without researcher bias. It was believed a computer-assisted approach would provide better and clearer outcomes from the consultation.

Living Kingston 2035 Program Leader Dr. Robyn Cochrane stated “We could quickly and confidently report to our management team and Councilors that all of the data had been carefully sorted using NVivo 10. There was little room for researcher bias and loss of data – we could confidently say that every idea was coded, sorted and reported.

Sarah Bishop, Manager – Communications and Public Affairs commented “It was crucial for us to quickly turnaround the consultation data so our Councilors could develop their draft 2013-17 Council Plan. By using NVivo our project team could quickly generate visualizations, frequency tables and customized detailed reports. When these reports were considered alongside the participant profiles for each data set and other information about the municipality of Kingston, Councilors could confidently use the consultation data to make decisions and plan for the future."

Word frequency query for the Arts, Culture and Heritage WISH ideas displayed visually as a tag cloud. The size and density of each word reflects the frequency reported in the WISH ideas. For the purposes of this example, the 100 most frequently reported words (comprising four or more letters) were selected.

Dr. Cochrane added “Sharing in-depth and high-level descriptive versions of the consultation findings with key stakeholders and the community is important to the integrity of the project and is best practice in the public participation sphere. Through this project, the City of Kingston has informed, consulted, involved, collaborated and empowered its community (see iap2.org.au)”.

Related products:

About City of Kingston:

City of Kingston is responsible for ensuring that its 142,000 residents are well governed and serviced. To find out more: kingston.vic.gov.au. The Victorian Government has created a guide which details the roles of local Councils. To find out more: localgovernment.vic.gov.au

About Dr. Robyn Cochrane:
Dr. Robyn Cochrane was employed by the City of Kingston as Project Leader of Living Kingston 2035. Robyn works as a Research/Teaching Associate with Monash University and is the lead researcher with Cochrane Research Solutions. Robyn holds a PhD in Management and has over 25 years of experience in the local government, vocational education and training, higher education and community sectors. Robyn has published articles in academic and practitioner journals and produced a range of research reports for industry partners. Dr. Cochrane’s interests include: qualitative and quantitative research designs and analysis; online and traditional surveying approaches and community strengthening. Contact details: robyncoc@bigpond.net.au

Exploring how we understand data differently

Bruce is comparing how experts and novices understand data for his PhD thesis.

 

Who used NVivo?

Bruce Tsuji, PhD student, Carleton University’s Department of Psychology, Ontario, Canada

What was the project?

For his PhD thesis, Bruce is comparing how experts and novices understand data – in particular, how they make sense of graphs and other kinds of visualizations.

As part of his research, Bruce asked members of Carleton University’s business and psychology faculty, to explain the content of a variety of graphs that were cast in either a business or a psychology context. Undergraduate students were asked to do the same. This helped Bruce to understand how it is that experts (for example, faculty professors) seem to be able to draw so much more information from graphs than university undergraduates.

The explanations were recorded and transcribed in order to identify the different components of the explanations provided.

The value of NVivo and how it was used

In his initial analyses, Bruce identified 23 different components to the explanations provided, and eventually settled on nine. NVivo helped him to identify those nine components and to subsequently understand how frequently they occurred and to determine the serial order in which they appeared in his participants’ explanations. This provided Bruce with clues about the strategies in use by novices and experts.

“When I first started working with NVivo I don’t mind admitting to a little fear and some scepticism. The fear had to do with how much time and effort learning another piece of software was going to cost me and the scepticism had to do with the fact that simply tabulating the components of my verbal transcripts was something that I could easily do with a spread sheet. I believe that a relatively small investment of effort will help determine if NVivo is the appropriate tool for you.”

“Queries was my favourite feature. After coding hour upon hour of verbal transcript data, the ability to quickly construct a query allowed me to step back and look at a bigger picture that often surprised me.”

“Apart from my dissertation, I often find myself producing small surveys of 10 or so multiple-choice or Likert scale questions along with one or two free response questions. I can readily imagine using NVivo 9 to get a better handle on those free responses. Similarly, I can foresee that an analysis of responses to the essay-type questions I produce in my university and college teaching could be helpful in understanding what students have acquired and what needs further work.”

Outcomes from using NVivo

By using NVivo, Bruce was able to get a “picture” of experts and novices completing the simple task of explaining a graph. As a result, this allowed him to identify the qualitative and quantitative differences amongst his experimental participants.

“I don’t think I would have been able to complete my analyses at all without NVivo. Obviously, I am aware of other competing qualitative analysis software packages but what I wanted was the lowest possible learning time. I also wanted to not lose sight of the transcript data itself. On both fronts, NVivo did the trick!”

Informing decision making in healthcare backed by a new standard of academic rigor

NVivo is helping the team at the Strategy Unit at NHS Midlands and Lancashire Commissioning Unit interrogate their data on a deeper level and make informed recommendations.

Background

The Strategy Unit hosted by NHS Midlands and Lancashire Commissioning Support Unit exists to improve outcomes through providing expert support and advice to public and third sector organizations. They work especially closely with Sustainability and Transformation Partnerships, emerging Integrated Care Systems and Clinical Commissioning Groups.  

Mahmoda Begum is a Senior Consultant within the unit. She works within a multidisciplinary team to support delivery of projects with a focus on high-quality research and evaluation. She contributes to generating, analyzing and interpreting evidence through mixed methods research approaches to evaluate change in health and social care.

While completing her Master’s degree in public health, Mahmoda had her first experience with NVivo. “Getting that experience allowed me to explore different research methods, and ways of critically looking at things. That’s now the approach I tend to take with qualitative research – that rigor and systematic academic way of doing things” she said.

Introducing academic rigor

The team at The Strategy Unit work with large amounts of data as part of their qualitative fieldwork, which includes transcriptions of interviews, focus groups and workshops, and researcher reflections. Access to data management and analyses tools in The Strategy Unit, such as NVivo, have allowed a new standard to be introduced to increase the quality of the outputs.

“Tools like NVivo have allowed us to support decision makers in the healthcare systems with the objective evidence of ‘what works’ and ‘what can be made to work’. We can be confident that it’s through a thorough analysis of all the data that has been collected in a timely and relevant way” she said.

The advantage of working to this standard is in being able to partner with academic organizations and publishing in peer-review journals. “Our approach wasn’t necessarily academically focused; it was more centered around meeting the needs of our health and social care leaders and practitioners that we were reporting to. Now we increasingly recognize the dual purpose of learning from and contributing to the evidence-base” said Mahmoda.

“It’s much more of a holistic approach than we had previously. There’s a few ideas that we’re actually working on now that could translate into academic papers, and NVivo has facilitated that” she said.

Showing NVivo’s benefits

Prior to Mahmoda joining the unit, the team had some initial exposure to NVivo, but she has been a driving force behind its adoption in the organization. For the past 18 months, the team have been working in NVivo, and utilizing its features beyond their standard data management activities. “We often try out new things that we think we might need for other projects, for example we’ve tried features like the visualizations, so we’ve tried to use it flexibly” said Mahmoda.

Showcasing the benefits of NVivo to her colleagues who have taken a more traditional approach to research has been a task Mahmoda has taken on with success.

“I’ve been able to show people in the organization who have come from a more traditional research background the benefits of NVivo by applying it to a dataset where they already had their own theories and notions around what the data was going to say” said Mahmoda.

“I was able to interrogate the data and say yes, these theories might match up to what you already think, but also there’s all these other ideas that you’ve missed out on. It’s been helpful to be able to give them the evidence to confirm their theories, but also demonstrate that some theories are simply their own preconceived ideas” she said.

NVivo has helped the team improve their processes for report writing methodology design and efficient analysis.  Looking to the future, the team want to be able to use it further in evidence synthesis, and to inform data triangulation. “The more people in The Strategy Unit who see the rigor and efficiency that NVivo provides, the more they value the benefit of using it” said Mahmoda.

NVivo in project work

The Strategy Unit were commissioned by NHS England to undertake a research project into appointment booking and other working arrangements covering the 10 High Impact Actions. The team interviewed practice managers of general practices, and looked at evidence that was already available with the intention to bring these pieces of information together in one place and explore the variations in access in general practice.

“We came up with a series of recommendations for the commissioners, national policy makers, and general practices on how general practices can improve appointment systems and other working arrangements for the access needs of their specific populations.  It was a substantial piece of work and hugely informative” said Mahmoda.

Overall, NVivo has been a welcome addition to the resources at The Strategy Unit, and has assisted in broadening the scope of what’s possible in their research and evaluation work. “Everything we do is evidence based, and NVivo really helps us practice what we preach” said Mahmoda.

“It’s something we want to use more widely, and we’ve got a lot of support internally to do that, we’re always looking for ways to improve and develop our skills” she said.

About the Strategy Unit at NHS Midlands and Lancashire Commissioning Unit and Mahmoda Begum

The Strategy Unit is a consultancy leading research, analysis and change from within the NHS.  They aim to support NHS, Public sector and third sector clients to understand the challenges they face, and make the best possible decisions, using evidence informed analysis and advice.

They work on a broad national level in the United Kingdom, and are made up of a team that have expertise and skills spanning from complex data analysis, decision support, research and development and strategic service transformation to executive-level strategic advice and evaluation.  They also work with a number of specialist associates and key partner organizations to deliver their services, including: ICF International, The Transformation Unit and the Health Services Management Center at the University of Birmingham. 

Mahmoda Begum is a consultant within The Strategy Unit, and has been a part of the team for just over 18 months. She comes from a research background and joined the team in a research and evaluation capacity and has helped drive the implementation of NVivo within the unit.  Mahmoda graduated university with a Bachelor’s Degree in Biomedical Science, but quickly decided a life of lab work wasn’t for her and moved to working in an applied social research setting, with a predominantly qualitative research focus. She has worked in the public sector for over 9 years, of which the last 4 years have been within the NHS.

ABOUT THE AUTHOR

QSR International with Mahmoda Begum, the Strategy Unit at NHS Midlands and Lancashire Commissioning Unit

The Strategy Unit is a consultancy leading research, analysis and change from within the NHS. They aim to support NHS, Public sector and third sector clients to understand the challenges they face, and make the best possible decisions, using evidence informed analysis and advice. Mahmoda Begum is a consultant within The Strategy unit, and has been a part of the team for just over 18 months. She comes from a research background and joined the team in and research and evaluation capacity and has helped drive the implementation of NVivo within the Unit.