It is time to learn about the effects of your EE program! What do the patterns, differences, and relationships in the data suggest about how well the program is achieving its objectives?
Table of Contents
Related Topics: Participatory Evaluation
What type of analysis do I need?
There are many different types of analyses that vary in complexity. To help you determine what type of analysis to choose, consider the following:
The paragraphs below discuss types of analyses according to whether you collected quantitative or qualitative data, and point you to software that can be used to analyze these data. Based on the information and resources in this section you should gain insight into what type of analysis to conduct for your evaluation and how to conduct at least some of these analyses yourself. In addition, you should have an improved understanding of what questions to ask of an evaluation expert to help you conduct your own or clarify her/his analysis.
How do I analyze quantitative data?
After reading this page and reviewing its resources, you may still feel the need for help. Consider consulting with an expert. You should be able to hire someone to show you how to conduct particular analyses - or - you can ask an expert to carry out complex or time consuming analyses.
Before you can analyze quantitative data, you will need to code data and enter data into a spreadsheet or a statistical analysis program (see Step 5).
You will want to begin with descriptive analyses of your data. These analyses are called descriptive because they allow you to summarize large amounts of information. Descriptive statistics include frequencies (counts), percents, ranks, measures of central tendency (e.g., mean, median, and mode), and measures of variability (e.g., range and standard deviation). In many cases, descriptive statistics will be sufficient to answer most stakeholders’ questions (Fitzpatrick et al, 2004). For example, you can calculate the mean response to a question or the percent of respondents who answered a question in a particular way. If you asked respondents to indicate how likely it was they will recycle batteries as a result of your program (on a scale from 1=extremely unlikely to 7=extremely likely), you may be able to report that the mean response was 6.5 or that 85% indicated that they will (by adding those that selected 5 or above).
Statistical significance is not the same as substantive significance!
When a difference is statistically significant, it means that the difference is probably not due to chance. This does not tell you if the difference is meaningful or trivial! Effect sizes provide you with a quantitative way to assess to what extent a significant difference may also be substantively important. To learn more about effect sizes, review the section on power analysis, statistical significance, and effect size
After conducting descriptive analyses, you may want to conduct more complex inferential analyses. These analyses include testing for significant differences. For example, you can test whether participants in your program scored higher on a multiple choice exam than individuals in a control group. You can even conduct these analyses in such a way that you can “control” for influences other than your program such as how participants performed before your program or based on their gender. For example, were female participants more likely to correctly answer the questions than male participants?
The resources below can help you learn more about how to analyze quantitative data:
Questions about the effectiveness of education programs, including EE programs, tend to be multi-level in nature. That is, although scores are generated by individuals, these individuals are grouped with others, for example, in classes and schools. It is at these levels (e.g. class, school) where the program is usually administered. In these cases, analyses should control for the shared experiences of children in the same classes, grade level or schools. Advanced statistical software programs are generally able to conduct analyses of this type.
- Analyzing Quantitative Data (.pdf)
Taylor-Powell, E. (1996). University of Wisconsin Extension Program, Development and Evaluation Unit
This guide presents an introduction to descriptive statistics and explains how to use them in evaluation.
- Inferential Statistics
Trochim, W. K. (2006).
This page describes how inferential statistics can be used in outcome evaluations. It includes links to descriptions of many types of procedures used in inferential statistics; including t-tests, ANOVA, Analysis of Covariance (ANCOVA), regression analysis, cluster analysis, and regression.
- Guide to Program Evaluation for Aquatic Educators (.pdf)
Recreational Boating and Fishing Foundation. (2006).
The section in Chapter 5, entitled “Analyze the Evaluation Data,” provides an introduction to analyzing quantitative EE evaluation data. It presents the steps involved in analysis, summarizes the purpose of statistical analysis, and explains how to reach conclusions from the analysis. A table gives an overview of the different types of statistical analyses and their uses in evaluations of EE and aquatic education programs.
- Hyper Stat Online Statistics Textbook
This online statistics textbook, available for free, offers a detailed discussion of how to use both descriptive and inferential statistics.
What software can I use to analyze quantitative data?
Software programs are invaluable in carrying out most quantitative statistical procedures. In fact, many of the procedures would be impossible for most individuals to carry out without such software. Fortunately there are many software programs to choose from including inexpensive or free ones:
- Using Excel for Analyzing Survey Questionnaires (.pdf)
This guide provides step-by-step instructions on using Microsoft Excel to:
- Create a database of responses
- Code data
- Display frequencies and percentages
- Calculate an average (mean), range, standard deviation, etc.
- Create cross-tabulations and pivot tables
Beginner Intermediate Advanced
This online clearinghouse offers links to hundreds of free resources for statistical analysis, including downloadable software packages, online statistics books, tutorials, and hundreds of pages that perform statistical calculations. This is a good place to go once you have a basic understanding of quantitative analysis and an idea of what you need for your own analysis.
- Open Stat
Open Stat is a free software package for performing statistical analyses. It is relatively simple to use and you can paste data into it from Microsoft Excel. There are some unique aspects to how it works, but it supports a variety of simple to complex analysis procedures, including frequencies, t-test, ANOVA, multiple regression, factor analysis, etc.
SPSS is one of the most popular software packages for statistical analysis and data management, though it is not free. It includes tools to help you organize, prepare, and understand your data, perform statistical analyses including basic descriptive statistics, regression, and many advanced procedures. It also helps you prepare results for presentation and reporting.
- AM by AIR
This software package was designed for "analyzing data from complex samples, especially large-scale assessments." It has many uses, from simple statistics such as frequencies, to more complex procedures such as regression and complex weighting schemes. Users can import data from a variety of sources (e.g., Excel, SPSS, etc.) and output can be displayed in a text file or a web browser.
How do I analyze qualitative data?
Though sample sizes in qualitative data tend to be smaller, the data sets themselves are typically large, complicated, and often messy to organize and analyze. In qualitative analysis, there are also fewer rules and standard procedures to guide the process than in quantitative analysis. However, good qualitative analysis, like good quantitative analysis, is highly systematic and disciplined.
Most of the qualitative analysis you are likely to conduct for an evaluation is what is called content analysis. Content analysis involves systematically analyzing the content of data, breaking it into meaningful pieces, and organizing those pieces in a way that allows their characteristics and meaning to be better understood.
Miles and Huberman (1994) propose breaking qualitative analysis into the following steps:
- Data reduction: This step involves selecting, focusing, condensing, and transforming data. The process should be guided by thinking about which data best answer the evaluation questions.
- Data display: This involves creating an organized, compressed way of arranging data (such as through a diagram, chart, matrix, or text). The display should help facilitate identifying themes, patterns, and connections that help answer your evaluation questions. This step usually involves coding, where you mark passages of text (or parts of images or sections of a video, etc.) that have the same message or are connected in some way, and you write an accompanying explanation of what the selected passages have in common.
- Conclusion drawing and verification: During this last step, revisit the data many times to verify, test, or confirm the themes and patterns you have identified.
Or at least lighter work, and certainly better work! Not only will qualitative analysis be more fun with more than one person, it will probably be richer, more thorough, and more objective. Though it may take longer, including others will corroborate patterns and yield additional insight. Multiple people coming to similar conclusions strengthen the interpretation of each individual reviewer. If conclusions differ, the data needs further examination. Consider involving individuals such as program participants and staff who may see the data from a different perspective. Involving others in analysis is particularly important for participatory evaluations.
While the three steps provide a good overall guideline, you may need to cycle through the steps repeatedly. Qualitative analysis is a cyclical and iterative process, with many rounds of investigating evidence, modifying hypotheses, and revisiting the data from a new light. You will need to reexamine data repeatedly as new questions and connections emerge and as you gain a more thorough understanding of the information you collected. Throughout the process of examining and reexamining data, concentrate on the following:
- Patterns, recurring themes, similarities, and differences
- Ways in which these patterns (or lack thereof) help answer evaluation questions
- Any deviations from these patterns and possible explanations for these
- Interesting or particularly insightful stories
- Specific language people use to describe phenomena
- To what extent patterns are supported by past studies or other evaluations (and if not, what might explain the differences)
- To what extent patterns suggest that additional data may need to be collected
The following resources will help guide you through the process of qualitative data analysis:
- Analyzing Qualitative Data (.pdf)
Taylor-Powell, E. and Renner, M. (2003). University of Wisconsin Extension Program, Development and Evaluation Unit
This guide walks you through the steps of content analysis. The steps include focusing the analysis, categorizing data, identifying patterns and connections, and interpretation. The guide offers many examples, useful tips, and pitfalls to avoid.
- User-Friendly Handbook for Mixed Method Evaluations
Frechtling, J. and L. Sharp, (1997). National Science Foundation.
Chapter 4, “Analyzing Qualitative Data,” provides a detailed description of qualitative analysis based on the Miles and Huberman (1994) three-step process (1. data reduction, 2. data display, and 3. conclusion drawing and verification). The chapter also provides advice on how to judge the quality of qualitative analysis, and includes a list of tips.
- Online QDA website (Quantitative Data Analysis)
Beginner Intermediate Advanced
This website provides a wealth of information to support qualitative data analysis, including introductory background and extensive information on different software packages.
What software can I use to analyze qualitative data?
Whereas computer programs are invaluable in almost any quantitative analysis, think carefully about whether you need software for qualitative analysis. While such software can be of great assistance in some evaluations, make sure you weigh the likely benefits against the cost of the software and the time it takes to become proficient in using it.
Using a computer program can enrich and improve the process of qualitative analysis by speeding up some analysis tasks (e.g. marking, coding, and moving data segments) and facilitating complex questioning of the data (Barry, 1998). Software cannot, however, help you pick out the meaningful codes, themes, or categories, or eliminate the challenging intellectual work of qualitative analysis (Frechtling and Sharp, 1997).
There are many software programs available for purchase and there are some that can be downloaded for free. Not surprisingly, free programs tend not to be as powerful or perform as many functions. They also are not updated as frequently. Nevertheless, such programs may be worth looking into if their capabilities meet your needs.
Two popular free programs are:
AnSWR is described as “a software system for coordinating and conducting large-scale, team-based analysis projects that integrate qualitative and quantitative techniques.” The website provides more details, an overview of what the program can do, manuals, installation instructions, and more.
- CDC EZ-Text
According to the website, CDC EZ-Text “is a software program developed to help researchers create, manage, and analyze semi-structured qualitative databases.” The website provides further description of the program, a frequently asked questions section (including information on how this program differs from AnSWR), and installation instructions.
The following resources will help you sift through many of the qualitative analysis software programs available to find one that will best meet your needs:
A sophisticated software program may be more than you actually need or may simply not be cost-effective. Pen and paper, scissors and photocopies, index cards, or even working within the limits of a word processing program and a spreadsheet - these are all techniques that can be used for qualitative data analysis. They may not fancy, but they can help to get the job done.
- CAQDAS – A Primer
Thomas Konig. Loughborough University
This article focuses on what it calls Computer-Assisted Qualitative Data Analysis Software (CAQDAS), describing in general what it can and cannot do, when to use it and what methodologies it works well with. A link will take you to a review of several popular programs.
- CAQDAS Networking Project
Beginner Intermediate Advanced
This site contains a variety of resources on computer assisted data analysis for qualitative research. The link to “Qualitative Resources” may be especially useful if you are looking into software. It includes access to a list of online articles and a qualitative software discussion list amongst other options.
- Choosing Qualitative Data Analysis Software: Atlas/ti and Nudist Compared
Christine A. Barry. (1998) Sociological Research Online
This article discusses the advantages and disadvantages of computer assisted qualitative data analysis software, reviews the primary differences between various software programs, and compares the two most popular programs, Nudist and Atlas/ti, in detail.
- A Software Sourcebook: Computer Programs for Qualitative Data Analysis.
Weitzman, E.A., and Miles, M.B. (1995). Thousand Oaks, CA: Sage.
This book provides a comprehensive review of your options for qualitative analysis software. The authors present strengths and weaknesses of the programs, which are grouped into six types depending on their primary function. The book helps you determine what software is most appropriate for your analysis based on the amount, types, and sources of data you have to analyze and the types of analyses you want to conduct.
Barry, C. (1998). "Choosing Qualitative Data Analysis Software: Atlas/ti and Nudist Compared." Retrieved September 2006 from: http://www.socresonline.org.uk/3/3/4.html.
Fitzpatrick, J.L., J.R. Sanders, and B.R. Worthen. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines. Boston, MA: Pearson Education, Inc.
Frechtling, J. and L. Sharp. (1997). User-Friendly Handbook for Mixed Method Evaluations. Downloaded September 17, 2006 from http://www.nsf.gov/pubs/1997/nsf97153/chap_4.htm.
Seidel, J. (1998). Qualitative Data Analysis. The Ethnograph v5 Manual, Appendix E. Downloaded September 18, 2006 from: http://www.qualisresearch.com/.
Miles, M.B, and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed., p. 10-12. Newbury Park, CA: Sage.