1.1 Planning steps of research

STEP 1: Identify the problem or topic

Identify a research problem or area of interest from everyday life experiences, practical issues, past research, or theory.   Pay attention to the feasibility of your research problem or topic and whether it can be researched systematically.  Determine the resources needed to conduct the study, your interest level, its size and complexity, as well as the value of your results or solution for both theory and practice.

To thoroughly describe the research problem or topic, create a statement that includes the educational topic or specific problem and the justification for research.

STEP 2: Review prior research

Explore the research literature to gain an understanding of the current state of knowledge pertaining to your research problem.  A review of prior research will inform you if your research problem has already been explored (and if a revision or replication is needed), how to design your study, what data collection methods to use, and how to make sense of the findings of your study once data analysis is complete. Reviewing prior research can also help with creating research questions, what population to explore, and laying the theoretical groundwork for your study.

If you are conducting qualitative research, this step is sometimes used throughout the research process or after data is collected (e.g., grounded theory research).

STEP 3: Determine the Research Purpose, Research Questions, or Hypotheses

Identifying a clear purpose and creating a purpose statement helps determine how the research should be conducted, what research design to use, and the research question(s) or hypothesis(es) of your study. Four general purposes for conducting educational research are to explore, describe, predict, or explain the relation between two or more educational variables.  

Explore – an attempt to generate ideas about educational phenomenon

Describe – an attempt to describe the characteristics of educational phenomenon

Predict – an attempt to forecast an educational phenomenon

Explain – an attempt to show why and how an educational phenomenon operates

The purpose of your study will help you determine which research design you should follow

1.2 Introduction to computer research

Qualitative research methods are being  used  increasingly  in  evaluation studies,  including  evaluations of computer systems  and  information technology. This chapter provides  an overview  of the  nature and  appropriate uses of qualitative methods and of key considerations in conducting qualitative  research.

The  goal  of  qualitative research is understanding issues  or  particular situations by investigating the  perspectives and  behavior of the  people  in these  situations and the context  within which they act. To accomplish this, qualitative research is conducted in natural settings  and  uses data  in the form of words rather than  numbers. Qualitative data  are gathered primarily from  observations, interviews,  and  documents, and  are  analyzed  by a variety  of systematic  techniques. This approach is useful in understanding causal processes,  and in facilitating  action  based  on the research results.

Qualitative methods are primarily  inductive.  Hypotheses are developed during the study so as to take into account  what is being learned about  the setting  and  the  people  in it. Qualitative methods may be combined with quantitative methods in conducting a study. Validity threats are addressed primarily  during  data  collection  and analysis.

Reasons for Qualitative Research

The reasons  for using qualitative methods in evaluating computer  information systems:

1.  Understanding how a system’s users perceive  and evaluate that  system and what meanings  the system has for them Users’ perspectives generally  are not known  in advance.  It is difficult to ascertain or understand these  through purely  quantitative approaches. By allowing researchers to investigate users’ perspectives in depth,  qualitative methods can contribute to the explanation of users’ behavior with respect to the system, and thus to the system’s successes and failures  and even of what is considered a “success” or “failure”.
2.  Understanding the  influence  of  social  and  organizational context  on systems use Computer information systems  do  not  exist  in a vacuum;  their  implementation, use,  and  success  or  failure  occur  in  a  social  and  organizational  context  that  shapes  what  happens when  that  system  is introduced. Some researchers consider  this so important as to treat “context” as intrinsically part of the object of study rather than as external to the information system.  Because  of “context,” in important respects,  a system  is not  the same system when it is introduced into different settings. As is true for users’ perspectives, the  researcher usually  does  not  know  in advance what all the important contextual influences  are. Qualitative methods are useful  for  discovering   and  understanding these  influences,  and  also  for developing testable hypotheses and theories.
3.  Providing  formative evaluation that  is aimed  at  improving  a program under  development, rather than  assessing an existing one. Although quantitative and  experimental designs  often  are  valuable  in assessing  outcomes, they  are  less helpful  in giving those  responsible for systems design and implementation timely feedback on their actions. Qualitative evaluation can help  both  in system  design  as well as in studies  of system use

Research Questions and Evaluation Goals

Qualitative methods typically are used to understand the perception of an information system  by  its  users,  the  context  within  which  the  system  is implemented or developed, and  the  processes  by which changes  occur  or outcomes are generated. They usually focus on the description, interpretation, and explanation of events, situations, processes,  and outcomes, rather than  the correlation of variables,  and tend  to be used for understanding a particular case or for comparison of a small number of cases, rather than for generalization to a specified population. They are useful for systematically collecting so-called “anecdotal” evidence  and turning  the experiences they describe  into data  that  can be rigorously  collected  and analyzed.

Thus, the  questions posed  in a qualitative study  are  initially  framed  as “what,”  “how,”  and  “why”  queries,  rather than  as  whether a  particular hypothesis  is true  or false. The fundamental question is “What  is going on here?” This question is progressively narrowed, focused,  and  made  more detailed as the  evaluation proceeds. Qualitative studies  may  begin  with specific concerns  or even  suppositions about  what  is going on, but  major strengths of qualitative methods are avoiding  tunnel  vision, seeing the un- expected, disconfirming  one’s  assumptions, and  discovering  new  ways of making  sense  of what  is going on. Qualitative evaluators typically  begin with questions such as:

• What  is happening here?
• Why is it happening?
• How has it come to happen in this particular way?
• What  do the people  involved  think  is happening?
• How are these  people  responding to what is happening?
• Why are these  people  responding that  way?

1.3 Other considerations to make in planning a research project

Designing

The design stage constitutes the step where the methodological procedure is planned and prepared. What is the time schedule and how do the different steps interrelate? When the chosen technique is interviews, designing the research project will be to determine which kind of interviews to use—personal, collective (focus group), expert, etc.—and how many interviews to perform.

Reporting

It goes without saying that reporting covers the part of the research process where the researcher writes a report to present his findings. 

2.0 What do we mean by collecting data?

Essentially, collecting data means putting your design for collecting information into operation. You’ve decided how you’re going to get information – whether by direct observation, interviews, surveys, experiments and testing, or other methods – and now you and/or other observers have to implement your plan. There’s a bit more to collecting data, however. If you are conducting observations, for example, you’ll have to define what you’re observing and arrange to make observations at the right times, so you actually observe what you need to. You’ll have to record the observations in appropriate ways and organize them so they’re optimally useful.

Recording and organizing data may take different forms, depending on the kind of information you’re collecting. The way you collect your data should relate to how you’re planning to analyze and use it. Regardless of what method you decide to use, recording should be done concurrent with data collection if possible, or soon afterwards, so that nothing gets lost and memory doesn’t fade.

Some of the things you might do with the information you collect include:

• Gathering together information from all sources and observations
• Making photocopies of all recording forms, records, audio or video recordings, and any other collected materials, to guard against loss, accidental erasure, or other problems
• Entering narratives, numbers, and other information into a computer program, where they can be arranged and/or worked on in various ways
• Performing any mathematical or similar operations needed to get quantitative information ready for analysis.  These might, for instance, include entering numerical observations into a chart, table, or spread sheet, or figuring the mean (average), median (midpoint), and/or mode (most frequently occurring) of a set of numbers.
• Transcribing (making an exact, word-for-word text version of) the contents of audio or video recordings
• Coding data (translating data, particularly qualitative data that isn’t expressed in numbers, into a form that allows it to be processed by a specific software program or subjected to statistical analysis)
• Organizing data in ways that make them easier to work with.  How you do this will depend on your research design and your evaluation questions. You might group observations by the dependent variable (indicator of success) they relate to, by individuals or groups of participants, by time, by activity, etc. You might also want to group observations in several different ways, so that you can study interactions among different variables.

2.1 Ethics for gathering data

Consider several ethical issues related to professionalism, and the collection and storing of data from human subjects when conducting educational research.

Professional issues

One important issue for researchers is to present truthful results.  There is never any justification for misrepresentation or fraud, and the cost is enormous, both to the researcher and professional community.  

Human subject research

Many professional organizations have prepared ethical guideless for educational research with human subjects such as the Department of Health, Education, and Welfare

2.2 Data Collection in computer science

The  most  important principle  of qualitative data  collection  is that  every- thing is potential data. The evaluator does not rigidly restrict  the scope of data  collection  in advance,  nor  use formal  rules  to decide  that  some  data are inadmissible or irrelevant. However, this approach creates  two potential problems:  validity and data  overload.

Validity  issues are addressed below. The problem of data  overload is in some ways more intractable. The evaluator must continually make decisions about  what  data  are  relevant and  may  change  these  decisions  over  the course of the project. The evaluator must work to focus the data collection process, but not to focus it so narrowly  as to miss or ignore data that would contribute important insights or evidence.

Qualitative evaluators use three  main sources  for data: (1) observation, (2)  open-ended interviews  and  survey  questions, and   documents and texts.  Qualitative studies  generally  collect  data  by using  several  of these methods to  give a wider  range  of coverage. Data  collection  almost always involves  the researcher’s direct  engagement in the setting  studied, what often  is called “fieldwork.” Thus, the researcher is the instrument for collecting  and  analyzing  data;  the  researcher’s impressions, observations, thoughts, and ideas also are data sources. The researcher incorporates these when  recording qualitative data  in detailed, often  verbatim form  as field notes or interview  transcripts. Such detail is essential  for the types of analy- sis that  are  used  in  qualitative research. We  discuss  each  of  these  data sources  in turn, drawing  again on Kaplan  and Duchon’s  study and several other  studies  for examples.

Observation

Observation in qualitative studies typically  involves  the  observer’s  active involvement in the setting studied; it is usually called “participant observation” to distinguish  it from passive or non-interactive observation. Participant observation allows the  observer to ask questions for clarification  of what is taking place and to engage in informal  discussion with system users, as well as to record  on-going activities and descriptions of the setting. It produces detailed descriptive accounts  of what was going on (including  verbal interaction), as well as eliciting the system users’ own explanations, evaluations, and perspectives in the immediate context  of use, rather than retrospectively. Such observation often  is crucial to the assessment of a system

Open-Ended Interviews  and Survey Questions

Open-ended interviewing requires a skilful  and  systematic  approach to questioning participants. This can range from informal  and conversational interviews  to ones with a specific agenda. There  are two distinctive  feature of open-ended interviewing. First, the goal is to elicit the respondent’s views and experiences in his or her own terms, rather than to collect data that are simply  a  choice  among  pre-established  response categories. Second,  the interviewer is not bound  to a rigid interview  format  or set of questions, but should  elaborate on what  is being  asked  if a question is not  understood, follow up on unanticipated and potentially valuable  information with additional  questions, and probe  for further explanation.

Another way to investigate the views of groups of respondents is through focus groups. This involves interviewing several  people  together, and adds an  opportunity for  those  present to  react  and  respond to  each  others’ remarks

Documents and Texts

Documents, texts, pictures  or photographs, and  artifacts  also can be valuable  sources  of qualitative data.  

2.3 Summary of data collection methods in research

Methods

From the list below, select the data gathering method(s) you wish to use and learn the best practices way to implement it.

Content analysis

The systematic examination of oral, written, or visual communication.

Experiment

A variety of research designs that use before and after, and/or group comparisons, to measure cause and effect relations.

Focus group

A group of similar individuals who provide information during a directed and moderated interactive group discussion.

Interview

A directed conversation with an individual using a list of questions designed to gather extended responses.

Observation

The systematic observation of behavior using checklists, scaled ratings, or narrative comments.

Survey

An ordered series of questions administered to individuals in a systematic manner.

2.4 When and by whom should data be collected and analyzed?

• You can hire or find a volunteer outside evaluator, such as from a nearby college or university, to take care of data collection and/or analysis for you.
• You can conduct a less formal evaluation. Your results may not be as sophisticated as if you subjected them to rigorous statistical procedures, but they can still tell you a lot about your program.  Just the numbers – the number of dropouts (and when most dropped out), for instance, or the characteristics of the people you serve – can give you important and usable information.
• You can try to learn enough about statistics and statistical software to conduct a formal evaluation yourself. (Take a course, for example.)
• You can collect the data and then send it off to someone – a university program, a friendly statistician or researcher, or someone you hire – to process it for you.
• You can collect and rely largely on qualitative data.  Whether this is an option depends to a large extent on what your program is about. You wouldn’t want to conduct a formal evaluation of effectiveness of a new medication using only qualitative data, but you might be able to draw some reasonable conclusions about use or compliance patterns from qualitative information.
• If possible, use a randomized or closely matched control group for comparison

3.1 Data Analysis using computer technology

The basic goal of qualitative data analysis is understanding: the search  for coherence and order. The purpose of data analysis is to develop  an under- standing  or interpretation that  answers  the basic question of what is going on here. This is done through an iterative process that  starts by developing an initial understanding of the setting and perspectives of the people  being studied. That  understanding then  is tested  and modified  through cycles of additional data  collection  and analysis until an adequately coherent interpretation is reached Thus,  in qualitative research, data  analysis  is an  ongoing  activity  that should  start  as soon as the project  begins and continue through the entire course  of the  research [5]. The  processes  of data  collection,  data  analysis, interpretation, and  even  research design  are  intertwined and  depend on each other.

We briefly discuss each of the four techniques.

Coding

The purpose of coding,  in qualitative research, is different from  that  in experimental or survey research or content analysis. Instead of applying  a pre-established set  of categories to  the  data  according  to  explicit,  unambiguous  rules, with the primary  goal being to generate frequency counts of the items in each category, it instead  involves selecting particular segments of data and sorting these  into categories that facilitate  insight, comparison, and the development of theory  [46]. While some coding categories may be drawn from the evaluation questions, existing theory, or prior knowledge of the  setting  and system, others  are developed inductively  by the evaluator during  the analysis, and still others  are taken  from the language  and conceptual  structure of the people  studied. The key feature of most qualitative coding is that it is grounded in the data  (i.e., it is developed in interaction with, and is tailored to the understanding of, the particular data  being analyzed).

Analytical Memos

An analytical  memo  is anything  that  a researcher writes in relationship to the research, other than direct field notes or transcription. It can range from a brief marginal comment on a transcript, or a theoretical idea incorporated into  field notes,  to a full-fledged  analytical  essay. All of these  are  ways of getting  ideas  down  on  paper,  and  of using  writing  as a way to  facilitate reflection   and   analytical   insight.   Memos   are   a   way   to   convert   the researcher’s perceptions and thoughts into a visible form that allows reflection and further manipulation. Writing memos is an important analysis technique, as well as being  valuable  for  many  other  purposes in the research, and should begin early in the study, perhaps even before  starting the study.

Displays

Displays,  such  as  matrices,  flowcharts,  and  concept  maps,  are  similar  to memos  in that  they make  ideas, data, and analysis visible and permanent. They also serve two other  key functions: data  reduction, and the presentation  of data  or analysis  in a form  that  allows it to be grasped  as a whole. These  analytical  tools have been  given their  most detailed elaboration by Miles and  Huberman , but  are  employed less self-consciously  by many other  researchers. Such displays  can be primarily  conceptual, as a way of developing theory,  or they  can be primarily  data  oriented. Data-oriented displays,  such  as matrices,  can  be  used  as an  elaboration of coding;  the coding  categories are  presented in a single display  in conjunction with a reduced subset  of the data  in each category.  Other types  of displays, such as concept  maps, flowcharts, causal networks, and organizational diagrams, display connections among  categories.

Contextual  and Narrative Analysis

Contextual and  narrative analysis  has developed mainly  as an alternative to coding. Instead of segmenting the data into discrete  elements and  resorting these  into  categories, these  approaches to  analysis  seek  to understand the  relationships between elements in a particular text, situation,  or  sequence  of  events.  Methods  such  as  discourse   analysis  , narrative analysis  , conversation analysis ; profiles, or ethnographic microanalysis identify  the  relationships among  the  different  elements in that particular interview or situation, and their meanings for  the  persons   involved,  rather than  aggregating data  across  contexts.

Software

Qualitative methods produce large amounts of data that may not be readily amenable to manipulation, analysis, or data reduction by hand. Computer software  is available  that  can facilitate  the  process  of qualitative analysis. Such programs perform some of the mechanical tasks of storing and coding data, retrieving and aggregating previously  coded data, and making connections among  coding categories, but do not “analyze” the data  in the sense  that   statistical   software   does.  All  of  the  conceptual  and  analytical work of making sense of the data still needs to be done by the evaluator. There  are different types of programs, some developed specifically for data analysis, and others (including word processors, textbase managers, and network builders) that  can be used  for some  of the  tasks  of analysis. For relatively  small-scale  projects,  some  qualitative researchers advocate not using  any  software  besides  a good  word  processor. A  very  sophisticated and powerful program may be difficult to use if it has unneeded features, so it is advisable  to carefully  consider  what  the  program needs  to do before committing to its use.

Validity

Validity   in  qualitative  research  addresses  the   necessarily   “subjective” nature of data collection  and analysis. Because  the researcher is the instrument  for collecting  and analyzing data, the study is subjective  in the sense of  being  different for  different  researchers.  Different researchers  may approach the  same  research question by collecting  different data  or  by interpreting the same data  differently.

Qualitative researchers acknowledge their  role  as research instruments by making  it an explicit part  of data  collection,  analysis, and reporting. As in collecting and analyzing any data, what the evaluator brings to the task— his or her biases, interests, perceptions, observations, knowledge, and critical faculties—all  play a role in the study. Qualitative researchers include  in their  studies  specific ways to under- stand  and control  the effects of their  background and role. They recognize that  the relationships they develop  with those  studied  have a major  effect on the data that can be gathered and the interpretations that can be developed.

Rich Data

Rich data are data that are detailed and varied enough  that they provide  a full and revealing  picture  of what is going on, and of the processes  involved. Collecting  rich data  makes  it difficult for the  researcher to see only what  supports his or her  prejudices and  expectations and  thus  provides  a test of one’s developing theories, as well as provides  a basis for generating, developing, and supporting such theories.

Feedback or Member Checking

This is the single most important way of ruling  out  the possibility  of misinterpreting the  meaning   of  what  participants say  and  do  or  what  the researcher observed, and the perspective the participants have on what is going on. Feedback, or member checking, involves systematically gathering feedback about one’s conclusions from participants in the setting studied  and  from  others  familiar  with the  setting. The  researcher checks  that  the interpretation makes sense to those who know the setting especially well. In addition, this is an important way of identifying  the researcher’s biases and affords the possibility for collecting additional important data

Searching for Discrepant Evidence and Negative  Cases

Identifying and analyzing  discrepant data  and negative  cases is a key part of the logic of validity testing in qualitative research. Instances that cannot be accounted for by a particular interpretation or explanation can point up important defects in that account. There  are strong pressures to ignore data that do not fit prior theories or conclusions, and it is important to rigorously examine  both supporting and discrepant data. In particularly difficult cases, the  only  solution   may  be  to  report the  discrepant evidence   and  allow readers to draw their  own conclusions

3.2.1 Reporting observation results

Analyze the data

Analyze the observational data by reviewing the completed observation form and any written comments. You should also have a face-to-face debriefing discussion with all observers if you did not conduct all observations yourself. This will give the observers an opportunity to explain the data and provide additional feedback. Begin reviewing the data after the first observation. Look for patterns in the data using the research questions or hypotheses of your study to focus analysis. Other questions or issues may emerge as you discuss and review the information.

Determine findings

Summarize findings based on your analysis and in relation to your research questions and previous research findings. Verify that findings are grounded in what was observed.

Report results

How you report your findings depends on the types of observations you used.  If you used qualitative observations (e.g., narrative comments), present repeating ideas that lead to major themes that, in turn, inform conclusions and implications.  Provide one or two examples of a repeating idea. You may also want to note an exception to a trend in order to highlight a noteworthy idea.

For quantitative observations (e.g., rating scales), present the statistical findings graphically and with the level of detail useful to the audience. For all observation types, make sure to discuss what practical or theoretical implications can be drawn for the findings, any major shortcomings or limitations of the methodology used, and directions or suggestions for future research.

3.2.2 Reporting interview results

Transcribe and analyze the data

Interviews generate large quantities of data and tape transcription typically takes four to six hours for each hour of speech, although using a transcription machine or having good typing skills can reduce the time. It is important, therefore, to have a clear plan to guide this phase of the study.  In addition, condensing, organizing, and making meaning of interviews is often the most time-consuming and expensive part of analysis. 

 Determine findings

View analyzed data from a distance until you see a larger picture and understand how this picture relates to your research question(s). Similar research may help you make sense of repeating ideas and larger themes. For example, you might identify underlying factors that explain the themes you have observed and then construct a logical chain of evidence. You might also describe an adaptive or maladaptive process that captures the behavior of respondents. If there are respondents who do not follow the usual pattern, it may be important to understand why. Qualitative researchers need to be flexible and open to the unexpected.  Drawing on repeating ideas and themes, summarize the findings in relation to your research question(s) and to previous research. 

When interpreting qualitative data, verify your findings. Review your data repeatedly to check that your findings are grounded in what was said. Look at independent evidence from other sources and use other methods, such as surveys, focus groups, or experiments, to triangulate your findings. To improve the study’s reliability and validity , show your results to some of the interviewees and ask them if you have accurately recorded what they meant.

Report results

To report qualitative results, present repeating ideas that lead to major themes that, in turn, inform conclusions and implications. Conclusions are statements that interpret and evaluate the results found from the study.  Make sure to give primary emphasis to the results that relate to the research questions of your study. 

Quote one or two responses that exemplify a repeating idea. Quotations, which capture the words, emotions, experiences, and perceptions of interviewees, are not easily dismissed by readers. You may also want to quote a response that was an exception to a trend in order to illustrate a minority opinion or highlight a noteworthy idea. If so, report that it is one person’s response. Finally, make sure to discuss what practical or theoretical implications can be drawn for your findings, any major shortcomings or limitations of the methodology used, and directions or suggestions for future research

3.2.3 Reporting focus group results

Transcribe and analyze the data

Data analysis may be relatively simple, involving a summary of major themes, or may call for more complex content analyses and comparisons of groups (Goldenkoff, 2004). A brief summary and analysis, highlighting major themes, is sufficient when the results are readily apparent or the purpose of the focus group is supplemental. On the other hand, to get an in-depth understanding of a complex issue you should conduct a systematic analysis using full transcripts [more] and a formalized coding scheme.

Determine findings

View analyzed data from a distance until you see a larger picture and understand how this picture relates to your research question(s). Similar research may help you make sense of repeating ideas and larger themes. For example, you might identify underlying factors that explain the themes you have observed and then construct a logical chain of evidence. You might also describe an adaptive or maladaptive process that captures the behavior of respondents. If there are respondents who do not follow the usual pattern, it may be important to understand why. Qualitative researchers need to be flexible and open to the unexpected.  Drawing on repeating ideas and themes, summarize the findings in relation to your research question(s) and to previous research.

Report results

To report qualitative results, present repeating ideas that lead to major themes that, in turn, inform conclusions and implications. Conclusions are statements that interpret and evaluate the results found from the study.  Make sure to give primary emphasis to the results that relate to the research questions of your study.  Quote one or two responses that exemplify a repeating idea. Quotations, which capture the words, emotions, experiences, and perceptions of interviewees, are not easily dismissed by readers. You may also want to quote a response that was an exception to a trend in order to illustrate a minority opinion or highlight a noteworthy idea. If so, report that it is one person’s response. Finally, make sure to discuss what practical or theoretical implications can be drawn for your findings, any major shortcomings or limitations of the methodology used, and directions or suggestions for future research.  

3.2.4 Reporting experiment results

Analyze the data

Calculate descriptive statistics on outcome measures and determine if the variables are normally distributed, a requirement for many statistical tests. If a variable is not normally distributed, consult with a statistician to determine if you need to transform the variable. While comparing group means and standard deviations will give you a rough sense of group differences on outcome measures, you must use statistical tests to demonstrate that these differences are unlikely to have occurred by chance.

Determine findings

From your data analysis, summarize the findings in relation to your research question(s) or hypotheses and to previous research findings.

Report results

Conclusions are statements that interpret and evaluate the results found from the study.  Make sure to give primary emphasis to the results that relate to the research questions of your study such as the effect of an intervention.  One way to represent the magnitude of an intervention effect is with a percentage change. For example, you might report that scores increased 35% for an intervention group compared with 15% for a control group that did not receive the intervention.  Also discuss what practical or theoretical implications can be drawn for your findings, any major shortcomings or limitations of the methodology used, and directions or suggestions for future research.

3.2.5 Quantitative content analysis findings

Determining the findings of your content analysis involves more than simply reporting initial results. Instead, it is important to critically examine results and check for statistical pitfalls to develop accurate findings upon which you can make reliable conclusions.

Critically examine results

• No matter what your results, ask some critical questions:
• Were the criteria you selected valid indicators of content quality? Did you omit important criteria or include unnecessary ones?
• If you implemented an intervention and are comparing content between/among groups or periods of time,
• were there significant differences between/among groups on the content before the intervention started?
• were conditions for groups roughly the same (for example, equivalent classrooms, instruction, and assistance outside of class)?
• did anything happen other than your instructional intervention that would have affected study results?
• was there any difference in motivation between/among groups before or during the study?

Check for statistical pitfalls

While any conclusive findings should be statistically significant, having statistically significant results does not mean, they are important or valuable; it just indicates that the difference you found is unlikely to be due to chance.

If you used multiple raters/coders, is the level of interrater reliability acceptable (e.g., .70 or higher)? Do results indicate any type of bias on the part of one or more of the raters/coders? If you find poor reliability or suspected bias, your results are possibly unreliable and data should be regathered and/or reanalyzed.

If you are comparing content between/among groups, could there be any errors due to sample size? If you have fewer than 25 cases per group, you may lack adequate statistical power to detect differences between groups. On the other hand, if you have very large groups, almost any difference, even a trivial one, will be statistically significant, and could lead you to make unwarranted conclusions.  For this reason, you should indicate effect sizes, which allow the readers to judge how meaningful the differences are between/among groups.  

Other statistical pitfalls

Consult with a statistician if you are unable to resolve statistical problems on your own.

Make conclusions

• Evaluate your results based on how well they answer your research questions or confirm your hypotheses.
• Statistically significant causal, predictive, or correlational findings, as well as important qualitative findings, should form the basis of your main conclusions. Emphasize your strongest findings.
• If you are evaluating an intervention using content analysis, consider all possible explanations for results before concluding an intervention definitely worked or did not work.
• Verify (triangulate ) findings from your content analysis with results from other data sources such as interviews or surveys that can provide additional insight. Finding similar results using different methods strengthens conclusions. On the other hand, differing results call for further analysis.

Definitions

Member  checking: Getting feedback from  participants in  the  study  to check the researchers’ interpretation.

Narrative analysis:.

Open-ended  interviewing: A form of interviewing that  does not employ  a fixed interview  schedule,  but allows the researcher to follow the respondent’s  lead  by exploring  topics  in greater depth  and  also  by pursuing unanticipated topics.

Open-ended   questions:  Interview  or  survey   questions  that   are   to  be answered in the  respondent’s own words, rather than  by selecting  pre- formulated responses.

Participant observation:  A  form  of observation in which  the  researcher participates in the  activities  going on in a natural setting  and  interacts with people  in that  setting,  rather than  simply recording their  behavior as an outside  observer.

Qualitative research: A strategy for empirical research that is conducted in natural settings,  that  uses data  in the  form  of words  (generally, though pictures,  artifacts,  and  other  non-quantitative data  may be used)  rather than  numbers, that  inductively  develops  categories and hypotheses, and that seeks to understand the perspectives of the participants in the setting studied, the context  of that setting, and the events and processes  that are taking  place there.

Rich data: Data  that  are detailed, comprehensive, and holistic.

Robustness:  Interpretations, results, or data that can withstand a variety of validity threats because  they hold up even if some of the underpinnings are removed or prove  incorrect.

Summative evaluation:  Evaluation that is aimed at assessing the value of a developed program for the purpose of administrative or policy decisions. This evaluation often  is done  by testing  the impact  of the program after it has been  implemented. See formative  evaluation.

Triangulation: The cross-checking  of inferences by using multiple methods, sources, or forms of data  for drawing conclusions.

Validity: The truth  or correctness of one’s descriptions, interpretations, or conclusions.

Validity threat: A way in which one’s description, interpretation, or con- clusion might be invalid, also known as “rival hypothesis” or “alternative explanation.”