CRITIQUE

The critique should take at least 75% of the time you spend on this assignment. This is where you show me that you have a thorough grasp of the concepts we have studied and that you can use the research design literature to improve your own work. Remember that experiments have one purpose – to test for direct cause and effect relationships. Think through what you set out to do. Review the research question(s) and the objectives. Review your answers and think about the strengths in your design and the weaknesses. Do not try to redo the assignment by repeatedly changing the flow chart. The objective is for you to analyze your own work critically. Treat your work as though YOU were the reader, not the researcher. Try to answer three questions. What aspects of your design would convince you of the value of the work – specifically your ability to test direct cause and effect in this case? (2) What weaknesses in the design would you spot? (3) How could you address weaknesses?

Step 1: Explain your decisions. I know what you plan to do. That is in the flow chart. Now I want you to explain why you made the choices. What was your logic? For example, you may have decided to use a Solomon four-group design because you want to minimize the effects of testing bias in the results. You might decide to use a switching replications design because you want all of the participants to receive the treatment for ethical reasons or because you want to know if the effect of the treatment is persistent. You need to provide these kinds of explanations in the critique – not in the flow chart. I will address several specific considerations below – based mostly on the kinds of issues that have caused problems for students in the past.

Step 2: Identify the strengths and weaknesses in your design with regard to internal validity (establishing direct cause and effect), external validity (nature of the generalization – theoretical or statistical), and explanatory power (contributions to the body of knowledge, focusing on explanation and/or theory building). Identifying weaknesses will not have a negative effect on your grade. I want you to put the flow chart together quickly – not spend days on it. This is your first attempt at building a research design. Being able to examine your decisions before you start to collect data is critical for all research so that you can identify weaknesses and correct them. Do not spend time “redoing” that chart. Explain your decisions (point 1) and what you think may need to be changed based on further consideration (point 2).

ALWAYS: Use, cite and reference the research design literature. This is an absolute requirement for success and accounts for 40 points on this assignment. Do this in the critique, in both steps 1 and 2.

1.Consider your design and protocol as a whole, including sampling and planned statistical analyses, with regard to internal validity. Remember that internal validity, for a realist, refers to the confidence that the conclusions that you reach are valid and justified. Internal validity involves both providing evidence that the explanation you propose (my idea that by increasing perceived power I can increase change in behavior) is valid and by your ability to show that other things probably did not cause any change you observe (eliminating alternatives).

What are the greatest strengths of your design and protocol with regard to internal validity? Put another way, what aspects of your design and protocol would reassure a reader that your conclusions are justified? Refer to the specific threats to internal validity that we have discussed in class. Consider each of them carefully as you answer this question. Be specific in your responses. Do not repeat broad generalities about the kinds of strengths your design has in general. Rather focus on your specific design decisions.

For example, in my proposed study I did four things that would give me some confidence in my conclusions. First, I set up a pre- and post-test of behavior for each. Any change that occurred should be the result of the training, and the treatment group should show a greater change in behavior. Second, I tested perceived power pre- and post-test for both treatment and control groups. I am proposing that I can change this condition by training. Any difference between pre- and post-training scores should be due to the training. Further, I should see a greater difference in the treatment than in the control group. So including this variable increased the internal validity of my study. Third, I also collected data about the intervening variable (behavioral intent). This helped, too. Finally, I plan to run regression analysis where I can confirm (or not) the proposed linkages between all these variables and, more important, I can see if perceived power is a more important (higher beta value) predictor of behavior in the treatment than in the control group. If so, this is one more piece of evidence that my idea is valid and justified. To eliminate other variables, I included a pre- and post-test of knowledge (this is something I really do usually do). Knowledge is not a construct in the theory of planned behavior, but we do know that knowledge is a pre-requisite for behavior. Therefore, I need to determine whether prior knowledge affected the outcome. Maybe all my participants were very knowledgeable before the training – resulting in no affect. Worst yet, perhaps there was a pre-training difference in knowledge for the two groups. By including knowledge as a construct and testing pre- and post-training, I should be able to eliminate or at least evaluate the importance of knowledge in the outcome.

What are the greatest weaknesses of your design with regard to internal validity? Consider each of them carefully as you answer this question. Be specific in your responses. Do not just repeat broad generalities about the kinds of weaknesses your design has in general. Respond in terms of your study. For example, in my study I have a potential problem because my sample is essentially a volunteer one. I cannot force people into my training sessions. They sign up because they are interested. For example, people who want to lose weight sign up in weight loss classes. If my training were “Diet Management for Weight Loss,” I could very possibly attract people who really want to change their eating habits. This could mean that almost any training approach will work, that I will not see a difference due to training method. On the other hand, since I conduct my training in a one-time, short-duration setting, I probably will not have to worry about mortality (dropouts) and that is a strength.

2.Consider your design and protocol as a whole, including sampling and planned statistical analyses, with regard to external validity. Remember that external validity, for a realist, refers to the degree to which we can extend our findings, and especially our conclusions, to people, places, conditions, or circumstances other than those under which the study was conducted. Sometimes we want to generalize statistically and we usually want to generalize theoretically. Identify clearly your planned generalization.

Identify the key strengths of your study with regard to external validity. Consider both statistical and theoretical generalization. Be specific. Do not just include generalities about the design in general. Refer to what you plan to do. For example, unlike some experiments, my study is not at all contrived, so artificiality should not be a problem. By including a pre- and post-training test of knowledge and including this in the regression model, I strengthen theoretical generalization because I can compare the significance of knowledge (a key construct in other theories) and perceived power as predictors of change in behavior. If it turns out (which it does when I do this) that knowledge does change, but that it does not predict change in behavior, I go a long way toward saying that the theory of planned behavior is a good one when you have to deal with these short-term exposures to trainees.

Identify the key weaknesses of your study with regard to external validity. Consider both statistical and theoretical generalization. Be specific. Do not just include generalities about the design in general. Refer to what you plan to do. People volunteer for my study because they want the training – they presumably have high motivation. This may pose some problems for statistical generalization. I probably can generalize the general conclusion that discovery-based learning will, in general, with lots of kinds of people, topics, places, etc., be superior to the traditional lecture-based learning in creating behavioral change. However, given the potential for high motivation on the part of my participants, I probably cannot generalize the degree of change in behavior from my study to other settings. In situations where motivation is not nearly as high, the amount of change might be considerably less. I also need to be careful about extending my conclusions to other kinds of training settings. I have a one-time, short-duration exposure to trainees. We cannot, based on my findings alone, conclude that the focus on discovery learning would result in similar behavioral changes under very different conditions. This is an opportunity for more research – for example, in classroom settings where there is longer-term, repeated exposure.

3.Discuss the contributions to explanatory power of your study – your contributions to the body of knowledge. Remember that explanatory power rests on the degree to which the researcher can explain the phenomenon under study in a complete and robust way. Explanatory power rests, to a large degree, on accumulating multiple kinds of evidence to support (or not) a theoretical explanation. Contributions to explanatory power often come from providing a different kind of evidence. For example, if most of the research about a specific topic has relied heavily on a single type of design, the scientific realist would argue that evidence from the other design groups is needed to enhance explanatory power. Contributions may also come from addressing parts of the theory that have been largely ignored in the research base. This may include, for example, verification of some aspects of the theory, extension of the theory to areas where it has not been tested before, or building new components into the theory, to name a few. What are the contributions of your study to explanatory power? What kinds of evidence will we have that we have not had before? What will we know or understand after you complete your study that we did not know or understand before?