Article Review on "Gall's "Figuring Out the Importance of Research"

Quoted Instructions for "Gall's "Figuring Out the Importance of Research" Assignment:

The development of null-hypothesis significance testing by Karl Pearson and RA Fisher over a hundred years ago ranks as one of the major advances in statistical methodology, and has paved the way for much of modern science and technology. Useful as this tool is, however, it is not the be-all and end-all of analytical methods, nor even necessarily the most effective way to tease meaning out of data. In recent years, there has developed a significant critique of significance testing and its applications, and a body of new tools intended to supplement if not necessarily replace significance testing in the pursuit of meaning. These new tools revolve largely around the development and interpretation of measures of effect size, as well as the application of rubrics and other more objective standards intended to clarify the utility of detected differences and associations in the data. It would be too early to say that there is a consensus one way or another; rather, we are in some stage of a dialectical process in which proponents and opponents of significance testing throw arguments back and forth at each other, and make the most of each opportunity to castigate one another*****'s views (and occasionally personal character). In this module, we will examine some aspect of this debate and consider how different sorts of metrics related to *****"significance*****" reveal different aspects of the problem.

A key concept in this debate is what is called *****"statistical power*****", or the *****"power of a test*****". Underlying all significance testing, and indeed almost all of statistics, is the mathematics of probability. Probability is in turn based on a series of axioms or assumptions about the nature of underlying reality. The fact is that those assumptions describe enough of our practical world that in most cases the tools derived from the mathematics of probability are extremely useful in increasing the predictability, and hence the manageability, of the world. Most of us have a pretty good sense of intuitive probability -- good enough, that is, to keep us alive. Most of the time, However, Daniel Kahnemann and Amos Tversky won the Nobel Prize for economics (well, actually, it was just Kahnemann, Tversky having cleverly managed to die a few years before the Nobel Committee got around to noticing them) by demonstrating the limits of our sense of intuitive probability, and the potentially catastrophic consequences ensuing on failed inference. One reason that statistics has achieved such a predominant position in modern science is precisely that, as we have said earlier, it*****'s a dandy tool for protecting us against the tendency to fail at inference. If significance testing is one side of the coin -- the part that keeps us from making incorrect inferences about relationships or tendencies in populations of any sort on the basis of small samples -- then statistical power is the other side -- the part that allows us to be sure that relationships and tendencies we notice in our samples are really characteristic of the wider population.

Statistical power can be increased in several ways. One is to study phenomena with more consistent effects. The larger the real-world effect size (that is, the greater the chance that a given cause will result in a given effect), the greater the chance of observing its occurrence; power thus is higher in studies where the relationships are more regular, and highest of all in the physical sciences where cause-and-effect relationships above the quantum level tend to be deterministic. Another way to boost power is to increase the size of the sample; in large enough samples, even relatively weak effects can usually be teased out. A third is to use more inherently powerful statistical tests; the various flavors of the General Linear Model (regression, ANOVA, etc.) that operate on interval data conforming to certain underlying assumptions are more powerful than parametric ordinal tests, which are in turn more powerful than the nonparametric tests that are commonly used with categorical data. Each of these alternatives is of course easier said than done. We can*****'t only study things that are obvious; costs and availability limit our access to large samples; and many interesting sets of data barely approximate the requirements for applying the GLM. Statistical power is, thus, in many cases far less flexible than its calculatable components would suggest and far closer to an absolute limit on the effectiveness of research in the behavioral sciences than many researchers would like to believe.

We*****'re assuming that you have a general acquaintance with the idea of sampling and what is done with it; if you need some review in this area, the background information contains a variety of useful sources. Two good sources for understanding the idea of power and its interpretation are:

Park, H.M. (2005). Hypothesis testing and statistical power of a test. UITS Center for Statistical and Mathematical Computing. Retrieved from http://www.indiana.edu/~statmath/stat/all/power/power.pdf

Trochim, W. (2006). Statistical power. Research Methods Knowledge Base. Retrieved from http://www.socialresearchmethods.net/kb/power.php

When you*****'re reasonably comfortable with the vocabulary and concepts of statistical power and significance testing, then you should take a look at two articles debating the issue of significance testing and its applications; between them, the debate is well framed:

Gall, M.D. (2001). Figuring out the importance of research results: Statistical significance versus practical significance. Paper presented at the 2001 annual meeting of the American Educational Research Association. Retrieved from http://www.uoregon.edu/~mgall/statistical_significance_v.htm

Levin, J.L. (1998). What if there were no more bickering about statistical significance tests? Research in the schools. 5(2),43-53. Retrieved from http://www.personal.psu.edu/users/d/m/dmr/sigtest/6mspdf.pdf

Assignment

Read these two articles in light of the technical material presented above.

Then write a 3 page critique of statistical significance testing, based on the arguments presented in these two papers.

Case assignment expectations

As noted, a critique is a review and commentary on a particular article or piece of research. It is not necessarily critical in the negative sense, although you may need to comment negatively on some aspects; both positive and negative aspects should be treated. Just because something appears in print, even in an A-list journal, does not make it free from possible errors or beyond criticism; nothing should be necessarily taken at face value. Your informed commentary and analysis is as important as your summary of the material in the article -- simply repeating what the article says does not constitute an adequate critique. You are also expected to use the terminology of path analysis and regression correctly and clearly.

In this case, your critique should address at least the following issues, as well as any other points that you find relevant and worthy of comment:

*****¢ The kinds of problems where significance testing can be helpfully applied, and the kinds of problems where it can*****'t

*****¢ The kinds of populations of respondents usually studied in business and organizational research

*****¢ Kinds of sampling strategies employed in data collection and the difficulties of obtaining suitable samples

*****¢ How and why assessment of statistical power ought to be included in research planning

*****¢ Why it is so seldom discussed in published research

*****¢ What researchers can and should do to improve statistical power

*****¢ Implications of thinking about sampling and power analysis for your own planned research interests

Remember, this is an applied statistics course. Thus, explaining the statistical tools, interpreting coefficients, and understanding the properties of the data analysis are particularly important, and need your careful thought and comment, not just general or generic observations.

You are expected to present your critique in appropriate academic form and language, with citations to the readings where needed.

Please include a REFERENCE Page.......she is a stickler for APA Format

*****