Java applets for power and sample size

Introduction

Via this page, you may download a Java application that allows you to do power and sample-size calculations for a number of standard statistical models.

What happened to the website I was expecting?

The previous edition of this website had the application embedded as a Java applet; however, many browsers no longer support the type of plug-in that is needed. However, if you still have the capability, and you want to run it as an applet, it is still available here.

Software features

The software is intended to be useful in planning statistical studies. It is not intended to be used for analysis of data that have already been collected.

Each menu selection provides a graphical interface for studying the power of one or more tests. The dialog windows include sliders (convertible to number-entry fields) for varying parameters, and a simple provision for graphing one variable against another. Each dialog window also offers a Help menu (on Macs, the Options and Help menus are added at the top of the screen). Please read the Help menus before contacting me with questions.

Two-sample t dialog

The “Balanced ANOVA” option provides another dialog with a list of several popular experimental designs, plus a provision for specifying your own model.

Download the software

The software is a Java application. Click this link to the file piface.jar and ask your browser to save the file. It encapsulates all of the Java code needed to run the app.

Note: Some browser software that thinks it is smarter than you renames this file piface.zip. If this happens, simply rename it piface.jar; do not unzip the file.

You may also want the icon file piface.ico if you put it on your desktop or a toolbar.

What else you need

You will need to have the Java Runtime Environment (JRE) or the Java Development Kit (JDK) installed on your system. You may already have it; but if not, the JRE may be downloaded from Oracle at https://java.com.

Running the software

Once you have JDK or JRE installed, then you can probably run the application just by double-clicking on piface.jar, or on a shortcut to it. Otherwise, you may run it from the command line in a terminal or DOS window, using a command like

java -jar piface.jar

This will bring up a selector list of analyses. A particular dialog can also be run directly from the command line, if you know its name (can be discovered by browsing piface.jar with a zip file utility). For example, the two-sample t-test dialog may be run using

java -cp piface.jar rvl.piface.apps.TwoTGUI

Citing the software

If you use this software in preparing a research paper, grant proposal, or other publication, I would appreciate your acknowledging it by citing it in the references. Here is a suggested bibliography entry in APA or “author (date)” style:

Lenth, R. V. (2006-9). Java Applets for Power and Sample Size [Computer software]. Retrieved month, day, year from http://www.stat.uiowa.edu/~rlenth/Power.

This form of the citation is appropriate whether you run it online (give the date you ran it) or the stand-alone version (give the date you downloaded it).

Frequently asked questions

What formula(s) do you use in these calculations? In most cases, power is an exact calculation based on the distributional situation in question. Typically it is a probability associated with a non-central distribution. In a few cases, an approximation is used, and is labeled as such. Sample sizes are calculated using root-finding methods in conjunction with power calculations. There are usually not nice neat formulas. That’s why we need software.
I need help on using a particular applet I am willing to provide minimal support if you truly don’t understand what inputs are required. However, each applet has a help menu, and I do request that you carefully read that before you e-mail me with questions. Also, this is getting to be pretty old software. Nothing has really changed here in a number of years, except for making the website more consistent with what’s happened to Java.
I need consulting help I am providing this software for free, but that does not obligate me to also answer substantive questions on power/sample size for your research project. If you need statistical advice on your research problem, you should contact a statistical consultant; and if you want expert advice, you should expect to pay for it. Most universities with statistics departments or statistics programs also offer a consulting service. If you think your research is important, then it is also important to get good advice on the statistical design and analysis (do this before you start collecting data).
How to do… Retrospective power … Cohen’s effect sizes
I recommend against these (see Advice section below). I have been asked why the Options menu in every single applet has links for retrospective power and Cohen effect sizes. It seems to some to be placing undue emphasis on methods I don’t like. The technical answer to the question is that these menu items are inherited from a base class, along with some other things (e.g., the graphics capabilities). The other answer is that people ask me about this all the time, in spite of everything I say on this site. If you follow those menu links, you get explanations of why not to do it. I’m especially proud of the dialog for retrospective power.
I’m not much of a stats person, but I tried [details …] – am I doing it right? Please compare this with: “I don’t know much about brain surgery, but my wife is suffering from [details …] and I plan to operate; can you advise me?”

Folks, just because you can plug numbers into a program doesn’t change the fact that if you don’t know what you’re doing, you’re almost guaranteed to get meaningless results – if not dangerously misleading ones. Statistics really is like rocket science; it isn’t easy, even to us who have studied it for a long time. Anybody who thinks it’s easy surely lacks a deep enough knowledge to understand why it isn’t! If your scientific integrity matters, and statistics is a mystery to you, then you need expert help. Find a statistician in your company or at a nearby university, and talk to her face-to-face if possible. It may well cost money. It’s worth it.
What other resources are available? If you have questions about sample size or power, I suggest going to Cross Validated which is frequented by a large number of people. The answer you need may already be available – so do a search first! (That said, per the caution above, sometimes it is the blind leading the blind.)

Advice

Here are two very wrong things that people try to do with this software:

Retrospective power (a.k.a. observed power, post hoc power). You’ve got the data, did the analysis, and did not achieve “significance.” So you compute power retrospectively to see if the test was powerful enough or not. This is an empty question. Of course it wasn’t powerful enough – that’s why the result isn’t significant. Power calculations are useful for design, not analysis. (Note: These comments refer to power computed based on the observed effect size and sample size. Considering a different sample size is obviously prospective in nature. Considering a different effect size might make sense, but probably what you really need to do instead is an equivalence test; see Hoenig and Heisey, 2001.)
Specify T-shirt effect sizes (“small”, “medium”, and “large”).
This is an elaborate way to arrive at the same sample size that has been used in past social science studies of large, medium, and small size (respectively). The method uses a standardized effect size as the goal. Think about it: for a “medium” effect size, you’ll choose the same n regardless of the accuracy or reliability of your instrument, or the narrowness or diversity of your subjects. Clearly, important considerations are being ignored here. “Medium” is definitely not the message!

Here are three very right things you can do:

Use power prospectively for planning future studies.
Software such as is provided on this website is useful for determining an appropriate sample size, or for evaluating a planned study to see if it is likely to yield useful information.
Put science before statistics.
It is easy to get caught up in statistical significance and such; but studies should be designed to meet scientific goals, and you need to keep those in sight at all times (in planning and analysis). The appropriate inputs to power/sample-size calculations are effect sizes that are deemed clinically important, based on careful considerations of the underlying scientific (not statistical) goals of the study. Statistical considerations are used to identify a plan that is effective in meeting scientific goals – not the other way around.
Do pilot studies.
Investigators tend to try to answer all the world’s questions with one study. However, you usually cannot do a definitive study in one step. It is far better to work incrementally. A pilot study helps you establish procedures, understand and protect against things that can go wrong, and obtain variance estimates needed in determining sample size. A pilot study with 20-30 degrees of freedom for error is generally quite adequate for obtaining reasonably reliable sample-size estimates.

Many funding agencies require a power/sample-size section in grant proposals. Following the above guidelines is good for improving your chances of being funded. You will have established that you have thought through the scientific issues, that your procedures are sound, and that you have a defensible sample size based on realistic variance estimates and scientifically tenable effect-size goals.

For your amusement (or despair), check out this video on how not to ask a statistician about sample-size. (Thanks to Susan Geyer, Morsani College of Medicine, Health Informatics Institute University of South Florida – aka JavaMama926. Dr. Geyer created it for use in a workshop she teaches at the American Society of Hematology’s Clinical Research Training Institute.)