Steps for Applying the Benjamini-Hochberg "False-Discovery-Rate" (FDR) Procedure for Multiple-Hypothesis-Test Correction

1. Identify a group of hypotheses among which you would shift emphasis according to how good the results look. Note: anything that you would want to talk about if it had strong statistical results counts!

2. How do you identify groupings of hypotheses? Groups of hypotheses can be handled separately if you will keep the same level of emphasis between groups. For example, you will focus in on Group A and focus in on Group B and discuss them equally regardless of what the results are.

3. Now, focusing in on one particular group, call the number of hypotheses in this set n. Multiply all reported p-values by n. Let’s call these “adjusted p-values.”

4. Order all the hypotheses from the smallest adjusted p-value at the top to the largest adjusted p-value at the bottom. (Note that having the numbers in Excel makes this easy to do and then later restore your original order.) Call the hypothesis with the smallest adjusted p-value #1, the one with the second-smallest adjusted p-value #2, etc. And call the m with the smallest adjusted p-values the “top m hypotheses.”

5. The conventional significant level for a False Discovery Rate is .1 (=10%). Let’s go with that, although it is easy to use other values.

6. If the #m hypothesis has an adjusted p-value less than .1 m, then the top m hypotheses are all significant at a false discovery rate (FDR) threshold of 10%.

7. Find the largest m for which the #m hypothesis had an adjusted p-value less than .1 m. This gives you the set of hypotheses within this group that are significant at the FDR 10% level.