What Can the Smallest P-values Tell Us? A Theory of Memoryless Conversion

Abstract

P-value quantifies the strength of statistical evidence against a null/ uninteresting/default hypothesis- the smaller, the more significant. In data- abundance areas such as genomics and functional MRI, researchers often encounter numerous hypothesis testing problems at a time and need to compute an array of p-values for multiple decision making. However, many fundamental questions concerning extreme p-values such as ?€?what is the probability for the smallest p-value to come from a problem of which the null hypothesis holds?" (Q1) or "among the 10 problems with most significant p-values, how many false positives are expected?" (Q2), remain unanswered. In this paper, we show that simple answers can be obtained by converting p-value from p to Y=-log(1-p). This conversion transforms a uniform distribution to a memoryless exponential distribution. In particular, mY(1) gives a conservative estimate for question Q1, where m is the total number of the hypotheses and Y(1) is the one converted from the smallest p-value p(1). Likewise, for question Q2, a conservative estimate is (m-9) Y(10)+ Y(1) + ...+Y(9), where Y(i) is converted from the ith smallest p-value p(i) . Using a martingale theory, we show how our results can complement the growing literature on familywise error rate and false discovery rate. We further demonstrate how our method can help scientists to prioritize their problem lists in designing follow-up studies.