15th February 2024

Hypothesis testing

The previous section discussed descriptive statistics that describe a sample using mean, median, range and standard deviation. However, descriptive statistics cannot infer further information regarding the population from which the sample is derived. This is where inferential statistics come in.

Inferential statistics are statistics used to make generalised predictions for a population using information obtained from a population sample. This involves making assumptions and hence deals with errors and significance.

Null and alternate hypotheses

Statistical tests are designed to detect any differences between two comparators. Hence, it is convention for us to hypothesise that there will be no differences between the two comparators, null hypothesis. This hypothesis is then tested using statistical tests.

The null hypothesis states no difference between two comparators – e.g., the treatment and control arm.

An alternate hypothesis states that there is a difference between the two comparators. This hypothesis can go both ways, e.g., the treatment arm has a better outcome than the control arm, or the treatment arm has a worse outcome than the control arm. This hypothesis will be accepted if the null hypothesis is rejected.

p-value

Statistical tests are used to test the null hypothesis. This is done by calculating the p-value to determine statistical significance.

p-value is the probability of getting the observed values you obtained in your dataset or something more extreme, provided that the null hypothesis is true. To put things in another way, let us look at an example:

You are examining the difference in patient outcomes between drug A and standard care.

Null hypothesis: There is no difference in patient outcomes between patients given drug A and standard care.

Your observed values showed the following:

Patients given drug A have a survival of 90%

Patients on standard care have a survival of 10%

This in itself would be reasonable to think that your findings are significant. However, how can you be sure? The result could be obtained by pure chance with small sample sizes or sampling bias, etc. This is where statistical tests and p-values come in.

Statistical tests were performed, and the p-value turned out to be 0.03.

This means:

If the null hypothesis is true: In other words, in reality, there is no actual difference between patient survival between both groups, and the probability of you obtaining the results above is 0.03 (3%), which is a very low probability. The differences in survival in your results are likely a true finding (a true positive).

If the p-value is 0.5: This means that the probability of you obtaining the results above if the null hypothesis is true is 50%. This is a very high probability. In other words, this finding is likely untrue (a false positive).

By convention, the p-value of <0.05 is chosen to mean statistical significance. However, there is no golden ‘cut-off’ for p-value; the choice of what cut-off for p-value is statistically significant depends on multiple factors – the area being researched, the sample size, the differences being detected, etc.

A statistically significant finding may not be clinically significant. To demonstrate the point, let us look at another example:

You are doing a study examining the difference in patient outcomes between intervention B and standard care.

Null hypothesis: There is no difference in patient outcomes between patients given drug A and standard care.

Your observed values showed the following:

Patients receiving intervention B have a survival of 10.2%

Patients on standard care have a survival of 10%

Statistical tests performed gave a p-value of 0.0001.

With a p-value of <0.05, the findings are statistically significant.

However, is the survival difference of 0.2% clinically significant? This is another question to consider when reviewing medical literature.

Errors

Setting a lower significance level (a lower p-value cut-off to accept as statistically significant) reduces the risk of type 1 error but increases the risk of type 2 error.

Statistical power

Statistical power is the probability that the difference will be detected from the study if an actual difference exists between intervention and control groups.

Power = 1 – type 2 error

Type 2 errors can occur due to insufficient sample sizes. Hence, having a large sample size will reduce type 2 errors and increase the power.

References

Sainani, K. (2013). Interpreting “Null” Results. PM&R, 5(6), 520–523. https://doi.org/10.1016/j.pmrj.2013.05.003

Sainani, K. L. (2009). Putting P Values in Perspective. PM&R, 1(9), 873–877. https://doi.org/10.1016/j.pmrj.2009.07.003

Sainani, K. L. (2012). Clinical Versus Statistical Significance. PM&R, 4(6), 442–445. https://doi.org/10.1016/j.pmrj.2012.04.014