15th February 2024

Hypothesis Testing and Tests of Significance

Hypothesis testing

The previous section discussed descriptive statistics that describe a sample using mean, median, range and standard deviation. However, descriptive statistics cannot infer further information regarding the population from which the sample is derived. This is where inferential statistics come in.

Inferential statistics are statistics used to make generalised predictions for a population using information obtained from a population sample. This involves making assumptions and hence deals with errors and significance.

Null and alternate hypotheses

Statistical tests are designed to detect any differences between two comparators. Hence, it is convention for us to hypothesise that there will be no differences between the two comparators, null hypothesis. This hypothesis is then tested using statistical tests.

The null hypothesis states no difference between two comparators – e.g., the treatment and control arm.

An alternate hypothesis states that there is a difference between the two comparators. This hypothesis can go both ways, e.g., the treatment arm has a better outcome than the control arm, or the treatment arm has a worse outcome than the control arm. This hypothesis will be accepted if the null hypothesis is rejected.

p-value

Statistical tests are used to test the null hypothesis. This is done by calculating the p-value to determine statistical significance.

p-value is the probability of getting the observed values you obtained in your dataset or something more extreme, provided that the null hypothesis is true. To put things in another way, let us look at an example:

You are examining the difference in patient outcomes between drug A and standard care.

Null hypothesis: There is no difference in patient outcomes between patients given drug A and standard care.

Your observed values showed the following:

Patients given drug A have a survival of 90%

Patients on standard care have a survival of 10%

This in itself would be reasonable to think that your findings are significant. However, how can you be sure? The result could be obtained by pure chance with small sample sizes or sampling bias, etc. This is where statistical tests and p-values come in.

Statistical tests were performed, and the p-value turned out to be 0.03.

This means:

If the null hypothesis is true: In other words, in reality, there is no actual difference between patient survival between both groups, and the probability of you obtaining the results above is 0.03 (3%), which is a very low probability. The differences in survival in your results are likely a true finding (a true positive).

If the p-value is 0.5: This means that the probability of you obtaining the results above if the null hypothesis is true is 50%. This is a very high probability. In other words, this finding is likely untrue (a false positive).

By convention, the p-value of <0.05 is chosen to mean statistical significance. However, there is no golden ‘cut-off’ for p-value; the choice of what cut-off for p-value is statistically significant depends on multiple factors – the area being researched, the sample size, the differences being detected, etc.

A statistically significant finding may not be clinically significant. To demonstrate the point, let us look at another example:

You are doing a study examining the difference in patient outcomes between intervention B and standard care.

Null hypothesis: There is no difference in patient outcomes between patients given drug A and standard care.

Your observed values showed the following:

Patients receiving intervention B have a survival of 10.2%

Patients on standard care have a survival of 10%

Statistical tests performed gave a p-value of 0.0001.

With a p-value of <0.05, the findings are statistically significant.

However, is the survival difference of 0.2% clinically significant? This is another question to consider when reviewing medical literature.

Errors

Setting a lower significance level (a lower p-value cut-off to accept as statistically significant) reduces the risk of type 1 error but increases the risk of type 2 error.

Statistical power

Statistical power is the probability that the difference will be detected from the study if an actual difference exists between intervention and control groups.

Power = 1 – type 2 error

Type 2 errors can occur due to insufficient sample sizes. Hence, having a large sample size will reduce type 2 errors and increase the power.

References

Sainani, K. (2013). Interpreting “Null” Results. PM&R, 5(6), 520–523. https://doi.org/10.1016/j.pmrj.2013.05.003

Sainani, K. L. (2009). Putting P Values in Perspective. PM&R, 1(9), 873–877. https://doi.org/10.1016/j.pmrj.2009.07.003

Sainani, K. L. (2012). Clinical Versus Statistical Significance. PM&R, 4(6), 442–445. https://doi.org/10.1016/j.pmrj.2012.04.014

Tests of significance

Tests of significance are statistical tests involving mathematical equations that compute experimental data to give a p-value. A few factors determine the type of statistical test used:

1. Type of data – categorical vs. continuous
2. Whether the data compared are independent or correlated
3. Distribution of data – normal vs. non-normal distributions.

The table below summarises the statistical tests for different types of data and distributions.

Parametric vs. non-parametric tests

Statistical tests can also be subcategorised into parametric and non-parametric tests.

In parametric tests, the sample data distribution is known and based on fixed parameters. For example, the Student’s t-test for two independent samples is only reliable if both samples have a normal distribution with homogenous variance. A parametric test has more statistical power when compared to its non-parametric equivalent.

Non-parametric tests do not assume any distribution of sample data. Non-parametric tests are more robust than parametric equivalent because non-parametric tests are valid in a broader range of situations.

Statistical tests for continuous data

Unpaired student T-test or Independent T-test

• Parametric test
• Compares the means of two independent groups of values

Paired Student t-test

• Parametric test
• Compares the means of two correlated groups, e.g. studies where the same people are measured at two different time points

Analysis of variance (ANOVA) test

• Parametric test
• Compares the means of more than two groups of values
• Better test when compared to performing multiple unpaired Student t-tests. Type 1 error (probability of false positive) increases when multiple unpaired Student t-tests are performed on the same dataset.

Mann Whitney U-test

• Also known as Wilcoxon rank-sum test
• Non-parametric equivalent to unpaired Student t-test

Wilcoxon sign-rank test

• Also referred to as the Wilcoxon matched pairs test
• Non-parametric equivalent to paired Student t-test

Kruskal-Wallis test

• Non-parametric equivalent to ANOVA

Statistical tests for categorical data

Chi-squared test

• Non-parametric test
• Compares proportions of two or more groups

McNemar’s Chi-squared test

• Non-parametric test
• Equivalent of Chi-squared test for paired data

Fisher’s exact test (aka Fisher’s exact probability test)

• Non-parametric test
• The equivalent of the Chi-squared test for sparse data.

Tests of correlation

Correlation is the measure of the linear relationship between two variables. It is measured using correlation coefficients. 

Pearson’s correlation coefficient

Pearson’s correlation coefficient assumes that one or both variables are normally distributed. The correlation coefficient ranges from -1 to +1.

• Values below 0 denote a negative correlation between variables (one variable decreases as another increases).
• A value equal to 0 denotes no correlation between the two variables.
• Values above 0 denote a positive correlation between variables (one variable increases as another increases).

A correlation coefficient may have a high value (strong correlation) but is statistically insignificant. The statistical significance of a correlation coefficient is based on its corresponding p-value.

Spearman’s rank correlation/Kendall’s rank correlation

Spearman’s rank correlation and Kendall’s rank correlation are non-parametric equivalents of Pearson’s correlation coefficient.

Regression analyses

In correlation coefficients discussed in the previous section, both variables are treated as equals.

In regression analyses, one variable is considered the independent variable. The other variable is considered the dependent variable. There are two techniques of regression analyses:

• Multiple linear regression
  • They are used to predict a dependent variable using multiple independent variables.
• Logistic regression
  • They are used to predict the probability of a binary outcome using multiple predictor variables.

References

The t-tests | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/7-t-tests

The Chi-squared tests | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/8-chi-squared-tests

Rank score tests | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/10-rank-score-tests

Correlation and regression | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/11-correlation-and-regression