15th February 2024

Tests of significance

Tests of significance are statistical tests involving mathematical equations that compute experimental data to give a p-value. A few factors determine the type of statistical test used:

1. Type of data – categorical vs. continuous
2. Whether the data compared are independent or correlated
3. Distribution of data – normal vs. non-normal distributions.

The table below summarises the statistical tests for different types of data and distributions.

Parametric vs. non-parametric tests

Statistical tests can also be subcategorised into parametric and non-parametric tests.

In parametric tests, the sample data distribution is known and based on fixed parameters. For example, the Student’s t-test for two independent samples is only reliable if both samples have a normal distribution with homogenous variance. A parametric test has more statistical power when compared to its non-parametric equivalent.

Non-parametric tests do not assume any distribution of sample data. Non-parametric tests are more robust than parametric equivalent because non-parametric tests are valid in a broader range of situations.

Statistical tests for continuous data

Unpaired student T-test or Independent T-test

• Parametric test
• Compares the means of two independent groups of values

Paired Student t-test

• Parametric test
• Compares the means of two correlated groups, e.g. studies where the same people are measured at two different time points

Analysis of variance (ANOVA) test

• Parametric test
• Compares the means of more than two groups of values
• Better test when compared to performing multiple unpaired Student t-tests. Type 1 error (probability of false positive) increases when multiple unpaired Student t-tests are performed on the same dataset.

Mann Whitney U-test

• Also known as Wilcoxon rank-sum test
• Non-parametric equivalent to unpaired Student t-test

Wilcoxon sign-rank test

• Also referred to as the Wilcoxon matched pairs test
• Non-parametric equivalent to paired Student t-test

Kruskal-Wallis test

• Non-parametric equivalent to ANOVA

Statistical tests for categorical data

Chi-squared test

• Non-parametric test
• Compares proportions of two or more groups

McNemar’s Chi-squared test

• Non-parametric test
• Equivalent of Chi-squared test for paired data

Fisher’s exact test (aka Fisher’s exact probability test)

• Non-parametric test
• The equivalent of the Chi-squared test for sparse data.

Tests of correlation

Correlation is the measure of the linear relationship between two variables. It is measured using correlation coefficients. 

Pearson’s correlation coefficient

Pearson’s correlation coefficient assumes that one or both variables are normally distributed. The correlation coefficient ranges from -1 to +1.

• Values below 0 denote a negative correlation between variables (one variable decreases as another increases).
• A value equal to 0 denotes no correlation between the two variables.
• Values above 0 denote a positive correlation between variables (one variable increases as another increases).

A correlation coefficient may have a high value (strong correlation) but is statistically insignificant. The statistical significance of a correlation coefficient is based on its corresponding p-value.

Spearman’s rank correlation/Kendall’s rank correlation

Spearman’s rank correlation and Kendall’s rank correlation are non-parametric equivalents of Pearson’s correlation coefficient.

Regression analyses

In correlation coefficients discussed in the previous section, both variables are treated as equals.

In regression analyses, one variable is considered the independent variable. The other variable is considered the dependent variable. There are two techniques of regression analyses:

• Multiple linear regression
  • They are used to predict a dependent variable using multiple independent variables.
• Logistic regression
  • They are used to predict the probability of a binary outcome using multiple predictor variables.

References

The t-tests | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/7-t-tests

The Chi-squared tests | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/8-chi-squared-tests

Rank score tests | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/10-rank-score-tests

Correlation and regression | The BMJ. (2020, October 28). The BMJ | The BMJ: Leading General Medical Journal. Research. Education. Comment. https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/11-correlation-and-regression