13th February 2024

Study Design

Study design

Primary versus secondary research

Medical research can be divided into two main forms: primary and secondary research. Primary research involves collecting new data, which can be further divided into experimental studies (including lab-based), clinical, and epidemiological studies. Secondary research involves drawing conclusions from pre-existing primary data and summarising them, most commonly in systematic reviews or meta-analyses.

Hierarchy of study design

In general, there is a recognised hierarchy of studies for obtaining evidence, with a more rigorous methodology and reduced risk of bias as one moves up the pyramid:

That being said, the “hierarchy” of evidence also depends on the type of question being answered, e.g. treatment benefits, prognosis, and cost-effectiveness. Here is a summary:

Randomised controlled trials (RCTs)

RCTs, particularly when double-blinded and with placebo controls where possible, are typically regarded as the gold standard of clinical research. Participants are randomly assigned to two or more groups to maximise homogeneity between groups to test a specific intervention, ideally with one group as a control or placebo group. Blinding participants and researchers to the intervention can also reduce the risk of bias. Participants are then followed up to determine the effectiveness of the experimental intervention over the control, and outcomes are measured accordingly.

Cohort studies

Also known as longitudinal studies, cohort studies follow groups of people over specified periods to ascertain how a chosen variable, e.g. occupation, dietary habits, or smoking status, may lead to different outcomes. They can also be divided into prospective and retrospective studies. Well-designed prospective studies are less likely to lead to bias than retrospective ones, but they can take years to perform and incur more significant costs. Famous examples of prospective studies include the British Doctors’ Study, which ran for 50 years from 1951 to 2001, and the ongoing Framingham Heart Study, now in its third generation of participants. Common statistical outcomes of cohort studies include cumulative incidence and incidence rates and ratios.

Cross-sectional studies

In cross-sectional studies, participants are selected based on a pre-defined set of inclusion and exclusion criteria. Then, the investigator measures the exposures and the outcomes of the selected participants at the same time. Therefore, this study design is ideal for assessing the prevalence of an outcome, exposure or disease in a specified population. Associations between an exposure/risk factor and an outcome may be explored, but it is difficult to establish causal relationships from a cross-sectional study. It is also important to remember that the prevalence of a disease (total number of cases, including both old and new) is dependent on its incidence (number of new cases) and length of survival. Hence, a cross-sectional study is not always sufficient to understand the trend of a disease. Important statistical outcomes used for analysis include prevalence and odds ratios.

Case-control studies

Case-control studies are retrospective observational studies typically used to determine the causality or exposure(s) related to a disease or outcome by comparing a group of people with the disease to another similarly matched group without the disease (controls). By knowing the outcome and assigning subjects to their respective groups, one can investigate exposures by tracing them back in time. Hence, they are particularly advantageous when investigating disease outbreaks or rare diseases. Odds ratios are typically used for comparing statistical outcomes in these studies.

Systematic reviews and meta-analyses

When primary data is pooled together, secondary research can be performed in the form of systematic reviews. By pooling data from multiple smaller trials, a meta-analysis could statistically combine results from ten trials of 500 patients each and be used to draw appropriate conclusions as if it were analysing a trial with 5,000 patients. A well-conducted systematic review, such as those published by the Cochrane Library, addresses a focussed, well-formulated clinical question, draws from multiple primary sources, synthesises material in the form of meta-analyses and conducts a rigorous critical appraisal to summarise the results of its included studies. In this way, it could provide a high level of evidence on the effectiveness of an intervention, which can be used to inform recommendations and guidelines.

References

Röhrig B, du Prel JB, Wachtlin D, Blettner M (2009). Types of study in medical research: Part 3 of a series on evaluation of scientific publications. Dtsch Arztebl Int, 106(15), 262-268.

Kapoor MC (2016). Types of studies and research design. Indian J Anaesth, 60(9), 626–630.

Setia MS (2016). Methodology Series Module 1: Cohort Studies. Indian J Dermatol, 61(1), 21-25.

Setia MS (2016). Methodology Series Module 3: Cross-sectional Studies. Indian J Dermatol, 61(3), 261-264.

Lewallen S, Courtright P (1998). Epidemiology in practice: case-control studies. Community Eye Health, 11(28), 57–58.

Deeks JJ, Higgins JPT, Altamn DG (ed.). Chapter 10: Analysing data and undertaking meta-analyses. In: Higgs JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (ed.) (2021). Cochrane Handbook for Systematic Reviews of Interventions. Accessed on 2022 Jan 11. Available from: https://training.cochrane.org/handbook/current/chapter-10.

Møller AM, Myles PS (2016). What makes a good systematic review and meta-analysis? British Journal of Anaesthesia, 117(4), 428-430.

Interpretation of different statistical outcomes

Different studies will present different statistical outcomes. We will now define and discuss common statistical terminology that appears in Part 1 questions, with accompanying sample questions.

Prevalence and incidence

These two concepts are often quoted in literature but can be easily confused. Simply put, the prevalence represents the existing cases within a defined population at a particular time point, which can reflect the population’s disease burden. Incidence, on the other hand, represents the number of new cases within a defined period. 

Relative risks, odds ratios and number needed to treat

Being asked to calculate any of the above from a given scenario is not uncommon in Part 1. We can measure the strength of an observed association between a risk factor and an outcome in relative terms (using risk ratios, incidence rate ratios or odds ratios) or in absolute terms (using risk differences and incidence rate differences).

Remember that the null value for risk/incidence/odds ratios is 1. This is 0 for absolute measurements (i.e., risk and incidence rate differences). The result is statistically insignificant if the confidence interval crosses the null value. 

It is easy to get caught out by a question asking to calculate an odds ratio from a case-control study. This is calculated by dividing the odds of one group by the odds of another group. For example, in this 2×2 table, to calculate the odds ratio of developing mesothelioma in those exposed to asbestos compared to those without asbestos exposure:

Odds of mesothelioma in individuals exposed to asbestos: 10/50 = 0.20

Odds of mesothelioma in individuals unexposed to asbestos: 2/50 = 0.04

Odds ratio = 0.20/0.04 = 5 (i.e., the odds of developing mesothelioma is five times higher in those exposed to asbestos compared to those without)

Number needed to treat (NNT) and number needed to harm (NNH) 

Rather than a relative risk or risk reduction or an odds ratio, an NNT can be a helpful way to summarise the results of a clinical therapeutic trial. An NNT is the number of people who undergo a specific treatment or take a medication in order to prevent an additional adverse outcome. For example, suppose a newly developed antiplatelet has an NNT of 15. In that case, one has to treat 15 people with the antiplatelet to prevent an additional adverse outcome such as a myocardial infarction.

where the ARR =control event rate CER-experimental event rate (EER).

Conversely, the NNH represents the number of people needed to be exposed to a risk factor to result in additional harm.

Evaluation of diagnostic tests

Another study you could be assessed on involves determining the adequacy and appropriateness of a diagnostic or screening test. These can be assessed by examining the sensitivity, specificity, positive and negative predictive values and likelihood ratios.

The simplest way to understand how to determine these ratios is by using the following 2×2 table:

Adapted from: Shreffler J, Huecker MR (2021). Diagnostic Testing Accuracy: Sensitivity, Specificity, Predictive Values and Likelihood Ratios. In: StatPearls. United States: StatPearls Publishing. Available from: https://www.ncbi.nlm.nih.gov/books/NBK557491/ 

Sensitivity and specificity

Sensitivity defines the proportion of true positives in the population, i.e., it is the test’s ability to yield a positive result for someone who has the disease.

On the other hand, specificity defines the proportion of true negatives in the population, i.e., it is the test’s ability to yield a negative result for someone without the disease. 

Both sensitivity and specificity are used together to determine the validity of a diagnostic test. They are inversely related such that as the sensitivity of a test increases, its specificity tends to decrease. A test with higher sensitivity is more beneficial to exclude a disease if a person tests negative. A highly specific test, however, is helpful to “rule in” a diagnosis when the person tests positive.

Positive and negative predictive values

In a clinical setting with the test result in front of you, the positive predictive value (PPV) and negative predictive value (NPV) help the clinician understand how to interpret a result and explain it to the patient in front of them.

Generally, the greater the value of the PPV and NPV, the closer the test becomes a “gold standard”. Notably, the prevalence of a disease also affects the PPV and NPV. The higher the disease prevalence, the greater the PPV; conversely, the NPV would decrease.

Positive and negative likelihood ratios

The literature increasingly quotes likelihood ratios as they can be clinically more relevant than sensitivity and specificity. They are also unaffected by disease prevalence, unlike predictive values, but they can comment on how much the utilisation of a test can alter its probability. Therefore, it can be used to calculate an individual’s disease probability. They are calculated as follows:

An LR+ is the probability of a person with the disease having a positive test result divided by the probability of a person without the disease having a positive test result. If the LR+ is >1, a positive result is more likely to occur in individuals with the disease than in those without. If the LR+ is between 0 and 1, then the converse is true: a positive result is less likely to occur in individuals with the disease than in those without.

An LR- is the probability of a person with the disease having a negative result divided by the probability of a person without the disease having a negative result. Therefore, an LR- >1 means a negative result is more likely to occur in those with the disease than in those without, and an LR- between 0 and 1 indicates a negative result is less likely to occur in those with the disease than those without.

References

Noordzij M, Dekker FW, Zoccali C, Jager KJ (2010). Measures of Disease Frequency: Prevalence and Incidence. Nephron Clin Pract, 115, c17-c20. doi: 10.1159/000286345

Tripepi G, Jager KJ, Dekker FW, Zoccali C (2010). Measures of Effect in Epidemiological Research. Nephron Clin Pract, 115, c91-c93. doi: 10.1159/000312870

Centre for Evidence-Based Medicine (2022). Number Needed to Treat (NNT). Accessed on 2022 Apr 1. Available from: https://www.cebm.ox.ac.uk/resources/ebm-tools/number-needed-to-treat-nnt 

Shreffler J, Huecker MR (2021). Diagnostic Testing Accuracy: Sensitivity, Specificity, Predictive Values and Likelihood Ratios. In: StatPearls. United States: StatPearls Publishing. Available from: https://www.ncbi.nlm.nih.gov/books/NBK557491/ 

Akobeng, A.K. (2007). Understanding diagnostic tests 1: sensitivity, specificity and predictive values. Acta Pædiatrica, 96, 338-341. https://doi.org/10.1111/j.1651-2227.2006.00180.x

Akobeng, A.K. (2007). Understanding diagnostic tests 2: likelihood ratios, pre- and post-test probabilities and their use in clinical practice. Acta Pædiatrica, 96, 487-491. https://doi.org/10.1111/j.1651-2227.2006.00179.x