statistical methods for quasi experimental design

• Search Menu
• Sign in through your institution
• Advance articles
• Editor's Choice
• Supplement Archive
• Cover Archive
• IDSA Guidelines
• IDSA Journals
• The Journal of Infectious Diseases
• Open Forum Infectious Diseases
• Photo Quizzes
• State-of-the-Art Reviews
• Voices of ID
• Author Guidelines
• Open Access
• Why Publish
• IDSA Journals Calls for Papers
• Advertising and Corporate Services
• Advertising
• Journals Career Network
• Reprints and ePrints
• Sponsored Supplements
• Branded Books
• About Clinical Infectious Diseases
• About the Infectious Diseases Society of America
• About the HIV Medicine Association
• IDSA COI Policy
• Editorial Board
• Self-Archiving Policy
• For Reviewers
• For Press Offices
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Two-group tests, regression analysis, time-series analysis, adding a control group, acknowledgments.

• < Previous

Statistical Analysis and Application of Quasi Experiments to Antimicrobial Resistance Intervention Studies

• Article contents
• Figures & tables
• Supplementary Data

George M. Eliopoulos, Michelle Shardell, Anthony D. Harris, Samer S. El-Kamary, Jon P. Furuno, Ram R. Miller, Eli N. Perencevich, Statistical Analysis and Application of Quasi Experiments to Antimicrobial Resistance Intervention Studies, Clinical Infectious Diseases , Volume 45, Issue 7, 1 October 2007, Pages 901–907, https://doi.org/10.1086/521255

• Permissions Icon Permissions

Quasi-experimental study designs are frequently used to assess interventions that aim to limit the emergence of antimicrobial-resistant pathogens. However, previous studies using these designs have often used suboptimal statistical methods, which may result in researchers making spurious conclusions. Methods used to analyze quasi-experimental data include 2-group tests, regression analysis, and time-series analysis, and they all have specific assumptions, data requirements, strengths, and limitations. An example of a hospital-based intervention to reduce methicillin-resistant Staphylococcus aureus infection rates and reduce overall length of stay is used to explore these methods.

Choosing the appropriate study design is critical when performing antimicrobial resistance intervention studies. When randomized studies in single hospitals or multihospital cluster-randomized trials are infeasible, investigators often choose before-and-after quasi-experimental designs [ 1 , 2 ]. Quasi-experimental studies can assess interventions applied at the hospital or unit level (e.g., hygiene education program in the medical intensive care unit [MICU] [ 3 ]) or individual level (e.g., methicillin-resistant Staphylococcus aureus [MRSA] decolonization programs [ 4 ]), in which data are collected at equally spaced time intervals (e.g., monthly) before and after the intervention.

Nonrandomization and the resulting data structure of quasi experiments impart several methodological challenges for analysis. First, common statistical methods, including 2-group Student's t tests and linear regression, were developed to analyze independent, individual-level observations, whereas quasi-experimental data are typically correlated unit-level observations; for example, MRSA counts (defined as the number of MRSA infections at multiple time intervals) collected 1 month apart are likely more similar than MRSA counts collected 2 months apart. Second, nonrandom assignment of the intervention often necessitates analytical control for potential confounders.

Unfortunately, application of statistical techniques to quasi experiments is rarely described in introductory biostatistics texts and courses. We aim to provide a resource for bridging the gap between clinician researchers and biostatisticians by introducing clinicians to statistical analysis of quasi experiments while guiding biostatisticians regarding design-related challenges of intervention studies for controlling antimicrobial resistance, thereby improving conduct and reporting of these studies, as recently outlined [ 5 , 6 ]. Strength of evidence from quasi-experimental data depends on the study design [ 1 , 2 , 7 ]. Studies with a concurrent nonequivalent control group provide stronger evidence about effectiveness of an intervention than do studies without a control group. Also, studies with several preintervention observations provide stronger evidence than do studies with few or no preintervention observations. As discussed below, the internal validity of quasi experiments is partially related to study design elements that affect researchers' ability to control for correlation, confounding, and time trends. Thus, before a study is initiated, hypotheses should be clearly stated, and design and analysis plans should be carefully developed.

We discuss several statistical techniques using the following example (motivated by a study by Pittet et al. [ 3 ]). After several months of abnormally high MRSA infection rates in the MICU, a hospital epidemiologist launches an education-based intervention to increase compliance with hand-disinfection procedures. The epidemiologist aims to compare rates of positivity for MRSA in clinical cultures before and after implementing the intervention. A secondary aim is to assess whether the intervention decreases overall length of stay (LOS) in the MICU. For both aims, data from 36 months before the intervention (2003–2005) are compared with data from 12 months after the intervention (2006). For ease of explanation, we first describe statistical methods for this example without a control group. We then discuss adaptations of methods for studies with a nonequivalent control group.

We discuss 2-group tests (e.g., Student's t test and χ 2 test), regression analysis (including segmented models), and time-series analysis in application to quasi-experimental studies of interventions to control antibiotic-resistant bacterial pathogens. We use simulated data for illustration and review data requirements, software, strengths, and limitations for each statistical method (tables 1 and 2 ). Persons seeking additional resources on statistics or quasi experiments are urged to consult a statistics primer [ 8 ] and literature regarding quasi-experimental studies, respectively [ 1 , 2 , 7 ].

Statistical method and software commands by outcome type.

Characteristics of each statistical method.

Two-group (i.e., bivariate) tests make crude comparisons (i.e., unadjusted for confounders) of MRSA infection rates and mean LOS in pre- and postintervention periods. We specifically discuss Student's t tests for continuous outcomes (e.g., LOS) and 2-rate χ 2 tests for count outcomes (e.g., number of MRSA infections).

Continuous outcomes. For continuous outcomes, 2 mean values are compared using Student's t test. In our example, we test the equality of the mean LOS before and after the education-based hand disinfection intervention. When data from several preintervention and postintervention periods are collected, as in interrupted time-series study designs [ 1 , 2 , 7 ], data from multiple periods before and after implementation of the intervention are pooled to produce 2 grand mean values. For example, 2300 patients per year (6900 total) with a mean LOS of 3.0 days during 2003–2005 (preintervention period) and 2800 patients with a mean LOS of 2.5 days in 2006 (postintervention period) can be compared. However, Student's t tests are sensitive to outlying values. If some patients have atypically long LOS, the median value is the preferred measurement of central tendency. Transformation (e.g., natural logarithm) of individual patients' LOS or a nonparametric test to compare median values (e.g., Wilcoxon rank-sum test) can be used.

Count outcomes. Crude comparisons can be made for count outcomes (e.g., number of MRSA infections) by performing a 2-rate χ 2 test. In our example, because the number of hospital admissions varies over time, comparing numbers of pre- and postintervention MRSA infections may produce invalid results. Summarizing data as a proportion, with the number of MRSA infections divided by the number of hospital admissions (e.g., 150 infections/6900 hospital admissions [2.2%], compared with 40 infections/2800 hospital admissions [1.4%]; P = .009), is appropriate if all patients are observed for the same duration of follow-up, when the proportion is interpreted as risk of infection for that particular follow-up period (e.g., 3-day risk of MRSA infection). However, observation of patients in infection-control studies is typically limited to hospital stays that vary in duration. The 2-rate χ 2 test accommodates this difference by comparing rates (number of infections per unit of person-time) between pre- and postintervention periods [ 6 ]. Given 150 and 40 infections before and after the intervention, respectively, if 6700 preintervention person-days per year (20,100 total) and 6600 postintervention person-days are observed, then the rates are 7.5 and 6.1 infections per 1000 person-days before and after the intervention, respectively ( P = .21). Thus, correcting for person-time using rates may produce conclusions different from those using proportions.

The 2-rate χ 2 test assumes that infection counts follow a Poisson distribution [ 9 , 10–11 ]. The Poisson assumption implies that the mean infection count per person-time equals the variance in the infection count for that person-time. If this assumption is violated, then incorrect SE estimates are calculated, resulting in incorrect confidence intervals and P values.

In interrupted time-series study designs, rates are collected at several periods, allowing the variance of infection counts per person per unit of time to be empirically estimated and compared with the mean value. If the “mean equals variance” assumption is not valid, a test using “robust” SEs on the basis of empirically estimated variances is recommended [ 12 , 13 ]. Consider 12 months of data on MRSA infection rates with a mean rate of 2.8 cases per 1000 person-days and a variance of 2.2. Thus, the Poisson assumption appears valid. In contrast, consider MRSA infection rates with a mean rate of 4.4 cases per 1000 person-days and a variance of 6.6. This latter example is typical such that satisfying the Poisson assumption is rare in practical applications. Therefore, researchers should perform both 2-rate χ 2 tests (with and without robust SEs) to evaluate whether confidence intervals and P values vary across assumptions. If conclusions differ between the 2 methods, test results using the more conservative robust SEs should be reported.

Strengths and limitations. Strengths of 2-group tests include simplicity, interpretability of results, and minimal data requirements (2 observation periods) ( table 2 ). These tests can accommodate >2 groups (e.g., before intervention, after intervention, and after intervention plus change in antimicrobial prescribing), using analysis of variance for continuous outcomes and χ 2 tests for count outcomes.

Two-group tests are limited by several assumptions. One assumption, independence between patients admitted to the hospital in the same period, is implausible because infectious organisms are transmissible. Independence of observations between periods is also implausible, because patients admitted to the hospital in different months may be exposed to constant antibiotic prescribing patterns. Also, without multiple levels of stratification, the ability to adjust for potential confounders (e.g., differences in severity of illness) is limited. Last, 2-group tests can detect changes in outcome levels but not changes in trends (e.g., monthly increases or decreases in the MRSA infection rate). If we use the 2-rate χ 2 test with data in figure 1 , the MRSA infection rates for 36 months before and 12 months after an intervention are 6.8 and 6.6 cases per 1000 person-days, respectively ( P = .87). However, figure 1 shows rates increasing by 0.25 cases per 1000 person-days per month until implementation of the intervention, then decreasing by 0.75 cases per 1000 person-days per month. By pooling counts into single pre- and postintervention rates, the 2-rate χ 2 test cannot detect this change in slope or trend, incorrectly finding no evidence of effectiveness of the intervention. To detect changes in slopes, a different statistical method, such as segmented regression, is needed.

Changes in rate of infection with methicillin-resistant Staphylococcus aureus (MRSA) over time before and after an intervention implemented at month 36, showing a change in slope that would not be detected by 2-group tests. Preintervention and postintervention rates are 6.8 and 6.6 infections per 1000 person-days, respectively ( P = .87, by 2-rate χ 2 test). Preintervention and postintervention slopes are 0.25 and -0.75 infections per 1000 person-days per month, respectively.

Regression analysis quantifies the relationship between an outcome (e.g., LOS or MRSA infection) and an intervention, allowing for statistical control of known confounders. Linear regression is used for continuous normally distributed outcomes (e.g., average monthly LOS or log-transformed individual LOS). Other outcome types, including MRSA counts, require analysis using generalized linear models [ 14 ]. In our example, MRSA infections are considered as MRSA counts per time period with an assumed Poisson distribution; thus, the appropriate method is Poisson regression.

Unlike in statistical literature, in clinical literature, “segmented regression” means regression analysis in which changes in mean outcome levels and trends before and after an intervention are estimated [ 15 ]. If changes in slopes are not estimated (e.g., nonsegmented regression model is fit), then estimates of the slopes may be biased, and changes in time trends attributable to the intervention would be undetected. Segmented regression models can be fit to estimate changes in levels and trends. In our example below, we estimate pre- and postintervention changes in LOS and MRSA levels and trends.

Continuous outcomes. Although individual LOS is usually skewed, mean monthly LOS is approximately normally distributed for large sample sizes (i.e., >30 patients per month). If LOS increases over time secondary to a steady increase in MRSA infection rates, regression analysis can model this pattern and estimate the effect of an intervention controlling for potential confounders (e.g., age and reasons for hospitalization). Given intervention status and potential confounders, the outcome variable (in this case, LOS) must satisfy the assumption of having constant variance.

Using the same data, we estimate changes in mean LOS, controlling for trends, using 2 different models ( figure 2 ). Figure 2A shows the results of nonsegmented linear regression, which cannot assess a change in time trend (i.e., slope). Figure 2B shows the results of segmented linear regression, which allows the slopes to differ before and after the intervention. Compared with the model in figure 2 A , the estimated time trend using segmented linear regression in figure 2 B is flatter after the intervention. Forcing equal slopes before and after the intervention when they are unequal can lead to spurious conclusions about an intervention's effectiveness.

Interrupted time-series data regarding length of hospital stay (LOS) simulated from a segmented linear regression model with a change in slope (before vs. after the intervention), fit with a nonsegmented linear regression model that cannot estimate a change in slope (A) and a segmented linear regression model that can estimate a change in slope (B). The intervention was implemented at month 36.

Count outcomes. Poisson regression is preferred over linear regression for estimating the association between the intervention and monthly MRSA infection rates, controlling for time trend, because counts are not normally distributed ( figure 3 ). Differences estimated from this model are summarized as incident rate ratios of MRSA infections.

Figure 3. Interrupted time-series methicillin-resistant Staphylococcus aureus (MRSA) infection data simulated from a segmented Poisson regression model with a change in slope (before vs. after the intervention), fit with a nonsegmented Poisson regression model that cannot estimate a change in slope (A) and a segmented Poisson regression model that can estimate a change in slope (B). The intervention was implemented at month 36.

Using the same data, we estimate changes in MRSA infection rates, controlling for trends, using 2 models ( figure 3 ). Figure 3A shows the results of nonsegmented Poisson regression, which precludes estimation of changes in time trend (i.e., slope), whereas figure 3 B shows the results of segmented Poisson regression, which allows different slopes before and after the intervention.

SE estimates of Poisson regression models are constrained by the “mean equals variance” assumption. This assumption is relaxed by fitting an overdispersed Poisson regression model [ 14 , 16 ]. Allowing overdispersion can affect SE estimates if the Poisson assumption is false without changing estimated regression parameters, producing more valid inferences. Poisson regression and overdispersed Poisson regression result in equal incident rate ratio estimates but different confidence intervals.

Strengths and limitations. Regression allows estimation of associations between the intervention and outcome while controlling for potential confounders, which is particularly important in nonrandomized quasi-experimental studies ( table 2 ). Segmented regression models estimate changes in mean outcome levels (i.e., intercepts) and trends (i.e., slopes), unlike standard regression models. However, some limitations previously discussed with 2-group tests remain. Specifically, independence between individuals and time periods is assumed. Additionally, regression analysis, in contrast to 2-group tests, requires data from multiple pre- and postintervention time intervals to estimate the slope. General guidelines suggest the use of at least 10 observations per model parameter to avoid overfitting [ 17 ]. The models in figures 2B and 3B contained 5 parameters; thus, they should be used only for studies with at least 50 total observations (in our example, months). For intervention studies, data from at least 10 observations before and after the intervention should be used. However, using at least 24 observations (in our example, 12 months before and after the intervention) would capture potential seasonal changes. Data from shorter intervals can be used (e.g., biweekly); however, choice of time interval is a compromise between maximizing the number of observations and maintaining sufficient data within each interval to provide interpretable summary measures [ 15 , 18 ]. In SAS, the command PROC GENMOD can estimate Poisson and linear regression models ( table 1 ) [ 19 ].

Time-series analysis consists of advanced statistical techniques that require understanding of regression and correlation. Whereas “interrupted time-series design” refers to studies consisting of equally spaced pre- and postintervention observations, “time-series analysis” refers to statistical methods for analyzing time-series design data. Two-group tests and regression analysis assume that monthly LOS and MRSA infection rates are independent over time. In contrast, time-series analysis estimates regression models while relaxing the independence assumption by estimating the autocorrelation between observations collected at different times (e.g., MRSA infection counts among MICU patients across different periods). To estimate autocorrelation, a correlation model is specified along with the regression model, resulting in more accurate SE estimates and improved statistical inference.

Continuous outcomes. Time-series analysis accommodates the previously discussed regression models; however, the challenge is how to correctly model correlation. In linear regression, monthly LOS measurements are assumed to be independent. However, autocorrelation may take one of several forms. For example, if correlation between 2 observations gradually decreases as time between them increases (e.g., correlation between months 1 and 2 is 0.5, correlation between months 1 and 3 is 0.25, and correlation between months 1 and 4 is 0.12), autocorrelation is likely autoregressive. However, if autocorrelation between 2 observations is initially strong but abruptly decreases to ∼0 (e.g., correlation between months 1 and 2 is 0.5 and correlation between months 1 and 3 is 0.05), a moving-average model is more appropriate. Occasionally, autocorrelation is strong for observations close in time and then sharply decreases to a nonzero level after some time threshold. In this case, autoregressive or moving-average models would be inadequate, and autoregressive moving-average (ARMA) models should be used. When correlation between observations does not decrease with duration of time, autoregressive, integrated, moving-average (ARIMA) models may be appropriate. In SAS, PROC AUTOREG estimates autoregressive models, and PROC ARIMA estimates autoregressive, moving-average, ARMA, and ARIMA models.

Count outcomes. Although most time-series software assume that outcomes are normally distributed, methods for Poisson counts are available [ 20 , 21 , 22–23 ]. One approach is to transform counts into monthly rates and use time-series methods for normal data (rates are approximately normally distributed if they are based on large numbers). In addition, Autoregressive [ 22 , 23 ], moving-average [ 21 ], and ARMA [ 20 ] models have been extended for generalized linear models (including Poisson models), called generalized ARMA models. The “garma” command in the R software library VGAM estimates generalized ARMA models [ 24 ].

Strengths and limitations. Time-series methods estimate dependence (i.e., correlation) between observations over time, lessening a common threat to valid inferences. They also accommodate segmented models. Thus, time-series methods generalize regression by relaxing the assumption of independent observations. However, the large data requirements often preclude its use. A general guideline is having ∼50 time points (e.g., 3 years of monthly preintervention data and 1 year of monthly postintervention data) to estimate complex correlation structures [ 25 ]. If fewer observations are available, only simple correlation structures can be reliably estimated [ 15 ].

Another limitation of time-series analysis is difficulty in building and interpreting correlation models. Several technical resources are available to guide analysts [ 26 , 27–28 ]. Review articles [ 25 , 29 , 30 ] and biomedical examples are also available [ 18 , 31 , 32 ]. Bootstrapping circumvents the problem of specifying and estimating an autocorrelation model. Bootstrap SEs can be calculated by estimating regression parameters assuming independence (i.e., linear or Poisson regression). Resulting SEs account for autocorrelation by sampling the data multiple (e.g., 1000 times) with replacement and estimating the parameters with each sample [ 33 ]. Thus, the bootstrap with regression is an alternative to time-series analysis when too few time intervals are observed.

Each method can easily accommodate comparison with a nonequivalent control group, a preferred epidemiological quasi-experimental design, because regression to the mean and maturation effects are common threats in these studies [ 1 , 7 ]. In our example, the intervention could be implemented in the MICU, and the nonequivalent control group could be the surgical intensive care unit. A 2-group t test would then compare changes in the mean LOS in the MICU and surgical intensive care unit (mean LOS after the intervention minus mean LOS before the intervention). Regression analysis (e.g., linear and Poisson) controlling for confounding variables can be performed by fitting separate trends for the MICU and surgical intensive care unit and comparing differences in changes in levels (i.e., intercepts) and trends (i.e., slopes) between the 2 units ( figure 4 ). In our example, the MRSA infection rate in the MICU decreases by 0.8 cases per 1000 person-days immediately on implementation of the intervention, suggesting a large impact of the intervention. However, the MRSA infection rate in the surgical intensive care unit decreases by 0.6 cases per 1000 person-days, suggesting that the decrease in the MRSA infection rate is partially attributable to nonintervention factors, which could not have been identified without a control group. Hence, including a control group is recommended to identify the true impact of an intervention.

Segmented Poisson regression analysis of interrupted time-series methicillin-resistant Staphylococcus aureus (MRSA) infection data, comparing infection rates in the medical intensive care unit (MICU; intervention group) and surgical intensive care unit (SICU; control group) before and after the intervention (implemented at month 36). The reduction of 0.6 infections per 1000 person-days in the SICU suggests that the reduction of 0.8 infections per 1000 person-days in the MICU was not solely due to the intervention.

In summary, 2-group tests, regression analysis, and time-series analysis can accommodate interrupted time-series quasi-experimental data. However, statistical validity depends on using appropriate methods for the study question, meeting data requirements, and verifying modeling assumptions. This last step requires premodeling exploratory data analysis and postmodeling diagnostics not addressed here [ 14 , 17 , 26 , 27 ].

Obtaining high-quality results depends on performing a well-designed study, because statistics cannot correct for a poor initial design [ 1 , 7 , 34 ], nor can they compensate for poor reporting of methods [ 5 , 6 ]. Results from analyses can only provide valid inference on the level of intervention. We provide guidelines of minimal data requirements for using each statistical method ( table 2 ). However, larger sample sizes may be needed to obtain a desired precision for estimating measures of association (e.g., mean difference or rate ratio) or power for statistical tests. A simulation study can determine required sample size using model-generated data analyzed with an appropriate method [ 35 ]. Investigators are encouraged to report sample size calculations in addition to statistical analysis methods [ 5 , 6 ]. Analyzing quasi-experimental data is challenging; therefore, we recommend collaboration between investigators, epidemiologists, and statisticians.

Financial support. National Institute of Health (grants R37 AG09901, 1 R01 AI6085901A1, and P30 AG028747-01 to M.S.; P60 AG12583 to R.R.M.; and institutional grant 1K12RR023250-01 to J.P.F.), Centers for Disease Control and Prevention (grant 1 R01 CI000369-01 to A.D.H. and E.N.P.), and Department of Veterans Affairs Health Services Research and Development Service (grants IIR 04-123-2 and Level 2 Advanced Career Development Award to E.N.P.).

Potential conflicts of interest. All authors: no conflicts.

Google Scholar

Google Preview

• drug resistance, microbial
• length of stay
• pathogenic organism
• antimicrobials
• methicillin-resistant staphylococcus aureus infections

Email alerts

More on this topic, related articles in pubmed, citing articles via, looking for your next opportunity.

• Recommend to your Library

Affiliations

• Online ISSN 1537-6591
• Print ISSN 1058-4838
• Copyright © 2024 Infectious Diseases Society of America
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Quasi-Experimental Design | Definition, Types & Examples

Quasi-Experimental Design | Definition, Types & Examples

Published on July 31, 2020 by Lauren Thomas . Revised on January 22, 2024.

Like a true experiment , a quasi-experimental design aims to establish a cause-and-effect relationship between an independent and dependent variable .

However, unlike a true experiment, a quasi-experiment does not rely on random assignment . Instead, subjects are assigned to groups based on non-random criteria.

Quasi-experimental design is a useful tool in situations where true experiments cannot be used for ethical or practical reasons.

Table of contents

Differences between quasi-experiments and true experiments, types of quasi-experimental designs, when to use quasi-experimental design, advantages and disadvantages, other interesting articles, frequently asked questions about quasi-experimental designs.

There are several common differences between true and quasi-experimental designs.

Example of a true experiment vs a quasi-experiment

However, for ethical reasons, the directors of the mental health clinic may not give you permission to randomly assign their patients to treatments. In this case, you cannot run a true experiment.

Instead, you can use a quasi-experimental design.

You can use these pre-existing groups to study the symptom progression of the patients treated with the new therapy versus those receiving the standard course of treatment.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Many types of quasi-experimental designs exist. Here we explain three of the most common types: nonequivalent groups design, regression discontinuity, and natural experiments.

Nonequivalent groups design

In nonequivalent group design, the researcher chooses existing groups that appear similar, but where only one of the groups experiences the treatment.

In a true experiment with random assignment , the control and treatment groups are considered equivalent in every way other than the treatment. But in a quasi-experiment where the groups are not random, they may differ in other ways—they are nonequivalent groups .

When using this kind of design, researchers try to account for any confounding variables by controlling for them in their analysis or by choosing groups that are as similar as possible.

This is the most common type of quasi-experimental design.

Regression discontinuity

Many potential treatments that researchers wish to study are designed around an essentially arbitrary cutoff, where those above the threshold receive the treatment and those below it do not.

Near this threshold, the differences between the two groups are often so minimal as to be nearly nonexistent. Therefore, researchers can use individuals just below the threshold as a control group and those just above as a treatment group.

However, since the exact cutoff score is arbitrary, the students near the threshold—those who just barely pass the exam and those who fail by a very small margin—tend to be very similar, with the small differences in their scores mostly due to random chance. You can therefore conclude that any outcome differences must come from the school they attended.

Natural experiments

In both laboratory and field experiments, researchers normally control which group the subjects are assigned to. In a natural experiment, an external event or situation (“nature”) results in the random or random-like assignment of subjects to the treatment group.

Even though some use random assignments, natural experiments are not considered to be true experiments because they are observational in nature.

Although the researchers have no control over the independent variable , they can exploit this event after the fact to study the effect of the treatment.

However, as they could not afford to cover everyone who they deemed eligible for the program, they instead allocated spots in the program based on a random lottery.

Although true experiments have higher internal validity , you might choose to use a quasi-experimental design for ethical or practical reasons.

Sometimes it would be unethical to provide or withhold a treatment on a random basis, so a true experiment is not feasible. In this case, a quasi-experiment can allow you to study the same causal relationship without the ethical issues.

The Oregon Health Study is a good example. It would be unethical to randomly provide some people with health insurance but purposely prevent others from receiving it solely for the purposes of research.

However, since the Oregon government faced financial constraints and decided to provide health insurance via lottery, studying this event after the fact is a much more ethical approach to studying the same problem.

True experimental design may be infeasible to implement or simply too expensive, particularly for researchers without access to large funding streams.

At other times, too much work is involved in recruiting and properly designing an experimental intervention for an adequate number of subjects to justify a true experiment.

In either case, quasi-experimental designs allow you to study the question by taking advantage of data that has previously been paid for or collected by others (often the government).

Quasi-experimental designs have various pros and cons compared to other types of studies.

• Higher external validity than most true experiments, because they often involve real-world interventions instead of artificial laboratory settings.
• Higher internal validity than other non-experimental types of research, because they allow you to better control for confounding variables than other types of studies do.
• Lower internal validity than true experiments—without randomization, it can be difficult to verify that all confounding variables have been accounted for.
• The use of retrospective data that has already been collected for other purposes can be inaccurate, incomplete or difficult to access.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

A quasi-experiment is a type of research design that attempts to establish a cause-and-effect relationship. The main difference with a true experiment is that the groups are not randomly assigned.

In experimental research, random assignment is a way of placing participants from your sample into different groups using randomization. With this method, every member of the sample has a known or equal chance of being placed in a control group or an experimental group.

Quasi-experimental design is most useful in situations where it would be unethical or impractical to run a true experiment .

Quasi-experiments have lower internal validity than true experiments, but they often have higher external validity  as they can use real-world interventions instead of artificial laboratory settings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Thomas, L. (2024, January 22). Quasi-Experimental Design | Definition, Types & Examples. Scribbr. Retrieved August 5, 2024, from https://www.scribbr.com/methodology/quasi-experimental-design/

Is this article helpful?

Lauren Thomas

Other students also liked, guide to experimental design | overview, steps, & examples, random assignment in experiments | introduction & examples, control variables | what are they & why do they matter, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

• Skip to secondary menu
• Skip to main content
• Skip to primary sidebar

Statistics By Jim

Making statistics intuitive

Quasi Experimental Design Overview & Examples

By Jim Frost Leave a Comment

What is a Quasi Experimental Design?

A quasi experimental design is a method for identifying causal relationships that does not randomly assign participants to the experimental groups. Instead, researchers use a non-random process. For example, they might use an eligibility cutoff score or preexisting groups to determine who receives the treatment.

Quasi-experimental research is a design that closely resembles experimental research but is different. The term “quasi” means “resembling,” so you can think of it as a cousin to actual experiments. In these studies, researchers can manipulate an independent variable — that is, they change one factor to see what effect it has. However, unlike true experimental research, participants are not randomly assigned to different groups.

Learn more about Experimental Designs: Definition & Types .

When to Use Quasi-Experimental Design

Researchers typically use a quasi-experimental design because they can’t randomize due to practical or ethical concerns. For example:

• Practical Constraints : A school interested in testing a new teaching method can only implement it in preexisting classes and cannot randomly assign students.
• Ethical Concerns : A medical study might not be able to randomly assign participants to a treatment group for an experimental medication when they are already taking a proven drug.

Quasi-experimental designs also come in handy when researchers want to study the effects of naturally occurring events, like policy changes or environmental shifts, where they can’t control who is exposed to the treatment.

Quasi-experimental designs occupy a unique position in the spectrum of research methodologies, sitting between observational studies and true experiments. This middle ground offers a blend of both worlds, addressing some limitations of purely observational studies while navigating the constraints often accompanying true experiments.

A significant advantage of quasi-experimental research over purely observational studies and correlational research is that it addresses the issue of directionality, determining which variable is the cause and which is the effect. In quasi-experiments, an intervention typically occurs during the investigation, and the researchers record outcomes before and after it, increasing the confidence that it causes the observed changes.

However, it’s crucial to recognize its limitations as well. Controlling confounding variables is a larger concern for a quasi-experimental design than a true experiment because it lacks random assignment.

In sum, quasi-experimental designs offer a valuable research approach when random assignment is not feasible, providing a more structured and controlled framework than observational studies while acknowledging and attempting to address potential confounders.

Types of Quasi-Experimental Designs and Examples

Quasi-experimental studies use various methods, depending on the scenario.

Natural Experiments

This design uses naturally occurring events or changes to create the treatment and control groups. Researchers compare outcomes between those whom the event affected and those it did not affect. Analysts use statistical controls to account for confounders that the researchers must also measure.

Natural experiments are related to observational studies, but they allow for a clearer causality inference because the external event or policy change provides both a form of quasi-random group assignment and a definite start date for the intervention.

For example, in a natural experiment utilizing a quasi-experimental design, researchers study the impact of a significant economic policy change on small business growth. The policy is implemented in one state but not in neighboring states. This scenario creates an unplanned experimental setup, where the state with the new policy serves as the treatment group, and the neighboring states act as the control group.

Researchers are primarily interested in small business growth rates but need to record various confounders that can impact growth rates. Hence, they record state economic indicators, investment levels, and employment figures. By recording these metrics across the states, they can include them in the model as covariates and control them statistically. This method allows researchers to estimate differences in small business growth due to the policy itself, separate from the various confounders.

Nonequivalent Groups Design

This method involves matching existing groups that are similar but not identical. Researchers attempt to find groups that are as equivalent as possible, particularly for factors likely to affect the outcome.

For instance, researchers use a nonequivalent groups quasi-experimental design to evaluate the effectiveness of a new teaching method in improving students’ mathematics performance. A school district considering the teaching method is planning the study. Students are already divided into schools, preventing random assignment.

The researchers matched two schools with similar demographics, baseline academic performance, and resources. The school using the traditional methodology is the control, while the other uses the new approach. Researchers are evaluating differences in educational outcomes between the two methods.

They perform a pretest to identify differences between the schools that might affect the outcome and include them as covariates to control for confounding. They also record outcomes before and after the intervention to have a larger context for the changes they observe.

Regression Discontinuity

This process assigns subjects to a treatment or control group based on a predetermined cutoff point (e.g., a test score). The analysis primarily focuses on participants near the cutoff point, as they are likely similar except for the treatment received. By comparing participants just above and below the cutoff, the design controls for confounders that vary smoothly around the cutoff.

For example, in a regression discontinuity quasi-experimental design focusing on a new medical treatment for depression, researchers use depression scores as the cutoff point. Individuals with depression scores just above a certain threshold are assigned to receive the latest treatment, while those just below the threshold do not receive it. This method creates two closely matched groups: one that barely qualifies for treatment and one that barely misses out.

By comparing the mental health outcomes of these two groups over time, researchers can assess the effectiveness of the new treatment. The assumption is that the only significant difference between the groups is whether they received the treatment, thereby isolating its impact on depression outcomes.

Controlling Confounders in a Quasi-Experimental Design

Accounting for confounding variables is a challenging but essential task for a quasi-experimental design.

In a true experiment, the random assignment process equalizes confounders across the groups to nullify their overall effect. It’s the gold standard because it works on all confounders, known and unknown.

Unfortunately, the lack of random assignment can allow differences between the groups to exist before the intervention. These confounding factors might ultimately explain the results rather than the intervention.

Consequently, researchers must use other methods to equalize the groups roughly using matching and cutoff values or statistically adjust for preexisting differences they measure to reduce the impact of confounders.

A key strength of quasi-experiments is their frequent use of “pre-post testing.” This approach involves conducting initial tests before collecting data to check for preexisting differences between groups that could impact the study’s outcome. By identifying these variables early on and including them as covariates, researchers can more effectively control potential confounders in their statistical analysis.

Additionally, researchers frequently track outcomes before and after the intervention to better understand the context for changes they observe.

Statisticians consider these methods to be less effective than randomization. Hence, quasi-experiments fall somewhere in the middle when it comes to internal validity , or how well the study can identify causal relationships versus mere correlation . They’re more conclusive than correlational studies but not as solid as true experiments.

In conclusion, quasi-experimental designs offer researchers a versatile and practical approach when random assignment is not feasible. This methodology bridges the gap between controlled experiments and observational studies, providing a valuable tool for investigating cause-and-effect relationships in real-world settings. Researchers can address ethical and logistical constraints by understanding and leveraging the different types of quasi-experimental designs while still obtaining insightful and meaningful results.

Cook, T. D., & Campbell, D. T. (1979).  Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin

Share this:

Reader Interactions

Comments and questions cancel reply.

A Guide on Data Analysis

23 quasi-experimental.

In most cases, it means that you have pre- and post-intervention data.

Great resources for causal inference include Causal Inference Mixtape and Recent Advances in Micro , especially if you like to read about the history of causal inference as a field as well (codes for Stata, R, and Python).

Libraries in R:

Econometrics

Causal Inference

Identification strategy for any quasi-experiment (No ways to prove or formal statistical test, but you can provide plausible argument and evidence)

• Where the exogenous variation comes from (by argument and institutional knowledge)
• The stable unit treatment value assumption (SUTVA) states that the treatment of unit \(i\) affect only the outcome of unit \(i\) (i.e., no spillover to the control groups)

All quasi-experimental methods involve a tradeoff between power and support for the exogeneity assumption (i.e., discard variation in the data that is not exogenous).

Consequently, we don’t usually look at \(R^2\) ( Ebbes, Papies, and Van Heerde 2011 ) . And it can even be misleading to use \(R^2\) as the basis for model comparison.

Clustering should be based on the design, not the expectations of correlation ( Abadie et al. 2023 ) . With a small sample , you should use the wild bootstrap procedure ( Cameron, Gelbach, and Miller 2008 ) to correct for the downward bias (see ( Cai et al. 2022 ) for additional assumptions).

Typical robustness check: recommended by ( Goldfarb, Tucker, and Wang 2022 )

Different controls: show models with and without controls. Typically, we want to see the change in the estimate of interest. See ( Altonji, Elder, and Taber 2005 ) for a formal assessment based on Rosenbaum bounds (i.e., changes in the estimate and threat of Omitted variables on the estimate). For specific applications in marketing, see ( Manchanda, Packard, and Pattabhiramaiah 2015 ) ( Shin, Sudhir, and Yoon 2012 )

Different functional forms

Different window of time (in longitudinal setting)

Different dependent variables (those that are related) or different measures of the dependent variables

Different control group size (matched vs. un-matched samples)

Placebo tests: see each placebo test for each setting below.

Showing the mechanism:

Mediation analysis

Moderation analysis

Estimate the model separately (for different groups)

Assess whether the three-way interaction between the source of variation (e.g., under DID, cross-sectional and time series) and group membership is significant.

External Validity:

Assess how representative your sample is

Explain the limitation of the design.

Use quasi-experimental results in conjunction with structural models: see ( J. E. Anderson, Larch, and Yotov 2015 ; Einav, Finkelstein, and Levin 2010 ; Chung, Steenburgh, and Sudhir 2014 )

• What is your identifying assumptions or identification strategy
• What are threats to the validity of your assumptions?
• What you do to address it? And maybe how future research can do to address it.

23.1 Natural Experiments

Reusing the same natural experiments for research, particularly when employing identical methods to determine the treatment effect in a given setting, can pose problems for hypothesis testing.

Simulations show that when \(N_{\text{Outcome}} >> N_{\text{True effect}}\) , more than 50% of statistically significant findings may be false positives ( Heath et al. 2023, 2331 ) .

Bonferroni correction

Romano and Wolf ( 2005 ) and Romano and Wolf ( 2016 ) correction: recommended

Benjamini and Yekutieli ( 2001 ) correction

Alternatively, refer to the rules of thumb from Table AI ( Heath et al. 2023, 2356 ) .

When applying multiple testing corrections, we can either use (but they will give similar results anyway ( Heath et al. 2023, 2335 ) ):

Chronological Sequencing : Outcomes are ordered by the date they were first reported, with multiple testing corrections applied in this sequence. This method progressively raises the statistical significance threshold as more outcomes are reviewed over time.

Best Foot Forward Policy : Outcomes are ordered from most to least likely to be rejected based on experimental data. Used primarily in clinical trials, this approach gives priority to intended treatment effects, which are subjected to less stringent statistical requirements. New outcomes are added to the sequence as they are linked to the primary treatment effect.

For all other tests, one can use multtest::mt.rawp2adjp which includes:

• Holm ( 1979 )
• Šidák ( 1967 )
• Hochberg ( 1988 )
• Benjamini and Hochberg ( 1995 )
• Benjamini and Yekutieli ( 2001 )
• Adaptive Benjamini and Hochberg ( 2000 )
• Two-stage Benjamini, Krieger, and Yekutieli ( 2006 )

Permutation adjusted p-values for simple multiple testing procedures

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7.3 Quasi-Experimental Research

Learning objectives.

• Explain what quasi-experimental research is and distinguish it clearly from both experimental and correlational research.
• Describe three different types of quasi-experimental research designs (nonequivalent groups, pretest-posttest, and interrupted time series) and identify examples of each one.

The prefix quasi means “resembling.” Thus quasi-experimental research is research that resembles experimental research but is not true experimental research. Although the independent variable is manipulated, participants are not randomly assigned to conditions or orders of conditions (Cook & Campbell, 1979). Because the independent variable is manipulated before the dependent variable is measured, quasi-experimental research eliminates the directionality problem. But because participants are not randomly assigned—making it likely that there are other differences between conditions—quasi-experimental research does not eliminate the problem of confounding variables. In terms of internal validity, therefore, quasi-experiments are generally somewhere between correlational studies and true experiments.

Quasi-experiments are most likely to be conducted in field settings in which random assignment is difficult or impossible. They are often conducted to evaluate the effectiveness of a treatment—perhaps a type of psychotherapy or an educational intervention. There are many different kinds of quasi-experiments, but we will discuss just a few of the most common ones here.

Nonequivalent Groups Design

Recall that when participants in a between-subjects experiment are randomly assigned to conditions, the resulting groups are likely to be quite similar. In fact, researchers consider them to be equivalent. When participants are not randomly assigned to conditions, however, the resulting groups are likely to be dissimilar in some ways. For this reason, researchers consider them to be nonequivalent. A nonequivalent groups design , then, is a between-subjects design in which participants have not been randomly assigned to conditions.

Imagine, for example, a researcher who wants to evaluate a new method of teaching fractions to third graders. One way would be to conduct a study with a treatment group consisting of one class of third-grade students and a control group consisting of another class of third-grade students. This would be a nonequivalent groups design because the students are not randomly assigned to classes by the researcher, which means there could be important differences between them. For example, the parents of higher achieving or more motivated students might have been more likely to request that their children be assigned to Ms. Williams’s class. Or the principal might have assigned the “troublemakers” to Mr. Jones’s class because he is a stronger disciplinarian. Of course, the teachers’ styles, and even the classroom environments, might be very different and might cause different levels of achievement or motivation among the students. If at the end of the study there was a difference in the two classes’ knowledge of fractions, it might have been caused by the difference between the teaching methods—but it might have been caused by any of these confounding variables.

Of course, researchers using a nonequivalent groups design can take steps to ensure that their groups are as similar as possible. In the present example, the researcher could try to select two classes at the same school, where the students in the two classes have similar scores on a standardized math test and the teachers are the same sex, are close in age, and have similar teaching styles. Taking such steps would increase the internal validity of the study because it would eliminate some of the most important confounding variables. But without true random assignment of the students to conditions, there remains the possibility of other important confounding variables that the researcher was not able to control.

Pretest-Posttest Design

In a pretest-posttest design , the dependent variable is measured once before the treatment is implemented and once after it is implemented. Imagine, for example, a researcher who is interested in the effectiveness of an antidrug education program on elementary school students’ attitudes toward illegal drugs. The researcher could measure the attitudes of students at a particular elementary school during one week, implement the antidrug program during the next week, and finally, measure their attitudes again the following week. The pretest-posttest design is much like a within-subjects experiment in which each participant is tested first under the control condition and then under the treatment condition. It is unlike a within-subjects experiment, however, in that the order of conditions is not counterbalanced because it typically is not possible for a participant to be tested in the treatment condition first and then in an “untreated” control condition.

If the average posttest score is better than the average pretest score, then it makes sense to conclude that the treatment might be responsible for the improvement. Unfortunately, one often cannot conclude this with a high degree of certainty because there may be other explanations for why the posttest scores are better. One category of alternative explanations goes under the name of history . Other things might have happened between the pretest and the posttest. Perhaps an antidrug program aired on television and many of the students watched it, or perhaps a celebrity died of a drug overdose and many of the students heard about it. Another category of alternative explanations goes under the name of maturation . Participants might have changed between the pretest and the posttest in ways that they were going to anyway because they are growing and learning. If it were a yearlong program, participants might become less impulsive or better reasoners and this might be responsible for the change.

Another alternative explanation for a change in the dependent variable in a pretest-posttest design is regression to the mean . This refers to the statistical fact that an individual who scores extremely on a variable on one occasion will tend to score less extremely on the next occasion. For example, a bowler with a long-term average of 150 who suddenly bowls a 220 will almost certainly score lower in the next game. Her score will “regress” toward her mean score of 150. Regression to the mean can be a problem when participants are selected for further study because of their extreme scores. Imagine, for example, that only students who scored especially low on a test of fractions are given a special training program and then retested. Regression to the mean all but guarantees that their scores will be higher even if the training program has no effect. A closely related concept—and an extremely important one in psychological research—is spontaneous remission . This is the tendency for many medical and psychological problems to improve over time without any form of treatment. The common cold is a good example. If one were to measure symptom severity in 100 common cold sufferers today, give them a bowl of chicken soup every day, and then measure their symptom severity again in a week, they would probably be much improved. This does not mean that the chicken soup was responsible for the improvement, however, because they would have been much improved without any treatment at all. The same is true of many psychological problems. A group of severely depressed people today is likely to be less depressed on average in 6 months. In reviewing the results of several studies of treatments for depression, researchers Michael Posternak and Ivan Miller found that participants in waitlist control conditions improved an average of 10 to 15% before they received any treatment at all (Posternak & Miller, 2001). Thus one must generally be very cautious about inferring causality from pretest-posttest designs.

Does Psychotherapy Work?

Early studies on the effectiveness of psychotherapy tended to use pretest-posttest designs. In a classic 1952 article, researcher Hans Eysenck summarized the results of 24 such studies showing that about two thirds of patients improved between the pretest and the posttest (Eysenck, 1952). But Eysenck also compared these results with archival data from state hospital and insurance company records showing that similar patients recovered at about the same rate without receiving psychotherapy. This suggested to Eysenck that the improvement that patients showed in the pretest-posttest studies might be no more than spontaneous remission. Note that Eysenck did not conclude that psychotherapy was ineffective. He merely concluded that there was no evidence that it was, and he wrote of “the necessity of properly planned and executed experimental studies into this important field” (p. 323). You can read the entire article here:

http://psychclassics.yorku.ca/Eysenck/psychotherapy.htm

Fortunately, many other researchers took up Eysenck’s challenge, and by 1980 hundreds of experiments had been conducted in which participants were randomly assigned to treatment and control conditions, and the results were summarized in a classic book by Mary Lee Smith, Gene Glass, and Thomas Miller (Smith, Glass, & Miller, 1980). They found that overall psychotherapy was quite effective, with about 80% of treatment participants improving more than the average control participant. Subsequent research has focused more on the conditions under which different types of psychotherapy are more or less effective.

In a classic 1952 article, researcher Hans Eysenck pointed out the shortcomings of the simple pretest-posttest design for evaluating the effectiveness of psychotherapy.

Wikimedia Commons – CC BY-SA 3.0.

Interrupted Time Series Design

A variant of the pretest-posttest design is the interrupted time-series design . A time series is a set of measurements taken at intervals over a period of time. For example, a manufacturing company might measure its workers’ productivity each week for a year. In an interrupted time series-design, a time series like this is “interrupted” by a treatment. In one classic example, the treatment was the reduction of the work shifts in a factory from 10 hours to 8 hours (Cook & Campbell, 1979). Because productivity increased rather quickly after the shortening of the work shifts, and because it remained elevated for many months afterward, the researcher concluded that the shortening of the shifts caused the increase in productivity. Notice that the interrupted time-series design is like a pretest-posttest design in that it includes measurements of the dependent variable both before and after the treatment. It is unlike the pretest-posttest design, however, in that it includes multiple pretest and posttest measurements.

Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows data from a hypothetical interrupted time-series study. The dependent variable is the number of student absences per week in a research methods course. The treatment is that the instructor begins publicly taking attendance each day so that students know that the instructor is aware of who is present and who is absent. The top panel of Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows how the data might look if this treatment worked. There is a consistently high number of absences before the treatment, and there is an immediate and sustained drop in absences after the treatment. The bottom panel of Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows how the data might look if this treatment did not work. On average, the number of absences after the treatment is about the same as the number before. This figure also illustrates an advantage of the interrupted time-series design over a simpler pretest-posttest design. If there had been only one measurement of absences before the treatment at Week 7 and one afterward at Week 8, then it would have looked as though the treatment were responsible for the reduction. The multiple measurements both before and after the treatment suggest that the reduction between Weeks 7 and 8 is nothing more than normal week-to-week variation.

Figure 7.5 A Hypothetical Interrupted Time-Series Design

The top panel shows data that suggest that the treatment caused a reduction in absences. The bottom panel shows data that suggest that it did not.

Combination Designs

A type of quasi-experimental design that is generally better than either the nonequivalent groups design or the pretest-posttest design is one that combines elements of both. There is a treatment group that is given a pretest, receives a treatment, and then is given a posttest. But at the same time there is a control group that is given a pretest, does not receive the treatment, and then is given a posttest. The question, then, is not simply whether participants who receive the treatment improve but whether they improve more than participants who do not receive the treatment.

Imagine, for example, that students in one school are given a pretest on their attitudes toward drugs, then are exposed to an antidrug program, and finally are given a posttest. Students in a similar school are given the pretest, not exposed to an antidrug program, and finally are given a posttest. Again, if students in the treatment condition become more negative toward drugs, this could be an effect of the treatment, but it could also be a matter of history or maturation. If it really is an effect of the treatment, then students in the treatment condition should become more negative than students in the control condition. But if it is a matter of history (e.g., news of a celebrity drug overdose) or maturation (e.g., improved reasoning), then students in the two conditions would be likely to show similar amounts of change. This type of design does not completely eliminate the possibility of confounding variables, however. Something could occur at one of the schools but not the other (e.g., a student drug overdose), so students at the first school would be affected by it while students at the other school would not.

Finally, if participants in this kind of design are randomly assigned to conditions, it becomes a true experiment rather than a quasi experiment. In fact, it is the kind of experiment that Eysenck called for—and that has now been conducted many times—to demonstrate the effectiveness of psychotherapy.

Key Takeaways

• Quasi-experimental research involves the manipulation of an independent variable without the random assignment of participants to conditions or orders of conditions. Among the important types are nonequivalent groups designs, pretest-posttest, and interrupted time-series designs.
• Quasi-experimental research eliminates the directionality problem because it involves the manipulation of the independent variable. It does not eliminate the problem of confounding variables, however, because it does not involve random assignment to conditions. For these reasons, quasi-experimental research is generally higher in internal validity than correlational studies but lower than true experiments.
• Practice: Imagine that two college professors decide to test the effect of giving daily quizzes on student performance in a statistics course. They decide that Professor A will give quizzes but Professor B will not. They will then compare the performance of students in their two sections on a common final exam. List five other variables that might differ between the two sections that could affect the results.

Discussion: Imagine that a group of obese children is recruited for a study in which their weight is measured, then they participate for 3 months in a program that encourages them to be more active, and finally their weight is measured again. Explain how each of the following might affect the results:

• regression to the mean
• spontaneous remission

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin.

Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16 , 319–324.

Posternak, M. A., & Miller, I. (2001). Untreated short-term course of major depression: A meta-analysis of studies using outcomes from studies using wait-list control groups. Journal of Affective Disorders, 66 , 139–146.

Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy . Baltimore, MD: Johns Hopkins University Press.

Research Methods in Psychology Copyright © 2016 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

The use and interpretation of quasi-experimental design

Last updated

6 February 2023

Reviewed by

Miroslav Damyanov

Short on time? Get an AI generated summary of this article instead

• What is a quasi-experimental design?

Commonly used in medical informatics (a field that uses digital information to ensure better patient care), researchers generally use this design to evaluate the effectiveness of a treatment – perhaps a type of antibiotic or psychotherapy, or an educational or policy intervention.

Even though quasi-experimental design has been used for some time, relatively little is known about it. Read on to learn the ins and outs of this research design.

Make research less tedious

Dovetail streamlines research to help you uncover and share actionable insights

• When to use a quasi-experimental design

A quasi-experimental design is used when it's not logistically feasible or ethical to conduct randomized, controlled trials. As its name suggests, a quasi-experimental design is almost a true experiment. However, researchers don't randomly select elements or participants in this type of research.

Researchers prefer to apply quasi-experimental design when there are ethical or practical concerns. Let's look at these two reasons more closely.

Ethical reasons

In some situations, the use of randomly assigned elements can be unethical. For instance, providing public healthcare to one group and withholding it to another in research is unethical. A quasi-experimental design would examine the relationship between these two groups to avoid physical danger.

Practical reasons

Randomized controlled trials may not be the best approach in research. For instance, it's impractical to trawl through large sample sizes of participants without using a particular attribute to guide your data collection .

Recruiting participants and properly designing a data-collection attribute to make the research a true experiment requires a lot of time and effort, and can be expensive if you don’t have a large funding stream.

A quasi-experimental design allows researchers to take advantage of previously collected data and use it in their study.

• Examples of quasi-experimental designs

Quasi-experimental research design is common in medical research, but any researcher can use it for research that raises practical and ethical concerns. Here are a few examples of quasi-experimental designs used by different researchers:

Example 1: Determining the effectiveness of math apps in supplementing math classes

A school wanted to supplement its math classes with a math app. To select the best app, the school decided to conduct demo tests on two apps before selecting the one they will purchase.

Scope of the research

Since every grade had two math teachers, each teacher used one of the two apps for three months. They then gave the students the same math exams and compared the results to determine which app was most effective.

Reasons why this is a quasi-experimental study

This simple study is a quasi-experiment since the school didn't randomly assign its students to the applications. They used a pre-existing class structure to conduct the study since it was impractical to randomly assign the students to each app.

Example 2: Determining the effectiveness of teaching modern leadership techniques in start-up businesses

A hypothetical quasi-experimental study was conducted in an economically developing country in a mid-sized city.

Five start-ups in the textile industry and five in the tech industry participated in the study. The leaders attended a six-week workshop on leadership style, team management, and employee motivation.

After a year, the researchers assessed the performance of each start-up company to determine growth. The results indicated that the tech start-ups were further along in their growth than the textile companies.

The basis of quasi-experimental research is a non-randomized subject-selection process. This study didn't use specific aspects to determine which start-up companies should participate. Therefore, the results may seem straightforward, but several aspects may determine the growth of a specific company, apart from the variables used by the researchers.

Example 3: A study to determine the effects of policy reforms and of luring foreign investment on small businesses in two mid-size cities

In a study to determine the economic impact of government reforms in an economically developing country, the government decided to test whether creating reforms directed at small businesses or luring foreign investments would spur the most economic development.

The government selected two cities with similar population demographics and sizes. In one of the cities, they implemented specific policies that would directly impact small businesses, and in the other, they implemented policies to attract foreign investment.

After five years, they collected end-of-year economic growth data from both cities. They looked at elements like local GDP growth, unemployment rates, and housing sales.

The study used a non-randomized selection process to determine which city would participate in the research. Researchers left out certain variables that would play a crucial role in determining the growth of each city. They used pre-existing groups of people based on research conducted in each city, rather than random groups.

• Advantages of a quasi-experimental design

Some advantages of quasi-experimental designs are:

Researchers can manipulate variables to help them meet their study objectives.

It offers high external validity, making it suitable for real-world applications, specifically in social science experiments.

Integrating this methodology into other research designs is easier, especially in true experimental research. This cuts down on the time needed to determine your outcomes.

• Disadvantages of a quasi-experimental design

Despite the pros that come with a quasi-experimental design, there are several disadvantages associated with it, including the following:

It has a lower internal validity since researchers do not have full control over the comparison and intervention groups or between time periods because of differences in characteristics in people, places, or time involved. It may be challenging to determine whether all variables have been used or whether those used in the research impacted the results.

There is the risk of inaccurate data since the research design borrows information from other studies.

There is the possibility of bias since researchers select baseline elements and eligibility.

• What are the different quasi-experimental study designs?

There are three distinct types of quasi-experimental designs:

Nonequivalent

Regression discontinuity, natural experiment.

This is a hybrid of experimental and quasi-experimental methods and is used to leverage the best qualities of the two. Like the true experiment design, nonequivalent group design uses pre-existing groups believed to be comparable. However, it doesn't use randomization, the lack of which is a crucial element for quasi-experimental design.

Researchers usually ensure that no confounding variables impact them throughout the grouping process. This makes the groupings more comparable.

Example of a nonequivalent group design

A small study was conducted to determine whether after-school programs result in better grades. Researchers randomly selected two groups of students: one to implement the new program, the other not to. They then compared the results of the two groups.

This type of quasi-experimental research design calculates the impact of a specific treatment or intervention. It uses a criterion known as "cutoff" that assigns treatment according to eligibility.

Researchers often assign participants above the cutoff to the treatment group. This puts a negligible distinction between the two groups (treatment group and control group).

Example of regression discontinuity

Students must achieve a minimum score to be enrolled in specific US high schools. Since the cutoff score used to determine eligibility for enrollment is arbitrary, researchers can assume that the disparity between students who only just fail to achieve the cutoff point and those who barely pass is a small margin and is due to the difference in the schools that these students attend.

Researchers can then examine the long-term effects of these two groups of kids to determine the effect of attending certain schools. This information can be applied to increase the chances of students being enrolled in these high schools.

This research design is common in laboratory and field experiments where researchers control target subjects by assigning them to different groups. Researchers randomly assign subjects to a treatment group using nature or an external event or situation.

However, even with random assignment, this research design cannot be called a true experiment since nature aspects are observational. Researchers can also exploit these aspects despite having no control over the independent variables.

Example of the natural experiment approach

An example of a natural experiment is the 2008 Oregon Health Study.

Oregon intended to allow more low-income people to participate in Medicaid.

Since they couldn't afford to cover every person who qualified for the program, the state used a random lottery to allocate program slots.

Researchers assessed the program's effectiveness by assigning the selected subjects to a randomly assigned treatment group, while those that didn't win the lottery were considered the control group.

• Differences between quasi-experiments and true experiments

There are several differences between a quasi-experiment and a true experiment:

Participants in true experiments are randomly assigned to the treatment or control group, while participants in a quasi-experiment are not assigned randomly.

In a quasi-experimental design, the control and treatment groups differ in unknown or unknowable ways, apart from the experimental treatments that are carried out. Therefore, the researcher should try as much as possible to control these differences.

Quasi-experimental designs have several "competing hypotheses," which compete with experimental manipulation to explain the observed results.

Quasi-experiments tend to have lower internal validity (the degree of confidence in the research outcomes) than true experiments, but they may offer higher external validity (whether findings can be extended to other contexts) as they involve real-world interventions instead of controlled interventions in artificial laboratory settings.

Despite the distinct difference between true and quasi-experimental research designs, these two research methodologies share the following aspects:

Both study methods subject participants to some form of treatment or conditions.

Researchers have the freedom to measure some of the outcomes of interest.

Researchers can test whether the differences in the outcomes are associated with the treatment.

• An example comparing a true experiment and quasi-experiment

Imagine you wanted to study the effects of junk food on obese people. Here's how you would do this as a true experiment and a quasi-experiment:

How to carry out a true experiment

In a true experiment, some participants would eat junk foods, while the rest would be in the control group, adhering to a regular diet. At the end of the study, you would record the health and discomfort of each group.

This kind of experiment would raise ethical concerns since the participants assigned to the treatment group are required to eat junk food against their will throughout the experiment. This calls for a quasi-experimental design.

How to carry out a quasi-experiment

In quasi-experimental research, you would start by finding out which participants want to try junk food and which prefer to stick to a regular diet. This allows you to assign these two groups based on subject choice.

In this case, you didn't assign participants to a particular group, so you can confidently use the results from the study.

When is a quasi-experimental design used?

Quasi-experimental designs are used when researchers don’t want to use randomization when evaluating their intervention.

What are the characteristics of quasi-experimental designs?

Some of the characteristics of a quasi-experimental design are:

Researchers don't randomly assign participants into groups, but study their existing characteristics and assign them accordingly.

Researchers study the participants in pre- and post-testing to determine the progress of the groups.

Quasi-experimental design is ethical since it doesn’t involve offering or withholding treatment at random.

Quasi-experimental design encompasses a broad range of non-randomized intervention studies. This design is employed when it is not ethical or logistically feasible to conduct randomized controlled trials. Researchers typically employ it when evaluating policy or educational interventions, or in medical or therapy scenarios.

How do you analyze data in a quasi-experimental design?

You can use two-group tests, time-series analysis, and regression analysis to analyze data in a quasi-experiment design. Each option has specific assumptions, strengths, limitations, and data requirements.

Should you be using a customer insights hub?

Do you want to discover previous research faster?

Do you share your research findings with others?

Do you analyze research data?

Start for free today, add your research, and get to key insights faster

Editor’s picks

Last updated: 18 April 2023

Last updated: 27 February 2023

Last updated: 6 February 2023

Last updated: 6 October 2023

Last updated: 5 February 2023

Last updated: 16 April 2023

Last updated: 9 March 2023

Last updated: 12 December 2023

Last updated: 11 March 2024

Last updated: 4 July 2024

Last updated: 6 March 2024

Last updated: 5 March 2024

Last updated: 13 May 2024

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next, log in or sign up.

Get started for free

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Statistical analysis and application of quasi experiments to antimicrobial resistance intervention studies

Affiliation.

• 1 Department of Epidemiology and Preventive Medicine, University of Maryland School of Medicine, Baltimore, MD 21201, USA.
• PMID: 17806059
• DOI: 10.1086/521255

Quasi-experimental study designs are frequently used to assess interventions that aim to limit the emergence of antimicrobial-resistant pathogens. However, previous studies using these designs have often used suboptimal statistical methods, which may result in researchers making spurious conclusions. Methods used to analyze quasi-experimental data include 2-group tests, regression analysis, and time-series analysis, and they all have specific assumptions, data requirements, strengths, and limitations. An example of a hospital-based intervention to reduce methicillin-resistant Staphylococcus aureus infection rates and reduce overall length of stay is used to explore these methods.

PubMed Disclaimer

Similar articles

• An integrative review of infection prevention and control programs for multidrug-resistant organisms in acute care hospitals: a socio-ecological perspective. Backman C, Taylor G, Sales A, Marck PB. Backman C, et al. Am J Infect Control. 2011 Jun;39(5):368-378. doi: 10.1016/j.ajic.2010.07.017. Epub 2011 Mar 23. Am J Infect Control. 2011. PMID: 21429622 Review.
• Effectiveness of simple control measures on methicillin-resistant Staphylococcus aureus infection status and characteristics with susceptibility patterns in a teaching hospital in Peshawar. Rafiq MS, Rafiq MI, Khan T, Rafiq M, Khan MM. Rafiq MS, et al. J Pak Med Assoc. 2015 Sep;65(9):915-20. J Pak Med Assoc. 2015. PMID: 26338732
• Evaluation of an intervention designed to decrease the rate of nosocomial methicillin-resistant Staphylococcus aureus infection by encouraging decreased fluoroquinolone use. Madaras-Kelly KJ, Remington RE, Lewis PG, Stevens DL. Madaras-Kelly KJ, et al. Infect Control Hosp Epidemiol. 2006 Feb;27(2):155-69. doi: 10.1086/500060. Epub 2006 Feb 8. Infect Control Hosp Epidemiol. 2006. PMID: 16465632
• Impact of Discontinuing Contact Precautions for Methicillin-Resistant Staphylococcus aureus and Vancomycin-Resistant Enterococcus: An Interrupted Time Series Analysis. Bearman G, Abbas S, Masroor N, Sanogo K, Vanhoozer G, Cooper K, Doll M, Stevens MP, Edmond MB. Bearman G, et al. Infect Control Hosp Epidemiol. 2018 Jun;39(6):676-682. doi: 10.1017/ice.2018.57. Epub 2018 Mar 27. Infect Control Hosp Epidemiol. 2018. PMID: 29580304
• Common approaches to the control of multidrug-resistant organisms other than methicillin-resistant Staphylococcus aureus (MRSA). Hebert C, Weber SG. Hebert C, et al. Infect Dis Clin North Am. 2011 Mar;25(1):181-200. doi: 10.1016/j.idc.2010.11.006. Epub 2010 Dec 17. Infect Dis Clin North Am. 2011. PMID: 21316000 Review.
• Impact of SARS-CoV-2 Preventive Measures against Healthcare-Associated Infections from Antibiotic-Resistant ESKAPEE Pathogens: A Two-Center, Natural Quasi-Experimental Study in Greece. Bolikas E, Astrinaki E, Panagiotaki E, Vitsaxaki E, Saplamidou S, Drositis I, Stafylaki D, Chamilos G, Gikas A, Kofteridis DP, Kritsotakis EI. Bolikas E, et al. Antibiotics (Basel). 2023 Jun 22;12(7):1088. doi: 10.3390/antibiotics12071088. Antibiotics (Basel). 2023. PMID: 37508184 Free PMC article.
• Evaluating a pilot, structured, face-to-face, antimicrobial stewardship, prospective audit-and-feedback program in emergency general surgery service in a community hospital. Chan AJ, Tsang ME, Langford BJ, Nisenbaum R, Wan M, Downing MA. Chan AJ, et al. Antimicrob Steward Healthc Epidemiol. 2023 Jun 5;3(1):e96. doi: 10.1017/ash.2023.168. eCollection 2023. Antimicrob Steward Healthc Epidemiol. 2023. PMID: 37325681 Free PMC article.
• Effects of post-mortem inspection techniques change on the detection capability of low public health impact diseases of slaughtered pigs: A quasi-experimental study. Villani C, Piccioni R. Villani C, et al. Ital J Food Saf. 2022 Dec 12;11(4):10761. doi: 10.4081/ijfs.2022.10761. eCollection 2022 Dec 5. Ital J Food Saf. 2022. PMID: 36590018 Free PMC article.
• A Threshold Logistic Modelling Approach for Identifying Thresholds between Antibiotic Use and Methicillin-Resistant Staphylococcus aureus Incidence Rates in Hospitals. Aldeyab MA, Bond SE, Conway BR, Lee-Milner J, Sarma JB, Lattyak WJ. Aldeyab MA, et al. Antibiotics (Basel). 2022 Sep 15;11(9):1250. doi: 10.3390/antibiotics11091250. Antibiotics (Basel). 2022. PMID: 36140029 Free PMC article.
• Identifying Targets for Antibiotic Use for the Management of Carbapenem-Resistant Acinetobacter baumannii (CRAb) in Hospitals-A Multi-Centre Nonlinear Time-Series Study. Al-Hashimy ZS, Conway BR, Al-Yaqoobi M, Khamis F, Al Mawali GZ, Al Maashani AM, Al Hadhrami YS, Al Alawi SS, Al Mamari MS, Lattyak WJ, Lattyak EA, Aldiab M, Gould I, López-Lozano JM, Aldeyab MA. Al-Hashimy ZS, et al. Antibiotics (Basel). 2022 Jun 7;11(6):775. doi: 10.3390/antibiotics11060775. Antibiotics (Basel). 2022. PMID: 35740181 Free PMC article.

Publication types

• Search in MeSH

Related information

• Cited in Books

Grants and funding

• 1 R01 AI6085901A1/AI/NIAID NIH HHS/United States
• 1 R01 CI000369-01/CI/NCPDCID CDC HHS/United States
• 1K12RR023250-01/RR/NCRR NIH HHS/United States
• P30 AG028747-01/AG/NIA NIH HHS/United States
• P60 AG12583/AG/NIA NIH HHS/United States
• R37 AG09901/AG/NIA NIH HHS/United States

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.
• Silverchair Information Systems
• MedlinePlus Health Information

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Experimental and Quasi-Experimental Methods

• Reference work entry
• First Online: 01 January 2014
• Cite this reference work entry

• Roger J. R. Levesque 2  

183 Accesses

Research designs are central to research projects in that they constitute the projects’ basic structure that will permit researchers to address their main research questions. Designs include, for example, the selection of relevant samples or groups, measures, treatments or programs, and methods of assignment. The two key designs that help researchers address whether a program or treatment causes an outcome are the experimental design, which uses random assignment to groups or programs, and quasi-experimental designs, which do not use random assignment (see Shadish et al. 2002 ; Bell 2010 ; Trochim 2006 ). These two methods are important to consider in that even the experimental design may not prove causation, and causation is what researchers often aim to show when they analyze data (e.g., they try to show that an outcome is likely to follow given a certain set of conditions). Still, the general rule tends to be that studies unable to determine causality are classified as...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Bell, S. H. (2010). The urban institute research of record: Quasi-experimental methods. Washington, DC: The Urban Institute. Retrieved Nov. 20, 2010, from http://www.urban.org/toolkit/data-methods/quasi-experimental.cfm

Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research . Chicago: Rand McNally.

Google Scholar  

Harris, A. D., McGregor, J. C., Perencevich, E. N., Furuno, J. P., Zhu, J., Peterson, D. E., & Finkelstein, J. (2006). The use and interpretation of quasi-experimental studies in medical informatics. The Journal of American Medical Informatics Association, 13 , 16–23.

Shadish, W. R., Cook, T. D., & Campbell, T. D. (2002). Experimental and quasi-experimental designs for generalized causal inference . Boston: Houghton-Mifflin.

Trochim, W. M. (2006). The research methods knowledge base (2nd ed.). Cincinnati: Atomic Dog. Retrieved Nov. 20, 2011, from http://www.socialresearchmethods.net/kb/

Download references

Author information

Authors and affiliations.

Indiana University, 302 Sycamore Hall, Bloomington, IN, 47405, USA

Roger J. R. Levesque

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Roger J. R. Levesque .

Editor information

Editors and affiliations, rights and permissions.

Reprints and permissions

Copyright information

© 2011 Springer Science+Business Media, LLC

About this entry

Cite this entry.

Levesque, R.J.R. (2011). Experimental and Quasi-Experimental Methods. In: Levesque, R.J.R. (eds) Encyclopedia of Adolescence. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-1695-2_655

Download citation

DOI : https://doi.org/10.1007/978-1-4419-1695-2_655

Published : 12 August 2014

Publisher Name : Springer, New York, NY

Print ISBN : 978-1-4419-1694-5

Online ISBN : 978-1-4419-1695-2

eBook Packages : Behavioral Science

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Am Med Inform Assoc
• v.13(1); Jan-Feb 2006

The Use and Interpretation of Quasi-Experimental Studies in Medical Informatics

Associated data.

Quasi-experimental study designs, often described as nonrandomized, pre-post intervention studies, are common in the medical informatics literature. Yet little has been written about the benefits and limitations of the quasi-experimental approach as applied to informatics studies. This paper outlines a relative hierarchy and nomenclature of quasi-experimental study designs that is applicable to medical informatics intervention studies. In addition, the authors performed a systematic review of two medical informatics journals, the Journal of the American Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics (IJMI), to determine the number of quasi-experimental studies published and how the studies are classified on the above-mentioned relative hierarchy. They hope that future medical informatics studies will implement higher level quasi-experimental study designs that yield more convincing evidence for causal links between medical informatics interventions and outcomes.

Quasi-experimental studies encompass a broad range of nonrandomized intervention studies. These designs are frequently used when it is not logistically feasible or ethical to conduct a randomized controlled trial. Examples of quasi-experimental studies follow. As one example of a quasi-experimental study, a hospital introduces a new order-entry system and wishes to study the impact of this intervention on the number of medication-related adverse events before and after the intervention. As another example, an informatics technology group is introducing a pharmacy order-entry system aimed at decreasing pharmacy costs. The intervention is implemented and pharmacy costs before and after the intervention are measured.

In medical informatics, the quasi-experimental, sometimes called the pre-post intervention, design often is used to evaluate the benefits of specific interventions. The increasing capacity of health care institutions to collect routine clinical data has led to the growing use of quasi-experimental study designs in the field of medical informatics as well as in other medical disciplines. However, little is written about these study designs in the medical literature or in traditional epidemiology textbooks. 1 , 2 , 3 In contrast, the social sciences literature is replete with examples of ways to implement and improve quasi-experimental studies. 4 , 5 , 6

In this paper, we review the different pretest-posttest quasi-experimental study designs, their nomenclature, and the relative hierarchy of these designs with respect to their ability to establish causal associations between an intervention and an outcome. The example of a pharmacy order-entry system aimed at decreasing pharmacy costs will be used throughout this article to illustrate the different quasi-experimental designs. We discuss limitations of quasi-experimental designs and offer methods to improve them. We also perform a systematic review of four years of publications from two informatics journals to determine the number of quasi-experimental studies, classify these studies into their application domains, determine whether the potential limitations of quasi-experimental studies were acknowledged by the authors, and place these studies into the above-mentioned relative hierarchy.

The authors reviewed articles and book chapters on the design of quasi-experimental studies. 4 , 5 , 6 , 7 , 8 , 9 , 10 Most of the reviewed articles referenced two textbooks that were then reviewed in depth. 4 , 6

Key advantages and disadvantages of quasi-experimental studies, as they pertain to the study of medical informatics, were identified. The potential methodological flaws of quasi-experimental medical informatics studies, which have the potential to introduce bias, were also identified. In addition, a summary table outlining a relative hierarchy and nomenclature of quasi-experimental study designs is described. In general, the higher the design is in the hierarchy, the greater the internal validity that the study traditionally possesses because the evidence of the potential causation between the intervention and the outcome is strengthened. 4

We then performed a systematic review of four years of publications from two informatics journals. First, we determined the number of quasi-experimental studies. We then classified these studies on the above-mentioned hierarchy. We also classified the quasi-experimental studies according to their application domain. The categories of application domains employed were based on categorization used by Yearbooks of Medical Informatics 1992–2005 and were similar to the categories of application domains employed by Annual Symposiums of the American Medical Informatics Association. 11 The categories were (1) health and clinical management; (2) patient records; (3) health information systems; (4) medical signal processing and biomedical imaging; (5) decision support, knowledge representation, and management; (6) education and consumer informatics; and (7) bioinformatics. Because the quasi-experimental study design has recognized limitations, we sought to determine whether authors acknowledged the potential limitations of this design. Examples of acknowledgment included mention of lack of randomization, the potential for regression to the mean, the presence of temporal confounders and the mention of another design that would have more internal validity.

All original scientific manuscripts published between January 2000 and December 2003 in the Journal of the American Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics (IJMI) were reviewed. One author (ADH) reviewed all the papers to identify the number of quasi-experimental studies. Other authors (ADH, JCM, JF) then independently reviewed all the studies identified as quasi-experimental. The three authors then convened as a group to resolve any disagreements in study classification, application domain, and acknowledgment of limitations.

Results and Discussion

What is a quasi-experiment.

Quasi-experiments are studies that aim to evaluate interventions but that do not use randomization. Similar to randomized trials, quasi-experiments aim to demonstrate causality between an intervention and an outcome. Quasi-experimental studies can use both preintervention and postintervention measurements as well as nonrandomly selected control groups.

Using this basic definition, it is evident that many published studies in medical informatics utilize the quasi-experimental design. Although the randomized controlled trial is generally considered to have the highest level of credibility with regard to assessing causality, in medical informatics, researchers often choose not to randomize the intervention for one or more reasons: (1) ethical considerations, (2) difficulty of randomizing subjects, (3) difficulty to randomize by locations (e.g., by wards), (4) small available sample size. Each of these reasons is discussed below.

Ethical considerations typically will not allow random withholding of an intervention with known efficacy. Thus, if the efficacy of an intervention has not been established, a randomized controlled trial is the design of choice to determine efficacy. But if the intervention under study incorporates an accepted, well-established therapeutic intervention, or if the intervention has either questionable efficacy or safety based on previously conducted studies, then the ethical issues of randomizing patients are sometimes raised. In the area of medical informatics, it is often believed prior to an implementation that an informatics intervention will likely be beneficial and thus medical informaticians and hospital administrators are often reluctant to randomize medical informatics interventions. In addition, there is often pressure to implement the intervention quickly because of its believed efficacy, thus not allowing researchers sufficient time to plan a randomized trial.

For medical informatics interventions, it is often difficult to randomize the intervention to individual patients or to individual informatics users. So while this randomization is technically possible, it is underused and thus compromises the eventual strength of concluding that an informatics intervention resulted in an outcome. For example, randomly allowing only half of medical residents to use pharmacy order-entry software at a tertiary care hospital is a scenario that hospital administrators and informatics users may not agree to for numerous reasons.

Similarly, informatics interventions often cannot be randomized to individual locations. Using the pharmacy order-entry system example, it may be difficult to randomize use of the system to only certain locations in a hospital or portions of certain locations. For example, if the pharmacy order-entry system involves an educational component, then people may apply the knowledge learned to nonintervention wards, thereby potentially masking the true effect of the intervention. When a design using randomized locations is employed successfully, the locations may be different in other respects (confounding variables), and this further complicates the analysis and interpretation.

In situations where it is known that only a small sample size will be available to test the efficacy of an intervention, randomization may not be a viable option. Randomization is beneficial because on average it tends to evenly distribute both known and unknown confounding variables between the intervention and control group. However, when the sample size is small, randomization may not adequately accomplish this balance. Thus, alternative design and analytical methods are often used in place of randomization when only small sample sizes are available.

What Are the Threats to Establishing Causality When Using Quasi-experimental Designs in Medical Informatics?

The lack of random assignment is the major weakness of the quasi-experimental study design. Associations identified in quasi-experiments meet one important requirement of causality since the intervention precedes the measurement of the outcome. Another requirement is that the outcome can be demonstrated to vary statistically with the intervention. Unfortunately, statistical association does not imply causality, especially if the study is poorly designed. Thus, in many quasi-experiments, one is most often left with the question: “Are there alternative explanations for the apparent causal association?” If these alternative explanations are credible, then the evidence of causation is less convincing. These rival hypotheses, or alternative explanations, arise from principles of epidemiologic study design.

Shadish et al. 4 outline nine threats to internal validity that are outlined in ▶ . Internal validity is defined as the degree to which observed changes in outcomes can be correctly inferred to be caused by an exposure or an intervention. In quasi-experimental studies of medical informatics, we believe that the methodological principles that most often result in alternative explanations for the apparent causal effect include (a) difficulty in measuring or controlling for important confounding variables, particularly unmeasured confounding variables, which can be viewed as a subset of the selection threat in ▶ ; (b) results being explained by the statistical principle of regression to the mean . Each of these latter two principles is discussed in turn.

Threats to Internal Validity

Adapted from Shadish et al. 4

An inability to sufficiently control for important confounding variables arises from the lack of randomization. A variable is a confounding variable if it is associated with the exposure of interest and is also associated with the outcome of interest; the confounding variable leads to a situation where a causal association between a given exposure and an outcome is observed as a result of the influence of the confounding variable. For example, in a study aiming to demonstrate that the introduction of a pharmacy order-entry system led to lower pharmacy costs, there are a number of important potential confounding variables (e.g., severity of illness of the patients, knowledge and experience of the software users, other changes in hospital policy) that may have differed in the preintervention and postintervention time periods ( ▶ ). In a multivariable regression, the first confounding variable could be addressed with severity of illness measures, but the second confounding variable would be difficult if not nearly impossible to measure and control. In addition, potential confounding variables that are unmeasured or immeasurable cannot be controlled for in nonrandomized quasi-experimental study designs and can only be properly controlled by the randomization process in randomized controlled trials.

Example of confounding. To get the true effect of the intervention of interest, we need to control for the confounding variable.

Another important threat to establishing causality is regression to the mean. 12 , 13 , 14 This widespread statistical phenomenon can result in wrongly concluding that an effect is due to the intervention when in reality it is due to chance. The phenomenon was first described in 1886 by Francis Galton who measured the adult height of children and their parents. He noted that when the average height of the parents was greater than the mean of the population, the children tended to be shorter than their parents, and conversely, when the average height of the parents was shorter than the population mean, the children tended to be taller than their parents.

In medical informatics, what often triggers the development and implementation of an intervention is a rise in the rate above the mean or norm. For example, increasing pharmacy costs and adverse events may prompt hospital informatics personnel to design and implement pharmacy order-entry systems. If this rise in costs or adverse events is really just an extreme observation that is still within the normal range of the hospital's pharmaceutical costs (i.e., the mean pharmaceutical cost for the hospital has not shifted), then the statistical principle of regression to the mean predicts that these elevated rates will tend to decline even without intervention. However, often informatics personnel and hospital administrators cannot wait passively for this decline to occur. Therefore, hospital personnel often implement one or more interventions, and if a decline in the rate occurs, they may mistakenly conclude that the decline is causally related to the intervention. In fact, an alternative explanation for the finding could be regression to the mean.

What Are the Different Quasi-experimental Study Designs?

In the social sciences literature, quasi-experimental studies are divided into four study design groups 4 , 6 :

• Quasi-experimental designs without control groups
• Quasi-experimental designs that use control groups but no pretest
• Quasi-experimental designs that use control groups and pretests
• Interrupted time-series designs

There is a relative hierarchy within these categories of study designs, with category D studies being sounder than categories C, B, or A in terms of establishing causality. Thus, if feasible from a design and implementation point of view, investigators should aim to design studies that fall in to the higher rated categories. Shadish et al. 4 discuss 17 possible designs, with seven designs falling into category A, three designs in category B, and six designs in category C, and one major design in category D. In our review, we determined that most medical informatics quasi-experiments could be characterized by 11 of 17 designs, with six study designs in category A, one in category B, three designs in category C, and one design in category D because the other study designs were not used or feasible in the medical informatics literature. Thus, for simplicity, we have summarized the 11 study designs most relevant to medical informatics research in ▶ .

Relative Hierarchy of Quasi-experimental Designs

O = Observational Measurement; X = Intervention Under Study. Time moves from left to right.

The nomenclature and relative hierarchy were used in the systematic review of four years of JAMIA and the IJMI. Similar to the relative hierarchy that exists in the evidence-based literature that assigns a hierarchy to randomized controlled trials, cohort studies, case-control studies, and case series, the hierarchy in ▶ is not absolute in that in some cases, it may be infeasible to perform a higher level study. For example, there may be instances where an A6 design established stronger causality than a B1 design. 15 , 16 , 17

Quasi-experimental Designs without Control Groups

Here, X is the intervention and O is the outcome variable (this notation is continued throughout the article). In this study design, an intervention (X) is implemented and a posttest observation (O1) is taken. For example, X could be the introduction of a pharmacy order-entry intervention and O1 could be the pharmacy costs following the intervention. This design is the weakest of the quasi-experimental designs that are discussed in this article. Without any pretest observations or a control group, there are multiple threats to internal validity. Unfortunately, this study design is often used in medical informatics when new software is introduced since it may be difficult to have pretest measurements due to time, technical, or cost constraints.

This is a commonly used study design. A single pretest measurement is taken (O1), an intervention (X) is implemented, and a posttest measurement is taken (O2). In this instance, period O1 frequently serves as the “control” period. For example, O1 could be pharmacy costs prior to the intervention, X could be the introduction of a pharmacy order-entry system, and O2 could be the pharmacy costs following the intervention. Including a pretest provides some information about what the pharmacy costs would have been had the intervention not occurred.

The advantage of this study design over A2 is that adding a second pretest prior to the intervention helps provide evidence that can be used to refute the phenomenon of regression to the mean and confounding as alternative explanations for any observed association between the intervention and the posttest outcome. For example, in a study where a pharmacy order-entry system led to lower pharmacy costs (O3 < O2 and O1), if one had two preintervention measurements of pharmacy costs (O1 and O2) and they were both elevated, this would suggest that there was a decreased likelihood that O3 is lower due to confounding and regression to the mean. Similarly, extending this study design by increasing the number of measurements postintervention could also help to provide evidence against confounding and regression to the mean as alternate explanations for observed associations.

This design involves the inclusion of a nonequivalent dependent variable ( b ) in addition to the primary dependent variable ( a ). Variables a and b should assess similar constructs; that is, the two measures should be affected by similar factors and confounding variables except for the effect of the intervention. Variable a is expected to change because of the intervention X, whereas variable b is not. Taking our example, variable a could be pharmacy costs and variable b could be the length of stay of patients. If our informatics intervention is aimed at decreasing pharmacy costs, we would expect to observe a decrease in pharmacy costs but not in the average length of stay of patients. However, a number of important confounding variables, such as severity of illness and knowledge of software users, might affect both outcome measures. Thus, if the average length of stay did not change following the intervention but pharmacy costs did, then the data are more convincing than if just pharmacy costs were measured.

The Removed-Treatment Design

This design adds a third posttest measurement (O3) to the one-group pretest-posttest design and then removes the intervention before a final measure (O4) is made. The advantage of this design is that it allows one to test hypotheses about the outcome in the presence of the intervention and in the absence of the intervention. Thus, if one predicts a decrease in the outcome between O1 and O2 (after implementation of the intervention), then one would predict an increase in the outcome between O3 and O4 (after removal of the intervention). One caveat is that if the intervention is thought to have persistent effects, then O4 needs to be measured after these effects are likely to have disappeared. For example, a study would be more convincing if it demonstrated that pharmacy costs decreased after pharmacy order-entry system introduction (O2 and O3 less than O1) and that when the order-entry system was removed or disabled, the costs increased (O4 greater than O2 and O3 and closer to O1). In addition, there are often ethical issues in this design in terms of removing an intervention that may be providing benefit.

The Repeated-Treatment Design

The advantage of this design is that it demonstrates reproducibility of the association between the intervention and the outcome. For example, the association is more likely to be causal if one demonstrates that a pharmacy order-entry system results in decreased pharmacy costs when it is first introduced and again when it is reintroduced following an interruption of the intervention. As for design A5, the assumption must be made that the effect of the intervention is transient, which is most often applicable to medical informatics interventions. Because in this design, subjects may serve as their own controls, this may yield greater statistical efficiency with fewer numbers of subjects.

Quasi-experimental Designs That Use a Control Group but No Pretest

An intervention X is implemented for one group and compared to a second group. The use of a comparison group helps prevent certain threats to validity including the ability to statistically adjust for confounding variables. Because in this study design, the two groups may not be equivalent (assignment to the groups is not by randomization), confounding may exist. For example, suppose that a pharmacy order-entry intervention was instituted in the medical intensive care unit (MICU) and not the surgical intensive care unit (SICU). O1 would be pharmacy costs in the MICU after the intervention and O2 would be pharmacy costs in the SICU after the intervention. The absence of a pretest makes it difficult to know whether a change has occurred in the MICU. Also, the absence of pretest measurements comparing the SICU to the MICU makes it difficult to know whether differences in O1 and O2 are due to the intervention or due to other differences in the two units (confounding variables).

Quasi-experimental Designs That Use Control Groups and Pretests

The reader should note that with all the studies in this category, the intervention is not randomized. The control groups chosen are comparison groups. Obtaining pretest measurements on both the intervention and control groups allows one to assess the initial comparability of the groups. The assumption is that if the intervention and the control groups are similar at the pretest, the smaller the likelihood there is of important confounding variables differing between the two groups.

The use of both a pretest and a comparison group makes it easier to avoid certain threats to validity. However, because the two groups are nonequivalent (assignment to the groups is not by randomization), selection bias may exist. Selection bias exists when selection results in differences in unit characteristics between conditions that may be related to outcome differences. For example, suppose that a pharmacy order-entry intervention was instituted in the MICU and not the SICU. If preintervention pharmacy costs in the MICU (O1a) and SICU (O1b) are similar, it suggests that it is less likely that there are differences in the important confounding variables between the two units. If MICU postintervention costs (O2a) are less than preintervention MICU costs (O1a), but SICU costs (O1b) and (O2b) are similar, this suggests that the observed outcome may be causally related to the intervention.

In this design, the pretests are administered at two different times. The main advantage of this design is that it controls for potentially different time-varying confounding effects in the intervention group and the comparison group. In our example, measuring points O1 and O2 would allow for the assessment of time-dependent changes in pharmacy costs, e.g., due to differences in experience of residents, preintervention between the intervention and control group, and whether these changes were similar or different.

With this study design, the researcher administers an intervention at a later time to a group that initially served as a nonintervention control. The advantage of this design over design C2 is that it demonstrates reproducibility in two different settings. This study design is not limited to two groups; in fact, the study results have greater validity if the intervention effect is replicated in different groups at multiple times. In the example of a pharmacy order-entry system, one could implement or intervene in the MICU and then at a later time, intervene in the SICU. This latter design is often very applicable to medical informatics where new technology and new software is often introduced or made available gradually.

Interrupted Time-Series Designs

An interrupted time-series design is one in which a string of consecutive observations equally spaced in time is interrupted by the imposition of a treatment or intervention. The advantage of this design is that with multiple measurements both pre- and postintervention, it is easier to address and control for confounding and regression to the mean. In addition, statistically, there is a more robust analytic capability, and there is the ability to detect changes in the slope or intercept as a result of the intervention in addition to a change in the mean values. 18 A change in intercept could represent an immediate effect while a change in slope could represent a gradual effect of the intervention on the outcome. In the example of a pharmacy order-entry system, O1 through O5 could represent monthly pharmacy costs preintervention and O6 through O10 monthly pharmacy costs post the introduction of the pharmacy order-entry system. Interrupted time-series designs also can be further strengthened by incorporating many of the design features previously mentioned in other categories (such as removal of the treatment, inclusion of a nondependent outcome variable, or the addition of a control group).

Systematic Review Results

The results of the systematic review are in ▶ . In the four-year period of JAMIA publications that the authors reviewed, 25 quasi-experimental studies among 22 articles were published. Of these 25, 15 studies were of category A, five studies were of category B, two studies were of category C, and no studies were of category D. Although there were no studies of category D (interrupted time-series analyses), three of the studies classified as category A had data collected that could have been analyzed as an interrupted time-series analysis. Nine of the 25 studies (36%) mentioned at least one of the potential limitations of the quasi-experimental study design. In the four-year period of IJMI publications reviewed by the authors, nine quasi-experimental studies among eight manuscripts were published. Of these nine, five studies were of category A, one of category B, one of category C, and two of category D. Two of the nine studies (22%) mentioned at least one of the potential limitations of the quasi-experimental study design.

Systematic Review of Four Years of Quasi-designs in JAMIA

JAMIA = Journal of the American Medical Informatics Association; IJMI = International Journal of Medical Informatics.

In addition, three studies from JAMIA were based on a counterbalanced design. A counterbalanced design is a higher order study design than other studies in category A. The counterbalanced design is sometimes referred to as a Latin-square arrangement. In this design, all subjects receive all the different interventions but the order of intervention assignment is not random. 19 This design can only be used when the intervention is compared against some existing standard, for example, if a new PDA-based order entry system is to be compared to a computer terminal–based order entry system. In this design, all subjects receive the new PDA-based order entry system and the old computer terminal-based order entry system. The counterbalanced design is a within-participants design, where the order of the intervention is varied (e.g., one group is given software A followed by software B and another group is given software B followed by software A). The counterbalanced design is typically used when the available sample size is small, thus preventing the use of randomization. This design also allows investigators to study the potential effect of ordering of the informatics intervention.

Although quasi-experimental study designs are ubiquitous in the medical informatics literature, as evidenced by 34 studies in the past four years of the two informatics journals, little has been written about the benefits and limitations of the quasi-experimental approach. As we have outlined in this paper, a relative hierarchy and nomenclature of quasi-experimental study designs exist, with some designs being more likely than others to permit causal interpretations of observed associations. Strengths and limitations of a particular study design should be discussed when presenting data collected in the setting of a quasi-experimental study. Future medical informatics investigators should choose the strongest design that is feasible given the particular circumstances.

UCCS Community

• Current Students
• Faculty Staff
• Alumni & Friends
• Parents & Families

Schools and Colleges

• College of Business
• College of Education
• College of Engineering and Applied Science
• College of Letters, Arts & Sciences
• College of Public Service
• Graduate School
• Helen and Arthur E. Johnson Beth-El College of Nursing and Health Sciences

Quick Links

• Search for Programs & Careers
• Academic Advising
• Ent Center for the Arts
• Kraemer Family Library
• Military and Veteran Affairs
• myUCCS Portal
• Campus Email
• Microsoft 365
• Mountain Lion Connect
• Support Network: Students
• Support Network: Faculty
• Account Help

Effect Size Calculators

Dr. Lee A. Becker

• Content, Part 1
• Content, Part 2
• Research Tools

Statistical Analysis of Quasi-Experimental Designs:

I. apriori selection techniques.

Content, part II

I. Overview

Random assignment is used in experimental designs to help assure that different treatment groups are equivalent prior to treatment. With small n 's randomization is messy, the groups may not be equivalent on some important characteristic.

In general, matching is used when you want to make sure that members of the various groups are equivalent on one or more characteristics. If you are want to make absolutely sure that the treatment groups are equivalent on some attribute you can use matched random assignment.

When you can't randomly assign to conditions you can still use matching techniques to try to equate groups on important characteristics. This set of notes makes the distinction between normative group matching and normative group equivalence. In normative group matching you select an exact match from normative comparison group for each participant in the treatment group. In normative group equivalence you select a comparison group that has approximately equivalent characteristics to the treatment group.

II. Matching in Experimental Designs: Matched Random Assignment

In an experimental design, matched random sampling can be used to equate the groups on one or more characteristics. Whitley (in chapter 8) uses an example of matching on IQ.

The Matching Process

1. Obtain scores on the variable of interest (e.g., IQ) and rank order participants according to that score. The scores in Table 1 have been ranked according to IQ scores. 2. Take the people with the top two scores (Block 1) and randomly assign them the control and experimental conditions. Take the people with next highest scores (Block 2) and randomly assign them to the control and experimental group.  Continue until all participants have been assigned to conditions.  Expand as necessary according to the design of your study.  For example, in a 2 x 3 factorial design, take the people with the top 6 IQ scores (Block 1) and randomly assign them to each of the six cells in the design.   This procedure will assure that each of the treatment and control groups are equivalent on IQ (or whatever characteristic(s) is(are) matched).

IQ Block Condition 133 1 Tx 132 1 Ctl 130 2 Ctl 130 2 Tx 129 3 Ctl 128 3 Tx 128 4 Tx 125 4 Ctl Note: Tx = Treatment group, Ctl = Control Group. Analysis of a Matched Random Assignment Design If the matching variable is related to the dependent variable, (e.g., IQ is related to almost all studies of memory and learning), then you can incorporate the matching variable as a blocking variable in your analysis of variance. That is, in the 2 x 3 example, the first 6 participants can be entered as IQ block #1, the second 6 participants as IQ block #2. This removes the variance due to IQ from the error term, increasing the power of the study. The analysis is treated as a repeated measures design where the measures for each block of participants are considered to be repeated measures. For example, in setting up the data for a two-group design (experimental vs. control) the data would look like this: 1 132 133 27 32 2 130 130 19 17 3 129 128 30 35 4 125 128 . 37 etc          Note: tx  = Treatment Group; ctl = Control Group  The analysis would be run as a repeated measures design with group (control vs. experimental) as a within-subjects factor. If you were interested in analyzing the equivalence of the groups on the IQ score variable you could enter the IQ scores as separate variables.  An analysis of variance of  the IQ scores with treatment group (Treatment vs. Control) as a within-subjects factor should show no mean differences between the two groups. Entering the IQ data would allow you to find the correlation between IQ and performance scores within each treatment group. One of the problems with this type of analysis is that if any score is missing then the entire block is set to missing.  None of the performance data from Block 4 in Table 2 would be included in the analysis because the performance score is missing for the person in the control group. If you had a 6 cells in your design you would loose the data on all 6 people in a block that had only one missing data point. I understand that Dr. Klebe has been writing a new data analysis program to take care of this kind of missing data problem. SPSS Note  The SPSS syntax commands for running the data in Table 2 as a repeated measures analysis of variance are shown in Table 3.  The SPSS syntax commands for running the data in Table 2 as a paired t test are shown in Table 4.  GLM ps_ctl ps_tx /WSFACTOR = group 2  /EMMEANS = TABLES(group) /PRINT = DESCRIPTIVE /WSDESIGN = group . T-TEST PAIRS= ps_ctl WITH ps_tx (PAIRED). III. Matching in Quasi-Experimental Designs: Normative Group Matching Suppose that you have a quasi-experiment where you want to compare an experimental group (e.g., people who have suffered mild head injury) with a sample from a normative population. Suppose that there are several hundred people in the normative population. One strategy is to randomly select the same number of people from the normative population as you have in your experimental group. If the demographic characteristics of the normative group approximate those of your experimental group, then this process may be appropriate. But, what if the normative group contains equal numbers of males and females ranging in age from 6 to 102, and people in your experimental condition are all males ranging in age from 18 to 35? Then it is unlikely that the demographic characteristics of the people sampled from the normative group will match those of your experimental group. For that reason, simple random selection is rarely appropriate when sampling from a normative population. The Normative Group Matching Procedure Determine the relevant characteristics (e.g., age, gender, SES, etc.) of each person in your experimental group. E.g., Exp person #1 is a 27 year-old male. Then randomly select one of the 27 year-old males from the normative population as a match for Exp person #1. Exp person #2 is a 35 year-old male, then randomly select one of the 35 year-old males as a match for Exp person #2. If you have done randomize normative group matching then the matching variable should be used as a blocking factor in the ANOVA. If you have a limited number of people in the normative group then you can do caliper matching . In caliper matching you select the matching person based a range of scores, for example, you can caliper match within a range of 3 years. Exp person #1 would be randomly selected from males whose age ranged from 26 to 27 years. If you used a five year caliper for age then for exp person #1 you randomly select a males from those whose age ranged from 25 to 29 years old. You would want a narrower age caliper for children and adolescents than for adults. This procedure becomes very difficult to accomplish when you try to start matching on more than one variable. Think of the problems of finding exact matches when several variables are used, e.g., an exact match for a 27-year old, white female with an IQ score of 103 and 5 children. Analysis of a Normative Group Matching Design The analysis is the same as for a matched random assignment design. If the matching variable is related to the dependent variable, then you can incorporate the matching variable as a blocking variable in your analysis of variance. III. Matching in Quasi-Experimental Designs: Normative Group Equivalence Because of the problems in selecting people in a normative group matching design and the potential problems with the data analysis of that design, you may want to make the normative comparison group equivalent on selected demographic characteristics. You might want the same proportion of males and females, and the mean age (and SD) of the normative group should be the same as those in the experimental group. If the ages of the people in the experimental group ranged from 18 to 35, then your normative group might contain an equal number of participants randomly selected from those in the age range from 18 to 35 in the normative population. Analysis of a Normative Group Equivalence Design In the case of normative group equivalence there is no special ANOVA procedure as there is in Normative Group Matching. In general, demographic characteristics themselves rarely predict the d.v., so you haven’t lost anything by using the group equivalence method. A Semantic Caution The term "matching" implies a one-to-one matching and it implies that you have incorporated that matched variable into your ANOVA design. Please don’t use the term "matching" when you mean mere "equivalence." 13. Study design and choosing a statistical test Sample size. University of Lincoln What statistical analysis should I use for a quasi experimental design? Top contributors to discussions in this field. Portland State University Tikrit University Ain Shams University SEGi University College Ninevah University Get help with your research Join ResearchGate to ask questions, get input, and advance your work. All Answers (3) Similar questions and discussions Asked 1 May 2019 Asked 12 September 2018 Asked 6 July 2023 Asked 26 October 2022 Asked 8 March 2022 Asked 16 May 2019 Asked 27 October 2020 Asked 6 June 2017 Asked 14 March 2017 Related Publications Recruit researchers Join for free Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up Physical Review Accelerators and Beams Special Editions Editorial Team Open Access Bayesian optimization algorithms for accelerator physics Ryan roussel et al., phys. rev. accel. beams 27 , 084801 – published 6 august 2024. No Citing Articles INTRODUCTION BACKGROUND AND MOTIVATION GAUSSIAN PROCESS MODELING ACQUISITION FUNCTION DEFINITION ACQUISITION FUNCTION OPTIMIZATION ACKNOWLEDGMENTS Accelerator physics relies on numerical algorithms to solve optimization problems in online accelerator control and tasks such as experimental design and model calibration in simulations. The effectiveness of optimization algorithms in discovering ideal solutions for complex challenges with limited resources often determines the problem complexity these methods can address. The accelerator physics community has recognized the advantages of Bayesian optimization algorithms, which leverage statistical surrogate models of objective functions to effectively address complex optimization challenges, especially in the presence of noise during accelerator operation and in resource-intensive physics simulations. In this review article, we offer a conceptual overview of applying Bayesian optimization techniques toward solving optimization problems in accelerator physics. We begin by providing a straightforward explanation of the essential components that make up Bayesian optimization techniques. We then give an overview of current and previous work applying and modifying these techniques to solve accelerator physics challenges. Finally, we explore practical implementation strategies for Bayesian optimization algorithms to maximize their performance, enabling users to effectively address complex optimization challenges in real-time beam control and accelerator design. Received 9 December 2023 Accepted 3 June 2024 DOI: https://doi.org/10.1103/PhysRevAccelBeams.27.084801 Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Published by the American Physical Society Physics Subject Headings (PhySH) Research Areas Authors & Affiliations Article text. Vol. 27, Iss. 8 — August 2024 Authorization Required Other options. Buy Article » Find an Institution with the Article » Download & Share Overview of challenges in using optimization algorithms for online accelerator control. Accelerator control algorithms make decisions about setting a wide variety of accelerator parameters in order to control beam parameters at target locations. Optimal decision making takes into account limited online accelerator measurements, as well as various sources of prior knowledge about the accelerator, including previous measurements, physics simulations, and physics principals. Optimization must also consider complicated aspects of realistic accelerator operation including external conditions, feedback systems, safety constraints, and repeatability errors. Overview of optimization challenges in accelerator physics simulations. Ideal algorithms aim to minimize the computational cost of performing optimization by orchestrating parallel simulation evaluations at multiple fidelities ranging from analytical models to high-fidelity (computationally expensive) simulations. Correlations between simulation predictions at different fidelities can be leveraged to reduce the number of high-fidelity simulation evaluations needed to find an ideal solution at the highest-fidelity level. Illustration of the Bayesian optimization process to find the maximum of a simple function. A Gaussian process (GP) model makes predictions of the function value (solid blue line) along with associated uncertainties (blue shading) based on previously collected data. An acquisition function then uses the GP model to predict the “value” of making potential future measurements, balancing both exploration and exploitation. The next observation is chosen by maximizing the acquisition function in parameter space. This process is repeated iteratively until optimization goals have been reached. Illustration of Bayesian regression using a linear model f ( x ) = w 1 x + w 0 . (a,d,g) Posterior probability density of the linear weights { w 1 , w 0 } conditioned on N observations of the function y = f ( x ) + ε . (b,e,h) Model predictions using random samples of { w 1 , w 0 } drawn from the posterior probability distribution. (c,f,i) Predictive mean (solid line) and 90% uncertainty intervals (shading) of the posterior model. Red cross and black dashes denote true parameters and values of the function f ( x ) , respectively. Reproduced with permission from [ 39 ]. Illustration of GP model predictions. (a) Prior model prediction of the function mean (solid blue line) and confidence interval (blue shading) at a set of test points in parameter space. The probability of the output value y at any given test point x * is a normal distribution. (b) The posterior GP model also predicts normal probability distributions at each test point, conditioned on the dataset D . (c) Individual function samples can also be drawn from the posterior GP model and can be used for Monte Carlo computations of function quantities. Visualization of how the length scale hyperparameter l effects GP modeling. Three GP models are trained on the same dataset using a Matérn kernel with fixed length scales of (a) 0.1, (b) 1, and (c) 10. Remaining hyperparameters are trained by maximizing the marginal log-likelihood. Examples of GP modeling with varying treatment of measurement noise. (a) Shows a GP model containing zero noise, forcing the GP prediction to fit experimental data exactly. (b) Shows a GP model trained on the same data with a fixed (homoskedastic) noise parameter. (c) Illustrates a GP model incorporating heteroskedastic noise, where the data variance for each point is explicitly specified. Illustration of the improvement in prediction accuracy that can be gained by including expected correlations, such as those that arise from adjacent quadrupoles, into the GP kernel design. Here, a 2D function has input correlations that are similar to what one might observe between adjacent quadrupoles (a). For a fixed set of training data points (shown in orange), a GP model using an uncorrelated kernel (b) produces less accurate posterior predictions of the true function than a model with an accurate correlated kernel (c). In the context of BO, learning a more accurate model with fewer data training points translates to faster convergence in optimization. Illustration of nonzero prior mean. In the absence of local data, the mean of the posterior distributions reverts to (a) zero or (b) the nonzero prior mean. The variance remains unchanged. Transmission optimization at ATLAS subsection using different prior mean functions. Solid and dashed lines depict the medians and the shaded areas depict the corresponding 90% confidence levels across 10 to 20 runs. Reproduced from [ 24 ]. Example of using log transformations in GP modeling for strictly positive output values. Data in real space (a) are transformed to log space before fitting a GP model (b). Samples drawn from the GP model in log space can then be transformed back into real space to make GP predictions. The resulting likelihood in real space is then a log-normal distribution which is strictly positive. Simulated application of standard and time-aware BO in a drifting trajectory stabilization problem. Simple BO settles on the mean value of the oscillations. ABO-ISO (isotropic) follows the changes but lags them because it only uses isotropic (local) kernel. ABO-SM (spectral mixture) captures long-range correlations and eventually correctly predicts necessary future changes in phase. By default, ABO-SM continues to explore around maximum value for optimization, producing a small step jitter. It can be eliminated by using the posterior mean as the acquisition function at the cost of convergence speed. Illustration of the prediction of a multifidelity Gaussian process, by comparing (a) a single-fidelity Gaussian process trained only on high-fidelity data, and (b),(c) a multifidelity Gaussian process trained on both high-fidelity and low-fidelity data, in the case where (b) high-fidelity and low-fidelity data are highly correlated, as well as (c) high-fidelity data and low-fidelity data are largely uncorrelated. In this particular example, the multifidelity GP is a multitask GP [ 61 ], as implemented in the library b o t orch. Dashed lines denote ground truth values of the low- and high-fidelity functions. Demonstration of combining GP models with a differentiable physics model of magnetic hysteresis. (a) Measured beam charge after passing through an aperture in the APS injector is plotted over three cycles of varying the current in an upstream quadrupole. Transmitted beam charge measurements are not repeatable due to hysteresis effects in the upstream quadrupole. (b) GP modeling with differentiable hysteresis model included accurately predicts beam charge over multiple hysteresis cycles with improved (reduced) uncertainty predictions. Reproduced from [ 28 ]. Examples of the EI and UCB acquisition functions for objective function maximization given the same GP model and training data. (a) EI acquisition function, where the dashed horizontal line denotes the best previously observed value f ( x * ) . (b) UCB acquisition function. Example of sampling behavior of Bayesian exploration (BE). (a) The BE acquisition function is maximized at locations in parameter space where the model uncertainty is highest, usually at locations farthest away from previous measurements. (b) In cases where the function is less sensitive to one parameter ( x 2 in this example), the model uncertainty is smaller along that axis, resulting in less frequent sampling along that dimension. Comparison between different constrained Bayesian optimization algorithms. (a) Weighting the acquisition function by the probability of satisfying the constraining function [ 73 ]. (b) Acquisition function optimization within a safe set using MoSaOpt in exploitation mode [ 32 ] and (c) SafeOpt [ 75 ]. (d) The constraint function, where valid regions satisfy c ( x ) > 0 . Summary of multiobjective BO (MOBO) using expected hypervolume improvement (EHVI). (a) Given Pareto front P and corresponding hypervolume H , the increase in hypervolume H I due to a new measurement y is given by the shaded green area. (b) Comparison between multiobjective optimization algorithms for optimizing the AWA injector problem. NSGA-II is a standard evolutionary algorithm [ 81 ], I-NN is surrogate model assisted NSGA-II [ 54 ]. (c) Projected hypervolume after a set number of MOBO iterations with insets showing hypervolume improvement due to fill in points (i) and measurement of newly dominant points (ii). Reproduced from [ 34 ]. Visualization of the BAX process for beam steering through quadrupole magnets. (a) Experimental measurements are used to build a GP model of the horizontal beam centroid position at a downstream screen C x as a function of the quadrupole strength and steering parameter. Note that the GP model is built with a first-order polynomial kernel, constraining predictions to planar surfaces. Dashed lines denote cross sections of the GP model shown in (b). (c) The BAX acquisition function that predicts the information gained about the ideal steering current by making future measurements. Demonstration of proximal biasing effects during Bayesian exploration (BE) of the constrained TNK test problem. (a) Normal BE. (b) BE using proximal biasing with l = 0.1 . The green arrow highlights a step where a larger jump in parameter space was allowed by proximal biasing. Reproduced from [ 86 ]. One-dimensional visualization of trust region BO (TuRBO) applied to a minimization problem with the UCB acquisition function. (a)–(d) Sequential evolution of the GP model and sampling pattern. Orange circles denote objective function measurements and green circles denote the most recent sequential measurement at each step. Trust region BO (TuRBO), simplex, and UCB applied to the minimization of total losses (maximization of lifetime) at the ESRF-EBS storage ring. Adapted from [ 96 ]. Comparison of optimization performance between a local optimization algorithm (Nelder-Mead simplex), BO using the UCB acquisition function ( β = 2 ), and BO using the UCB acquisition strongly weighted toward exploration ( β = 100 ). All algorithms are initialized with a single observation at x = 0.75 and aim to minimize the objective function. (a)–(c) Observations of the objective function in parameter space for each algorithm. The dashed line denotes the true objective function. (d)–(f) Objective function values as a function of algorithm iteration. Note that simplex terminates after reaching a convergence criteria. Comparison between GP modeling of hard and soft constraining functions. (a) GP modeling of a heaviside constraining function does not accurately predict constraint values due to a single sharp feature that cannot be learned without dense sampling on either side of the constraint boundary. (b) Smooth constraining functions with a single characteristic length scale are more accurately modeled with GP modeling. Inset: Visualization of bounding box constraint function f ( x ) = max i { | | C − S i ( x ) | | } used to keep beam distributions inside an ROI, where r is the radius of a circular ROI, C is the center coordinates of the ROI, and S i are corner coordinates of a bounding box around the beam. Comparison between GP modeling of the two-dimensional sphere function f ( x 1 , x 2 ) = x 1 2 + x 2 2 with and without interpolated measurements. (a) Shows the posterior mean of the GP model with four measurements taken sequentially. (b) Shows the same four measurements taken sequentially but with interpolated points in between each measurement. Incorporating interpolated points in the dataset leads to higher modeling accuracy, leading to accurate identification of the sphere function minimum at the origin. Performance scaling with dataset size for b o t orch/ gp y t orch ( 0.9.4 / 1.11 ) libraries on a single-objective optimization run. Synthetic five-variable quadratic objective was used with Monte Carlo version of UCB acquisition function and 100 Adam optimizer iterations. GPU memory usage is only applicable to GPU runs. Sign up to receive regular email alerts from Physical Review Accelerators and Beams Reuse & Permissions It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures. Forgot your username/password? Create an account Article Lookup Paste a citation or doi, enter a citation. 218 IMAGES PPT PPT PPT Advantages Of Quasi Experimental Research Methodology of quasi-experimental Design PPT VIDEO quantitative research design Types of Quasi Experimental Research Design Methods 36 Statistical Methods in Experimental Design EXPERIMENTAL DESIGNS: TRUE AND QUASI DESIGNS Types of true Experimental Research Design COMMENTS Statistical Analysis and Application of Quasi Experiments to However, previous studies using these designs have often used suboptimal statistical methods, which may result in researchers making spurious conclusions. Methods used to analyze quasi-experimental data include 2-group tests, regression analysis, and time-series analysis, and they all have specific assumptions, data requirements, strengths, and ... Quasi-Experimental Design Revised on January 22, 2024. Like a true experiment, a quasi-experimental design aims to establish a cause-and-effect relationship between an independent and dependent variable. However, unlike a true experiment, a quasi-experiment does not rely on random assignment. Instead, subjects are assigned to groups based on non-random criteria. Quasi Experimental Design Overview & Examples A quasi experimental design is a method for identifying causal relationships that does not randomly assign participants to the experimental groups. Instead, researchers use a non-random process. For example, they might use an eligibility cutoff score or preexisting groups to determine who receives the treatment. Selecting and Improving Quasi-Experimental Designs in Effectiveness and Quasi-experimental designs (QEDs) are increasingly employed to achieve a better balance between internal and external validity. Although these designs are often referred to and summarized in terms of logistical benefits versus threats to internal validity, there is still uncertainty about: (1) how to select from among various QEDs, and (2 ... Quasi-Experimental Research Design Quasi-experimental design is a research method that seeks to evaluate the causal relationships between variables, but without the full control over the independent variable(s) that is available in a true experimental design. ... Inferential Statistics. This method involves using statistical tests to determine whether the results of a study are ... Chapter 23 Quasi-experimental 23 Quasi-experimental. 23. Quasi-experimental. In most cases, it means that you have pre- and post-intervention data. Great resources for causal inference include Causal Inference Mixtape and Recent Advances in Micro, especially if you like to read about the history of causal inference as a field as well (codes for Stata, R, and Python ... 1.5: Common Quasi-Experimental Designs 1.5: Common Quasi-Experimental Designs. Recall that when participants in a between-subjects designs are randomly assigned to treatment conditions, the resulting groups are likely to be quite similar. In fact, researchers consider them to be equivalent. When participants are not randomly assigned to conditions, however, the resulting groups are ... Quasi-experimentation: A guide to design and analysis. Design and statistical techniques for a full coverage of quasi-experimentation are collected in an accessible format, in a single volume. The book begins with a general overview of quasi-experimentation. Chapter 2 defines a treatment effect and the hurdles over which one must leap to draw credible causal inferences. 7.3 Quasi-Experimental Research Describe three different types of quasi-experimental research designs (nonequivalent groups, pretest-posttest, and interrupted time series) and identify examples of each one. The prefix quasi means "resembling.". Thus quasi-experimental research is research that resembles experimental research but is not true experimental research. How to Use and Interpret Quasi-Experimental Design A quasi-experimental study (also known as a non-randomized pre-post intervention) is a research design in which the independent variable is manipulated, but participants are not randomly assigned to conditions. Commonly used in medical informatics (a field that uses digital information to ensure better patient care), researchers generally use ... Quasi-Experimental Designs for Causal Inference This article discusses four of the strongest quasi-experimental designs for identifying causal effects: regression discontinuity design, instrumental variable design, matching and propensity score designs, and the comparative interrupted time series design. For each design we outline the strategy and assumptions for identifying a causal effect ... Statistical analysis and application of quasi experiments to Quasi-experimental study designs are frequently used to assess interventions that aim to limit the emergence of antimicrobial-resistant pathogens. However, previous studies using these designs have often used suboptimal statistical methods, which may result in researchers making spurious conclusions. Experimental and Quasi-Experimental Methods Experimental and Quasi-Experimental Methods. Research designs are central to research projects in that they constitute the projects' basic structure that will permit researchers to address their main research questions. Designs include, for example, the selection of relevant samples or groups, measures, treatments or programs, and methods of ... PDF EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH Sources of Invalidity for Designs 1 through 6 8 2. Sources of Invalidity for Quasi-Experimental Designs 7 through 12 40 3. Sources of Invalidity for Quasi-Experimental Designs 13 through 16 S6 FIGURES 1. Regression in the Prediction of Posttest Scores from Pretest, and Vice Versa 10 2. Some Possible Outcomes of a 3 X 3 Factorial Design 28 3. Quasi-experiment A quasi-experiment is an empirical interventional study used to estimate the causal impact of an intervention on target population without random assignment. Quasi-experimental research shares similarities with the traditional experimental design or randomized controlled trial, but it specifically lacks the element of random assignment to ... The Use and Interpretation of Quasi-Experimental Studies in Medical In medical informatics, the quasi-experimental, sometimes called the pre-post intervention, design often is used to evaluate the benefits of specific interventions. The increasing capacity of health care institutions to collect routine clinical data has led to the growing use of quasi-experimental study designs in the field of medical ... Quasi-Experimental Design: Types, Examples, Pros, and Cons See why leading organizations rely on MasterClass for learning & development. A quasi-experimental design can be a great option when ethical or practical concerns make true experiments impossible, but the research methodology does have its drawbacks. Learn all the ins and outs of a quasi-experimental design. Quasi-Experimentation : A Guide to Design and Analysis Featuring engaging examples from diverse disciplines, this book explains how to use modern approaches to quasi-experimentation to derive credible estimates of treatment effects under the demanding constraints of field settings. Foremost expert Charles S. Reichardt provides an in-depth examination of the design and statistical analysis of pretest-posttest, nonequivalent groups, regression ... Statistical Analysis of Quasi-Experimental Designs: III. Matching in Quasi-Experimental Designs: Normative Group Equivalence. Because of the problems in selecting people in a normative group matching design and the potential problems with the data analysis of that design, you may want to make the normative comparison group equivalent on selected demographic characteristics. You might want the same proportion of males and females, and the mean ... PDF Quasi-Experimental Evaluation Designs Quasi-experimental designs do not randomly assign participants to treatment and control groups. Quasi-experimental designs identify a comparison group that is as similar as possible to the treatment group in terms of pre-intervention (baseline) characteristics. There are different types of quasi -experimental designs and they use different ... 13. Study design and choosing a statistical test A quasi experimental design is one in which treatment allocation is not random. An example of this is given in table 9.1 in which injuries are compared in two dropping zones. This is subject to potential biases in that the reason why a person is allocated to a particular dropping zone may be related to their risk of a sprained ankle. What statistical analysis should I use for a quasi experimental design All Answers (3) First, a quasi-experimental design requires a comparisons between those who were exposed to some factor versus those in a "non-equivalent control group" who did not have any ... Bayesian optimization algorithms for accelerator physics Accelerator physics relies on numerical algorithms to solve optimization problems in online accelerator control and tasks such as experimental design and model calibration in simulations. The effectiveness of optimization algorithms in discovering ideal solutions for complex challenges with limited resources often determines the problem complexity these methods can address. The accelerator ... Versatile Method for Preparing Two-Dimensional Metal Dihalides Ever since the ground-breaking isolation of graphene, numerous two-dimensional (2D) materials have emerged with 2D metal dihalides gaining significant attention due to their intriguing electrical and magnetic properties. In this study, we introduce an innovative approach via anhydrous solvent-induced recrystallization of bulk powders to obtain crystals of metal dihalides (MX2, with M = Cu, Ni ... essay homework paper phd project resume review study thesis