experimental design quasi experimental or non experimental design

Experimental vs Quasi-Experimental Design: Which to Choose?

Here’s a table that summarizes the similarities and differences between an experimental and a quasi-experimental study design:

What is a quasi-experimental design?

A quasi-experimental design is a non-randomized study design used to evaluate the effect of an intervention. The intervention can be a training program, a policy change or a medical treatment.

Unlike a true experiment, in a quasi-experimental study the choice of who gets the intervention and who doesn’t is not randomized. Instead, the intervention can be assigned to participants according to their choosing or that of the researcher, or by using any method other than randomness.

Having a control group is not required, but if present, it provides a higher level of evidence for the relationship between the intervention and the outcome.

(for more information, I recommend my other article: Understand Quasi-Experimental Design Through an Example ) .

Examples of quasi-experimental designs include:

• One-Group Posttest Only Design
• Static-Group Comparison Design
• One-Group Pretest-Posttest Design
• Separate-Sample Pretest-Posttest Design

What is an experimental design?

An experimental design is a randomized study design used to evaluate the effect of an intervention. In its simplest form, the participants will be randomly divided into 2 groups:

• A treatment group: where participants receive the new intervention which effect we want to study.
• A control or comparison group: where participants do not receive any intervention at all (or receive some standard intervention).

Randomization ensures that each participant has the same chance of receiving the intervention. Its objective is to equalize the 2 groups, and therefore, any observed difference in the study outcome afterwards will only be attributed to the intervention – i.e. it removes confounding.

(for more information, I recommend my other article: Purpose and Limitations of Random Assignment ).

Examples of experimental designs include:

• Posttest-Only Control Group Design
• Pretest-Posttest Control Group Design
• Solomon Four-Group Design
• Matched Pairs Design
• Randomized Block Design

When to choose an experimental design over a quasi-experimental design?

Although many statistical techniques can be used to deal with confounding in a quasi-experimental study, in practice, randomization is still the best tool we have to study causal relationships.

Another problem with quasi-experiments is the natural progression of the disease or the condition under study — When studying the effect of an intervention over time, one should consider natural changes because these can be mistaken with changes in outcome that are caused by the intervention. Having a well-chosen control group helps dealing with this issue.

So, if losing the element of randomness seems like an unwise step down in the hierarchy of evidence, why would we ever want to do it?

This is what we’re going to discuss next.

When to choose a quasi-experimental design over a true experiment?

The issue with randomness is that it cannot be always achievable.

So here are some cases where using a quasi-experimental design makes more sense than using an experimental one:

• If being in one group is believed to be harmful for the participants , either because the intervention is harmful (ex. randomizing people to smoking), or the intervention has a questionable efficacy, or on the contrary it is believed to be so beneficial that it would be malevolent to put people in the control group (ex. randomizing people to receiving an operation).
• In cases where interventions act on a group of people in a given location , it becomes difficult to adequately randomize subjects (ex. an intervention that reduces pollution in a given area).
• When working with small sample sizes , as randomized controlled trials require a large sample size to account for heterogeneity among subjects (i.e. to evenly distribute confounding variables between the intervention and control groups).

Further reading

• Statistical Software Popularity in 40,582 Research Papers
• Checking the Popularity of 125 Statistical Tests and Models
• Objectives of Epidemiology (With Examples)
• 12 Famous Epidemiologists and Why

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Quasi-Experimental Design | Definition, Types & Examples

Quasi-Experimental Design | Definition, Types & Examples

Published on July 31, 2020 by Lauren Thomas . Revised on January 22, 2024.

Like a true experiment , a quasi-experimental design aims to establish a cause-and-effect relationship between an independent and dependent variable .

However, unlike a true experiment, a quasi-experiment does not rely on random assignment . Instead, subjects are assigned to groups based on non-random criteria.

Quasi-experimental design is a useful tool in situations where true experiments cannot be used for ethical or practical reasons.

Table of contents

Differences between quasi-experiments and true experiments, types of quasi-experimental designs, when to use quasi-experimental design, advantages and disadvantages, other interesting articles, frequently asked questions about quasi-experimental designs.

There are several common differences between true and quasi-experimental designs.

Example of a true experiment vs a quasi-experiment

However, for ethical reasons, the directors of the mental health clinic may not give you permission to randomly assign their patients to treatments. In this case, you cannot run a true experiment.

Instead, you can use a quasi-experimental design.

You can use these pre-existing groups to study the symptom progression of the patients treated with the new therapy versus those receiving the standard course of treatment.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Many types of quasi-experimental designs exist. Here we explain three of the most common types: nonequivalent groups design, regression discontinuity, and natural experiments.

Nonequivalent groups design

In nonequivalent group design, the researcher chooses existing groups that appear similar, but where only one of the groups experiences the treatment.

In a true experiment with random assignment , the control and treatment groups are considered equivalent in every way other than the treatment. But in a quasi-experiment where the groups are not random, they may differ in other ways—they are nonequivalent groups .

When using this kind of design, researchers try to account for any confounding variables by controlling for them in their analysis or by choosing groups that are as similar as possible.

This is the most common type of quasi-experimental design.

Regression discontinuity

Many potential treatments that researchers wish to study are designed around an essentially arbitrary cutoff, where those above the threshold receive the treatment and those below it do not.

Near this threshold, the differences between the two groups are often so minimal as to be nearly nonexistent. Therefore, researchers can use individuals just below the threshold as a control group and those just above as a treatment group.

However, since the exact cutoff score is arbitrary, the students near the threshold—those who just barely pass the exam and those who fail by a very small margin—tend to be very similar, with the small differences in their scores mostly due to random chance. You can therefore conclude that any outcome differences must come from the school they attended.

Natural experiments

In both laboratory and field experiments, researchers normally control which group the subjects are assigned to. In a natural experiment, an external event or situation (“nature”) results in the random or random-like assignment of subjects to the treatment group.

Even though some use random assignments, natural experiments are not considered to be true experiments because they are observational in nature.

Although the researchers have no control over the independent variable , they can exploit this event after the fact to study the effect of the treatment.

However, as they could not afford to cover everyone who they deemed eligible for the program, they instead allocated spots in the program based on a random lottery.

Although true experiments have higher internal validity , you might choose to use a quasi-experimental design for ethical or practical reasons.

Sometimes it would be unethical to provide or withhold a treatment on a random basis, so a true experiment is not feasible. In this case, a quasi-experiment can allow you to study the same causal relationship without the ethical issues.

The Oregon Health Study is a good example. It would be unethical to randomly provide some people with health insurance but purposely prevent others from receiving it solely for the purposes of research.

However, since the Oregon government faced financial constraints and decided to provide health insurance via lottery, studying this event after the fact is a much more ethical approach to studying the same problem.

True experimental design may be infeasible to implement or simply too expensive, particularly for researchers without access to large funding streams.

At other times, too much work is involved in recruiting and properly designing an experimental intervention for an adequate number of subjects to justify a true experiment.

In either case, quasi-experimental designs allow you to study the question by taking advantage of data that has previously been paid for or collected by others (often the government).

Quasi-experimental designs have various pros and cons compared to other types of studies.

• Higher external validity than most true experiments, because they often involve real-world interventions instead of artificial laboratory settings.
• Higher internal validity than other non-experimental types of research, because they allow you to better control for confounding variables than other types of studies do.
• Lower internal validity than true experiments—without randomization, it can be difficult to verify that all confounding variables have been accounted for.
• The use of retrospective data that has already been collected for other purposes can be inaccurate, incomplete or difficult to access.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

A quasi-experiment is a type of research design that attempts to establish a cause-and-effect relationship. The main difference with a true experiment is that the groups are not randomly assigned.

In experimental research, random assignment is a way of placing participants from your sample into different groups using randomization. With this method, every member of the sample has a known or equal chance of being placed in a control group or an experimental group.

Quasi-experimental design is most useful in situations where it would be unethical or impractical to run a true experiment .

Quasi-experiments have lower internal validity than true experiments, but they often have higher external validity  as they can use real-world interventions instead of artificial laboratory settings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Thomas, L. (2024, January 22). Quasi-Experimental Design | Definition, Types & Examples. Scribbr. Retrieved September 18, 2024, from https://www.scribbr.com/methodology/quasi-experimental-design/

Is this article helpful?

Lauren Thomas

Other students also liked, guide to experimental design | overview, steps, & examples, random assignment in experiments | introduction & examples, control variables | what are they & why do they matter, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Quantitative Research Designs: Non-Experimental vs. Experimental

While there are many types of quantitative research designs, they generally fall under one of three umbrellas: experimental research, quasi-experimental research, and non-experimental research.

Experimental research designs are what many people think of when they think of research; they typically involve the manipulation of variables and random assignment of participants to conditions. A traditional experiment may involve the comparison of a control group to an experimental group who receives a treatment (i.e., a variable is manipulated). When done correctly, experimental designs can provide evidence for cause and effect. Because of their ability to determine causation, experimental designs are the gold-standard for research in medicine, biology, and so on. However, such designs can also be used in the “soft sciences,” like social science. Experimental research has strict standards for control within the research design and for establishing validity. These designs may also be very resource and labor intensive. Additionally, it can be hard to justify the generalizability of the results in a very tightly controlled or artificial experimental setting. However, if done well, experimental research methods can lead to some very convincing and interesting results.

Need help with your research?

Schedule a time to speak with an expert using the calendar below.

Non-experimental research, on the other hand, can be just as interesting, but you cannot draw the same conclusions from it as you can with experimental research. Non-experimental research is usually descriptive or correlational, which means that you are either describing a situation or phenomenon simply as it stands, or you are describing a relationship between two or more variables, all without any interference from the researcher. This means that you do not manipulate any variables (e.g., change the conditions that an experimental group undergoes) or randomly assign participants to a control or treatment group. Without this level of control, you cannot determine any causal effects. While validity is still a concern in non-experimental research, the concerns are more about the validity of the measurements, rather than the validity of the effects.

Finally, a quasi-experimental design is a combination of the two designs described above. For quasi-experimental designs you still can manipulate a variable in the experimental group, but there is no random assignment into groups. Quasi-experimental designs are the most common when the researcher uses a convenience sample to recruit participants. For example, let’s say you were interested in studying the effect of stress on student test scores at the school that you work for. You teach two separate classes so you decide to just use each class as a different group. Class A becomes the experimental group who experiences the stressor manipulation and class B becomes the control group. Because you are sampling from two different pre-existing groups, without any random assignment, this would be known as a quasi-experimental design. These types of designs are very useful for when you want to find a causal relationship between variables but cannot randomly assign people to groups for practical or ethical reasons, such as working with a population of clinically depressed people or looking for gender differences (we can’t randomly assign people to be clinically depressed or to be a different gender). While these types of studies sometimes have higher external validity than a true experimental design, since they involve real world interventions and group rather than a laboratory setting, because of the lack of random assignment in these groups, the generalizability of the study is severely limited.

So, how do you choose between these designs? This will depend on your topic, your available resources, and desired goal. For example, do you want to see if a particular intervention relieves feelings of anxiety? The most convincing results for that would come from a true experimental design with random sampling and random assignment to groups. Ultimately, this is a decision that should be made in close collaboration with your advisor. Therefore, I recommend discussing the pros and cons of each type of research, what it might mean for your personal dissertation process, and what is required of each design before making a decision.

Take the Course: Experimental and Quasi-Experimental Research Design

• Skip to secondary menu
• Skip to main content
• Skip to primary sidebar

Statistics By Jim

Making statistics intuitive

Quasi Experimental Design Overview & Examples

By Jim Frost Leave a Comment

What is a Quasi Experimental Design?

A quasi experimental design is a method for identifying causal relationships that does not randomly assign participants to the experimental groups. Instead, researchers use a non-random process. For example, they might use an eligibility cutoff score or preexisting groups to determine who receives the treatment.

Quasi-experimental research is a design that closely resembles experimental research but is different. The term “quasi” means “resembling,” so you can think of it as a cousin to actual experiments. In these studies, researchers can manipulate an independent variable — that is, they change one factor to see what effect it has. However, unlike true experimental research, participants are not randomly assigned to different groups.

Learn more about Experimental Designs: Definition & Types .

When to Use Quasi-Experimental Design

Researchers typically use a quasi-experimental design because they can’t randomize due to practical or ethical concerns. For example:

• Practical Constraints : A school interested in testing a new teaching method can only implement it in preexisting classes and cannot randomly assign students.
• Ethical Concerns : A medical study might not be able to randomly assign participants to a treatment group for an experimental medication when they are already taking a proven drug.

Quasi-experimental designs also come in handy when researchers want to study the effects of naturally occurring events, like policy changes or environmental shifts, where they can’t control who is exposed to the treatment.

Quasi-experimental designs occupy a unique position in the spectrum of research methodologies, sitting between observational studies and true experiments. This middle ground offers a blend of both worlds, addressing some limitations of purely observational studies while navigating the constraints often accompanying true experiments.

A significant advantage of quasi-experimental research over purely observational studies and correlational research is that it addresses the issue of directionality, determining which variable is the cause and which is the effect. In quasi-experiments, an intervention typically occurs during the investigation, and the researchers record outcomes before and after it, increasing the confidence that it causes the observed changes.

However, it’s crucial to recognize its limitations as well. Controlling confounding variables is a larger concern for a quasi-experimental design than a true experiment because it lacks random assignment.

In sum, quasi-experimental designs offer a valuable research approach when random assignment is not feasible, providing a more structured and controlled framework than observational studies while acknowledging and attempting to address potential confounders.

Types of Quasi-Experimental Designs and Examples

Quasi-experimental studies use various methods, depending on the scenario.

Natural Experiments

This design uses naturally occurring events or changes to create the treatment and control groups. Researchers compare outcomes between those whom the event affected and those it did not affect. Analysts use statistical controls to account for confounders that the researchers must also measure.

Natural experiments are related to observational studies, but they allow for a clearer causality inference because the external event or policy change provides both a form of quasi-random group assignment and a definite start date for the intervention.

For example, in a natural experiment utilizing a quasi-experimental design, researchers study the impact of a significant economic policy change on small business growth. The policy is implemented in one state but not in neighboring states. This scenario creates an unplanned experimental setup, where the state with the new policy serves as the treatment group, and the neighboring states act as the control group.

Researchers are primarily interested in small business growth rates but need to record various confounders that can impact growth rates. Hence, they record state economic indicators, investment levels, and employment figures. By recording these metrics across the states, they can include them in the model as covariates and control them statistically. This method allows researchers to estimate differences in small business growth due to the policy itself, separate from the various confounders.

Nonequivalent Groups Design

This method involves matching existing groups that are similar but not identical. Researchers attempt to find groups that are as equivalent as possible, particularly for factors likely to affect the outcome.

For instance, researchers use a nonequivalent groups quasi-experimental design to evaluate the effectiveness of a new teaching method in improving students’ mathematics performance. A school district considering the teaching method is planning the study. Students are already divided into schools, preventing random assignment.

The researchers matched two schools with similar demographics, baseline academic performance, and resources. The school using the traditional methodology is the control, while the other uses the new approach. Researchers are evaluating differences in educational outcomes between the two methods.

They perform a pretest to identify differences between the schools that might affect the outcome and include them as covariates to control for confounding. They also record outcomes before and after the intervention to have a larger context for the changes they observe.

Regression Discontinuity

This process assigns subjects to a treatment or control group based on a predetermined cutoff point (e.g., a test score). The analysis primarily focuses on participants near the cutoff point, as they are likely similar except for the treatment received. By comparing participants just above and below the cutoff, the design controls for confounders that vary smoothly around the cutoff.

For example, in a regression discontinuity quasi-experimental design focusing on a new medical treatment for depression, researchers use depression scores as the cutoff point. Individuals with depression scores just above a certain threshold are assigned to receive the latest treatment, while those just below the threshold do not receive it. This method creates two closely matched groups: one that barely qualifies for treatment and one that barely misses out.

By comparing the mental health outcomes of these two groups over time, researchers can assess the effectiveness of the new treatment. The assumption is that the only significant difference between the groups is whether they received the treatment, thereby isolating its impact on depression outcomes.

Controlling Confounders in a Quasi-Experimental Design

Accounting for confounding variables is a challenging but essential task for a quasi-experimental design.

In a true experiment, the random assignment process equalizes confounders across the groups to nullify their overall effect. It’s the gold standard because it works on all confounders, known and unknown.

Unfortunately, the lack of random assignment can allow differences between the groups to exist before the intervention. These confounding factors might ultimately explain the results rather than the intervention.

Consequently, researchers must use other methods to equalize the groups roughly using matching and cutoff values or statistically adjust for preexisting differences they measure to reduce the impact of confounders.

A key strength of quasi-experiments is their frequent use of “pre-post testing.” This approach involves conducting initial tests before collecting data to check for preexisting differences between groups that could impact the study’s outcome. By identifying these variables early on and including them as covariates, researchers can more effectively control potential confounders in their statistical analysis.

Additionally, researchers frequently track outcomes before and after the intervention to better understand the context for changes they observe.

Statisticians consider these methods to be less effective than randomization. Hence, quasi-experiments fall somewhere in the middle when it comes to internal validity , or how well the study can identify causal relationships versus mere correlation . They’re more conclusive than correlational studies but not as solid as true experiments.

In conclusion, quasi-experimental designs offer researchers a versatile and practical approach when random assignment is not feasible. This methodology bridges the gap between controlled experiments and observational studies, providing a valuable tool for investigating cause-and-effect relationships in real-world settings. Researchers can address ethical and logistical constraints by understanding and leveraging the different types of quasi-experimental designs while still obtaining insightful and meaningful results.

Cook, T. D., & Campbell, D. T. (1979).  Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin

Share this:

Reader Interactions

Comments and questions cancel reply.

• Privacy Policy

Home » Quasi-Experimental Research Design – Types, Methods

Quasi-Experimental Research Design – Types, Methods

Table of Contents

Quasi-Experimental Design

Quasi-experimental design is a research method that seeks to evaluate the causal relationships between variables, but without the full control over the independent variable(s) that is available in a true experimental design.

In a quasi-experimental design, the researcher uses an existing group of participants that is not randomly assigned to the experimental and control groups. Instead, the groups are selected based on pre-existing characteristics or conditions, such as age, gender, or the presence of a certain medical condition.

Types of Quasi-Experimental Design

There are several types of quasi-experimental designs that researchers use to study causal relationships between variables. Here are some of the most common types:

Non-Equivalent Control Group Design

This design involves selecting two groups of participants that are similar in every way except for the independent variable(s) that the researcher is testing. One group receives the treatment or intervention being studied, while the other group does not. The two groups are then compared to see if there are any significant differences in the outcomes.

Interrupted Time-Series Design

This design involves collecting data on the dependent variable(s) over a period of time, both before and after an intervention or event. The researcher can then determine whether there was a significant change in the dependent variable(s) following the intervention or event.

Pretest-Posttest Design

This design involves measuring the dependent variable(s) before and after an intervention or event, but without a control group. This design can be useful for determining whether the intervention or event had an effect, but it does not allow for control over other factors that may have influenced the outcomes.

Regression Discontinuity Design

This design involves selecting participants based on a specific cutoff point on a continuous variable, such as a test score. Participants on either side of the cutoff point are then compared to determine whether the intervention or event had an effect.

Natural Experiments

This design involves studying the effects of an intervention or event that occurs naturally, without the researcher’s intervention. For example, a researcher might study the effects of a new law or policy that affects certain groups of people. This design is useful when true experiments are not feasible or ethical.

Data Analysis Methods

Here are some data analysis methods that are commonly used in quasi-experimental designs:

Descriptive Statistics

This method involves summarizing the data collected during a study using measures such as mean, median, mode, range, and standard deviation. Descriptive statistics can help researchers identify trends or patterns in the data, and can also be useful for identifying outliers or anomalies.

Inferential Statistics

This method involves using statistical tests to determine whether the results of a study are statistically significant. Inferential statistics can help researchers make generalizations about a population based on the sample data collected during the study. Common statistical tests used in quasi-experimental designs include t-tests, ANOVA, and regression analysis.

Propensity Score Matching

This method is used to reduce bias in quasi-experimental designs by matching participants in the intervention group with participants in the control group who have similar characteristics. This can help to reduce the impact of confounding variables that may affect the study’s results.

Difference-in-differences Analysis

This method is used to compare the difference in outcomes between two groups over time. Researchers can use this method to determine whether a particular intervention has had an impact on the target population over time.

Interrupted Time Series Analysis

This method is used to examine the impact of an intervention or treatment over time by comparing data collected before and after the intervention or treatment. This method can help researchers determine whether an intervention had a significant impact on the target population.

Regression Discontinuity Analysis

This method is used to compare the outcomes of participants who fall on either side of a predetermined cutoff point. This method can help researchers determine whether an intervention had a significant impact on the target population.

Steps in Quasi-Experimental Design

Here are the general steps involved in conducting a quasi-experimental design:

• Identify the research question: Determine the research question and the variables that will be investigated.
• Choose the design: Choose the appropriate quasi-experimental design to address the research question. Examples include the pretest-posttest design, non-equivalent control group design, regression discontinuity design, and interrupted time series design.
• Select the participants: Select the participants who will be included in the study. Participants should be selected based on specific criteria relevant to the research question.
• Measure the variables: Measure the variables that are relevant to the research question. This may involve using surveys, questionnaires, tests, or other measures.
• Implement the intervention or treatment: Implement the intervention or treatment to the participants in the intervention group. This may involve training, education, counseling, or other interventions.
• Collect data: Collect data on the dependent variable(s) before and after the intervention. Data collection may also include collecting data on other variables that may impact the dependent variable(s).
• Analyze the data: Analyze the data collected to determine whether the intervention had a significant impact on the dependent variable(s).
• Draw conclusions: Draw conclusions about the relationship between the independent and dependent variables. If the results suggest a causal relationship, then appropriate recommendations may be made based on the findings.

Quasi-Experimental Design Examples

Here are some examples of real-time quasi-experimental designs:

• Evaluating the impact of a new teaching method: In this study, a group of students are taught using a new teaching method, while another group is taught using the traditional method. The test scores of both groups are compared before and after the intervention to determine whether the new teaching method had a significant impact on student performance.
• Assessing the effectiveness of a public health campaign: In this study, a public health campaign is launched to promote healthy eating habits among a targeted population. The behavior of the population is compared before and after the campaign to determine whether the intervention had a significant impact on the target behavior.
• Examining the impact of a new medication: In this study, a group of patients is given a new medication, while another group is given a placebo. The outcomes of both groups are compared to determine whether the new medication had a significant impact on the targeted health condition.
• Evaluating the effectiveness of a job training program : In this study, a group of unemployed individuals is enrolled in a job training program, while another group is not enrolled in any program. The employment rates of both groups are compared before and after the intervention to determine whether the training program had a significant impact on the employment rates of the participants.
• Assessing the impact of a new policy : In this study, a new policy is implemented in a particular area, while another area does not have the new policy. The outcomes of both areas are compared before and after the intervention to determine whether the new policy had a significant impact on the targeted behavior or outcome.

Applications of Quasi-Experimental Design

Here are some applications of quasi-experimental design:

• Educational research: Quasi-experimental designs are used to evaluate the effectiveness of educational interventions, such as new teaching methods, technology-based learning, or educational policies.
• Health research: Quasi-experimental designs are used to evaluate the effectiveness of health interventions, such as new medications, public health campaigns, or health policies.
• Social science research: Quasi-experimental designs are used to investigate the impact of social interventions, such as job training programs, welfare policies, or criminal justice programs.
• Business research: Quasi-experimental designs are used to evaluate the impact of business interventions, such as marketing campaigns, new products, or pricing strategies.
• Environmental research: Quasi-experimental designs are used to evaluate the impact of environmental interventions, such as conservation programs, pollution control policies, or renewable energy initiatives.

When to use Quasi-Experimental Design

Here are some situations where quasi-experimental designs may be appropriate:

• When the research question involves investigating the effectiveness of an intervention, policy, or program : In situations where it is not feasible or ethical to randomly assign participants to intervention and control groups, quasi-experimental designs can be used to evaluate the impact of the intervention on the targeted outcome.
• When the sample size is small: In situations where the sample size is small, it may be difficult to randomly assign participants to intervention and control groups. Quasi-experimental designs can be used to investigate the impact of an intervention without requiring a large sample size.
• When the research question involves investigating a naturally occurring event : In some situations, researchers may be interested in investigating the impact of a naturally occurring event, such as a natural disaster or a major policy change. Quasi-experimental designs can be used to evaluate the impact of the event on the targeted outcome.
• When the research question involves investigating a long-term intervention: In situations where the intervention or program is long-term, it may be difficult to randomly assign participants to intervention and control groups for the entire duration of the intervention. Quasi-experimental designs can be used to evaluate the impact of the intervention over time.
• When the research question involves investigating the impact of a variable that cannot be manipulated : In some situations, it may not be possible or ethical to manipulate a variable of interest. Quasi-experimental designs can be used to investigate the relationship between the variable and the targeted outcome.

Purpose of Quasi-Experimental Design

The purpose of quasi-experimental design is to investigate the causal relationship between two or more variables when it is not feasible or ethical to conduct a randomized controlled trial (RCT). Quasi-experimental designs attempt to emulate the randomized control trial by mimicking the control group and the intervention group as much as possible.

The key purpose of quasi-experimental design is to evaluate the impact of an intervention, policy, or program on a targeted outcome while controlling for potential confounding factors that may affect the outcome. Quasi-experimental designs aim to answer questions such as: Did the intervention cause the change in the outcome? Would the outcome have changed without the intervention? And was the intervention effective in achieving its intended goals?

Quasi-experimental designs are useful in situations where randomized controlled trials are not feasible or ethical. They provide researchers with an alternative method to evaluate the effectiveness of interventions, policies, and programs in real-life settings. Quasi-experimental designs can also help inform policy and practice by providing valuable insights into the causal relationships between variables.

Overall, the purpose of quasi-experimental design is to provide a rigorous method for evaluating the impact of interventions, policies, and programs while controlling for potential confounding factors that may affect the outcome.

Advantages of Quasi-Experimental Design

Quasi-experimental designs have several advantages over other research designs, such as:

• Greater external validity : Quasi-experimental designs are more likely to have greater external validity than laboratory experiments because they are conducted in naturalistic settings. This means that the results are more likely to generalize to real-world situations.
• Ethical considerations: Quasi-experimental designs often involve naturally occurring events, such as natural disasters or policy changes. This means that researchers do not need to manipulate variables, which can raise ethical concerns.
• More practical: Quasi-experimental designs are often more practical than experimental designs because they are less expensive and easier to conduct. They can also be used to evaluate programs or policies that have already been implemented, which can save time and resources.
• No random assignment: Quasi-experimental designs do not require random assignment, which can be difficult or impossible in some cases, such as when studying the effects of a natural disaster. This means that researchers can still make causal inferences, although they must use statistical techniques to control for potential confounding variables.
• Greater generalizability : Quasi-experimental designs are often more generalizable than experimental designs because they include a wider range of participants and conditions. This can make the results more applicable to different populations and settings.

Limitations of Quasi-Experimental Design

There are several limitations associated with quasi-experimental designs, which include:

• Lack of Randomization: Quasi-experimental designs do not involve randomization of participants into groups, which means that the groups being studied may differ in important ways that could affect the outcome of the study. This can lead to problems with internal validity and limit the ability to make causal inferences.
• Selection Bias: Quasi-experimental designs may suffer from selection bias because participants are not randomly assigned to groups. Participants may self-select into groups or be assigned based on pre-existing characteristics, which may introduce bias into the study.
• History and Maturation: Quasi-experimental designs are susceptible to history and maturation effects, where the passage of time or other events may influence the outcome of the study.
• Lack of Control: Quasi-experimental designs may lack control over extraneous variables that could influence the outcome of the study. This can limit the ability to draw causal inferences from the study.
• Limited Generalizability: Quasi-experimental designs may have limited generalizability because the results may only apply to the specific population and context being studied.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Survey Research – Types, Methods, Examples

Phenomenology – Methods, Examples and Guide

Focus Groups – Steps, Examples and Guide

Basic Research – Types, Methods and Examples

Experimental Design – Types, Methods, Guide

Transformative Design – Methods, Types, Guide

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7.3 Quasi-Experimental Research

Learning objectives.

• Explain what quasi-experimental research is and distinguish it clearly from both experimental and correlational research.
• Describe three different types of quasi-experimental research designs (nonequivalent groups, pretest-posttest, and interrupted time series) and identify examples of each one.

The prefix quasi means “resembling.” Thus quasi-experimental research is research that resembles experimental research but is not true experimental research. Although the independent variable is manipulated, participants are not randomly assigned to conditions or orders of conditions (Cook & Campbell, 1979). Because the independent variable is manipulated before the dependent variable is measured, quasi-experimental research eliminates the directionality problem. But because participants are not randomly assigned—making it likely that there are other differences between conditions—quasi-experimental research does not eliminate the problem of confounding variables. In terms of internal validity, therefore, quasi-experiments are generally somewhere between correlational studies and true experiments.

Quasi-experiments are most likely to be conducted in field settings in which random assignment is difficult or impossible. They are often conducted to evaluate the effectiveness of a treatment—perhaps a type of psychotherapy or an educational intervention. There are many different kinds of quasi-experiments, but we will discuss just a few of the most common ones here.

Nonequivalent Groups Design

Recall that when participants in a between-subjects experiment are randomly assigned to conditions, the resulting groups are likely to be quite similar. In fact, researchers consider them to be equivalent. When participants are not randomly assigned to conditions, however, the resulting groups are likely to be dissimilar in some ways. For this reason, researchers consider them to be nonequivalent. A nonequivalent groups design , then, is a between-subjects design in which participants have not been randomly assigned to conditions.

Imagine, for example, a researcher who wants to evaluate a new method of teaching fractions to third graders. One way would be to conduct a study with a treatment group consisting of one class of third-grade students and a control group consisting of another class of third-grade students. This would be a nonequivalent groups design because the students are not randomly assigned to classes by the researcher, which means there could be important differences between them. For example, the parents of higher achieving or more motivated students might have been more likely to request that their children be assigned to Ms. Williams’s class. Or the principal might have assigned the “troublemakers” to Mr. Jones’s class because he is a stronger disciplinarian. Of course, the teachers’ styles, and even the classroom environments, might be very different and might cause different levels of achievement or motivation among the students. If at the end of the study there was a difference in the two classes’ knowledge of fractions, it might have been caused by the difference between the teaching methods—but it might have been caused by any of these confounding variables.

Of course, researchers using a nonequivalent groups design can take steps to ensure that their groups are as similar as possible. In the present example, the researcher could try to select two classes at the same school, where the students in the two classes have similar scores on a standardized math test and the teachers are the same sex, are close in age, and have similar teaching styles. Taking such steps would increase the internal validity of the study because it would eliminate some of the most important confounding variables. But without true random assignment of the students to conditions, there remains the possibility of other important confounding variables that the researcher was not able to control.

Pretest-Posttest Design

In a pretest-posttest design , the dependent variable is measured once before the treatment is implemented and once after it is implemented. Imagine, for example, a researcher who is interested in the effectiveness of an antidrug education program on elementary school students’ attitudes toward illegal drugs. The researcher could measure the attitudes of students at a particular elementary school during one week, implement the antidrug program during the next week, and finally, measure their attitudes again the following week. The pretest-posttest design is much like a within-subjects experiment in which each participant is tested first under the control condition and then under the treatment condition. It is unlike a within-subjects experiment, however, in that the order of conditions is not counterbalanced because it typically is not possible for a participant to be tested in the treatment condition first and then in an “untreated” control condition.

If the average posttest score is better than the average pretest score, then it makes sense to conclude that the treatment might be responsible for the improvement. Unfortunately, one often cannot conclude this with a high degree of certainty because there may be other explanations for why the posttest scores are better. One category of alternative explanations goes under the name of history . Other things might have happened between the pretest and the posttest. Perhaps an antidrug program aired on television and many of the students watched it, or perhaps a celebrity died of a drug overdose and many of the students heard about it. Another category of alternative explanations goes under the name of maturation . Participants might have changed between the pretest and the posttest in ways that they were going to anyway because they are growing and learning. If it were a yearlong program, participants might become less impulsive or better reasoners and this might be responsible for the change.

Another alternative explanation for a change in the dependent variable in a pretest-posttest design is regression to the mean . This refers to the statistical fact that an individual who scores extremely on a variable on one occasion will tend to score less extremely on the next occasion. For example, a bowler with a long-term average of 150 who suddenly bowls a 220 will almost certainly score lower in the next game. Her score will “regress” toward her mean score of 150. Regression to the mean can be a problem when participants are selected for further study because of their extreme scores. Imagine, for example, that only students who scored especially low on a test of fractions are given a special training program and then retested. Regression to the mean all but guarantees that their scores will be higher even if the training program has no effect. A closely related concept—and an extremely important one in psychological research—is spontaneous remission . This is the tendency for many medical and psychological problems to improve over time without any form of treatment. The common cold is a good example. If one were to measure symptom severity in 100 common cold sufferers today, give them a bowl of chicken soup every day, and then measure their symptom severity again in a week, they would probably be much improved. This does not mean that the chicken soup was responsible for the improvement, however, because they would have been much improved without any treatment at all. The same is true of many psychological problems. A group of severely depressed people today is likely to be less depressed on average in 6 months. In reviewing the results of several studies of treatments for depression, researchers Michael Posternak and Ivan Miller found that participants in waitlist control conditions improved an average of 10 to 15% before they received any treatment at all (Posternak & Miller, 2001). Thus one must generally be very cautious about inferring causality from pretest-posttest designs.

Does Psychotherapy Work?

Early studies on the effectiveness of psychotherapy tended to use pretest-posttest designs. In a classic 1952 article, researcher Hans Eysenck summarized the results of 24 such studies showing that about two thirds of patients improved between the pretest and the posttest (Eysenck, 1952). But Eysenck also compared these results with archival data from state hospital and insurance company records showing that similar patients recovered at about the same rate without receiving psychotherapy. This suggested to Eysenck that the improvement that patients showed in the pretest-posttest studies might be no more than spontaneous remission. Note that Eysenck did not conclude that psychotherapy was ineffective. He merely concluded that there was no evidence that it was, and he wrote of “the necessity of properly planned and executed experimental studies into this important field” (p. 323). You can read the entire article here:

http://psychclassics.yorku.ca/Eysenck/psychotherapy.htm

Fortunately, many other researchers took up Eysenck’s challenge, and by 1980 hundreds of experiments had been conducted in which participants were randomly assigned to treatment and control conditions, and the results were summarized in a classic book by Mary Lee Smith, Gene Glass, and Thomas Miller (Smith, Glass, & Miller, 1980). They found that overall psychotherapy was quite effective, with about 80% of treatment participants improving more than the average control participant. Subsequent research has focused more on the conditions under which different types of psychotherapy are more or less effective.

In a classic 1952 article, researcher Hans Eysenck pointed out the shortcomings of the simple pretest-posttest design for evaluating the effectiveness of psychotherapy.

Wikimedia Commons – CC BY-SA 3.0.

Interrupted Time Series Design

A variant of the pretest-posttest design is the interrupted time-series design . A time series is a set of measurements taken at intervals over a period of time. For example, a manufacturing company might measure its workers’ productivity each week for a year. In an interrupted time series-design, a time series like this is “interrupted” by a treatment. In one classic example, the treatment was the reduction of the work shifts in a factory from 10 hours to 8 hours (Cook & Campbell, 1979). Because productivity increased rather quickly after the shortening of the work shifts, and because it remained elevated for many months afterward, the researcher concluded that the shortening of the shifts caused the increase in productivity. Notice that the interrupted time-series design is like a pretest-posttest design in that it includes measurements of the dependent variable both before and after the treatment. It is unlike the pretest-posttest design, however, in that it includes multiple pretest and posttest measurements.

Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows data from a hypothetical interrupted time-series study. The dependent variable is the number of student absences per week in a research methods course. The treatment is that the instructor begins publicly taking attendance each day so that students know that the instructor is aware of who is present and who is absent. The top panel of Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows how the data might look if this treatment worked. There is a consistently high number of absences before the treatment, and there is an immediate and sustained drop in absences after the treatment. The bottom panel of Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows how the data might look if this treatment did not work. On average, the number of absences after the treatment is about the same as the number before. This figure also illustrates an advantage of the interrupted time-series design over a simpler pretest-posttest design. If there had been only one measurement of absences before the treatment at Week 7 and one afterward at Week 8, then it would have looked as though the treatment were responsible for the reduction. The multiple measurements both before and after the treatment suggest that the reduction between Weeks 7 and 8 is nothing more than normal week-to-week variation.

Figure 7.5 A Hypothetical Interrupted Time-Series Design

The top panel shows data that suggest that the treatment caused a reduction in absences. The bottom panel shows data that suggest that it did not.

Combination Designs

A type of quasi-experimental design that is generally better than either the nonequivalent groups design or the pretest-posttest design is one that combines elements of both. There is a treatment group that is given a pretest, receives a treatment, and then is given a posttest. But at the same time there is a control group that is given a pretest, does not receive the treatment, and then is given a posttest. The question, then, is not simply whether participants who receive the treatment improve but whether they improve more than participants who do not receive the treatment.

Imagine, for example, that students in one school are given a pretest on their attitudes toward drugs, then are exposed to an antidrug program, and finally are given a posttest. Students in a similar school are given the pretest, not exposed to an antidrug program, and finally are given a posttest. Again, if students in the treatment condition become more negative toward drugs, this could be an effect of the treatment, but it could also be a matter of history or maturation. If it really is an effect of the treatment, then students in the treatment condition should become more negative than students in the control condition. But if it is a matter of history (e.g., news of a celebrity drug overdose) or maturation (e.g., improved reasoning), then students in the two conditions would be likely to show similar amounts of change. This type of design does not completely eliminate the possibility of confounding variables, however. Something could occur at one of the schools but not the other (e.g., a student drug overdose), so students at the first school would be affected by it while students at the other school would not.

Finally, if participants in this kind of design are randomly assigned to conditions, it becomes a true experiment rather than a quasi experiment. In fact, it is the kind of experiment that Eysenck called for—and that has now been conducted many times—to demonstrate the effectiveness of psychotherapy.

Key Takeaways

• Quasi-experimental research involves the manipulation of an independent variable without the random assignment of participants to conditions or orders of conditions. Among the important types are nonequivalent groups designs, pretest-posttest, and interrupted time-series designs.
• Quasi-experimental research eliminates the directionality problem because it involves the manipulation of the independent variable. It does not eliminate the problem of confounding variables, however, because it does not involve random assignment to conditions. For these reasons, quasi-experimental research is generally higher in internal validity than correlational studies but lower than true experiments.
• Practice: Imagine that two college professors decide to test the effect of giving daily quizzes on student performance in a statistics course. They decide that Professor A will give quizzes but Professor B will not. They will then compare the performance of students in their two sections on a common final exam. List five other variables that might differ between the two sections that could affect the results.

Discussion: Imagine that a group of obese children is recruited for a study in which their weight is measured, then they participate for 3 months in a program that encourages them to be more active, and finally their weight is measured again. Explain how each of the following might affect the results:

• regression to the mean
• spontaneous remission

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin.

Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16 , 319–324.

Posternak, M. A., & Miller, I. (2001). Untreated short-term course of major depression: A meta-analysis of studies using outcomes from studies using wait-list control groups. Journal of Affective Disorders, 66 , 139–146.

Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy . Baltimore, MD: Johns Hopkins University Press.

Research Methods in Psychology Copyright © 2016 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

5 Chapter 5: Experimental and Quasi-Experimental Designs

Case stu dy: the impact of teen court.

Research Study

An Experimental Evaluation of Teen Courts 1

Research Question

Is teen court more effective at reducing recidivism and improving attitudes than traditional juvenile justice processing?

Methodology

Researchers randomly assigned 168 juvenile offenders ages 11 to 17 from four different counties in Maryland to either teen court as experimental group members or to traditional juvenile justice processing as control group members. (Note: Discussion on the technical aspects of experimental designs, including random assignment, is found in detail later in this chapter.) Of the 168 offenders, 83 were assigned to teen court and 85 were assigned to regular juvenile justice processing through random assignment. Of the 83 offenders assigned to the teen court experimental group, only 56 (67%) agreed to participate in the study. Of the 85 youth randomly assigned to normal juvenile justice processing, only 51 (60%) agreed to participate in the study.

Upon assignment to teen court or regular juvenile justice processing, all offenders entered their respective sanction. Approximately four months later, offenders in both the experimental group (teen court) and the control group (regular juvenile justice processing) were asked to complete a post-test survey inquiring about a variety of behaviors (frequency of drug use, delinquent behavior, variety of drug use) and attitudinal measures (social skills, rebelliousness, neighborhood attachment, belief in conventional rules, and positive self-concept). The study researchers also collected official re-arrest data for 18 months starting at the time of offender referral to juvenile justice authorities.

Teen court participants self-reported higher levels of delinquency than those processed through regular juvenile justice processing. According to official re-arrests, teen court youth were re-arrested at a higher rate and incurred a higher average number of total arrests than the control group. Teen court offenders also reported significantly lower scores on survey items designed to measure their �belief in conventional rules� compared to offenders processed through regular juvenile justice avenues. Other attitudinal and opinion measures did not differ significantly between the experimental and control group members based on their post-test responses. In sum, those youth randomly assigned to teen court fared worse than control group members who were not randomly assigned to teen court.

Limitations with the Study Procedure

Limitations are inherent in any research study and those research efforts that utilize experimental designs are no exception. It is important to consider the potential impact that a limitation of the study procedure could have on the results of the study.

In the current study, one potential limitation is that teen courts from four different counties in Maryland were utilized. Because of the diversity in teen court sites, it is possible that there were differences in procedure between the four teen courts and such differences could have impacted the outcomes of this study. For example, perhaps staff members at one teen court were more punishment-oriented than staff members at the other county teen courts. This philosophical difference may have affected treatment delivery and hence experimental group members� belief in conventional attitudes and recidivism. Although the researchers monitored each teen court to help ensure treatment consistency between study sites, it is possible that differences existed in the day-to-day operation of the teen courts that may have affected participant outcomes. This same limitation might also apply to control group members who were sanctioned with regular juvenile justice processing in four different counties.

A researcher must also consider the potential for differences between the experimental and control group members. Although the offenders were randomly assigned to the experimental or control group, and the assumption is that the groups were equivalent to each other prior to program participation, the researchers in this study were only able to compare the experimental and control groups on four variables: age, school grade, gender, and race. It is possible that the experimental and control group members differed by chance on one or more factors not measured or available to the researchers. For example, perhaps a large number of teen court members experienced problems at home that can explain their more dismal post-test results compared to control group members without such problems. A larger sample of juvenile offenders would likely have helped to minimize any differences between the experimental and control group members. The collection of additional information from study participants would have also allowed researchers to be more confident that the experimental and control group members were equivalent on key pieces of information that could have influenced recidivism and participant attitudes.

Finally, while 168 juvenile offenders were randomly assigned to either the experimental or control group, not all offenders agreed to participate in the evaluation. Remember that of the 83 offenders assigned to the teen court experimental group, only 56 (67%) agreed to participate in the study. Of the 85 youth randomly assigned to normal juvenile justice processing, only 51 (60%) agreed to participate in the study. While this limitation is unavoidable, it still could have influenced the study. Perhaps those 27 offenders who declined to participate in the teen court group differed significantly from the 56 who agreed to participate. If so, it is possible that the differences among those two groups could have impacted the results of the study. For example, perhaps the 27 youths who were randomly assigned to teen court but did not agree to be a part of the study were some of the least risky of potential teen court participants�less serious histories, better attitudes to begin with, and so on. In this case, perhaps the most risky teen court participants agreed to be a part of the study, and as a result of being more risky, this led to more dismal delinquency outcomes compared to the control group at the end of each respective program. Because parental consent was required for the study authors to be able to compare those who declined to participate in the study to those who agreed, it is unknown if the participants and nonparticipants differed significantly on any variables among either the experimental or control group. Moreover, of the resulting 107 offenders who took part in the study, only 75 offenders accurately completed the post-test survey measuring offending and attitudinal outcomes.

Again, despite the experimental nature of this study, such limitations could have impacted the study results and must be considered.

Impact on Criminal Justice

Teen courts are generally designed to deal with nonserious first time offenders before they escalate to more serious and chronic delinquency. Innovative programs such as �Scared Straight� and juvenile boot camps have inspired an increase in teen court programs across the country, although there is little evidence regarding their effectiveness compared to traditional sanctions for youthful offenders. This study provides more specific evidence as to the effectiveness of teen courts relative to normal juvenile justice processing. Researchers learned that teen court participants fared worse than those in the control group. The potential labeling effects of teen court, including stigma among peers, especially where the offense may have been very minor, may be more harmful than doing less or nothing. The real impact of this study lies in the recognition that teen courts and similar sanctions for minor offenders may do more harm than good.

One important impact of this study is that it utilized an experimental design to evaluate the effectiveness of a teen court compared to traditional juvenile justice processing. Despite the study�s limitations, by using an experimental design it improved upon previous teen court evaluations by attempting to ensure any results were in fact due to the treatment, not some difference between the experimental and control group. This study also utilized both official and self-report measures of delinquency, in addition to self-report measures on such factors as self-concept and belief in conventional rules, which have been generally absent from teen court evaluations. The study authors also attempted to gauge the comparability of the experimental and control groups on factors such as age, gender, and race to help make sure study outcomes were attributable to the program, not the participants.

In This Chapter You Will Learn

The four components of experimental and quasi-experimental research designs and their function in answering a research question

The differences between experimental and quasi-experimental designs

The importance of randomization in an experimental design

The types of questions that can be answered with an experimental or quasi-experimental research design

About the three factors required for a causal relationship

That a relationship between two or more variables may appear causal, but may in fact be spurious, or explained by another factor

That experimental designs are relatively rare in criminal justice and why

About common threats to internal validity or alternative explanations to what may appear to be a causal relationship between variables

Why experimental designs are superior to quasi-experimental designs for eliminating or reducing the potential of alternative explanations

Introduction

The teen court evaluation that began this chapter is an example of an experimental design. The researchers of the study wanted to determine whether teen court was more effective at reducing recidivism and improving attitudes compared to regular juvenile justice case processing. In short, the researchers were interested in the relationship between variables �the relationship of teen court to future delinquency and other outcomes. When researchers are interested in whether a program, policy, practice, treatment, or other intervention impacts some outcome, they often utilize a specific type of research method/design called experimental design. Although there are many types of experimental designs, the foundation for all of them is the classic experimental design. This research design, and some typical variations of this experimental design, are the focus of this chapter.

Although the classic experiment may be appropriate to answer a particular research question, there are barriers that may prevent researchers from using this or another type of experimental design. In these situations, researchers may turn to quasi-experimental designs. Quasi-experiments include a group of research designs that are missing a key element found in the classic experiment and other experimental designs (hence the term �quasi� experiment). Despite this missing part, quasi-experiments are similar in structure to experimental designs and are used to answer similar types of research questions. This chapter will also focus on quasi-experiments and how they are similar to and different from experimental designs.

Uncovering the relationship between variables, such as the impact of teen court on future delinquency, is important in criminal justice and criminology, just as it is in other scientific disciplines such as education, biology, and medicine. Indeed, whereas criminal justice researchers may be interested in whether a teen court reduces recidivism or improves attitudes, medical field researchers may be concerned with whether a new drug reduces cholesterol, or an education researcher may be focused on whether a new teaching style leads to greater academic gains. Across these disciplines and topics of interest, the experimental design is appropriate. In fact, experimental designs are used in all scientific disciplines; the only thing that changes is the topic. Specific to criminal justice, below is a brief sampling of the types of questions that can be addressed using an experimental design:

Does participation in a correctional boot camp reduce recidivism?

What is the impact of an in-cell integration policy on inmate-on-inmate assaults in prisons?

Does police officer presence in schools reduce bullying?

Do inmates who participate in faith-based programming while in prison have a lower recidivism rate upon their release from prison?

Do police sobriety checkpoints reduce drunken driving fatalities?

What is the impact of a no-smoking policy in prisons on inmate-on-inmate assaults?

Does participation in a domestic violence intervention program reduce repeat domestic violence arrests?

A focus on the classic experimental design will demonstrate the usefulness of this research design for addressing criminal justice questions interested in cause and effect relationships. Particular attention is paid to the classic experimental design because it serves as the foundation for all other experimental and quasi-experimental designs, some of which are covered in this chapter. As a result, a clear understanding of the components, organization, and logic of the classic experimental design will facilitate an understanding of other experimental and quasi-experimental designs examined in this chapter. It will also allow the reader to better understand the results produced from those various designs, and importantly, what those results mean. It is a truism that the results of a research study are only as �good� as the design or method used to produce them. Therefore, understanding the various experimental and quasi-experimental designs is the key to becoming an informed consumer of research.

The Challenge of Establishing Cause and Effect

Researchers interested in explaining the relationship between variables, such as whether a treatment program impacts recidivism, are interested in causation or causal relationships. In a simple example, a causal relationship exists when X (independent variable) causes Y (dependent variable), and there are no other factors (Z) that can explain that relationship. For example, offenders who participated in a domestic violence intervention program (X�domestic violence intervention program) experienced fewer re-arrests (Y�re-arrests) than those who did not participate in the domestic violence program, and no other factor other than participation in the domestic violence program can explain these results. The classic experimental design is superior to other research designs in uncovering a causal relationship, if one exists. Before a causal relationship can be established, however, there are three conditions that must be met (see Figure 5.1). 2

FIGURE 5.1 | The Cause and Effect Relationship

Timing The first condition for a causal relationship is timing. For a causal relationship to exist, it must be shown that the independent variable or cause (X) preceded the dependent variable or outcome (Y) in time. A decrease in domestic violence re-arrests (Y) cannot occur before participation in a domestic violence reduction program (X ), if the domestic violence program is proposed to be the cause of fewer re-arrests. Ensuring that cause comes before effect is not sufficient to establish that a causal relationship exists, but it is one requirement that must be met for a causal relationship.

Association In addition to timing, there must also be an observable association between X and Y, the second necessary condition for a causal relationship. Association is also commonly referred to as covariance or correlation. When an association or correlation exits, this means there is some pattern of relationship between X and Y �as X changes by increasing or decreasing, Y also changes by increasing or decreasing. Here, the notion of X and Y increasing or decreasing can mean an actual increase/decrease in the quantity of some factor, such as an increase/decrease in the number of prison terms or days in a program or re-arrests. It can also refer to an increase/decrease in a particular category, for example, from nonparticipation in a program to participation in a program. For instance, subjects who participated in a domestic violence reduction program (X) incurred fewer domestic violence re-arrests (Y) than those who did not participate in the program. In this example, X and Y are associated�as X change s or increases from nonparticipation to participation in the domestic violence program, Y or the number of re-arrests for domestic violence decreases.

Associations between X and Y can occur in two different directions: positive or negative. A positive association means that as X increases, Y increases, or, as X decreases, Y decreases. A negative association means that as X increases, Y decreases, or, as X decreases, Y increases. In the example above, the association is negative�participation in the domestic violence program was associated with a reduction in re-arrests. This is also sometimes called an inverse relationship.

Elimination of Alternative Explanations Although participation in a domestic violence program may be associated with a reduction in re-arrests, this does not mean for certain that participation in the program was the cause of reduced re-arrests. Just as timing by itself does not imply a causal relationship, association by itself does not imply a causal relationship. For example, instead of the program being the cause of a reduction in re-arrests, perhaps several of the program participants died shortly after completion of the domestic violence program and thus were not able to engage in domestic violence (and their deaths were unknown to the researcher tracking re-arrests). Perhaps a number of the program participants moved out of state and domestic violence re-arrests occurred but were not able to be uncovered by the researcher. Perhaps those in the domestic violence program experienced some other event, such as the trauma of a natural disaster, and that experience led to a reduction in domestic violence, an event not connected to the domestic violence program. If any of these situations occurred, it might appear that the domestic violence program led to fewer re-arrests. However, the observed reduction in re-arrests can actually be attributed to a factor unrelated to the domestic violence program.

The previous discussion leads to the third and final necessary consideration in determining a causal relationship� elimination of alternative explanations. This means that the researcher must rule out any other potential explanation of the results, except for the experimental condition such as a program, policy, or practice. Accounting for or ruling out alternative explanations is much more difficult than ensuring timing and association. Ruling out all alternative explanations is difficult because there are so many potential other explanations that can wholly or partly explain the findings of a research study. This is especially true in the social sciences, where researchers are often interested in relationships explaining human behavior. Because of this difficulty, associations by themselves are sometimes mistaken as causal relationships when in fact they are spurious. A spurious relationship is one where it appears that X and Y are causally related, but the relationship is actually explained by something other than the independent variable, or X.

One only needs to go so far as the daily newspaper to find headlines and stories of mere associations being mistaken, assumed, or represented as causal relationships. For example, a newspaper headline recently proclaimed �Churchgoers live longer.� 3 An uninformed consumer may interpret this headline as evidence of a causal relationship�that going to church by itself will lead to a longer life�but the astute consumer would note possible alternative explanations. For example, people who go to church may live longer because they tend to live healthier lifestyles and tend to avoid risky situations. These are two probable alternative explanations to the relationship independent of simply going to church. In another example, researchers David Kalist and Daniel Yee explored the relationship between first names and delinquent behavior in their manuscript titled �First Names and Crime: Does Unpopularity Spell Trouble?� 4 Kalist and Lee (2009) found that unpopular names are associated with juvenile delinquency. In other words, those individuals with the most unpopular names were more likely to be delinquent than those with more popular names. According to the authors, is it not necessarily someone�s name that leads to delinquent behavior, but rather, the most unpopular names also tend to be correlated with individuals who come from disadvantaged home environments and experience a low socio-economic status of living. Rightly noted by the authors, these alternative explanations help to explain the link between someone�s name and delinquent behavior�a link that is not causal.

A frequently cited example provides more insight to the claim that an association by itself is not sufficient to prove causality. In certain cities in the United States, for example, as ice cream sales increase on a particular day or in a particular month so does the incidence of certain forms of crime. If this association were represented as a causal statement, it would be that ice cream or ice cream sales causes crime. There is an association, no doubt, and let us assume that ice cream sales rose before the increase in crime (timing). Surely, however, this relationship between ice cream sales and crime is spurious. The alternative explanation is that ice cream sales and crime are associated in certain parts of the country because of the weather. Ice cream sales tend to increase in warmer temperatures, and it just so happens that certain forms of crime tend to increase in warmer temperatures as well. This coincidence or association does not mean a causal relationship exists. Additionally, this does not mean that warm temperatures cause crime either. There are plenty of other alternative explanations for the increase in certain forms of crime and warmer temperatures. 6 For another example of a study subject to alternative explanations, read the June 2011 news article titled �Less Crime in U.S. Thanks to Videogames.� 7 Based on your reading, what are some other potential explanations for the crime drop other than videogames?

The preceding examples demonstrate how timing and association can be present, but the final needed condition for a causal relationship is that all alternative explanations are ruled out. While this task is difficult, the classic experimental design helps to ensure these additional explanatory factors are minimized. When other designs are used, such as quasi-experimental designs, the chance that alternative explanations emerge is greater. This potential should become clearer as we explore the organization and logic of the classic experimental design.

CLASSICS IN CJ RESEARCH

Minneapolis Domestic Violence Experiment

The Minneapolis Domestic Violence Experiment (MDVE) 5

Which police action (arrest, separation, or mediation) is most effective at deterring future misdemeanor domestic violence?

The experiment began on March 17, 1981, and continued until August 1, 1982. The experiment was conducted in two of Minneapolis�s four police precincts�the two with the highest number of domestic violence reports and arrests. A total of 314 reports of misdemeanor domestic violence were handled by the police during this time frame.

This study utilized an experimental design with the random assignment of police actions. Each police officer involved in the study was given a pad of report forms. Upon a misdemeanor domestic violence call, the officer�s action (arrest, separation, or mediation) was predetermined by the order and color of report forms in the officer�s notebook. Colored report forms were randomly ordered in the officer�s notebook and the color on the form determined the officer response once at the scene. For example, after receiving a call for domestic violence, an officer would turn to his or her report pad to determine the action. If the top form was pink, the action was arrest. If on the next call the top form was a different color, an action other than arrest would occur. All colored report forms were randomly ordered through a lottery assignment method. The result is that all police officer actions to misdemeanor domestic violence calls were randomly assigned. To ensure the lottery procedure was properly carried out, research staff participated in ride-alongs with officers to ensure that officers did not skip the order of randomly ordered forms. Research staff also made sure the reports were received in the order they were randomly assigned in the pad of report forms.

To examine the relationship of different officer responses to future domestic violence, the researchers examined official arrests of the suspects in a 6-month follow-up period. For example, the researchers examined those initially arrested for misdemeanor domestic violence and how many were subsequently arrested for domestic violence within a 6-month time frame. They did the same procedure for the police actions of separation and mediation. The researchers also interviewed the victim(s) of each incident and asked if a repeat domestic violence incident occurred with the same suspect in the 6-month follow-up period. This allowed researchers to examine domestic violence offenses that may have occurred but did not come to the official attention of police. The researchers then compared official arrests for domestic violence to self-reported domestic violence after the experiment.

Suspects arrested for misdemeanor domestic violence, as opposed to situations where separation or mediation was used, were significantly less likely to engage in repeat domestic violence as measured by official arrest records and victim interviews during the 6-month follow-up period. According to official police records, 10% of those initially arrested engaged in repeat domestic violence in the followup period, 19% of those who initially received mediation engaged in repeat domestic violence, and 24% of those who randomly received separation engaged in repeat domestic violence. According to victim interviews, 19% of those initially arrested engaged in repeat domestic violence, compared to 37% for separation and 33% for mediation. The general conclusion of the experiment was that arrest was preferable to separation or mediation in deterring repeat domestic violence across both official police records and victim interviews.

A few issues that affected the random assignment procedure occurred throughout the study. First, some officers did not follow the randomly assigned action (arrest, separation, or mediation) as a result of other circumstances that occurred at the scene. For example, if the randomly assigned action was separation, but the suspect assaulted the police officer during the call, the officer might arrest the suspect. Second, some officers simply ignored the assigned action if they felt a particular call for domestic violence required another action. For example, if the action was mediation as indicated by the randomly assigned report form, but the officer felt the suspect should be arrested, he or she may have simply ignored the randomly assigned response and substituted his or her own. Third, some officers forgot their report pads and did not know the randomly assigned course of action to take upon a call of domestic violence. Fourth and finally, the police chief also allowed officers to deviate from the randomly assigned action in certain circumstances. In all of these situations, the random assignment procedures broke down.

The results of the MDVE had a rapid and widespread impact on law enforcement practice throughout the United States. Just two years after the release of the study, a 1986 telephone survey of 176 urban police departments serving cities with populations of 100,000 or more found that 46 percent of the departments preferred to make arrests in cases of minor domestic violence, largely due to the effectiveness of this practice in the Minneapolis Domestic Violence Experiment. 8

In an attempt to replicate the findings of the Minneapolis Domestic Violence Experiment, the National Institute of Justice sponsored the Spouse Assault Replication Program. Replication studies were conducted in Omaha, Charlotte, Milwaukee, Miami, and Colorado Springs from 1986�1991. In three of the five replications, offenders randomly assigned to the arrest group had higher levels of continued domestic violence in comparison to other police actions during domestic violence situations. 9 Therefore, rather than providing results that were consistent with the Minneapolis Domestic Violence Experiment, the results from the five replication experiments produced inconsistent findings about whether arrest deters domestic violence. 10

Despite the findings of the replications, the push to arrest domestic violence offenders has continued in law enforcement. Today many police departments require officers to make arrests in domestic violence situations. In agencies that do not mandate arrest, department policy typically states a strong preference toward arrest. State legislatures have also enacted laws impacting police actions regarding domestic violence. Twenty-one states have mandatory arrest laws while eight have pro-arrest statutes for domestic violence. 11

The Classic Experimental Design

Table 5.1 provides an illustration of the classic experimental design. 12 It is important to become familiar with the specific notation and organization of the classic experiment before a full discussion of its components and their purpose.

Major Components of the Classic Experimental Design

The classic experimental design has four major components:

1. Treatment

2. Experimental Group and Control Group

3. Pre-Test and Post-Test

4. Random Assignment

Treatment The first component of the classic experimental design is the treatment, and it is denoted by X in the classic experimental design. The treatment can be a number of things�a program, a new drug, or the implementation of a new policy. In a classic experimental design, the primary goal is to determine what effect, if any, a particular treatment had on some outcome. In this way, the treatment can also be considered the independent variable.

TABLE 5.1 | The Classic Experimental Design

Experimental Group = Group that receives the treatment

Control Group = Group that does not receive the treatment

R = Random assignment

O 1 = Observation before the treatment, or the pre-test

X = Treatment or the independent variable

O 2 = Observation after the treatment, or the post-test

Experimental and Control Groups The second component of the classic experiment is an experimental group and a control group. The experimental group receives the treatment, and the control group does not receive the treatment. There will always be at least one group that receives the treatment in experimental and quasi-experimental designs. In some cases, experiments may have multiple experimental groups receiving multiple treatments.

Pre-Test and Post-Test The third component of the classic experiment is a pre-test and a post-test. A pretest is a measure of the dependent variable or outcome before the treatment. The post-test is a measure of the dependent variable after the treatment is administered. It is important to note that the post-test is defined based on the stated goals of the program. For example, if the stated goal of a particular program is to reduce re-arrests, the post-test will be a measure of re-arrests after the program. The dependent variable also defines the pre-test. For example, if a researcher wanted to examine the impact of a domestic violence reduction program (treatment or X) on the goal of reducing re-arrests (dependent variable or Y), the pre-test would be the number of domestic violence arrests incurred before the program. Program goals may be numerous and all can constitute a post-test, and hence, the pre-test. For example, perhaps the goal of the domestic violence program is also that participants learn of different pro-social ways to handle domestic conflicts other than resorting to violence. If researchers wanted to examine this goal, the post-test might be subjects� level of knowledge about pro-social ways to handle domestic conflicts other than violence. The pre-test would then be subjects� level of knowledge about these pro-social alternatives to violence before they received the treatment program.

Although all designs have a post-test, it is not always the case that designs have a pre-test. This is because researchers may not have access or be able to collect information constituting the pre-test. For example, researchers may not be able to determine subjects� level of knowledge about alternatives to domestic violence before the intervention program if the subjects are already enrolled in the domestic violence intervention program. In other cases, there may be financial barriers to collecting pre-test information. In the teen court evaluation that started this chapter, for example, researchers were not able to collect pre-test information on study participants due to the financial strain it would have placed on the agencies involved in the study. 13 There are a number of potential reasons why a pre-test might not be available in a research study. The defining feature, however, is that the pre-test is determined by the post-test.

Random Assignment The fourth component of the classic experiment is random assignment. Random assignment refers to a process whereby members of the experimental group and control group are assigned to the two groups through a random and unbiased process. Random assignment should not be mistaken for random selection as discussed in Chapter 3. Random selection refers to selecting a smaller but representative sample from a larger population. For example, a researcher may randomly select a sample from a larger city population for the purposes of sending sample members a mail survey to determine their attitudes on crime. The goal of random selection in this example is to make sure the sample, although smaller in size than the population, accurately represents the larger population.

Random assignment, on the other hand, refers to the process of assigning subjects to either the experimental or control group with the goal that the groups are similar or equivalent to each other in every way (see Figure 5.2). The exception to this rule is that one group gets the treatment and the other does not (see discussion below on why equivalence is so important). Although the concept of random is similar in each, the goals are different between random selection and random assignment. 14 Experimental designs all feature random assignment, but this is not true of other research designs, in particular quasi-experimental designs.

FIGURE 5.2 | Random Assignment

The classic experimental design is the foundation for all other experimental and quasi-experimental designs because it retains all of the major components discussed above. As mentioned, sometimes designs do not have a pre-test, a control group, or random assignment. Because the pre-test, control group, and random assignment are so critical to the goal of uncovering a causal relationship, if one exists, we explore them further below.

The Logic of the Classic Experimental Design

Consider a research study using the classic experimental design where the goal is to determine if a domestic violence treatment program has any effect on re-arrests for domestic violence. The randomly assigned experimental and control groups are comprised of persons who had previously been arrested for domestic violence. The pretest is a measure of the number of domestic violence arrests before the program. This is because the goal of the program is to determine whether re-arrests are impacted after the treatment. The post-test is the number of re-arrests following the treatment program.

Once randomly assigned, the experimental group members receive the domestic violence program, and the control group members do not. After the program, the researcher will compare the pre-test arrests for domestic violence of the experimental group to post-test arrests for domestic violence to determine if arrests increased, decreased, or remained constant since the start of the program. The researcher will also compare the post-test re-arrests for domestic violence between the experimental and control groups. With this example, we explore the usefulness of the classic experimental design, and the contribution of the pre-test, random assignment, and the control group to the goal of determining whether a domestic violence program reduces re-arrests.

The Pre-Test As a component of the classic experiment, the pre-test allows an examination of change in the dependent variable from before the domestic violence program to after the domestic violence program. In short, a pre-test allows the researcher to determine if re-arrests increased, decreased, or remained the same following the domestic violence program. Without a pre-test, researchers would not be able to determine the extent of change, if any, from before to after the program for either the experimental or control group.

Although the pre-test is a measure of the dependent variable before the treatment, it can also be thought of as a measure whereby the researcher can compare the experimental group to the control group before the treatment is administered. For example, the pre-test helps researchers to make sure both groups are similar or equivalent on previous arrests for domestic violence. The importance of equivalence between the experimental and control groups on previous arrests is discussed below with random assignment.

Random Assignment Random assignment helps to ensure that the experimental and control groups are equivalent before the introduction of the treatment. This is perhaps one of the most critical aspects of the classic experiment and all experimental designs. Although the experimental and control groups will be made up of different people with different characteristics, assigning them to groups via a random assignment process helps to ensure that any differences or bias between the groups is eliminated or minimized. By minimizing bias, we mean that the groups will balance each other out on all factors except the treatment. If they are balanced out on all factors prior to the administration of the treatment, any differences between the groups at the post-test must be due to the treatment�the only factor that differs between the experimental group and the control group. According to Shadish, Cook, and Campbell: �If implemented correctly, random assignment creates two or more groups of units that are probabilistically similar to each other on the average. Hence, any outcome differences that are observed between those groups at the end of a study are likely to be due to treatment, not to differences between the groups that already existed at the start of the study.� 15 Considered in another way, if the experimental and control group differed significantly on any relevant factor other than the treatment, the researcher would not know if the results observed at the post-test are attributable to the treatment or to the differences between the groups.

Consider an example where 500 domestic abusers were randomly assigned to the experimental group and 500 were randomly assigned to the control group. Because they were randomly assigned, we would likely find more frequent domestic violence arrestees in both groups, older and younger arrestees in both groups, and so on. If random assignment was implemented correctly, it would be highly unlikely that all of the experimental group members were the most serious or frequent arrestees and all of the control group members were less serious and/or less frequent arrestees. While there are no guarantees, we know the chance of this happening is extremely small with random assignment because it is based on known probability theory. Thus, except for a chance occurrence, random assignment will result in equivalence between the experimental and control group in much the same way that flipping a coin multiple times will result in heads approximately 50% of the time and tails approximately 50% of the time. Over 1,000 tosses of a coin, for example, should result in roughly 500 heads and 500 tails. While there is a chance that flipping a coin 1,000 times will result in heads 1,000 times, or some other major imbalance between heads and tails, this potential is small and would only occur by chance.

The same logic from above also applies with randomly assigning people to groups, and this can even be done by flipping a coin. By assigning people to groups through a random and unbiased process, like flipping a coin, only by chance (or researcher error) will one group have more of one characteristic than another, on average. If there are no major (also called statistically significant) differences between the experimental and control group before the treatment, the most plausible explanation for the results at the post-test is the treatment.

As mentioned, it is possible by some chance occurrence that the experimental and control group members are significantly different on some characteristic prior to administration of the treatment. To confirm that the groups are in fact similar after they have been randomly assigned, the researcher can examine the pre-test if one is present. If the researcher has additional information on subjects before the treatment is administered, such as age, or any other factor that might influence post-test results at the end of the study, he or she can also compare the experimental and control group on those measures to confirm that the groups are equivalent. Thus, a researcher can confirm that the experimental and control groups are equivalent on information known to the researcher.

Being able to compare the groups on known measures is an important way to ensure the random assignment process �worked.� However, perhaps most important is that randomization also helps to ensure similarity across unknown variables between the experimental and control group. Because random assignment is based on known probability theory, there is a much higher probability that all potential differences between the groups that could impact the post-test should balance out with random assignment�known or unknown. Without random assignment, it is likely that the experimental and control group would differ on important but unknown factors and such differences could emerge as alternative explanations for the results. For example, if a researcher did not utilize random assignment and instead took the first 500 domestic abusers from an ordered list and assigned them to the experimental group and the last 500 domestic abusers and assigned them to the control group, one of the groups could be �lopsided� or imbalanced on some important characteristic that could impact the outcome of the study. With random assignment, there is a much higher likelihood that these important characteristics among the experimental and control groups will balance out because no individual has a different chance of being placed into one group versus the other. The probability of one or more characteristics being concentrated into one group and not the other is extremely small with random assignment.

To further illustrate the importance of random assignment to group equivalence, suppose the first 500 domestic violence abusers who were assigned to the experimental group from the ordered list had significantly fewer domestic violence arrests before the program than the last 500 domestic violence abusers on the list. Perhaps this is because the ordered list was organized from least to most chronic domestic abusers. In this instance, the control group would be lopsided concerning number of pre-program domestic violence arrests�they would be more chronic than the experimental group. The arrest imbalance then could potentially explain the post-test results following the domestic violence program. For example, the �less risky� offenders in the experimental group might be less likely to be re-arrested regardless of their participation in the domestic violence program, especially compared to the more chronic domestic abusers in the control group. Because of imbalances between the experimental and control group on arrests before the program was implemented, it would not be known for certain whether an observed reduction in re-arrests after the program for the experimental group was due to the program or the natural result of having less risky offenders in the experimental group. In this instance, the results might be taken to suggest that the program significantly reduces re-arrests. This conclusion might be spurious, however, for the association may simply be due to the fact that the offenders in the experimental group were much different (less frequent offenders) than the control group. Here, the program may have had no effect�the experimental group members may have performed the same regardless of the treatment because they were low-level offenders.

The example above suggests that differences between the experimental and control groups based on previous arrest records could have a major impact on the results of a study. Such differences can arise with the lack of random assignment. If subjects were randomly assigned to the experimental and control group, however, there would be a much higher probability that less frequent and more frequent domestic violence arrestees would have been found in both the experimental and control groups and the differences would have balanced out between the groups�leaving any differences between the groups at the post-test attributable to the treatment only.

In summary, random assignment helps to ensure that the experimental and control group members are balanced or equivalent on all factors that could impact the dependent variable or post-test�known or unknown. The only factor they are not balanced or equal on is the treatment. As such, random assignment helps to isolate the impact of the treatment, if any, on the post-test because it increases confidence that the only difference between the groups should be that one group gets the treatment and the other does not. If that is the only difference between the groups, any change in the dependent variable between the experimental and control group must be attributed to the treatment and not an alternative explanation, such as significant arrest history imbalance between the groups (refer to Figure 5.2). This logic also suggests that if the experimental group and control group are imbalanced on any factor that may be relevant to the outcome, that factor then becomes a potential alternative explanation for the results�an explanation that reduces the researcher�s ability to isolate the real impact of the treatment.

WHAT RESEARCH SHOWS: IMPACTING CRIMINAL JUSTICE OPERATIONS

Scared Straight

The 1978 documentary Scared Straight introduced to the public the �Lifer�s Program� at Rahway State Prison in New Jersey. This program sought to decrease juvenile delinquency by bringing at-risk and delinquent juveniles into the prison where they would be �scared straight� by inmates serving life sentences. Participants in the program were talked to and yelled at by the inmates in an effort to scare them. It was believed that the fear felt by the participants would lead to a discontinuation of their problematic behavior so that they would not end up in prison themselves. Although originally touted as a success based on anecdotal evidence, subsequent evaluations of the program and others like it proved otherwise.

Using a classic experimental design, Finckenauer evaluated the original �Lifer�s Program� at Rahway State Prison. 16 Participating juveniles were randomly assigned to the experimental group or the control group. Results of the evaluation were not positive. Post-test measures revealed that juveniles who were assigned to the experimental group and participated in the program were actually more seriously delinquent afterwards than those who did not participate in the program. Also using an experimental design with random assignment, Yarborough evaluated the �Juvenile Offenders Learn Truth� (JOLT) program at the State Prison of Southern Michigan at Jackson. 17 This program was similar to that of the �Lifer�s Program� only with fewer obscenities used by the inmates. Post-test measurements were taken at two intervals, 3 and 6 months after program completion. Again, results were not positive. Findings revealed no significant differences between those juveniles who attended the program and those who did not.

Other experiments conducted on Scared Straight -like programs further revealed their inability to deter juveniles from future criminality. 18 Despite the intuitive popularity of these programs, these evaluations proved that such programs were not successful. In fact, it is postulated that these programs may have actually done more harm than good.

The Control Group The presence of an equivalent control group (created through random assignment) also gives the researcher more confidence that the findings at the post-test are due to the treatment and not some other alternative explanation. This logic is perhaps best demonstrated by considering how interpretation of results is affected without a control group. Absent an equivalent control group, it cannot be known whether the results of the study are due to the program or some other factor. This is because the control group provides a baseline of comparison or a �control.� For example, without a control group, the researcher may find that domestic violence arrests declined from pre-test to post-test. But the researcher would not be able to definitely attribute that finding to the program without a control group. Perhaps the single experimental group incurred fewer arrests because they matured over their time in the program, regardless of participation in the domestic violence program. Having a randomly assigned control group would allow this consideration to be eliminated, because the equivalent control group would also have naturally matured if that was the case.

Because the control group is meant to be similar to the experimental group on all factors with the exception that the experimental group receives the treatment, the logic is that any differences between the experimental and control group after the treatment must then be attributable only to the treatment itself�everything else occurs equally in both the experimental and control groups and thus cannot be the cause of results. The bottom line is that a control group allows the researcher more confidence to attribute any change in the dependent variable from pre- to post-test and between the experimental and control groups to the treatment�and not another alternative explanation. Absent a control group, the researcher would have much less confidence in the results.

Knowledge about the major components of the classic experimental design and how they contribute to an understanding of cause and effect serves as an important foundation for studying different types of experimental and quasi-experimental designs and their organization. A useful way to become familiar with the components of the experimental design and their important role is to consider the impact on the interpretation of results when one or more components are lacking. For example, what if a design lacked a pre-test? How could this impact the interpretation of post-test results and knowledge about the comparability of the experimental and control group? What if a design lacked random assignment? What are some potential problems that could occur and how could those potential problems impact interpretation of results? What if a design lacked a control group? How does the absence of an equivalent control group affect a researcher�s ability to determine the unique effects of the treatment on the outcomes being measured? The ability to discuss the contribution of a pre-test, random assignment, and a control group�and what is the impact when one or more of those components is absent from a research design�is the key to understanding both experimental and quasi-experimental designs that will be discussed in the remainder of this chapter. As designs lose these important parts and transform from a classic experiment to another experimental design or to a quasi-experiment, they become less useful in isolating the impact that a treatment has on the dependent variable and allow more room for alternative explanations of the results.

One more important point must be made before further delving into experimental and quasi-experimental designs. This point is that rarely, if ever, will the average consumer of research be exposed to the symbols or specific language of the classic experiment, or other experimental and quasi-experimental designs examined in this chapter. In fact, it is unlikely that the average consumer will ever be exposed to the terms pre-test, post-test, experimental group, or random assignment in the popular media, among other terms related to experimental and quasi-experimental designs. Yet, consumers are exposed to research results produced from these and other research designs every day. For example, if a national news organization or your regional newspaper reported a story about the effectiveness of a new drug to reduce cholesterol or the effects of different diets on weight loss, it is doubtful that the results would be reported as produced through a classic experimental design that used a control group and random assignment. Rather, these media outlets would use generally nonscientific terminology such as �results of an experiment showed� or �results of a scientific experiment indicated� or �results showed that subjects who received the new drug had greater cholesterol reductions than those who did not receive the new drug.� Even students who regularly search and read academic articles for use in course papers and other projects will rarely come across such design notation in the research studies they utilize. Depiction of the classic experimental design, including a discussion of its components and their function, simply illustrates the organization and notation of the classic experimental design. Unfortunately, the average consumer has to read between the lines to determine what type of design was used to produce the reported results. Understanding the key components of the classic experimental design allows educated consumers of research to read between those lines.

RESEARCH IN THE NEWS

�Swearing Makes Pain More Tolerable� 19

In 2009, Richard Stephens, John Atkins, and Andrew Kingston of the School of Psychology at Keele University conducted a study with 67 undergraduate students to determine if swearing affects an individual�s response to pain. Researchers asked participants to immerse their hand in a container filled with ice-cold water and repeat a preferred swear word. The researchers then asked the same participants to immerse their hand in ice-cold water while repeating a word used to describe a table (a non-swear word). The results showed that swearing increased pain tolerance compared to the non-swearing condition. Participants who used a swear word were able to hold their hand in ice-cold water longer than when they did not swear. Swearing also decreased participants� perception of pain.

1. This study is an example of a repeated measures design. In this form of experimental design, study participants are exposed to an experimental condition (swearing with hand in ice-cold water) and a control condition (non-swearing with hand in ice-cold water) while repeated outcome measures are taken with each condition, for example, the length of time a participant was able to keep his or her hand submerged in ice-cold water. Conduct an Internet search for �repeated measures design� and explore the various ways such a study could be conducted, including the potential benefits and drawbacks to this design.

2. After researching repeated measures designs, devise a hypothetical repeated measures study of your own.

3. Retrieve and read the full research study �Swearing as a Response to Pain� by Stephens, Atkins, and Kingston while paying attention to the design and methods (full citation information for this study is listed below). Has your opinion of the study results changed after reading the full study? Why or why not?

Full Study Source: Stephens, R., Atkins, J., and Kingston, A. (2009). �Swearing as a response to pain.� NeuroReport 20, 1056�1060.

Variations on the Experimental Design

The classic experimental design is the foundation upon which all experimental and quasi-experimental designs are based. As such, it can be modified in numerous ways to fit the goals (or constraints) of a particular research study. Below are two variations of the experimental design. Again, knowledge about the major components of the classic experiment, how they contribute to an explanation of results, and what the impact is when one or more components are missing provides an understanding of all other experimental designs.

Post-Test Only Experimental Design

The post-test only experimental design could be used to examine the impact of a treatment program on school disciplinary infractions as measured or operationalized by referrals to the principal�s office (see Table 5.2). In this design, the researcher randomly assigns a group of discipline problem students to the experimental group and control group by flipping a coin�heads to the experimental group and tails to the control group. The experimental group then enters the 3-month treatment program. After the program, the researcher compares the number of referrals to the principal�s office between the experimental and control groups over some period of time, for example, discipline referrals at 6 months after the program. The researcher finds that the experimental group has a much lower number of referrals to the principal�s office in the 6 month follow-up period than the control group.

TABLE 5.2 | Post-Test Only Experimental Design

Several issues arise in this example study. The researcher would not know if discipline problems decreased, increased, or stayed the same from before to after the treatment program because the researcher did not have a count of disciplinary referrals prior to the treatment program (e.g., a pre-test). Although the groups were randomly assigned and are presumed equivalent, the absence of a pre-test means the researcher cannot confirm that the experimental and control groups were equivalent before the treatment was administered, particularly on the number of referrals to the principal�s office. The groups could have differed by a chance occurrence even with random assignment, and any such differences between the groups could potentially explain the post-test difference in the number of referrals to the principal�s office. For example, if the control group included much more serious or frequent discipline problem students than the experimental group by chance, this difference might explain the lower number of referrals for the experimental group, not that the treatment produced this result.

Experimental Design with Two Treatments and a Control Group

This design could be used to determine the impact of boot camp versus juvenile detention on post-release recidivism (see Table 5.3). Recidivism in this study is operationalized as re-arrest for delinquent behavior. First, a population of known juvenile delinquents is randomly assigned to either boot camp, juvenile detention, or a control condition where they receive no sanction. To accomplish random assignment to groups, the researcher places the names of all youth into a hat and assigns the groups in order. For example, the first name pulled goes into experimental group 1, the next into experimental group 2, and the next into the control group, and so on. Once randomly assigned, the experimental group youth receive either boot camp or juvenile detention for a period of 3 months, whereas members of the control group are released on their own recognizance to their parents. At the end of the experiment, the researcher compares the re-arrest activity of boot camp participants to detention delinquents to control group members during a 6-month follow-up period.

TABLE 5.3 | Experimental Design with Two Treatments and a Control Group

This design has several advantages. First, it includes all major components of the classic experimental design, and simply adds an additional treatment for comparison purposes. Random assignment was utilized and this means that the groups have a higher probability of being equivalent on all factors that could impact the post-test. Thus, random assignment in this example helps to ensure the only differences between the groups are the treatment conditions. Without random assignment, there is a greater chance that one group of youth was somehow different, and this difference could impact the post-test. For example, if the boot camp youth were much less serious and frequent delinquents than the juvenile detention youth or control group youth, the results might erroneously show that the boot camp reduced recidivism when in fact the youth in boot camp may have been the �best risks��unlikely to get re-arrested with or without boot camp. The pre-test in the example above allows the researcher to determine change in re-arrests from pretest to post-test. Thus, the researcher can determine if delinquent behavior, as measured by re-arrest, increased, decreased, or remained constant from pre- to post-test. The pre-test also allows the researcher to confirm that the random assignment process resulted in equivalent groups based on the pre-test. Finally, the presence of a control group allows the researcher to have more confidence that any differences in the post-test are due to the treatment. For example, if the control group had more re-arrests than the boot camp or juvenile detention experimental groups 6 months after their release from those programs, the researcher would have more confidence that the programs produced fewer re-arrests because the control group members were the same as the experimental groups; the only difference was that they did not receive a treatment.

The one key feature of experimental designs is that they all retain random assignment. This is why they are considered �experimental� designs. Sometimes, however, experimental designs lack a pre-test. Knowledge of the usefulness of a pre-test demonstrates the potential problems with those designs where it is missing. For example, in the post-test only experimental design, a researcher would not be able to make a determination of change in the dependent variable from pre- to post-test. Perhaps most importantly, the researcher would not be able to confirm that the experimental and control groups were in fact equivalent on a pre-test measure before the introduction of the treatment. Even though both groups were randomly assigned, and probability theory suggests they should be equivalent, without a pre-test measure the researcher could not confirm similarity because differences could occur by chance even with random assignment. If there were any differences at the post-test between the experimental group and control group, the results might be due to some explanation other than the treatment, namely that the groups differed prior to the administration of the treatment. The same limitation could apply in any form of experimental design that does not utilize a pre-test for conformational purposes.

Understanding the contribution of a pre-test to an experimental design shows that it is a critical component. It provides a measure of change and also gives the researcher more confidence that the observed results are due to the treatment, and not some difference between the experimental and control groups. Despite the usefulness of a pre-test, however, perhaps the most critical ingredient of any experimental design is random assignment. It is important to note that all experimental designs retain random assignment.

Experimental Designs Are Rare in Criminal Justice and Criminology

The classic experiment is the foundation for other types of experimental and quasi-experimental designs. The unfortunate reality, however, is that the classic experiment, or other experimental designs, are few and far between in criminal justice. 20 Recall that one of the major components of an experimental design is random assignment. Achieving random assignment is often a barrier to experimental research in criminal justice. Achieving random assignment might, for example, require the approval of the chief (or city council or both) of a major metropolitan police agency to allow researchers to randomly assign patrol officers to certain areas of a city and/or randomly assign police officer actions. Recall the MDVE. This experiment required the full cooperation of the chief of police and other decision-makers to allow researchers to randomly assign police actions. In another example, achieving random assignment might require a judge to randomly assign a group of youthful offenders to a certain juvenile court sanction (experimental group), and another group of similar youthful offenders to no sanction or an alternative sanction as a control group. 21 In sum, random assignment typically requires the cooperation of a number of individuals and sometimes that cooperation is difficult to obtain.

Even when random assignment can be accomplished, sometimes it is not implemented correctly and the random assignment procedure breaks down. This is another barrier to conducting experimental research. For example, in the MDVE, researchers randomly assigned officer responses, but the officers did not always follow the assigned course of action. Moreover, some believe that the random assignment of criminal justice programs, sentences, or randomly assigning officer responses may be unethical in certain circumstances, and even a violation of the rights of citizens. For example, some believe it is unfair when random assignment results in some delinquents being sentenced to boot camp while others get assigned to a control group without any sanction at all or a less restrictive sanction than boot camp. In the MDVE, some believe it is unfair that some suspects were arrested and received an official record whereas others were not arrested for the same type of behavior. In other cases, subjects in the experimental group may receive some benefit from the treatment that is essentially denied to the control group for a period of time and this can become an issue as well.

There are other important reasons why random assignment is difficult to accomplish. Random assignment may, for example, involve a disruption of the normal procedures of agencies and their officers. In the MDVE, officers had to adjust their normal and established routine, and this was a barrier at times in that study. Shadish, Cook, and Campbell also note that random assignment may not always be feasible or desirable when quick answers are needed. 22 This is because experimental designs sometimes take a long time to produce results. In addition to the time required in planning and organizing the experiment, and treatment delivery, researchers may need several months if not years to collect and analyze the data before they have answers. This is particularly important because time is often of the essence in criminal justice research, especially in research efforts testing the effect of some policy or program where it is not feasible to wait years for answers. Waiting for the results of an experimental design means that many policy-makers may make decisions without the results.

Quasi-Experimental Designs

In general terms, quasi-experiments include a group of designs that lack random assignment. Quasi-experiments may also lack other parts, such as a pre-test or a control group, just like some experimental designs. The absence of random assignment, however, is the ingredient that transforms an otherwise experimental design into a quasi-experiment. Lacking random assignment is a major disadvantage because it increases the chances that the experimental and control groups differ on relevant factors before the treatment�both known and unknown�differences that may then emerge as alternative explanations of the outcomes.

Just like experimental designs, quasi-experimental designs can be organized in many different ways. This section will discuss three types of quasi-experiments: nonequivalent group design, one-group longitudinal design, and two-group longitudinal design.

Nonequivalent Group Design

The nonequivalent group design is perhaps the most common type of quasi-experiment. 23 Notice that it is very similar to the classic experimental design with the exception that it lacks random assignment (see Table 5.4). Additionally, what was labeled the experimental group in an experimental design is sometimes called the treatment group in the nonequivalent group design. What was labeled the control group in the experimental design is sometimes called the comparison group in the nonequivalent group design. This terminological distinction is an indicator that the groups were not created through random assignment.

TABLE 5.4 | Nonequivalent Group Design

NR = Not Randomly assigned

One of the main problems with the nonequivalent group design is that it lacks random assignment, and without random assignment, there is a greater chance that the treatment and comparison groups may be different in some way that can impact study results. Take, for example, a nonequivalent group design where a researcher is interested in whether an aggression-reduction treatment program can reduce inmate-on-inmate assaults in a prison setting. Assume that the researcher asked for inmates who had previously been involved in assaultive activity to volunteer for the aggression-reduction program. Suppose the researcher placed the first 50 volunteers into the treatment group and the next 50 volunteers into the comparison group. Note that this method of assignment is not random but rather first come, first serve.

Because the study utilized volunteers and there was no random assignment, it is possible that the first 50 volunteers placed into the treatment group differed significantly from the last 50 volunteers who were placed in the comparison group. This can lead to alternative explanations for the results. For example, if the treatment group was much younger than the comparison group, the researcher may find at the end of the program that the treatment group still maintained a higher rate of infractions than the comparison group�even after the aggression-reduction program! The conclusion might be that the aggression program actually increased the level of violence among the treatment group. This conclusion would likely be spurious and may be due to the age differential between the treatment and comparison groups. Indeed, research has revealed that younger inmates are significantly more likely to engage in prison assaults than older inmates. The fact that the treatment group incurred more assaults than the comparison group after the aggression-reduction program may only relate to the age differential between the groups, not that the program had no effect or that it somehow may have increased aggression. The previous example highlights the importance of random assignment and the potential problems that can occur in its absence.

Although researchers who utilize a quasi-experimental design are not able to randomly assign their subjects to groups, they can employ other techniques in an attempt to make the groups as equivalent as possible on known or measured factors before the treatment is given. In the example above, it is likely that the researcher would have known the age of inmates, their prior assault record, and various other pieces of information (e.g., previous prison stays). Through a technique called matching, the researcher could make sure the treatment and comparison groups were �matched� on these important factors before administering the aggression reduction program to the treatment group. This type of matching can be done individual to individual (e.g., subject #1 in treatment group is matched to a selected subject #1 in comparison group on age, previous arrests, gender), or aggregately, such that the comparison group is similar to the treatment group overall (e.g., average ages between groups are similar, equal proportions of males and females). Knowledge of these and other important variables, for example, would allow the researcher to make sure that the treatment group did not have heavy concentrations of younger or more frequent or serious offenders than the comparison group�factors that are related to assaultive activity independent of the treatment program. In short, matching allows the researcher some control over who goes into the treatment and comparison groups so as to balance these groups on important factors absent random assignment. If unbalanced on one or more factors, these factors could emerge as alternative explanations of the results. Figure 5.3 demonstrates the logic of matching both at the individual and aggregate level in a quasi-experimental design.

Matching is an important part of the nonequivalent group design. By matching, the researcher can approximate equivalence between the groups on important variables that may influence the post-test. However, it is important to note that a researcher can only match subjects on factors that they have information about�a researcher cannot match the treatment and comparison group members on factors that are unmeasured or otherwise unknown but which may still impact outcomes. For example, if the researcher has no knowledge about the number of previous incarcerations, the researcher cannot match the treatment and comparison groups on this factor. Matching also requires that the information used for matching is valid and reliable, which is not always the case. Agency records, for example, are notorious for inconsistencies, errors, omissions, and for being dated, but are often utilized for matching purposes. Asking survey questions to generate information for matching (for example, how many times have you been incarcerated?) can also be problematic because some respondents may lie, forget, or exaggerate their behavior or experiences.

In addition to the above considerations, the more factors a researcher wishes to match the group members on, the more difficult it becomes to find appropriate matches. Matching on prior arrests or age is less complex than matching on several additional pieces of information. Finally, matching is never considered superior to random assignment when the goal is to construct equitable groups. This is because there is a much higher likelihood of equivalence with random assignment on factors that are both measured and unknown to the researcher. Thus, the results produced from a nonequivalent group design, even with matching, are at a greater risk of alternative explanations than an experimental design that features random assignment.

FIGURE 5.3 | (a) Individual Matching (b) Aggregate Matching

The previous discussion is not to suggest that the nonequivalent group design cannot be useful in answering important research questions. Rather, it is to suggest that the nonequivalent group design, and hence any quasi-experiment, is more susceptible to alternative explanations than the classic experimental design because of the absence of random assignment. As a result, a researcher must be prepared to rule out potential alternative explanations. Quasi-experimental designs that lack a pre-test or a comparison group are even less desirable than the nonequivalent group design and are subject to additional alternative explanations because of these missing parts. Although the quasi-experiment may be all that is available and still can serve as an important design in evaluating the impact of a particular treatment, it is not preferable to the classic experiment. Researchers (and consumers) must be attuned to the potential issues of this design so as to make informed conclusions about the results produced from such research studies.

The Effects of Red Light Camera (RLC) Enforcement

On March 15, 2009, an article appeared in the Santa Cruz Sentinel entitled �Ticket�s in the Mail: Red-Light Cameras Questioned.� The article stated �while studies show fewer T-bone crashes at lights with cameras and fewer drivers running red lights, the number of rear-end crashes increases.� 24 The study mentioned in the newspaper, which showed fewer drivers running red lights with cameras, was conducted by Richard Retting, Susan Ferguson, and Charles Farmer of the Insurance Institute for Highway Safety (IIHS). 25 They completed a quasi-experimental study in Philadelphia to determine the impact of red light cameras (RLC) on red light violations. In the study, the researchers selected nine intersections�six of which were experimental sites that utilized RLCs and three comparison sites that did not utilize RLCs. The six experimental sites were located in Philadelphia, Pennsylvania, and the three comparison sites were located in Atlantic County, New Jersey. The researchers chose the comparison sites based on the proximity to Philadelphia, the ability to collect data using the same methods as at experimental intersections (e.g., the use of cameras for viewing red light traffic), and the fact that police officials in Atlantic County had offered assistance selecting and monitoring the intersections.

The authors collected three phases of information in the RLC study at the experimental and comparison sites:

Phase 1 Data Collection: Baseline (pre-test) data collection at the experimental and comparison sites consisting of the number of vehicles passing through each intersection, the number of red light violations, and the rate of red light violations per 10,000 vehicles.

Phase 2 Data Collection: Number of vehicles traveling through experimental and comparison intersections, number of red light violations after a 1-second yellow light increase at the experimental sites (treatment 1), number of red light violations at comparison sites without a 1-second yellow light increase, and red light violations per 10,000 vehicles at both experimental and comparison sites.

Phase 3 Data Collection: Red light violations after a 1-second yellow light increase and RLC enforcement at the experimental sites (treatment 2), red light violations at comparison sites without a 1-second yellow increase or RLC enforcement, number of vehicles passing through the experimental and comparison intersections, and the rate of red light violations per 10,000 vehicles.

The researchers operationalized �red light violations� as those where the vehicle entered the intersection one-half of a second or more after the onset of the red signal where the vehicle�s rear tires had to be positioned behind the crosswalk or stop line prior to entering on red. Vehicles already in the intersection at the onset of the red light, or those making a right turn on red with or without stopping were not considered red light violations.

The researchers collected video data at each of the experimental and comparison sites during Phases 1�3. This allowed the researchers to examine red light violations before, during, and after the implementation of red light enforcement and yellow light time increases. Based on an analysis of data, the researchers revealed that the implementation of a 1-second yellow light increase led to reductions in the rate of red light violations from Phase 1 to Phase 2 in all of the experimental sites. In 2 out of 3 comparison sites, the rate of red light violations also decreased, despite no yellow light increase. From Phase 2 to Phase 3 (the enforcement of red light camera violations in addition to a 1-second yellow light increase at experimental sites), the authors noted decreases in the rate of red light violations in all experimental sites, and decreases among 2 of 3 comparison sites without red light enforcement in effect.

Concluding their study, the researchers noted that the study �found large and highly significant incremental reductions in red light running associated with increased yellow signal timing followed by the introduction of red light cameras.� Despite these findings, the researchers noted a number of potential factors to consider in light of the findings: the follow-up time periods utilized when counting red light violations before and after the treatment conditions were instituted; publicity about red light camera enforcement; and the size of fines associated with red light camera enforcement (the fine in Philadelphia was $100, higher than in many other cities), among others.

After reading about the study used in the newspaper article, has your impression of the newspaper headline and quote changed?

For more information and research on the effect of RLCs, visit the Insurance Institute for Highway Safety at http://www .iihs.org/research/topics/rlr.html .

One-Group Longitudinal Design

Like all experimental designs, the quasi-experimental design can come in a variety of forms. The second quasi-experimental design (above) is the one-group longitudinal design (also called a simple interrupted time series design). 26 An examination of this design shows that it lacks both random assignment and a comparison group (see Table 5.5). A major difference between this design and others we have covered is that it includes multiple pre-test and post-test observations.

TABLE 5.5 | One-Group Longitudinal Design

The one-group longitudinal design is useful when researchers are interested in exploring longer-term patterns. Indeed, the term longitudinal generally means �over time��repeated measurements of the pre-test and post-test over time. This is different from cross-sectional designs, which examine the pre-test and post-test at only one point in time (e.g., at a single point before the application of the treatment and at a single point after the treatment). For example, in the nonequivalent group design and the classic experimental design previously examined, both are cross-sectional because pre-tests and post-tests are measured at one point in time (e.g., at a point 6 months after the treatment). Yet, these designs could easily be considered longitudinal if researchers took repeated measures of the pre-test and post-test.

The organization of the one-group longitudinal design is to examine a baseline of several pre-test observations, introduce a treatment or intervention, and then examine the post-test at several different time intervals. As organized, this design is useful for gauging the impact that a particular program, policy, or law has, if any, and how long the treatment impact lasts. Consider an example whereby a researcher is interested in gauging the impact of a tobacco ban on inmate-on-inmate assaults in a prison setting. This is an important question, for recent years have witnessed correctional systems banning all tobacco products from prison facilities. Correctional administrators predicted that there would be a major increase of inmate-on-inmate violence once the bans took effect. The one-group longitudinal design would be one appropriate design to examine the impact of banning tobacco on inmate assaults.

To construct this study using the one-group longitudinal design, the researcher would first examine the rate of inmate-on-inmate assaults in the prison system (or at an individual prison, a particular cellblock, or whatever the unit of analysis) prior to the removal of tobacco. This is the pre-test, or a baseline of assault activity before the ban goes into effect. In the design presented above, perhaps the researcher would measure the level of assaults in the preceding four months prior to the tobacco ban. When establishing a pre-test baseline, the general rule is that, in a longitudinal design, the more time utilized, both in overall time and number of intervals, the better. For example, the rate of assaults in the preceding month is not as useful as an entire year of data on inmate assaults prior to the tobacco ban. Next, once the tobacco ban is implemented, the researcher would then measure the rate of inmate assaults in the coming months to determine what impact the ban had on inmate-on-inmate assaults. This is shown in Table 5.5 as the multiple post-test measures of assaults. Assaults may increase, decrease, or remain constant from the pre-test baseline over the term of the post-test.

If assaults increased at the same time as the ban went into effect, the researcher might conclude that the increase was due only to the tobacco ban. But, could there be alternative explanations? The answer to this question is yes, there may be other plausible explanations for the increase even with several months of pre-test data. Unfortunately, without a comparison group there is no way for the researcher to be certain if the increase in assaults was due to the tobacco ban, or some other factor that may have spurred the increase in assaults and happened at the same time as the tobacco ban. What if assaults decreased after the tobacco ban went into effect? In this scenario, because there is no comparison group, the researcher would still not know if the results would have happened anyway without the tobacco ban. In these instances, the lack of a comparison group prevents the researcher from confidently attributing the results to the tobacco ban, and interpretation is subject to numerous alternative explanations.

Two-Group Longitudinal Design

A remedy for the previous situation would be to introduce a comparison group (see Table 5.6). Prior to the full tobacco ban, suppose prison administrators conducted a pilot program at one prison to provide insight as to what would happen once the tobacco ban went into effect systemwide. To conduct this pilot, the researcher identified one prison. At this prison, the researcher identified two different cellblocks, C-Block and D-Block. C-Block constitutes the treatment group, or the cellblock of inmates who will have their tobacco taken away. D-Block is the comparison group�inmates in this cellblock will retain their tobacco privileges during the course of the study and during a determined follow-up period to measure post-test assaults (e.g., 12-months). This is a two-group longitudinal design (also sometimes called a multiple interrupted time series design), and adding a comparison group makes this design superior to the one-group longitudinal design.

TABLE 5.6 | Two-Group Longitudinal Design

The usefulness of adding a comparison group to the study means that the researcher can have more confidence that the results at the post-test are due to the tobacco ban and not some alternative explanation. This is because any difference in assaults at the post-test between the treatment and comparison group should be attributed to the only difference between them, the tobacco ban. For this interpretation to hold, however, the researcher must be sure that C-Block and D-Block are similar or equivalent on all factors that might influence the post-test. There are many potential factors that should be considered. For example, the researcher will want to make sure that the same types of inmates are housed in both cellblocks. If a chronic group of assaultive inmates constitutes members of C-Block, but not D-Block, this differential could explain the results, not the treatment.

The researcher might also want to make sure equitable numbers of tobacco and non-tobacco users are found in each cellblock. If very few inmates in C-Block are smokers, the real effect of removing tobacco may be hidden. The researcher might also examine other areas where potential differences might arise, for example, that both cellblocks are staffed with equal numbers of officers, that officers in each cellblock tend to resolve inmate disputes similarly, and other potential issues that could influence post-test measure of assaults. Equivalence could also be ensured by comparing the groups on additional evidence before the ban takes effect: number of prior prison sentences, time served in prison, age, seriousness of conviction crime, and other factors that might relate to assaultive behavior, regardless of the tobacco ban. Moreover, the researcher should ensure that inmates in C-Block do not know that their D-Block counterparts are still allowed tobacco during the pilot study, and vice versa. If either group knows about the pilot program being an experiment, they might act differently than normal, and this could become an explanation of results. Additionally, the researchers might also try to make sure that C-Block inmates are completely tobacco free after the ban goes into effect�that they do not hoard, smuggle, or receive tobacco from officers or other inmates during the tobacco ban in or outside of the cellblock. If these and other important differences are accounted for at the individual and cellblock level, the researcher will have more confidence that any differences in assaults at the post-test between the treatment and comparison groups are related to the tobacco ban, and not some other difference between the two groups or the two cellblocks.

The addition of a comparison group aids in the ability of the researcher to isolate the true impact of a tobacco ban on inmate-on-inmate assaults. All factors that influence the treatment group should also influence the comparison group because the groups are made up of equivalent individuals in equivalent circumstances, with the exception of the tobacco ban. If this is the only difference, the results can be attributed to the ban. Although the addition of the comparison group in the two-group longitudinal design provides more confidence that the findings are attributed to the tobacco ban, the fact that this design lacks randomization means that alternative explanations cannot be completely ruled out�but they can be minimized. This example also suggests that the quasi-experiment in this instance may actually be preferable to an experimental design�noting the realities of prison administration. For example, prison inmates are not typically randomly assigned to different cellblocks by prison officers. Moreover, it is highly unlikely that a prison would have two open cellblocks waiting for a researcher to randomly assign incoming inmates to the prison for a tobacco ban study. Therefore, it is likely there would be differences among the groups in the quasi-experiment.

Fortunately, if differences between the groups are present, the researcher can attempt to determine their potential impact before interpretation of results. The researcher can also use statistical models after the ban takes effect to determine the impact of any differences between the groups on the post-test. While the two-group longitudinal quasi-experiment just discussed could also take the form of an experimental design, if random assignment could somehow be accomplished, the previous discussion provides one situation where an experimental design might be appropriate and desired for a particular research question, but would not be realistic considering the many barriers.

The Threat of Alternative Explanations

Alternative explanations are those factors that could explain the post-test results, other than the treatment. Throughout this chapter, we have noted the potential for alternative explanations and have given several examples of explanations other than the treatment. It is important to know that potential alternative explanations can arise in any research design discussed in this chapter. However, alternative explanations often arise because some design part is missing, for example, random assignment, a pre-test, or a control or comparison group. This is especially true in criminal justice where researchers often conduct field studies and have less control over their study conditions than do researchers who conduct experiments under highly controlled laboratory conditions. A prime example of this is the tobacco ban study, where it would be difficult for researchers to ensure that C-Block inmates, the treatment group, were completely tobacco free during the course of the study.

Alternative explanations are typically referred to as threats to internal validity. In this context, if an experiment is internally valid, it means that alternative explanations have been ruled out and the treatment is the only factor that produced the results. If a study is not internally valid, this means that alternative explanations for the results exist or potentially exist. In this section, we focus on some common alternative explanations that may arise in experimental and quasi-experimental designs. 27

Selection Bias

One of the more common alternative explanations that may occur is selection bias. Selection bias generally indicates that the treatment group (or experimental group) is somehow different from the comparison group (or control group) on a factor that could influence the post-test results. Selection bias is more often a threat in quasi-experimental designs than experimental designs due to the lack of random assignment. Suppose in our study of the prison tobacco ban, members of C-Block were substantially younger than members of D-Block, the comparison group. Such an imbalance between the groups would mean the researcher would not know if the differences in assaults are real (meaning the result of the tobacco ban) or a result of the age differential. Recall that research shows that younger inmates are more assaultive than older inmates and so we would expect more assaults among the younger offenders independent of the tobacco ban.

In a quasi-experiment, selection bias is perhaps the most prevalent type of alternative explanation and can seriously compromise results. Indeed, many of the examples above have referred to potential situations where the groups are imbalanced or not equivalent on some important factor. Although selection bias is a common threat in quasi-experimental designs because of lack of random assignment, and can be a threat in experimental designs because the groups could differ by chance alone or the practice of randomization was not maintained throughout the study (see Classics in CJ Research-MDVE above), a researcher may be able to detect such differentials. For example, the researcher could detect such differences by comparing the groups on the pre-test or other types of information before the start of the study. If differences were found, the researcher could take measures to correct them. The researcher could also use a statistical model that could account or control for differences between the groups and isolate the impact of the treatment, if any. This discussion is beyond the scope of this text but would be a potential way to deal with selection bias and estimate the impact of this bias on study results. The researcher could also, if possible, attempt to re-match the groups in a quasi-experiment or randomly assign the groups a second time in an experimental design to ensure equivalence. At the least, the researcher could recognize the group differences and discuss their potential impact on the results. Without a pre-test or other pre-study information on study participants, however, such differences might not be able to be detected and, therefore, it would be more difficult to determine how the differences, as a result of selection bias, influenced the results.

Another potential alternative explanation is history. History refers to any event experienced differently by the treatment and comparison groups in the time between the pre-test and the post-test that could impact results. Suppose during the course of the tobacco ban study several riots occurred on D-Block, the comparison group. Because of the riots, prison officers �locked down� this cellblock numerous times. Because D-Block inmates were locked down at various times, this could have affected their ability to otherwise engage in inmate assaults. At the end of the study, the assaults in D-Block might have decreased from their pre-test levels because of the lockdowns, whereas in C-Block assaults may have occurred at their normal pace because there was not a lockdown, or perhaps even increased from the pretest because tobacco was also taken away. Even if the tobacco ban had no effect and assaults remained constant in C-Block from pre- to post-test, the lockdown in D-Block might make it appear that the tobacco ban led to increased assaults in C-Block. Thus, the researcher would not know if the post-test results for the C-Block treatment group were attributable to the tobacco ban or the simple fact that D-Block inmates were locked down and their assault activity was artificially reduced. In this instance, the comparison group becomes much less useful because the lockdown created a historical factor that imbalanced the groups during the treatment phase and nullified the comparison.

Another potential alternative explanation is maturation. Maturation refers to the natural biological, psychological, or emotional processes we all experience as time passes�aging, becoming more or less intelligent, becoming bored, and so on. For example, if a researcher was interested in the effect of a boot camp on recidivism for juvenile offenders, it is possible that over the course of the boot camp program the delinquents naturally matured as they aged and this produced the reduction in recidivism�not that the boot camp somehow led to this reduction. This threat is particularly applicable in situations that deal with populations that rapidly change over a relatively short period of time or when a treatment lasts a considerable period of time. However, this threat could be eliminated with a comparison group that is similar to the treatment group. This is because the maturation effects would occur in both groups and the effect of the boot camp, if any, could be isolated. This assumes, however, that the groups are matched and equitable on factors subject to the maturation process, such as age. If not, such differentials could be an alternative explanation of results. For example, if the treatment and comparison groups differ by age, on average, this could mean that one group changes or matures at a different rate than the other group. This differential rate of change or maturation as a result of the age differential could explain the results, not the treatment. This example demonstrates how selection bias and maturation can interact at the same time as alternative explanations. This example also suggests the importance of an equivalent control or comparison group to eliminate or minimize the impact of maturation as an alternative explanation.

Attrition or Subject Mortality

Attrition or subject mortality is another typical alternative explanation. Attrition refers to differential loss in the number or type of subjects between the treatment and comparison groups and can occur in both experimental and quasi-experimental designs. Suppose we wanted to conduct a study to determine who is the better research methods professor among the authors of this textbook. Let�s assume that we have an experimental design where students were randomly assigned to professor 1, professor 2, or professor 3. By randomly assigning students to each respective professor, there is greater probability that the groups are equivalent and thus there are no differences between the three groups with one exception�the professor they receive and his or her particular teaching and delivery style. This is the treatment. Let�s also assume that the professors will be administering the same tests and using the same textbook. After the group members are randomly assigned, a pre-treatment evaluation shows the groups are in fact equivalent on all important known factors that could influence post-test scores, such as grade point average, age, time in school, and exposure to research methods concepts. Additionally, all groups scored comparably on a pre-test of knowledge about research methods, thus there is more confidence that the groups are in fact equivalent.

At the conclusion of the study, we find that professor 2�s group has the lowest final test scores of the three. However, because professor 2 is such an outstanding professor, the results appear odd. At first glance, the researcher thinks the results could have been influenced by students dropping out of the class. For example, perhaps several of professor 2�s students dropped the course but none did from the classes of professor 1 or 3. It is revealed, however, that an equal number of students dropped out of all three courses before the post-test and, therefore, this could not be the reason for the low scores in professor 2�s course. Upon further investigation, however, the researcher finds that although an equal number of students dropped out of each class, the dropouts in professor 2�s class were some of his best students. In contrast, those who dropped out of professor 1�s and professor 3�s courses were some of their poorest students. In this example, professor 2 appears to be the least effective teacher. However, this result appears to be due to the fact that his best students dropped out, and this highly influenced the final test average for his group. Although there was not a differential loss of subjects in terms of numbers (which can also be an attrition issue), there was differential loss in the types of students. This differential loss, not the teaching style, is an alternative explanation of the results.

Testing or Testing Bias

Another potential alternative explanation is testing or testing bias. Suppose that after the pre-test of research methods knowledge, professor 1 and professor 3 reviewed the test with their students and gave them the correct answers. Professor 2 did not. The fact that professor l�s and professor 3�s groups did better on the post-test final exam may be explained by the finding that students in those groups remembered the answers to the pre-test, were thus biased at the pre-test, and this artificially inflated their post-test scores. Testing bias can explain the results because students in groups 1 and 3 may have simply remembered the answers from the pre-test review. In fact, the students in professor l�s and 3�s courses may have scored high on the post-test without ever having been exposed to the treatment because they were biased at the pre-test.

Instrumentation

Another alternative explanation that can arise is instrumentation. Instrumentation refers to changes in the measuring instrument from pre- to post-test. Using the previous example, suppose professors 1 and 3 did not give the same final exam as professor 2. For example, professors 1 and 3 changed the final exam and professor 2 kept the final exam the same as the pretest. Because professors 1 and 3 changed the exam, and perhaps made it easier or somehow different from the pre-test exam, results that showed lower scores for professor 2�s students may be related only to instrumentation changes from pre- to post-test. Obviously, to limit the influence of instrumentation, researchers should make sure that instruments remain consistent from pre- to post-test.

A final alternative explanation is reactivity. Reactivity occurs when members of the treatment or experimental group change their behavior simply as a result of being part of a study. This is akin to the finding that people tend to change their behavior when they are being watched or are aware they are being studied. If members of the experiment know they are part of an experiment and are being studied and watched, it is possible that their behavior will change independent of the treatment. If this occurs, the researcher will not know if the behavior change is the result of the treatment, or simply a result of being part of a study. For example, suppose a researcher wants to determine if a boot camp program impacts the recidivism of delinquent offenders. Members of the experimental group are sentenced to boot camp and members of the control group are released on their own recognizance to their parents. Because members of the experimental group know they are part of the experiment, and hence being watched closely after they exit boot camp, they may artificially change their behavior and avoid trouble. Their change of behavior may be totally unrelated to boot camp, but rather, to their knowledge of being part of an experiment.

Other Potential Alternative Explanations

The above discussion provided some typical alternative explanations that may arise with the designs discussed in this chapter. There are, however, other potential alternative explanations that may arise. These alternative explanations arise only when a control or comparison group is present.

One such alternative explanation is diffusion of treatment. Diffusion of treatment occurs when the control or comparison group learns about the treatment its members are being denied and attempts to mimic the behavior of the treatment group. If the control group is successful in mimicking the experimental group, for example, the results at the end of the study may show similarity in outcomes between groups and cause the researcher to conclude that the program had no effect. In fact, however, the finding of no effect can be explained by the comparison group mimicking the treatment group. 28 In reality, there may be no effect of the treatment, but the researcher would not know this for sure because the control group effectively transformed into another experimental group�there is then no baseline of comparison. Consider a study where a researcher wants to determine the impact of a training program on class behavior and participation. In this study, the experimental group is exposed to several sessions of training on how to act appropriately in class and how to engage in class participation. The control group does not receive such training, but they are aware that they are part of an experiment. Suppose after a few class sessions the control group starts to mimic the behavior of the experimental group, acting the same way and participating in class the same way. At the conclusion of the study, the researcher might determine that the program had no impact because the comparison group, which did not receive the new program, showed similar progress.

In a related explanation, sometimes the comparison or control group learns about the experiment and attempts to compete with the experimental or treatment group. This alternative explanation is called compensatory rivalry. For example, suppose a police chief wants to determine if a new training program will increase the endurance of SWAT team officers. The chief randomly assigns SWAT members to either an experimental or control group. The experimental group will receive the new endurance training program and the control group will receive the normal program that has been used for years. During the course of the study, suppose the control group learns that the treatment group is receiving the new endurance program and starts to compete with the experimental group. Perhaps the control group runs five more miles per day and works out an extra hour in the weight room, in addition to their normal endurance program. At the end of the study, and due to the control group�s extra and competing effort, the results might show no effect of the new endurance program, and at worst, experimental group members may show a decline in endurance compared to the control group. The rivalry or competing behavior actually explains the results, not that the new endurance program has no effect or a damaging effect. Although the new endurance program may in reality have no effect, this cannot be known because of the actions of the control group, who learned about the treatment and competed with the experimental group.

Closely related to compensatory rivalry is the alternative explanation of comparison or control group demoralization. 29 In this instance, instead of competing with the experimental or treatment group, the control or comparison group simply gives up and changes their normal behavior. Using the SWAT example, perhaps the control group simply quits their normal endurance program when they learn about the treatment group receiving the new endurance program. At the post-test, their endurance will likely drop considerably compared to the treatment group. Because of this, the new endurance program might emerge as a shining success. In reality, however, the researcher will not know if any changes in endurance between the experimental and control groups are a result of the new endurance program or the control group giving up. Due to their giving up, there is no longer a comparison group of equitable others, the change in endurance among the treatment group members could be attributed to a number of alternative explanations, for example, maturation. If the comparison group behaves normally, the researcher will be able to exclude maturation as a potential explanation. This is because any maturation effects will occur in both groups.

The previous discussion suggests that when the control or comparison group learns about the experiment and the treatment they are denied, potential alternative explanations can arise. Perhaps the best remedy to protect from the alternative explanations just discussed is to make sure the treatment and comparison groups do not have contact with one another. In laboratory experiments this can be ensured, but sometimes this is a problem in criminal justice studies, which are often conducted in the field.

The previous discussion also suggests that there are numerous alternative explanations that can impact the interpretation of results from a study. A careful researcher would know that alternative explanations must be ruled out before reaching a definitive conclusion about the impact of a particular program. The researcher must be attuned to these potential alternative explanations because they can influence results and how results are interpreted. Moreover, the discussion shows that several alternative explanations can occur at the same time. For example, it is possible that selection bias, maturation, attrition, and compensatory rivalry all emerge as alternative explanations in the same study. Knowing about these potential alternative explanations and how they can impact the results of a study is what distinguishes a consumer of research from an educated consumer of research.

Chapter Summary

The primary focus of this chapter was the classic experimental design, the foundation for other types of experimental and quasi-experimental designs. The classic experimental design is perhaps the most useful design when exploring causal relationships. Often, however, researchers cannot employ the classic experimental design to answer a research question. In fact, the classic experimental design is rare in criminal justice and criminology because it is often difficult to ensure random assignment for a variety of reasons. In circumstances where an experimental design is appropriate but not feasible, researchers may turn to one of many quasi-experimental designs. The most important difference between the two is that quasi-experimental designs do not feature random assignment. This can create potential problems for researchers. The main problem is that there is a greater chance the treatment and comparison groups may differ on important characteristics that could influence the results of a study. Although researchers can attempt to prevent imbalances between the groups by matching them on important known characteristics, it is still much more difficult to establish equivalence than it is in the classic experiment. As such, it becomes more difficult to determine what impact a treatment had, if any, as one moves from an experimental to a quasi-experimental design.

Perhaps the most important lesson to be learned in this chapter is that to be an educated consumer of research results requires an understanding of the type of design that produced the results. There are numerous ways experimental and quasi-experimental designs can be structured. This is why much attention was paid to the classic experimental design. In reality, all experimental and quasi-experimental designs are variations of the classic experiment in some way�adding or deleting certain components. If the components and organization and logic of the classic experimental design are understood, consumers of research will have a better understanding of the results produced from any sort of research design. For example, what problems in interpretation arise when a design lacks a pre-test, a control group, or random assignment? Having an answer to this question is a good start toward being an informed consumer of research results produced through experimental and quasi-experimental designs.

Critical Thinking Questions

1. Why is randomization/random assignment preferable to matching? Provide several reasons with explanation.

2. What are some potential reasons a researcher would not be able to utilize random assignment?

3. What is a major limitation of matching?

4. What is the difference between a longitudinal study and a cross-sectional study?

5. Describe a hypothetical study where maturation, and not the treatment, could explain the outcomes of the research.

association (or covariance or correlation): One of three conditions that must be met for establishing cause and effect, or a causal relationship. Association refers to the condition that X and Y must be related for a causal relationship to exist. Association is also referred to as covariance or correlation. Although two variables may be associated (or covary or be correlated), this does not automatically imply that they are causally related

attrition or subject mortality: A threat to internal validity, it refers to the differential loss of subjects between the experimental (treatment) and control (comparison) groups during the course of a study

cause and effect relationship: A cause and effect relationship occurs when one variable causes another, and no other explanation for that relationship exists

classic experimental design or experimental design: A design in a research study that features random assignment to an experimental or control group. Experimental designs can vary tremendously, but a constant feature is random assignment, experimental and control groups, and a post-test. For example, a classic experimental design features random assignment, a treatment, experimental and control groups, and pre- and post-tests

comparison group: The group in a quasi-experimental design that does not receive the treatment. In an experimental design, the comparison group is referred to as the control group

compensatory rivalry: A threat to internal validity, it occurs when the control or comparison group attempts to compete with the experimental or treatment group

control group: In an experimental design, the control group does not receive the treatment. The control group serves as a baseline of comparison to the experimental group. It serves as an example of what happens when a group equivalent to the experimental group does not receive the treatment

cross-sectional designs: A measurement of the pre-test and post-test at one point in time (e.g., six months before and six months after the program)

demoralization: A threat to internal validity closely associated with compensatory rivalry, it occurs when the control or comparison group gives up and changes their normal behavior. While in compensatory rivalry the group members compete, in demoralization, they simply quit. Both are not normal behavioral reactions

dependent variable: Also known as the outcome in a research study. A post-test is a measure of the dependent variable

diffusion of treatment: A threat to internal validity, it occurs when the control or comparison group members learn that they are not getting the treatment and attempt to mimic the behavior of the experimental or treatment group. This mimicking may make it seem as if the treatment is having no effect, when in fact it may be

elimination of alternative explanations: One of three conditions that must be met for establishing cause and effect. Elimination of alternative explanations means that the researcher has ruled out other explanations for an observed relationship between X and Y

experimental group: In an experimental design, the experimental group receives the treatment

history: A threat to internal validity, it refers to any event experienced differently by the treatment and comparison groups�an event that could explain the results other than the supposed cause

independent variable: Also called the cause

instrumentation: A threat to internal validity, it refers to changes in the measuring instrument from pre- to post-test

longitudinal: Refers to repeated measurements of the pre-test and post-test over time, typically for the same group of individuals. This is the opposite of cross-sectional

matching: A process sometimes utilized in some quasi-experimental designs that feature treatment and comparison groups. Matching is a process whereby the researcher attempts to ensure equivalence between the treatment and comparison groups on known information, in the absence of the ability to randomly assign the groups

maturation: A threat to internal validity, maturation refers to the natural biological, psychological, or emotional processes as time passes

negative association: Refers to a negative association between two variables. A negative association is demonstrated when X increases and Y decreases, or X decreases and Y increases. Also known as an inverse relationship�the variables moving in opposite directions

operationalized or operationalization: Refers to the process of assigning a working definition to a concept. For example, the concept of intelligence can be operationalized or defined as grade point average or score on a standardized exam, among others

pilot program or test: Refers to a smaller test study or pilot to work out problems before a larger study and to anticipate changes needed for a larger study. Similar to a test run

positive association: Refers to a positive association between two variables. A positive association means as X increases, Y increases, or as X decreases, Y decreases

post-test: The post-test is a measure of the dependent variable after the treatment has been administered

pre-test: The pre-test is a measure of the dependent variable or outcome before a treatment is administered

quasi-experiment: A quasi-experiment refers to any number of research design configurations that resemble an experimental design but primarily lack random assignment. In the absence of random assignment, quasi-experimental designs feature matching to attempt equivalence

random assignment: Refers to a process whereby members of the experimental group and control group are assigned to each group through a random and unbiased process

random selection: Refers to selecting a smaller but representative subset from a population. Not to be confused with random assignment

reactivity: A threat to internal validity, it occurs when members of the experimental (treatment) or control (comparison) group change their behavior unnaturally as a result of being part of a study

selection bias: A threat to internal validity, selection bias occurs when the experimental (treatment) group and control (comparison) group are not equivalent. The difference between the groups can be a threat to internal validity, or, an alternative explanation to the findings

spurious: A spurious relationship is one where X and Y appear to be causally related, but in fact the relationship is actually explained by a variable or factor other than X

testing or testing bias: A threat to internal validity, it refers to the potential of study members being biased prior to a treatment, and this bias, rather than the treatment, may explain study results

threat to internal validity: Also known as alternative explanation to a relationship between X and Y. Threats to internal validity are factors that explain Y, or the dependent variable, and are not X, or the independent variable

timing: One of three conditions that must be met for establishing cause and effect. Timing refers to the condition that X must come before Y in time for X to be a cause of Y. While timing is necessary for a causal relationship, it is not sufficient, and considerations of association and eliminating other alternative explanations must be met

treatment: A component of a research design, it is typically denoted by the letter X. In a research study on the impact of teen court on juvenile recidivism, teen court is the treatment. In a classic experimental design, the treatment is given only to the experimental group, not the control group

treatment group: The group in a quasi-experimental design that receives the treatment. In an experimental design, this group is called the experimental group

unit of analysis: Refers to the focus of a research study as being individuals, groups, or other units of analysis, such as prisons or police agencies, and so on

variable(s): A variable is a concept that has been given a working definition and can take on different values. For example, intelligence can be defined as a person�s grade point average and can range from low to high or can be defined numerically by different values such as 3.5 or 4.0

1 Povitsky, W., N. Connell, D. Wilson, & D. Gottfredson. (2008). �An experimental evaluation of teen courts.� Journal of Experimental Criminology, 4, 137�163.

2 Hirschi, T., and H. Selvin (1966). �False criteria of causality in delinquency.� Social Problems, 13, 254�268.

3 Robert Roy Britt, �Churchgoers Live Longer.� April, 3, 2006. http://www.livescience.com/health/060403_church_ good.html. Retrieved on September 30, 2008.

4 Kalist, D., and D. Yee (2009). �First names and crime: Does unpopularity spell trouble?� Social Science Quarterly, 90 (1), 39�48.

5 Sherman, L. (1992). Policing domestic violence. New York: The Free Press.

6 For historical and interesting reading on the effects of weather on crime and other disorder, see Dexter, E. (1899). �Influence of weather upon crime.� Popular Science Monthly, 55, 653�660 in Horton, D. (2000). Pioneering Perspectives in Criminology. Incline Village, NV: Copperhouse.

7 http://www.escapistmagazine.com/news/view/111191-Less-Crime-in-U-S-Thanks-to-Videogames , retrieved on September 13, 2011. This news article was in response to a study titled �Understanding the effects of violent videogames on violent crime.� See Cunningham, Scott, Engelst�tter, Benjamin, and Ward, (April 7, 2011). Available at SSRN: http://ssm.com/abstract= 1804959.

8 Cohn, E. G. (1987). �Changing the domestic violence policies of urban police departments: Impact of the Minneapolis experiment.� Response, 10 (4), 22�24.

9 Schmidt, Janell D., & Lawrence W. Sherman (1993). �Does arrest deter domestic violence?� American Behavioral Scientist, 36 (5), 601�610.

10 Maxwell, Christopher D., Joel H. Gamer, & Jeffrey A. Fagan. (2001). The effects of arrest on intimate partner violence: New evidence for the spouse assault replication program. Washington D.C.: National Institute of Justice.

11 Miller, N. (2005). What does research and evaluation say about domestic violence laws? A compendium of justice system laws and related research assessments. Alexandria, VA: Institute for Law and Justice.

12 The sections on experimental and quasi-experimental designs rely heavily on the seminal work of Campbell and Stanley (Campbell, D.T., & J. C. Stanley. (1963). Experimental and quasi-experimental designs for research. Chicago: RandMcNally) and more recently, Shadish, W., T. Cook, & D. Campbell. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.

13 Povitsky et al. (2008). p. 146, note 9.

14 Shadish, W., T. Cook, & D. Campbell. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin Company.

15 Ibid, 15.

16 Finckenauer, James O. (1982). Scared straight! and the panacea phenomenon. Englewood Cliffs, N.J.: Prentice Hall.

17 Yarborough, J.C. (1979). Evaluation of JOLT (Juvenile Offenders Learn Truth) as a deterrence program. Lansing, MI: Michigan Department of Corrections.

18 Petrosino, Anthony, Carolyn Turpin-Petrosino, & James O. Finckenauer. (2000). �Well-meaning programs can have harmful effects! Lessons from experiments of programs such as Scared Straight.� Crime and Delinquency, 46, 354�379.

19 �Swearing makes pain more tolerable� retrieved at http:// www.livescience.com/health/090712-swearing-pain.html (July 13, 2009). Also see �Bleep! My finger! Why swearing helps ease pain� by Tiffany Sharpies, retrieved at http://www.time.com/time/health/article /0,8599,1910691,00.html?xid=rss-health (July 16, 2009).

20 For an excellent discussion of the value of controlled experiments and why they are so rare in the social sciences, see Sherman, L. (1992). Policing domestic violence. New York: The Free Press, 55�74.

21 For discussion, see Weisburd, D., T. Einat, & M. Kowalski. (2008). �The miracle of the cells: An experimental study of interventions to increase payment of court-ordered financial obligations.� Criminology and Public Policy, 7, 9�36.

22 Shadish, Cook, & Campbell. (2002).

24 Kelly, Cathy. (March 15, 2009). �Tickets in the mail: Red-light cameras questioned.� Santa Cruz Sentinel.

25 Retting, Richard, Susan Ferguson, & Charles Farmer. (January 2007). �Reducing red light running through longer yellow signal timing and red light camera enforcement: Results of a field investigation.� Arlington, VA: Insurance Institute for Highway Safety.

26 Shadish, Cook, & Campbell. (2002).

27 See Shadish, Cook, & Campbell. (2002), pp. 54�61 for an excellent discussion of threats to internal validity. Also see Chapter 2 for an extended discussion of all forms of validity considered in research design.

28 Trochim, W. (2001). The research methods knowledge base, 2nd ed. Cincinnati, OH: Atomic Dog.

Applied Research Methods in Criminal Justice and Criminology Copyright © 2022 by University of North Texas is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Share This Book

Our systems are now restored following recent technical disruption, and we’re working hard to catch up on publishing. We apologise for the inconvenience caused. Find out more: https://www.cambridge.org/universitypress/about-us/news-and-blogs/cambridge-university-press-publishing-update-following-technical-disruption

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
• > Quasi-Experimental Research

Book contents

• The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
• Cambridge Handbooks in Psychology
• Copyright page
• Contributors
• Part I From Idea to Reality: The Basics of Research
• Part II The Building Blocks of a Study
• Part III Data Collection
• 13 Cross-Sectional Studies
• 14 Quasi-Experimental Research
• 15 Non-equivalent Control Group Pretest–Posttest Design in Social and Behavioral Research
• 16 Experimental Methods
• 17 Longitudinal Research: A World to Explore
• 18 Online Research Methods
• 19 Archival Data
• 20 Qualitative Research Design
• Part IV Statistical Approaches
• Part V Tips for a Successful Research Career

14 - Quasi-Experimental Research

from Part III - Data Collection

Published online by Cambridge University Press:  25 May 2023

In this chapter, we discuss the logic and practice of quasi-experimentation. Specifically, we describe four quasi-experimental designs – one-group pretest–posttest designs, non-equivalent group designs, regression discontinuity designs, and interrupted time-series designs – and their statistical analyses in detail. Both simple quasi-experimental designs and embellishments of these simple designs are presented. Potential threats to internal validity are illustrated along with means of addressing their potentially biasing effects so that these effects can be minimized. In contrast to quasi-experiments, randomized experiments are often thought to be the gold standard when estimating the effects of treatment interventions. However, circumstances frequently arise where quasi-experiments can usefully supplement randomized experiments or when quasi-experiments can fruitfully be used in place of randomized experiments. Researchers need to appreciate the relative strengths and weaknesses of the various quasi-experiments so they can choose among pre-specified designs or craft their own unique quasi-experiments.

Access options

Save book to kindle.

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service .

• Quasi-Experimental Research
• By Charles S. Reichardt , Daniel Storage , Damon Abraham
• Edited by Austin Lee Nichols , Central European University, Vienna , John Edlund , Rochester Institute of Technology, New York
• Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
• Online publication: 25 May 2023
• Chapter DOI: https://doi.org/10.1017/9781009010054.015

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox .

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive .

Instant insights, infinite possibilities

The use and interpretation of quasi-experimental design

Last updated

6 February 2023

Reviewed by

Miroslav Damyanov

Short on time? Get an AI generated summary of this article instead

• What is a quasi-experimental design?

Commonly used in medical informatics (a field that uses digital information to ensure better patient care), researchers generally use this design to evaluate the effectiveness of a treatment – perhaps a type of antibiotic or psychotherapy, or an educational or policy intervention.

Even though quasi-experimental design has been used for some time, relatively little is known about it. Read on to learn the ins and outs of this research design.

Make research less tedious

Dovetail streamlines research to help you uncover and share actionable insights

• When to use a quasi-experimental design

A quasi-experimental design is used when it's not logistically feasible or ethical to conduct randomized, controlled trials. As its name suggests, a quasi-experimental design is almost a true experiment. However, researchers don't randomly select elements or participants in this type of research.

Researchers prefer to apply quasi-experimental design when there are ethical or practical concerns. Let's look at these two reasons more closely.

Ethical reasons

In some situations, the use of randomly assigned elements can be unethical. For instance, providing public healthcare to one group and withholding it to another in research is unethical. A quasi-experimental design would examine the relationship between these two groups to avoid physical danger.

Practical reasons

Randomized controlled trials may not be the best approach in research. For instance, it's impractical to trawl through large sample sizes of participants without using a particular attribute to guide your data collection .

Recruiting participants and properly designing a data-collection attribute to make the research a true experiment requires a lot of time and effort, and can be expensive if you don’t have a large funding stream.

A quasi-experimental design allows researchers to take advantage of previously collected data and use it in their study.

• Examples of quasi-experimental designs

Quasi-experimental research design is common in medical research, but any researcher can use it for research that raises practical and ethical concerns. Here are a few examples of quasi-experimental designs used by different researchers:

Example 1: Determining the effectiveness of math apps in supplementing math classes

A school wanted to supplement its math classes with a math app. To select the best app, the school decided to conduct demo tests on two apps before selecting the one they will purchase.

Scope of the research

Since every grade had two math teachers, each teacher used one of the two apps for three months. They then gave the students the same math exams and compared the results to determine which app was most effective.

Reasons why this is a quasi-experimental study

This simple study is a quasi-experiment since the school didn't randomly assign its students to the applications. They used a pre-existing class structure to conduct the study since it was impractical to randomly assign the students to each app.

Example 2: Determining the effectiveness of teaching modern leadership techniques in start-up businesses

A hypothetical quasi-experimental study was conducted in an economically developing country in a mid-sized city.

Five start-ups in the textile industry and five in the tech industry participated in the study. The leaders attended a six-week workshop on leadership style, team management, and employee motivation.

After a year, the researchers assessed the performance of each start-up company to determine growth. The results indicated that the tech start-ups were further along in their growth than the textile companies.

The basis of quasi-experimental research is a non-randomized subject-selection process. This study didn't use specific aspects to determine which start-up companies should participate. Therefore, the results may seem straightforward, but several aspects may determine the growth of a specific company, apart from the variables used by the researchers.

Example 3: A study to determine the effects of policy reforms and of luring foreign investment on small businesses in two mid-size cities

In a study to determine the economic impact of government reforms in an economically developing country, the government decided to test whether creating reforms directed at small businesses or luring foreign investments would spur the most economic development.

The government selected two cities with similar population demographics and sizes. In one of the cities, they implemented specific policies that would directly impact small businesses, and in the other, they implemented policies to attract foreign investment.

After five years, they collected end-of-year economic growth data from both cities. They looked at elements like local GDP growth, unemployment rates, and housing sales.

The study used a non-randomized selection process to determine which city would participate in the research. Researchers left out certain variables that would play a crucial role in determining the growth of each city. They used pre-existing groups of people based on research conducted in each city, rather than random groups.

• Advantages of a quasi-experimental design

Some advantages of quasi-experimental designs are:

Researchers can manipulate variables to help them meet their study objectives.

It offers high external validity, making it suitable for real-world applications, specifically in social science experiments.

Integrating this methodology into other research designs is easier, especially in true experimental research. This cuts down on the time needed to determine your outcomes.

• Disadvantages of a quasi-experimental design

Despite the pros that come with a quasi-experimental design, there are several disadvantages associated with it, including the following:

It has a lower internal validity since researchers do not have full control over the comparison and intervention groups or between time periods because of differences in characteristics in people, places, or time involved. It may be challenging to determine whether all variables have been used or whether those used in the research impacted the results.

There is the risk of inaccurate data since the research design borrows information from other studies.

There is the possibility of bias since researchers select baseline elements and eligibility.

• What are the different quasi-experimental study designs?

There are three distinct types of quasi-experimental designs:

Nonequivalent

Regression discontinuity, natural experiment.

This is a hybrid of experimental and quasi-experimental methods and is used to leverage the best qualities of the two. Like the true experiment design, nonequivalent group design uses pre-existing groups believed to be comparable. However, it doesn't use randomization, the lack of which is a crucial element for quasi-experimental design.

Researchers usually ensure that no confounding variables impact them throughout the grouping process. This makes the groupings more comparable.

Example of a nonequivalent group design

A small study was conducted to determine whether after-school programs result in better grades. Researchers randomly selected two groups of students: one to implement the new program, the other not to. They then compared the results of the two groups.

This type of quasi-experimental research design calculates the impact of a specific treatment or intervention. It uses a criterion known as "cutoff" that assigns treatment according to eligibility.

Researchers often assign participants above the cutoff to the treatment group. This puts a negligible distinction between the two groups (treatment group and control group).

Example of regression discontinuity

Students must achieve a minimum score to be enrolled in specific US high schools. Since the cutoff score used to determine eligibility for enrollment is arbitrary, researchers can assume that the disparity between students who only just fail to achieve the cutoff point and those who barely pass is a small margin and is due to the difference in the schools that these students attend.

Researchers can then examine the long-term effects of these two groups of kids to determine the effect of attending certain schools. This information can be applied to increase the chances of students being enrolled in these high schools.

This research design is common in laboratory and field experiments where researchers control target subjects by assigning them to different groups. Researchers randomly assign subjects to a treatment group using nature or an external event or situation.

However, even with random assignment, this research design cannot be called a true experiment since nature aspects are observational. Researchers can also exploit these aspects despite having no control over the independent variables.

Example of the natural experiment approach

An example of a natural experiment is the 2008 Oregon Health Study.

Oregon intended to allow more low-income people to participate in Medicaid.

Since they couldn't afford to cover every person who qualified for the program, the state used a random lottery to allocate program slots.

Researchers assessed the program's effectiveness by assigning the selected subjects to a randomly assigned treatment group, while those that didn't win the lottery were considered the control group.

• Differences between quasi-experiments and true experiments

There are several differences between a quasi-experiment and a true experiment:

Participants in true experiments are randomly assigned to the treatment or control group, while participants in a quasi-experiment are not assigned randomly.

In a quasi-experimental design, the control and treatment groups differ in unknown or unknowable ways, apart from the experimental treatments that are carried out. Therefore, the researcher should try as much as possible to control these differences.

Quasi-experimental designs have several "competing hypotheses," which compete with experimental manipulation to explain the observed results.

Quasi-experiments tend to have lower internal validity (the degree of confidence in the research outcomes) than true experiments, but they may offer higher external validity (whether findings can be extended to other contexts) as they involve real-world interventions instead of controlled interventions in artificial laboratory settings.

Despite the distinct difference between true and quasi-experimental research designs, these two research methodologies share the following aspects:

Both study methods subject participants to some form of treatment or conditions.

Researchers have the freedom to measure some of the outcomes of interest.

Researchers can test whether the differences in the outcomes are associated with the treatment.

• An example comparing a true experiment and quasi-experiment

Imagine you wanted to study the effects of junk food on obese people. Here's how you would do this as a true experiment and a quasi-experiment:

How to carry out a true experiment

In a true experiment, some participants would eat junk foods, while the rest would be in the control group, adhering to a regular diet. At the end of the study, you would record the health and discomfort of each group.

This kind of experiment would raise ethical concerns since the participants assigned to the treatment group are required to eat junk food against their will throughout the experiment. This calls for a quasi-experimental design.

How to carry out a quasi-experiment

In quasi-experimental research, you would start by finding out which participants want to try junk food and which prefer to stick to a regular diet. This allows you to assign these two groups based on subject choice.

In this case, you didn't assign participants to a particular group, so you can confidently use the results from the study.

When is a quasi-experimental design used?

Quasi-experimental designs are used when researchers don’t want to use randomization when evaluating their intervention.

What are the characteristics of quasi-experimental designs?

Some of the characteristics of a quasi-experimental design are:

Researchers don't randomly assign participants into groups, but study their existing characteristics and assign them accordingly.

Researchers study the participants in pre- and post-testing to determine the progress of the groups.

Quasi-experimental design is ethical since it doesn’t involve offering or withholding treatment at random.

Quasi-experimental design encompasses a broad range of non-randomized intervention studies. This design is employed when it is not ethical or logistically feasible to conduct randomized controlled trials. Researchers typically employ it when evaluating policy or educational interventions, or in medical or therapy scenarios.

How do you analyze data in a quasi-experimental design?

You can use two-group tests, time-series analysis, and regression analysis to analyze data in a quasi-experiment design. Each option has specific assumptions, strengths, limitations, and data requirements.

Should you be using a customer insights hub?

Do you want to discover previous research faster?

Do you share your research findings with others?

Do you analyze research data?

Start for free today, add your research, and get to key insights faster

Editor’s picks

Last updated: 18 April 2023

Last updated: 27 February 2023

Last updated: 22 August 2024

Last updated: 5 February 2023

Last updated: 16 August 2024

Last updated: 9 March 2023

Last updated: 30 April 2024

Last updated: 12 December 2023

Last updated: 11 March 2024

Last updated: 4 July 2024

Last updated: 6 March 2024

Last updated: 5 March 2024

Last updated: 13 May 2024

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next, log in or sign up.

Get started for free

• Link to facebook
• Link to linkedin
• Link to twitter
• Link to youtube
• Writing Tips

An Introduction to Quasi-Experimental Design

• 3-minute read
• 9th January 2022

If you’re a researcher or student in the sciences, you’ll probably come across the term “quasi-experimental design” at some point. But what exactly does it mean?

In this post, we’ll guide you through the different forms of quasi-experimental design and how it compares to true experiments.

What is Quasi-Experimental Design?

Quasi-experimental design (QED) is a research design method that’s useful when regular experimental conditions are impractical or unethical.

Both quasi-experimental designs and true experiments show a cause-and-effect relationship between a dependent and independent variable . Participants in a true experiment are randomly assigned to different treatment groups. The quasi-experimental design, on the other hand, assigns groups based on criteria instead of randomly.

Quasi-Experimental Design Vs. True Experimental Design

The main difference between a quasi-experimental and true experimental design is that in the former, groups aren’t randomly assigned. There are also some other key differences between these research methods.

True experimental design involves:

●     Having control as a researcher over the design of the treatment or program that participants receive (i.e., the independent variable)

●     Control variables as a necessary component

In contrast, a quasi-experimental design involves:

●     The researcher studying groups after they’ve received a treatment or program

●     Control variables as a common element but they aren’t necessary for the experiment to work

Examples of Experimental Design

Perhaps the easiest way to understand quasi-experimental design is to look at how it might be used in practice.

Let’s say you hypothesize that having access to free art lessons will improve the mental health of children from low-income families.

In a true experiment, you’d randomly assign participants to two groups: one that receives free art lessons and another that doesn’t.

However, it’s ethically questionable to deny one group of children access to something that might benefit them.

Find this useful?

Subscribe to our newsletter and get writing tips from our editors straight to your inbox.

Instead, you might decide to compare the data from a community that’s already offered free art classes to these children with that of a community that’s not yet done so.

This second example would be a quasi-experimental design.

Advantages and Disadvantages of Quasi-Experimental Design

Quasi-experimental design has some advantages and disadvantages you’ll need to consider when designing your research.

On the plus side, quasi-experimental design:

●     Has a higher external validity than true experimental design, as it usually involves real-world scenarios

●     Allows you to control for unexpected, confounding variables, resulting in a higher internal validity than other non-experimental methods of research

●     Enables the study of cause-and-effect relationships without the ethical issue of denying a treatment to those who may benefit from it

●     Does not require access to large-scale funding and other practical concerns, as the treatment has already been issued by others

The disadvantages of quasi-experimental design, however, include:

●     Lower internal validity than found in true experiments, as it’s more difficult to account for all confounding variables without using random assignment

●     The necessary data required for research potentially being inaccurate, outdated, or difficult to access

Expert Proofreading for Researchers

We hope our guide has helped you understand the basics of quasi-experimental design.

If you need help with your research paper , our expert proofreaders are available 24/7. Try us out by submitting a free sample document today.

Share this article:

Post A New Comment

Got content that needs a quick turnaround? Let us polish your work. Explore our editorial business services.

5-minute read

Free Email Newsletter Template

Promoting a brand means sharing valuable insights to connect more deeply with your audience, and...

6-minute read

How to Write a Nonprofit Grant Proposal

If you’re seeking funding to support your charitable endeavors as a nonprofit organization, you’ll need...

9-minute read

How to Use Infographics to Boost Your Presentation

Is your content getting noticed? Capturing and maintaining an audience’s attention is a challenge when...

8-minute read

Why Interactive PDFs Are Better for Engagement

Are you looking to enhance engagement and captivate your audience through your professional documents? Interactive...

7-minute read

Seven Key Strategies for Voice Search Optimization

Voice search optimization is rapidly shaping the digital landscape, requiring content professionals to adapt their...

4-minute read

Five Creative Ways to Showcase Your Digital Portfolio

Are you a creative freelancer looking to make a lasting impression on potential clients or...

Make sure your writing is the best it can be with our expert English proofreading and editing.

Experimental and Quasi-Experimental Research

Guide Title: Experimental and Quasi-Experimental Research Guide ID: 64

You approach a stainless-steel wall, separated vertically along its middle where two halves meet. After looking to the left, you see two buttons on the wall to the right. You press the top button and it lights up. A soft tone sounds and the two halves of the wall slide apart to reveal a small room. You step into the room. Looking to the left, then to the right, you see a panel of more buttons. You know that you seek a room marked with the numbers 1-0-1-2, so you press the button marked "10." The halves slide shut and enclose you within the cubicle, which jolts upward. Soon, the soft tone sounds again. The door opens again. On the far wall, a sign silently proclaims, "10th floor."

You have engaged in a series of experiments. A ride in an elevator may not seem like an experiment, but it, and each step taken towards its ultimate outcome, are common examples of a search for a causal relationship-which is what experimentation is all about.

You started with the hypothesis that this is in fact an elevator. You proved that you were correct. You then hypothesized that the button to summon the elevator was on the left, which was incorrect, so then you hypothesized it was on the right, and you were correct. You hypothesized that pressing the button marked with the up arrow would not only bring an elevator to you, but that it would be an elevator heading in the up direction. You were right.

As this guide explains, the deliberate process of testing hypotheses and reaching conclusions is an extension of commonplace testing of cause and effect relationships.

Basic Concepts of Experimental and Quasi-Experimental Research

Discovering causal relationships is the key to experimental research. In abstract terms, this means the relationship between a certain action, X, which alone creates the effect Y. For example, turning the volume knob on your stereo clockwise causes the sound to get louder. In addition, you could observe that turning the knob clockwise alone, and nothing else, caused the sound level to increase. You could further conclude that a causal relationship exists between turning the knob clockwise and an increase in volume; not simply because one caused the other, but because you are certain that nothing else caused the effect.

Independent and Dependent Variables

Beyond discovering causal relationships, experimental research further seeks out how much cause will produce how much effect; in technical terms, how the independent variable will affect the dependent variable. You know that turning the knob clockwise will produce a louder noise, but by varying how much you turn it, you see how much sound is produced. On the other hand, you might find that although you turn the knob a great deal, sound doesn't increase dramatically. Or, you might find that turning the knob just a little adds more sound than expected. The amount that you turned the knob is the independent variable, the variable that the researcher controls, and the amount of sound that resulted from turning it is the dependent variable, the change that is caused by the independent variable.

Experimental research also looks into the effects of removing something. For example, if you remove a loud noise from the room, will the person next to you be able to hear you? Or how much noise needs to be removed before that person can hear you?

Treatment and Hypothesis

The term treatment refers to either removing or adding a stimulus in order to measure an effect (such as turning the knob a little or a lot, or reducing the noise level a little or a lot). Experimental researchers want to know how varying levels of treatment will affect what they are studying. As such, researchers often have an idea, or hypothesis, about what effect will occur when they cause something. Few experiments are performed where there is no idea of what will happen. From past experiences in life or from the knowledge we possess in our specific field of study, we know how some actions cause other reactions. Experiments confirm or reconfirm this fact.

Experimentation becomes more complex when the causal relationships they seek aren't as clear as in the stereo knob-turning examples. Questions like "Will olestra cause cancer?" or "Will this new fertilizer help this plant grow better?" present more to consider. For example, any number of things could affect the growth rate of a plant-the temperature, how much water or sun it receives, or how much carbon dioxide is in the air. These variables can affect an experiment's results. An experimenter who wants to show that adding a certain fertilizer will help a plant grow better must ensure that it is the fertilizer, and nothing else, affecting the growth patterns of the plant. To do this, as many of these variables as possible must be controlled.

Matching and Randomization

In the example used in this guide (you'll find the example below), we discuss an experiment that focuses on three groups of plants -- one that is treated with a fertilizer named MegaGro, another group treated with a fertilizer named Plant!, and yet another that is not treated with fetilizer (this latter group serves as a "control" group). In this example, even though the designers of the experiment have tried to remove all extraneous variables, results may appear merely coincidental. Since the goal of the experiment is to prove a causal relationship in which a single variable is responsible for the effect produced, the experiment would produce stronger proof if the results were replicated in larger treatment and control groups.

Selecting groups entails assigning subjects in the groups of an experiment in such a way that treatment and control groups are comparable in all respects except the application of the treatment. Groups can be created in two ways: matching and randomization. In the MegaGro experiment discussed below, the plants might be matched according to characteristics such as age, weight and whether they are blooming. This involves distributing these plants so that each plant in one group exactly matches characteristics of plants in the other groups. Matching may be problematic, though, because it "can promote a false sense of security by leading [the experimenter] to believe that [the] experimental and control groups were really equated at the outset, when in fact they were not equated on a host of variables" (Jones, 291). In other words, you may have flowers for your MegaGro experiment that you matched and distributed among groups, but other variables are unaccounted for. It would be difficult to have equal groupings.

Randomization, then, is preferred to matching. This method is based on the statistical principle of normal distribution. Theoretically, any arbitrarily selected group of adequate size will reflect normal distribution. Differences between groups will average out and become more comparable. The principle of normal distribution states that in a population most individuals will fall within the middle range of values for a given characteristic, with increasingly fewer toward either extreme (graphically represented as the ubiquitous "bell curve").

Differences between Quasi-Experimental and Experimental Research

Thus far, we have explained that for experimental research we need:

• a hypothesis for a causal relationship;
• a control group and a treatment group;
• to eliminate confounding variables that might mess up the experiment and prevent displaying the causal relationship; and
• to have larger groups with a carefully sorted constituency; preferably randomized, in order to keep accidental differences from fouling things up.

But what if we don't have all of those? Do we still have an experiment? Not a true experiment in the strictest scientific sense of the term, but we can have a quasi-experiment, an attempt to uncover a causal relationship, even though the researcher cannot control all the factors that might affect the outcome.

A quasi-experimenter treats a given situation as an experiment even though it is not wholly by design. The independent variable may not be manipulated by the researcher, treatment and control groups may not be randomized or matched, or there may be no control group. The researcher is limited in what he or she can say conclusively.

The significant element of both experiments and quasi-experiments is the measure of the dependent variable, which it allows for comparison. Some data is quite straightforward, but other measures, such as level of self-confidence in writing ability, increase in creativity or in reading comprehension are inescapably subjective. In such cases, quasi-experimentation often involves a number of strategies to compare subjectivity, such as rating data, testing, surveying, and content analysis.

Rating essentially is developing a rating scale to evaluate data. In testing, experimenters and quasi-experimenters use ANOVA (Analysis of Variance) and ANCOVA (Analysis of Co-Variance) tests to measure differences between control and experimental groups, as well as different correlations between groups.

Since we're mentioning the subject of statistics, note that experimental or quasi-experimental research cannot state beyond a shadow of a doubt that a single cause will always produce any one effect. They can do no more than show a probability that one thing causes another. The probability that a result is the due to random chance is an important measure of statistical analysis and in experimental research.

Example: Causality

Let's say you want to determine that your new fertilizer, MegaGro, will increase the growth rate of plants. You begin by getting a plant to go with your fertilizer. Since the experiment is concerned with proving that MegaGro works, you need another plant, using no fertilizer at all on it, to compare how much change your fertilized plant displays. This is what is known as a control group.

Set up with a control group, which will receive no treatment, and an experimental group, which will get MegaGro, you must then address those variables that could invalidate your experiment. This can be an extensive and exhaustive process. You must ensure that you use the same plant; that both groups are put in the same kind of soil; that they receive equal amounts of water and sun; that they receive the same amount of exposure to carbon-dioxide-exhaling researchers, and so on. In short, any other variable that might affect the growth of those plants, other than the fertilizer, must be the same for both plants. Otherwise, you can't prove absolutely that MegaGro is the only explanation for the increased growth of one of those plants.

Such an experiment can be done on more than two groups. You may not only want to show that MegaGro is an effective fertilizer, but that it is better than its competitor brand of fertilizer, Plant! All you need to do, then, is have one experimental group receiving MegaGro, one receiving Plant! and the other (the control group) receiving no fertilizer. Those are the only variables that can be different between the three groups; all other variables must be the same for the experiment to be valid.

Controlling variables allows the researcher to identify conditions that may affect the experiment's outcome. This may lead to alternative explanations that the researcher is willing to entertain in order to isolate only variables judged significant. In the MegaGro experiment, you may be concerned with how fertile the soil is, but not with the plants'; relative position in the window, as you don't think that the amount of shade they get will affect their growth rate. But what if it did? You would have to go about eliminating variables in order to determine which is the key factor. What if one receives more shade than the other and the MegaGro plant, which received more shade, died? This might prompt you to formulate a plausible alternative explanation, which is a way of accounting for a result that differs from what you expected. You would then want to redo the study with equal amounts of sunlight.

Methods: Five Steps

Experimental research can be roughly divided into five phases:

Identifying a research problem

The process starts by clearly identifying the problem you want to study and considering what possible methods will affect a solution. Then you choose the method you want to test, and formulate a hypothesis to predict the outcome of the test.

For example, you may want to improve student essays, but you don't believe that teacher feedback is enough. You hypothesize that some possible methods for writing improvement include peer workshopping, or reading more example essays. Favoring the former, your experiment would try to determine if peer workshopping improves writing in high school seniors. You state your hypothesis: peer workshopping prior to turning in a final draft will improve the quality of the student's essay.

Planning an experimental research study

The next step is to devise an experiment to test your hypothesis. In doing so, you must consider several factors. For example, how generalizable do you want your end results to be? Do you want to generalize about the entire population of high school seniors everywhere, or just the particular population of seniors at your specific school? This will determine how simple or complex the experiment will be. The amount of time funding you have will also determine the size of your experiment.

Continuing the example from step one, you may want a small study at one school involving three teachers, each teaching two sections of the same course. The treatment in this experiment is peer workshopping. Each of the three teachers will assign the same essay assignment to both classes; the treatment group will participate in peer workshopping, while the control group will receive only teacher comments on their drafts.

Conducting the experiment

At the start of an experiment, the control and treatment groups must be selected. Whereas the "hard" sciences have the luxury of attempting to create truly equal groups, educators often find themselves forced to conduct their experiments based on self-selected groups, rather than on randomization. As was highlighted in the Basic Concepts section, this makes the study a quasi-experiment, since the researchers cannot control all of the variables.

For the peer workshopping experiment, let's say that it involves six classes and three teachers with a sample of students randomly selected from all the classes. Each teacher will have a class for a control group and a class for a treatment group. The essay assignment is given and the teachers are briefed not to change any of their teaching methods other than the use of peer workshopping. You may see here that this is an effort to control a possible variable: teaching style variance.

Analyzing the data

The fourth step is to collect and analyze the data. This is not solely a step where you collect the papers, read them, and say your methods were a success. You must show how successful. You must devise a scale by which you will evaluate the data you receive, therefore you must decide what indicators will be, and will not be, important.

Continuing our example, the teachers' grades are first recorded, then the essays are evaluated for a change in sentence complexity, syntactical and grammatical errors, and overall length. Any statistical analysis is done at this time if you choose to do any. Notice here that the researcher has made judgments on what signals improved writing. It is not simply a matter of improved teacher grades, but a matter of what the researcher believes constitutes improved use of the language.

Writing the paper/presentation describing the findings

Once you have completed the experiment, you will want to share findings by publishing academic paper (or presentations). These papers usually have the following format, but it is not necessary to follow it strictly. Sections can be combined or not included, depending on the structure of the experiment, and the journal to which you submit your paper.

• Abstract : Summarize the project: its aims, participants, basic methodology, results, and a brief interpretation.
• Introduction : Set the context of the experiment.
• Review of Literature : Provide a review of the literature in the specific area of study to show what work has been done. Should lead directly to the author's purpose for the study.
• Statement of Purpose : Present the problem to be studied.
• Participants : Describe in detail participants involved in the study; e.g., how many, etc. Provide as much information as possible.
• Materials and Procedures : Clearly describe materials and procedures. Provide enough information so that the experiment can be replicated, but not so much information that it becomes unreadable. Include how participants were chosen, the tasks assigned them, how they were conducted, how data were evaluated, etc.
• Results : Present the data in an organized fashion. If it is quantifiable, it is analyzed through statistical means. Avoid interpretation at this time.
• Discussion : After presenting the results, interpret what has happened in the experiment. Base the discussion only on the data collected and as objective an interpretation as possible. Hypothesizing is possible here.
• Limitations : Discuss factors that affect the results. Here, you can speculate how much generalization, or more likely, transferability, is possible based on results. This section is important for quasi-experimentation, since a quasi-experiment cannot control all of the variables that might affect the outcome of a study. You would discuss what variables you could not control.
• Conclusion : Synthesize all of the above sections.
• References : Document works cited in the correct format for the field.

Experimental and Quasi-Experimental Research: Issues and Commentary

Several issues are addressed in this section, including the use of experimental and quasi-experimental research in educational settings, the relevance of the methods to English studies, and ethical concerns regarding the methods.

Using Experimental and Quasi-Experimental Research in Educational Settings

Charting causal relationships in human settings.

Any time a human population is involved, prediction of casual relationships becomes cloudy and, some say, impossible. Many reasons exist for this; for example,

• researchers in classrooms add a disturbing presence, causing students to act abnormally, consciously or unconsciously;
• subjects try to please the researcher, just because of an apparent interest in them (known as the Hawthorne Effect); or, perhaps
• the teacher as researcher is restricted by bias and time pressures.

But such confounding variables don't stop researchers from trying to identify causal relationships in education. Educators naturally experiment anyway, comparing groups, assessing the attributes of each, and making predictions based on an evaluation of alternatives. They look to research to support their intuitive practices, experimenting whenever they try to decide which instruction method will best encourage student improvement.

Combining Theory, Research, and Practice

The goal of educational research lies in combining theory, research, and practice. Educational researchers attempt to establish models of teaching practice, learning styles, curriculum development, and countless other educational issues. The aim is to "try to improve our understanding of education and to strive to find ways to have understanding contribute to the improvement of practice," one writer asserts (Floden 1996, p. 197).

In quasi-experimentation, researchers try to develop models by involving teachers as researchers, employing observational research techniques. Although results of this kind of research are context-dependent and difficult to generalize, they can act as a starting point for further study. The "educational researcher . . . provides guidelines and interpretive material intended to liberate the teacher's intelligence so that whatever artistry in teaching the teacher can achieve will be employed" (Eisner 1992, p. 8).

Bias and Rigor

Critics contend that the educational researcher is inherently biased, sample selection is arbitrary, and replication is impossible. The key to combating such criticism has to do with rigor. Rigor is established through close, proper attention to randomizing groups, time spent on a study, and questioning techniques. This allows more effective application of standards of quantitative research to qualitative research.

Often, teachers cannot wait to for piles of experimentation data to be analyzed before using the teaching methods (Lauer and Asher 1988). They ultimately must assess whether the results of a study in a distant classroom are applicable in their own classrooms. And they must continuously test the effectiveness of their methods by using experimental and qualitative research simultaneously. In addition to statistics (quantitative), researchers may perform case studies or observational research (qualitative) in conjunction with, or prior to, experimentation.

Relevance to English Studies

Situations in english studies that might encourage use of experimental methods.

Whenever a researcher would like to see if a causal relationship exists between groups, experimental and quasi-experimental research can be a viable research tool. Researchers in English Studies might use experimentation when they believe a relationship exists between two variables, and they want to show that these two variables have a significant correlation (or causal relationship).

A benefit of experimentation is the ability to control variables, such as the amount of treatment, when it is given, to whom and so forth. Controlling variables allows researchers to gain insight into the relationships they believe exist. For example, a researcher has an idea that writing under pseudonyms encourages student participation in newsgroups. Researchers can control which students write under pseudonyms and which do not, then measure the outcomes. Researchers can then analyze results and determine if this particular variable alone causes increased participation.

Transferability-Applying Results

Experimentation and quasi-experimentation allow for generating transferable results and accepting those results as being dependent upon experimental rigor. It is an effective alternative to generalizability, which is difficult to rely upon in educational research. English scholars, reading results of experiments with a critical eye, ultimately decide if results will be implemented and how. They may even extend that existing research by replicating experiments in the interest of generating new results and benefiting from multiple perspectives. These results will strengthen the study or discredit findings.

Concerns English Scholars Express about Experiments

Researchers should carefully consider if a particular method is feasible in humanities studies, and whether it will yield the desired information. Some researchers recommend addressing pertinent issues combining several research methods, such as survey, interview, ethnography, case study, content analysis, and experimentation (Lauer and Asher, 1988).

Advantages and Disadvantages of Experimental Research: Discussion

In educational research, experimentation is a way to gain insight into methods of instruction. Although teaching is context specific, results can provide a starting point for further study. Often, a teacher/researcher will have a "gut" feeling about an issue which can be explored through experimentation and looking at causal relationships. Through research intuition can shape practice .

A preconception exists that information obtained through scientific method is free of human inconsistencies. But, since scientific method is a matter of human construction, it is subject to human error . The researcher's personal bias may intrude upon the experiment , as well. For example, certain preconceptions may dictate the course of the research and affect the behavior of the subjects. The issue may be compounded when, although many researchers are aware of the affect that their personal bias exerts on their own research, they are pressured to produce research that is accepted in their field of study as "legitimate" experimental research.

The researcher does bring bias to experimentation, but bias does not limit an ability to be reflective . An ethical researcher thinks critically about results and reports those results after careful reflection. Concerns over bias can be leveled against any research method.

Often, the sample may not be representative of a population, because the researcher does not have an opportunity to ensure a representative sample. For example, subjects could be limited to one location, limited in number, studied under constrained conditions and for too short a time.

Despite such inconsistencies in educational research, the researcher has control over the variables , increasing the possibility of more precisely determining individual effects of each variable. Also, determining interaction between variables is more possible.

Even so, artificial results may result . It can be argued that variables are manipulated so the experiment measures what researchers want to examine; therefore, the results are merely contrived products and have no bearing in material reality. Artificial results are difficult to apply in practical situations, making generalizing from the results of a controlled study questionable. Experimental research essentially first decontextualizes a single question from a "real world" scenario, studies it under controlled conditions, and then tries to recontextualize the results back on the "real world" scenario. Results may be difficult to replicate .

Perhaps, groups in an experiment may not be comparable . Quasi-experimentation in educational research is widespread because not only are many researchers also teachers, but many subjects are also students. With the classroom as laboratory, it is difficult to implement randomizing or matching strategies. Often, students self-select into certain sections of a course on the basis of their own agendas and scheduling needs. Thus when, as often happens, one class is treated and the other used for a control, the groups may not actually be comparable. As one might imagine, people who register for a class which meets three times a week at eleven o'clock in the morning (young, no full-time job, night people) differ significantly from those who register for one on Monday evenings from seven to ten p.m. (older, full-time job, possibly more highly motivated). Each situation presents different variables and your group might be completely different from that in the study. Long-term studies are expensive and hard to reproduce. And although often the same hypotheses are tested by different researchers, various factors complicate attempts to compare or synthesize them. It is nearly impossible to be as rigorous as the natural sciences model dictates.

Even when randomization of students is possible, problems arise. First, depending on the class size and the number of classes, the sample may be too small for the extraneous variables to cancel out. Second, the study population is not strictly a sample, because the population of students registered for a given class at a particular university is obviously not representative of the population of all students at large. For example, students at a suburban private liberal-arts college are typically young, white, and upper-middle class. In contrast, students at an urban community college tend to be older, poorer, and members of a racial minority. The differences can be construed as confounding variables: the first group may have fewer demands on its time, have less self-discipline, and benefit from superior secondary education. The second may have more demands, including a job and/or children, have more self-discipline, but an inferior secondary education. Selecting a population of subjects which is representative of the average of all post-secondary students is also a flawed solution, because the outcome of a treatment involving this group is not necessarily transferable to either the students at a community college or the students at the private college, nor are they universally generalizable.

When a human population is involved, experimental research becomes concerned if behavior can be predicted or studied with validity. Human response can be difficult to measure . Human behavior is dependent on individual responses. Rationalizing behavior through experimentation does not account for the process of thought, making outcomes of that process fallible (Eisenberg, 1996).

Nevertheless, we perform experiments daily anyway . When we brush our teeth every morning, we are experimenting to see if this behavior will result in fewer cavities. We are relying on previous experimentation and we are transferring the experimentation to our daily lives.

Moreover, experimentation can be combined with other research methods to ensure rigor . Other qualitative methods such as case study, ethnography, observational research and interviews can function as preconditions for experimentation or conducted simultaneously to add validity to a study.

We have few alternatives to experimentation. Mere anecdotal research , for example is unscientific, unreplicatable, and easily manipulated. Should we rely on Ed walking into a faculty meeting and telling the story of Sally? Sally screamed, "I love writing!" ten times before she wrote her essay and produced a quality paper. Therefore, all the other faculty members should hear this anecdote and know that all other students should employ this similar technique.

On final disadvantage: frequently, political pressure drives experimentation and forces unreliable results. Specific funding and support may drive the outcomes of experimentation and cause the results to be skewed. The reader of these results may not be aware of these biases and should approach experimentation with a critical eye.

Advantages and Disadvantages of Experimental Research: Quick Reference List

Experimental and quasi-experimental research can be summarized in terms of their advantages and disadvantages. This section combines and elaborates upon many points mentioned previously in this guide.

Ethical Concerns

Experimental research may be manipulated on both ends of the spectrum: by researcher and by reader. Researchers who report on experimental research, faced with naive readers of experimental research, encounter ethical concerns. While they are creating an experiment, certain objectives and intended uses of the results might drive and skew it. Looking for specific results, they may ask questions and look at data that support only desired conclusions. Conflicting research findings are ignored as a result. Similarly, researchers, seeking support for a particular plan, look only at findings which support that goal, dismissing conflicting research.

Editors and journals do not publish only trouble-free material. As readers of experiments members of the press might report selected and isolated parts of a study to the public, essentially transferring that data to the general population which may not have been intended by the researcher. Take, for example, oat bran. A few years ago, the press reported how oat bran reduces high blood pressure by reducing cholesterol. But that bit of information was taken out of context. The actual study found that when people ate more oat bran, they reduced their intake of saturated fats high in cholesterol. People started eating oat bran muffins by the ton, assuming a causal relationship when in actuality a number of confounding variables might influence the causal link.

Ultimately, ethical use and reportage of experimentation should be addressed by researchers, reporters and readers alike.

Reporters of experimental research often seek to recognize their audience's level of knowledge and try not to mislead readers. And readers must rely on the author's skill and integrity to point out errors and limitations. The relationship between researcher and reader may not sound like a problem, but after spending months or years on a project to produce no significant results, it may be tempting to manipulate the data to show significant results in order to jockey for grants and tenure.

Meanwhile, the reader may uncritically accept results that receive validity by being published in a journal. However, research that lacks credibility often is not published; consequentially, researchers who fail to publish run the risk of being denied grants, promotions, jobs, and tenure. While few researchers are anything but earnest in their attempts to conduct well-designed experiments and present the results in good faith, rhetorical considerations often dictate a certain minimization of methodological flaws.

Concerns arise if researchers do not report all, or otherwise alter, results. This phenomenon is counterbalanced, however, in that professionals are also rewarded for publishing critiques of others' work. Because the author of an experimental study is in essence making an argument for the existence of a causal relationship, he or she must be concerned not only with its integrity, but also with its presentation. Achieving persuasiveness in any kind of writing involves several elements: choosing a topic of interest, providing convincing evidence for one's argument, using tone and voice to project credibility, and organizing the material in a way that meets expectations for a logical sequence. Of course, what is regarded as pertinent, accepted as evidence, required for credibility, and understood as logical varies according to context. If the experimental researcher hopes to make an impact on the community of professionals in their field, she must attend to the standards and orthodoxy's of that audience.

Related Links

Contrasts: Traditional and computer-supported writing classrooms. This Web presents a discussion of the Transitions Study, a year-long exploration of teachers and students in computer-supported and traditional writing classrooms. Includes description of study, rationale for conducting the study, results and implications of the study.

http://kairos.technorhetoric.net/2.2/features/reflections/page1.htm

Annotated Bibliography

A cozy world of trivial pursuits? (1996, June 28) The Times Educational Supplement . 4174, pp. 14-15.

A critique discounting the current methods Great Britain employs to fund and disseminate educational research. The belief is that research is performed for fellow researchers not the teaching public and implications for day to day practice are never addressed.

Anderson, J. A. (1979, Nov. 10-13). Research as argument: the experimental form. Paper presented at the annual meeting of the Speech Communication Association, San Antonio, TX.

In this paper, the scientist who uses the experimental form does so in order to explain that which is verified through prediction.

Anderson, Linda M. (1979). Classroom-based experimental studies of teaching effectiveness in elementary schools . (Technical Report UTR&D-R- 4102). Austin: Research and Development Center for Teacher Education, University of Texas.

Three recent large-scale experimental studies have built on a database established through several correlational studies of teaching effectiveness in elementary school.

Asher, J. W. (1976). Educational research and evaluation methods . Boston: Little, Brown.

Abstract unavailable by press time.

Babbie, Earl R. (1979). The Practice of Social Research . Belmont, CA: Wadsworth.

A textbook containing discussions of several research methodologies used in social science research.

Bangert-Drowns, R.L. (1993). The word processor as instructional tool: a meta-analysis of word processing in writing instruction. Review of Educational Research, 63 (1), 69-93.

Beach, R. (1993). The effects of between-draft teacher evaluation versus student self-evaluation on high school students' revising of rough drafts. Research in the Teaching of English, 13 , 111-119.

The question of whether teacher evaluation or guided self-evaluation of rough drafts results in increased revision was addressed in Beach's study. Differences in the effects of teacher evaluations, guided self-evaluation (using prepared guidelines,) and no evaluation of rough drafts were examined. The final drafts of students (10th, 11th, and 12th graders) were compared with their rough drafts and rated by judges according to degree of change.

Beishuizen, J. & Moonen, J. (1992). Research in technology enriched schools: a case for cooperation between teachers and researchers . (ERIC Technical Report ED351006).

This paper describes the research strategies employed in the Dutch Technology Enriched Schools project to encourage extensive and intensive use of computers in a small number of secondary schools, and to study the effects of computer use on the classroom, the curriculum, and school administration and management.

Borg, W. P. (1989). Educational Research: an Introduction . (5th ed.). New York: Longman.

An overview of educational research methodology, including literature review and discussion of approaches to research, experimental design, statistical analysis, ethics, and rhetorical presentation of research findings.

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research . Boston: Houghton Mifflin.

A classic overview of research designs.

Campbell, D.T. (1988). Methodology and epistemology for social science: selected papers . ed. E. S. Overman. Chicago: University of Chicago Press.

This is an overview of Campbell's 40-year career and his work. It covers in seven parts measurement, experimental design, applied social experimentation, interpretive social science, epistemology and sociology of science. Includes an extensive bibliography.

Caporaso, J. A., & Roos, Jr., L. L. (Eds.). Quasi-experimental approaches: Testing theory and evaluating policy. Evanston, WA: Northwestern University Press.

A collection of articles concerned with explicating the underlying assumptions of quasi-experimentation and relating these to true experimentation. With an emphasis on design. Includes a glossary of terms.

Collier, R. Writing and the word processor: How wary of the gift-giver should we be? Unpublished manuscript.

Unpublished typescript. Charts the developments to date in computers and composition and speculates about the future within the framework of Willie Sypher's model of the evolution of creative discovery.

Cook, T.D. & Campbell, D.T. (1979). Quasi-experimentation: design and analysis issues for field settings . Boston: Houghton Mifflin Co.

The authors write that this book "presents some quasi-experimental designs and design features that can be used in many social research settings. The designs serve to probe causal hypotheses about a wide variety of substantive issues in both basic and applied research."

Cutler, A. (1970). An experimental method for semantic field study. Linguistic Communication, 2 , N. pag.

This paper emphasizes the need for empirical research and objective discovery procedures in semantics, and illustrates a method by which these goals may be obtained.

Daniels, L. B. (1996, Summer). Eisenberg's Heisenberg: The indeterminancies of rationality. Curriculum Inquiry, 26 , 181-92.

Places Eisenberg's theories in relation to the death of foundationalism by showing that he distorts rational studies into a form of relativism. He looks at Eisenberg's ideas on indeterminacy, methods and evidence, what he is against and what we should think of what he says.

Danziger, K. (1990). Constructing the subject: Historical origins of psychological research. Cambridge: Cambridge University Press.

Danzinger stresses the importance of being aware of the framework in which research operates and of the essentially social nature of scientific activity.

Diener, E., et al. (1972, December). Leakage of experimental information to potential future subjects by debriefed subjects. Journal of Experimental Research in Personality , 264-67.

Research regarding research: an investigation of the effects on the outcome of an experiment in which information about the experiment had been leaked to subjects. The study concludes that such leakage is not a significant problem.

Dudley-Marling, C., & Rhodes, L. K. (1989). Reflecting on a close encounter with experimental research. Canadian Journal of English Language Arts. 12 , 24-28.

Researchers, Dudley-Marling and Rhodes, address some problems they met in their experimental approach to a study of reading comprehension. This article discusses the limitations of experimental research, and presents an alternative to experimental or quantitative research.

Edgington, E. S. (1985). Random assignment and experimental research. Educational Administration Quarterly, 21 , N. pag.

Edgington explores ways on which random assignment can be a part of field studies. The author discusses both non-experimental and experimental research and the need for using random assignment.

Eisenberg, J. (1996, Summer). Response to critiques by R. Floden, J. Zeuli, and L. Daniels. Curriculum Inquiry, 26 , 199-201.

A response to critiques of his argument that rational educational research methods are at best suspect and at worst futile. He believes indeterminacy controls this method and worries that chaotic research is failing students.

Eisner, E. (1992, July). Are all causal claims positivistic? A reply to Francis Schrag. Educational Researcher, 21 (5), 8-9.

Eisner responds to Schrag who claimed that critics like Eisner cannot escape a positivistic paradigm whatever attempts they make to do so. Eisner argues that Schrag essentially misses the point for trying to argue for the paradigm solely on the basis of cause and effect without including the rest of positivistic philosophy. This weakens his argument against multiple modal methods, which Eisner argues provides opportunities to apply the appropriate research design where it is most applicable.

Floden, R.E. (1996, Summer). Educational research: limited, but worthwhile and maybe a bargain. (response to J.A. Eisenberg). Curriculum Inquiry, 26 , 193-7.

Responds to John Eisenberg critique of educational research by asserting the connection between improvement of practice and research results. He places high value of teacher discrepancy and knowledge that research informs practice.

Fortune, J. C., & Hutson, B. A. (1994, March/April). Selecting models for measuring change when true experimental conditions do not exist. Journal of Educational Research, 197-206.

This article reviews methods for minimizing the effects of nonideal experimental conditions by optimally organizing models for the measurement of change.

Fox, R. F. (1980). Treatment of writing apprehension and tts effects on composition. Research in the Teaching of English, 14 , 39-49.

The main purpose of Fox's study was to investigate the effects of two methods of teaching writing on writing apprehension among entry level composition students, A conventional teaching procedure was used with a control group, while a workshop method was employed with the treatment group.

Gadamer, H-G. (1976). Philosophical hermeneutics . (D. E. Linge, Trans.). Berkeley, CA: University of California Press.

A collection of essays with the common themes of the mediation of experience through language, the impossibility of objectivity, and the importance of context in interpretation.

Gaise, S. J. (1981). Experimental vs. non-experimental research on classroom second language learning. Bilingual Education Paper Series, 5 , N. pag.

Aims on classroom-centered research on second language learning and teaching are considered and contrasted with the experimental approach.

Giordano, G. (1983). Commentary: Is experimental research snowing us? Journal of Reading, 27 , 5-7.

Do educational research findings actually benefit teachers and students? Giordano states his opinion that research may be helpful to teaching, but is not essential and often is unnecessary.

Goldenson, D. R. (1978, March). An alternative view about the role of the secondary school in political socialization: A field-experimental study of theory and research in social education. Theory and Research in Social Education , 44-72.

This study concludes that when political discussion among experimental groups of secondary school students is led by a teacher, the degree to which the students' views were impacted is proportional to the credibility of the teacher.

Grossman, J., and J. P. Tierney. (1993, October). The fallibility of comparison groups. Evaluation Review , 556-71.

Grossman and Tierney present evidence to suggest that comparison groups are not the same as nontreatment groups.

Harnisch, D. L. (1992). Human judgment and the logic of evidence: A critical examination of research methods in special education transition literature. In D. L. Harnisch et al. (Eds.), Selected readings in transition.

This chapter describes several common types of research studies in special education transition literature and the threats to their validity.

Hawisher, G. E. (1989). Research and recommendations for computers and composition. In G. Hawisher and C. Selfe. (Eds.), Critical Perspectives on Computers and Composition Instruction . (pp. 44-69). New York: Teacher's College Press.

An overview of research in computers and composition to date. Includes a synthesis grid of experimental research.

Hillocks, G. Jr. (1982). The interaction of instruction, teacher comment, and revision in teaching the composing process. Research in the Teaching of English, 16 , 261-278.

Hillock conducted a study using three treatments: observational or data collecting activities prior to writing, use of revisions or absence of same, and either brief or lengthy teacher comments to identify effective methods of teaching composition to seventh and eighth graders.

Jenkinson, J. C. (1989). Research design in the experimental study of intellectual disability. International Journal of Disability, Development, and Education, 69-84.

This article catalogues the difficulties of conducting experimental research where the subjects are intellectually disables and suggests alternative research strategies.

Jones, R. A. (1985). Research Methods in the Social and Behavioral Sciences. Sunderland, MA: Sinauer Associates, Inc..

A textbook designed to provide an overview of research strategies in the social sciences, including survey, content analysis, ethnographic approaches, and experimentation. The author emphasizes the importance of applying strategies appropriately and in variety.

Kamil, M. L., Langer, J. A., & Shanahan, T. (1985). Understanding research in reading and writing . Newton, Massachusetts: Allyn and Bacon.

Examines a wide variety of problems in reading and writing, with a broad range of techniques, from different perspectives.

Kennedy, J. L. (1985). An Introduction to the Design and Analysis of Experiments in Behavioral Research . Lanham, MD: University Press of America.

An introductory textbook of psychological and educational research.

Keppel, G. (1991). Design and analysis: a researcher's handbook . Englewood Cliffs, NJ: Prentice Hall.

This updates Keppel's earlier book subtitled "a student's handbook." Focuses on extensive information about analytical research and gives a basic picture of research in psychology. Covers a range of statistical topics. Includes a subject and name index, as well as a glossary.

Knowles, G., Elija, R., & Broadwater, K. (1996, Spring/Summer). Teacher research: enhancing the preparation of teachers? Teaching Education, 8 , 123-31.

Researchers looked at one teacher candidate who participated in a class which designed their own research project correlating to a question they would like answered in the teaching world. The goal of the study was to see if preservice teachers developed reflective practice by researching appropriate classroom contexts.

Lace, J., & De Corte, E. (1986, April 16-20). Research on media in western Europe: A myth of sisyphus? Paper presented at the annual meeting of the American Educational Research Association. San Francisco.

Identifies main trends in media research in western Europe, with emphasis on three successive stages since 1960: tools technology, systems technology, and reflective technology.

Latta, A. (1996, Spring/Summer). Teacher as researcher: selected resources. Teaching Education, 8 , 155-60.

An annotated bibliography on educational research including milestones of thought, practical applications, successful outcomes, seminal works, and immediate practical applications.

Lauer. J.M. & Asher, J. W. (1988). Composition research: Empirical designs . New York: Oxford University Press.

Approaching experimentation from a humanist's perspective to it, authors focus on eight major research designs: Case studies, ethnographies, sampling and surveys, quantitative descriptive studies, measurement, true experiments, quasi-experiments, meta-analyses, and program evaluations. It takes on the challenge of bridging language of social science with that of the humanist. Includes name and subject indexes, as well as a glossary and a glossary of symbols.

Mishler, E. G. (1979). Meaning in context: Is there any other kind? Harvard Educational Review, 49 , 1-19.

Contextual importance has been largely ignored by traditional research approaches in social/behavioral sciences and in their application to the education field. Developmental and social psychologists have increasingly noted the inadequacies of this approach. Drawing examples for phenomenology, sociolinguistics, and ethnomethodology, the author proposes alternative approaches for studying meaning in context.

Mitroff, I., & Bonoma, T. V. (1978, May). Psychological assumptions, experimentations, and real world problems: A critique and an alternate approach to evaluation. Evaluation Quarterly , 235-60.

The authors advance the notion of dialectic as a means to clarify and examine the underlying assumptions of experimental research methodology, both in highly controlled situations and in social evaluation.

Muller, E. W. (1985). Application of experimental and quasi-experimental research designs to educational software evaluation. Educational Technology, 25 , 27-31.

Muller proposes a set of guidelines for the use of experimental and quasi-experimental methods of research in evaluating educational software. By obtaining empirical evidence of student performance, it is possible to evaluate if programs are making the desired learning effect.

Murray, S., et al. (1979, April 8-12). Technical issues as threats to internal validity of experimental and quasi-experimental designs . San Francisco: University of California.

The article reviews three evaluation models and analyzes the flaws common to them. Remedies are suggested.

Muter, P., & Maurutto, P. (1991). Reading and skimming from computer screens and books: The paperless office revisited? Behavior and Information Technology, 10 (4), 257-66.

The researchers test for reading and skimming effectiveness, defined as accuracy combined with speed, for written text compared to text on a computer monitor. They conclude that, given optimal on-line conditions, both are equally effective.

O'Donnell, A., Et al. (1992). The impact of cooperative writing. In J. R. Hayes, et al. (Eds.). Reading empirical research studies: The rhetoric of research . (pp. 371-84). Hillsdale, NJ: Lawrence Erlbaum Associates.

A model of experimental design. The authors investigate the efficacy of cooperative writing strategies, as well as the transferability of skills learned to other, individual writing situations.

Palmer, D. (1988). Looking at philosophy . Mountain View, CA: Mayfield Publishing.

An introductory text with incisive but understandable discussions of the major movements and thinkers in philosophy from the Pre-Socratics through Sartre. With illustrations by the author. Includes a glossary.

Phelps-Gunn, T., & Phelps-Terasaki, D. (1982). Written language instruction: Theory and remediation . London: Aspen Systems Corporation.

The lack of research in written expression is addressed and an application on the Total Writing Process Model is presented.

Poetter, T. (1996, Spring/Summer). From resistance to excitement: becoming qualitative researchers and reflective practitioners. Teaching Education , 8109-19.

An education professor reveals his own problematic research when he attempted to institute a educational research component to a teacher preparation program. He encountered dissent from students and cooperating professionals and ultimately was rewarded with excitement towards research and a recognized correlation to practice.

Purves, A. C. (1992). Reflections on research and assessment in written composition. Research in the Teaching of English, 26 .

Three issues concerning research and assessment is writing are discussed: 1) School writing is a matter of products not process, 2) school writing is an ill-defined domain, 3) the quality of school writing is what observers report they see. Purves discusses these issues while looking at data collected in a ten-year study of achievement in written composition in fourteen countries.

Rathus, S. A. (1987). Psychology . (3rd ed.). Poughkeepsie, NY: Holt, Rinehart, and Winston.

An introductory psychology textbook. Includes overviews of the major movements in psychology, discussions of prominent examples of experimental research, and a basic explanation of relevant physiological factors. With chapter summaries.

Reiser, R. A. (1982). Improving the research skills of instructional designers. Educational Technology, 22 , 19-21.

In his paper, Reiser starts by stating the importance of research in advancing the field of education, and points out that graduate students in instructional design lack the proper skills to conduct research. The paper then goes on to outline the practicum in the Instructional Systems Program at Florida State University which includes: 1) Planning and conducting an experimental research study; 2) writing the manuscript describing the study; 3) giving an oral presentation in which they describe their research findings.

Report on education research . (Journal). Washington, DC: Capitol Publication, Education News Services Division.

This is an independent bi-weekly newsletter on research in education and learning. It has been publishing since Sept. 1969.

Rossell, C. H. (1986). Why is bilingual education research so bad?: Critique of the Walsh and Carballo study of Massachusetts bilingual education programs . Boston: Center for Applied Social Science, Boston University. (ERIC Working Paper 86-5).

The Walsh and Carballo evaluation of the effectiveness of transitional bilingual education programs in five Massachusetts communities has five flaws and the five flaws are discussed in detail.

Rubin, D. L., & Greene, K. (1992). Gender-typical style in written language. Research in the Teaching of English, 26.

This study was designed to find out whether the writing styles of men and women differ. Rubin and Green discuss the pre-suppositions that women are better writers than men.

Sawin, E. (1992). Reaction: Experimental research in the context of other methods. School of Education Review, 4 , 18-21.

Sawin responds to Gage's article on methodologies and issues in educational research. He agrees with most of the article but suggests the concept of scientific should not be regarded in absolute terms and recommends more emphasis on scientific method. He also questions the value of experiments over other types of research.

Schoonmaker, W. E. (1984). Improving classroom instruction: A model for experimental research. The Technology Teacher, 44, 24-25.

The model outlined in this article tries to bridge the gap between classroom practice and laboratory research, using what Schoonmaker calls active research. Research is conducted in the classroom with the students and is used to determine which two methods of classroom instruction chosen by the teacher is more effective.

Schrag, F. (1992). In defense of positivist research paradigms. Educational Researcher, 21, (5), 5-8.

The controversial defense of the use of positivistic research methods to evaluate educational strategies; the author takes on Eisner, Erickson, and Popkewitz.

Smith, J. (1997). The stories educational researchers tell about themselves. Educational Researcher, 33 (3), 4-11.

Recapitulates main features of an on-going debate between advocates for using vocabularies of traditional language arts and whole language in educational research. An "impasse" exists were advocates "do not share a theoretical disposition concerning both language instruction and the nature of research," Smith writes (p. 6). He includes a very comprehensive history of the debate of traditional research methodology and qualitative methods and vocabularies. Definitely worth a read by graduates.

Smith, N. L. (1980). The feasibility and desirability of experimental methods in evaluation. Evaluation and Program Planning: An International Journal , 251-55.

Smith identifies the conditions under which experimental research is most desirable. Includes a review of current thinking and controversies.

Stewart, N. R., & Johnson, R. G. (1986, March 16-20). An evaluation of experimental methodology in counseling and counselor education research. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.

The purpose of this study was to evaluate the quality of experimental research in counseling and counselor education published from 1976 through 1984.

Spector, P. E. (1990). Research Designs. Newbury Park, California: Sage Publications.

In this book, Spector introduces the basic principles of experimental and nonexperimental design in the social sciences.

Tait, P. E. (1984). Do-it-yourself evaluation of experimental research. Journal of Visual Impairment and Blindness, 78 , 356-363 .

Tait's goal is to provide the reader who is unfamiliar with experimental research or statistics with the basic skills necessary for the evaluation of research studies.

Walsh, S. M. (1990). The current conflict between case study and experimental research: A breakthrough study derives benefits from both . (ERIC Document Number ED339721).

This paper describes a study that was not experimentally designed, but its major findings were generalizable to the overall population of writers in college freshman composition classes. The study was not a case study, but it provided insights into the attitudes and feelings of small clusters of student writers.

Waters, G. R. (1976). Experimental designs in communication research. Journal of Business Communication, 14 .

The paper presents a series of discussions on the general elements of experimental design and the scientific process and relates these elements to the field of communication.

Welch, W. W. (March 1969). The selection of a national random sample of teachers for experimental curriculum evaluation. Scholastic Science and Math , 210-216.

Members of the evaluation section of Harvard project physics describe what is said to be the first attempt to select a national random sample of teachers, and list 6 steps to do so. Cost and comparison with a volunteer group are also discussed.

Winer, B.J. (1971). Statistical principles in experimental design , (2nd ed.). New York: McGraw-Hill.

Combines theory and application discussions to give readers a better understanding of the logic behind statistical aspects of experimental design. Introduces the broad topic of design, then goes into considerable detail. Not for light reading. Bring your aspirin if you like statistics. Bring morphine is you're a humanist.

Winn, B. (1986, January 16-21). Emerging trends in educational technology research. Paper presented at the Annual Convention of the Association for Educational Communication Technology.

This examination of the topic of research in educational technology addresses four major areas: (1) why research is conducted in this area and the characteristics of that research; (2) the types of research questions that should or should not be addressed; (3) the most appropriate methodologies for finding answers to research questions; and (4) the characteristics of a research report that make it good and ultimately suitable for publication.

Citation Information

Luann Barnes, Jennifer Hauser, Luana Heikes, Anthony J. Hernandez, Paul Tim Richard, Katherine Ross, Guo Hua Yang, and Mike Palmquist. (1994-2024). Experimental and Quasi-Experimental Research. The WAC Clearinghouse. Colorado State University. Available at https://wac.colostate.edu/repository/writing/guides/.

Copyright Information

Copyright © 1994-2024 Colorado State University and/or this site's authors, developers, and contributors . Some material displayed on this site is used with permission.

• Experimental Vs Non-Experimental Research: 15 Key Differences

There is a general misconception around research that once the research is non-experimental, then it is non-scientific, making it more important to understand what experimental and experimental research entails. Experimental research is the most common type of research, which a lot of people refer to as scientific research. 

Non experimental research, on the other hand, is easily used to classify research that is not experimental. It clearly differs from experimental research, and as such has different use cases. 

In this article, we will be explaining these differences in detail so as to ensure proper identification during the research process.

What is Experimental Research?  

Experimental research is the type of research that uses a scientific approach towards manipulating one or more control variables of the research subject(s) and measuring the effect of this manipulation on the subject. It is known for the fact that it allows the manipulation of control variables. 

This research method is widely used in various physical and social science fields, even though it may be quite difficult to execute. Within the information field, they are much more common in information systems research than in library and information management research.

Experimental research is usually undertaken when the goal of the research is to trace cause-and-effect relationships between defined variables. However, the type of experimental research chosen has a significant influence on the results of the experiment.

Therefore bringing us to the different types of experimental research. There are 3 main types of experimental research, namely; pre-experimental, quasi-experimental, and true experimental research.

Pre-experimental Research

Pre-experimental research is the simplest form of research, and is carried out by observing a group or groups of dependent variables after the treatment of an independent variable which is presumed to cause change on the group(s). It is further divided into three types.

• One-shot case study research 
• One-group pretest-posttest research 
• Static-group comparison

Quasi-experimental Research

The Quasi type of experimental research is similar to true experimental research, but uses carefully selected rather than randomized subjects. The following are examples of quasi-experimental research:

• Time series 
• No equivalent control group design
• Counterbalanced design.

True Experimental Research

True experimental research is the most accurate type,  and may simply be called experimental research. It manipulates a control group towards a group of randomly selected subjects and records the effect of this manipulation.

True experimental research can be further classified into the following groups:

• The posttest-only control group 
• The pretest-posttest control group 
• Solomon four-group 

Pros of True Experimental Research

• Researchers can have control over variables.
• It can be combined with other research methods.
• The research process is usually well structured.
• It provides specific conclusions.
• The results of experimental research can be easily duplicated.

Cons of True Experimental Research

• It is highly prone to human error.
• Exerting control over extraneous variables may lead to the personal bias of the researcher.
• It is time-consuming.
• It is expensive. 
• Manipulating control variables may have ethical implications.
• It produces artificial results.

What is Non-Experimental Research?  

Non-experimental research is the type of research that does not involve the manipulation of control or independent variable. In non-experimental research, researchers measure variables as they naturally occur without any further manipulation.

This type of research is used when the researcher has no specific research question about a causal relationship between 2 different variables, and manipulation of the independent variable is impossible. They are also used when:

• subjects cannot be randomly assigned to conditions.
• the research subject is about a causal relationship but the independent variable cannot be manipulated.
• the research is broad and exploratory
• the research pertains to a non-causal relationship between variables.
• limited information can be accessed about the research subject.

There are 3 main types of non-experimental research , namely; cross-sectional research, correlation research, and observational research.

Cross-sectional Research

Cross-sectional research involves the comparison of two or more pre-existing groups of people under the same criteria. This approach is classified as non-experimental because the groups are not randomly selected and the independent variable is not manipulated.

For example, an academic institution may want to reward its first-class students with a scholarship for their academic excellence. Therefore, each faculty places students in the eligible and ineligible group according to their class of degree.

In this case, the student’s class of degree cannot be manipulated to qualify him or her for a scholarship because it is an unethical thing to do. Therefore, the placement is cross-sectional.

Correlational Research

Correlational type of research compares the statistical relationship between two variables .Correlational research is classified as non-experimental because it does not manipulate the independent variables.

For example, a researcher may wish to investigate the relationship between the class of family students come from and their grades in school. A questionnaire may be given to students to know the average income of their family, then compare it with CGPAs. 

The researcher will discover whether these two factors are positively correlated, negatively corrected, or have zero correlation at the end of the research.

Observational Research

Observational research focuses on observing the behavior of a research subject in a natural or laboratory setting. It is classified as non-experimental because it does not involve the manipulation of independent variables.

A good example of observational research is an investigation of the crowd effect or psychology in a particular group of people. Imagine a situation where there are 2 ATMs at a place, and only one of the ATMs is filled with a queue, while the other is abandoned.

The crowd effect infers that the majority of newcomers will also abandon the other ATM.

You will notice that each of these non-experimental research is descriptive in nature. It then suffices to say that descriptive research is an example of non-experimental research.

Pros of Observational Research

• The research process is very close to a real-life situation.
• It does not allow for the manipulation of variables due to ethical reasons.
• Human characteristics are not subject to experimental manipulation.

Cons of Observational Research

• The groups may be dissimilar and nonhomogeneous because they are not randomly selected, affecting the authenticity and generalizability of the study results.
• The results obtained cannot be absolutely clear and error-free.

What Are The Differences Between Experimental and Non-Experimental Research?    

• Definitions

Experimental research is the type of research that uses a scientific approach towards manipulating one or more control variables and measuring their defect on the dependent variables, while non-experimental research is the type of research that does not involve the manipulation of control variables.

The main distinction in these 2 types of research is their attitude towards the manipulation of control variables. Experimental allows for the manipulation of control variables while non-experimental research doesn’t.

 Examples of experimental research are laboratory experiments that involve mixing different chemical elements together to see the effect of one element on the other while non-experimental research examples are investigations into the characteristics of different chemical elements.

Consider a researcher carrying out a laboratory test to determine the effect of adding Nitrogen gas to Hydrogen gas. It may be discovered that using the Haber process, one can create Nitrogen gas.

Non-experimental research may further be carried out on Ammonia, to determine its characteristics, behaviour, and nature.

There are 3 types of experimental research, namely; experimental research, quasi-experimental research, and true experimental research. Although also 3 in number, non-experimental research can be classified into cross-sectional research, correlational research, and observational research.

The different types of experimental research are further divided into different parts, while non-experimental research types are not further divided. Clearly, these divisions are not the same in experimental and non-experimental research.

• Characteristics

Experimental research is usually quantitative, controlled, and multivariable. Non-experimental research can be both quantitative and qualitative , has an uncontrolled variable, and also a cross-sectional research problem.

The characteristics of experimental research are the direct opposite of that of non-experimental research. The most distinct characteristic element is the ability to control or manipulate independent variables in experimental research and not in non-experimental research. 

In experimental research, a level of control is usually exerted on extraneous variables, therefore tampering with the natural research setting. Experimental research settings are usually more natural with no tampering with the extraneous variables.

• Data Collection/Tools

  The data used during experimental research is collected through observational study, simulations, and surveys while non-experimental data is collected through observations, surveys, and case studies. The main distinction between these data collection tools is case studies and simulations.

Even at that, similar tools are used differently. For example, an observational study may be used during a laboratory experiment that tests how the effect of a control variable manifests over a period of time in experimental research. 

However, when used in non-experimental research, data is collected based on the researcher’s discretion and not through a clear scientific reaction. In this case, we see a difference in the level of objectivity. 

The goal of experimental research is to measure the causes and effects of variables present in research, while non-experimental research provides very little to no information about causal agents.

Experimental research answers the question of why something is happening. This is quite different in non-experimental research, as they are more descriptive in nature with the end goal being to describe what .

 Experimental research is mostly used to make scientific innovations and find major solutions to problems while non-experimental research is used to define subject characteristics, measure data trends, compare situations and validate existing conditions.

For example, if experimental research results in an innovative discovery or solution, non-experimental research will be conducted to validate this discovery. This research is done for a period of time in order to properly study the subject of research.

Experimental research process is usually well structured and as such produces results with very little to no errors, while non-experimental research helps to create real-life related experiments. There are a lot more advantages of experimental and non-experimental research , with the absence of each of these advantages in the other leaving it at a disadvantage.

For example, the lack of a random selection process in non-experimental research leads to the inability to arrive at a generalizable result. Similarly, the ability to manipulate control variables in experimental research may lead to the personal bias of the researcher.

• Disadvantage

 Experimental research is highly prone to human error while the major disadvantage of non-experimental research is that the results obtained cannot be absolutely clear and error-free. In the long run, the error obtained due to human error may affect the results of the experimental research.

Some other disadvantages of experimental research include the following; extraneous variables cannot always be controlled, human responses can be difficult to measure, and participants may also cause bias.

  In experimental research, researchers can control and manipulate control variables, while in non-experimental research, researchers cannot manipulate these variables. This cannot be done due to ethical reasons. 

For example, when promoting employees due to how well they did in their annual performance review, it will be unethical to manipulate the results of the performance review (independent variable). That way, we can get impartial results of those who deserve a promotion and those who don’t.

Experimental researchers may also decide to eliminate extraneous variables so as to have enough control over the research process. Once again, this is something that cannot be done in non-experimental research because it relates more to real-life situations.

Experimental research is carried out in an unnatural setting because most of the factors that influence the setting are controlled while the non-experimental research setting remains natural and uncontrolled. One of the things usually tampered with during research is extraneous variables.

In a bid to get a perfect and well-structured research process and results, researchers sometimes eliminate extraneous variables. Although sometimes seen as insignificant, the elimination of these variables may affect the research results.

Consider the optimization problem whose aim is to minimize the cost of production of a car, with the constraints being the number of workers and the number of hours they spend working per day. 

In this problem, extraneous variables like machine failure rates or accidents are eliminated. In the long run, these things may occur and may invalidate the result.

• Cause-Effect Relationship

The relationship between cause and effect is established in experimental research while it cannot be established in non-experimental research. Rather than establish a cause-effect relationship, non-experimental research focuses on providing descriptive results.

Although it acknowledges the causal variable and its effect on the dependent variables, it does not measure how or the extent to which these dependent variables change. It, however, observes these changes, compares the changes in 2 variables, and describes them.

Experimental research does not compare variables while non-experimental research does. It compares 2 variables and describes the relationship between them.

The relationship between these variables can be positively correlated, negatively correlated or not correlated at all. For example, consider a case whereby the subject of research is a drum, and the control or independent variable is the drumstick.

Experimental research will measure the effect of hitting the drumstick on the drum, where the result of this research will be sound. That is, when you hit a drumstick on a drum, it makes a sound.

Non-experimental research, on the other hand, will investigate the correlation between how hard the drum is hit and the loudness of the sound that comes out. That is, if the sound will be higher with a harder bang, lower with a harder bang, or will remain the same no matter how hard we hit the drum.

• Quantitativeness

Experimental research is a quantitative research method while non-experimental research can be both quantitative and qualitative depending on the time and the situation where it is been used. An example of a non-experimental quantitative research method is correlational research .

Researchers use it to correlate two or more variables using mathematical analysis methods. The original patterns, relationships, and trends between variables are observed, then the impact of one of these variables on the other is recorded along with how it changes the relationship between the two variables.

Observational research is an example of non-experimental research, which is classified as a qualitative research method.

• Cross-section

Experimental research is usually single-sectional while non-experimental research is cross-sectional. That is, when evaluating the research subjects in experimental research, each group is evaluated as an entity.

For example, let us consider a medical research process investigating the prevalence of breast cancer in a certain community. In this community, we will find people of different ages, ethnicities, and social backgrounds. 

If a significant amount of women from a particular age are found to be more prone to have the disease, the researcher can conduct further studies to understand the reason behind it. A further study into this will be experimental and the subject won’t be a cross-sectional group. 

A lot of researchers consider the distinction between experimental and non-experimental research to be an extremely important one. This is partly due to the fact that experimental research can accommodate the manipulation of independent variables, which is something non-experimental research can not.

Therefore, as a researcher who is interested in using any one of experimental and non-experimental research, it is important to understand the distinction between these two. This helps in deciding which method is better for carrying out particular research. 

Connect to Formplus, Get Started Now - It's Free!

• examples of experimental research
• non experimental research
• busayo.longe

You may also like:

Simpson’s Paradox & How to Avoid it in Experimental Research

In this article, we are going to look at Simpson’s Paradox from its historical point and later, we’ll consider its effect in...

Experimental Research Designs: Types, Examples & Methods

Ultimate guide to experimental research. It’s definition, types, characteristics, uses, examples and methodolgy

What is Experimenter Bias? Definition, Types & Mitigation

In this article, we will look into the concept of experimental bias and how it can be identified in your research

Response vs Explanatory Variables: Definition & Examples

In this article, we’ll be comparing the two types of variables, what they both mean and see some of their real-life applications in research

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case AskWhy Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research Research Tools and Apps

Quasi-experimental Research: What It Is, Types & Examples

Much like an actual experiment, quasi-experimental research tries to demonstrate a cause-and-effect link between a dependent and an independent variable. A quasi-experiment, on the other hand, does not depend on random assignment, unlike an actual experiment. The subjects are sorted into groups based on non-random variables.

What is Quasi-Experimental Research?

“Resemblance” is the definition of “quasi.” Individuals are not randomly allocated to conditions or orders of conditions, even though the regression analysis is changed. As a result, quasi-experimental research is research that appears to be experimental but is not.

The directionality problem is avoided in quasi-experimental research since the regression analysis is altered before the multiple regression is assessed. However, because individuals are not randomized at random, there are likely to be additional disparities across conditions in quasi-experimental research.

As a result, in terms of internal consistency, quasi-experiments fall somewhere between correlational research and actual experiments.

The key component of a true experiment is randomly allocated groups. This means that each person has an equivalent chance of being assigned to the experimental group or the control group, depending on whether they are manipulated or not.

Simply put, a quasi-experiment is not a real experiment. A quasi-experiment does not feature randomly allocated groups since the main component of a real experiment is randomly assigned groups. Why is it so crucial to have randomly allocated groups, given that they constitute the only distinction between quasi-experimental and actual  experimental research ?

Let’s use an example to illustrate our point. Let’s assume we want to discover how new psychological therapy affects depressed patients. In a genuine trial, you’d split half of the psych ward into treatment groups, With half getting the new psychotherapy therapy and the other half receiving standard  depression treatment .

And the physicians compare the outcomes of this treatment to the results of standard treatments to see if this treatment is more effective. Doctors, on the other hand, are unlikely to agree with this genuine experiment since they believe it is unethical to treat one group while leaving another untreated.

A quasi-experimental study will be useful in this case. Instead of allocating these patients at random, you uncover pre-existing psychotherapist groups in the hospitals. Clearly, there’ll be counselors who are eager to undertake these trials as well as others who prefer to stick to the old ways.

These pre-existing groups can be used to compare the symptom development of individuals who received the novel therapy with those who received the normal course of treatment, even though the groups weren’t chosen at random.

If any substantial variations between them can be well explained, you may be very assured that any differences are attributable to the treatment but not to other extraneous variables.

As we mentioned before, quasi-experimental research entails manipulating an independent variable by randomly assigning people to conditions or sequences of conditions. Non-equivalent group designs, pretest-posttest designs, and regression discontinuity designs are only a few of the essential types.

What are quasi-experimental research designs?

Quasi-experimental research designs are a type of research design that is similar to experimental designs but doesn’t give full control over the independent variable(s) like true experimental designs do.

In a quasi-experimental design, the researcher changes or watches an independent variable, but the participants are not put into groups at random. Instead, people are put into groups based on things they already have in common, like their age, gender, or how many times they have seen a certain stimulus.

Because the assignments are not random, it is harder to draw conclusions about cause and effect than in a real experiment. However, quasi-experimental designs are still useful when randomization is not possible or ethical.

The true experimental design may be impossible to accomplish or just too expensive, especially for researchers with few resources. Quasi-experimental designs enable you to investigate an issue by utilizing data that has already been paid for or gathered by others (often the government). 

Because they allow better control for confounding variables than other forms of studies, they have higher external validity than most genuine experiments and higher  internal validity  (less than true experiments) than other non-experimental research.

Is quasi-experimental research quantitative or qualitative?

Quasi-experimental research is a quantitative research method. It involves numerical data collection and statistical analysis. Quasi-experimental research compares groups with different circumstances or treatments to find cause-and-effect links. 

It draws statistical conclusions from quantitative data. Qualitative data can enhance quasi-experimental research by revealing participants’ experiences and opinions, but quantitative data is the method’s foundation.

Quasi-experimental research types

There are many different sorts of quasi-experimental designs. Three of the most popular varieties are described below: Design of non-equivalent groups, Discontinuity in regression, and Natural experiments.

Design of Non-equivalent Groups

Example: design of non-equivalent groups, discontinuity in regression, example: discontinuity in regression, natural experiments, example: natural experiments.

However, because they couldn’t afford to pay everyone who qualified for the program, they had to use a random lottery to distribute slots.

Experts were able to investigate the program’s impact by utilizing enrolled people as a treatment group and those who were qualified but did not play the jackpot as an experimental group.

How QuestionPro helps in quasi-experimental research?

QuestionPro can be a useful tool in quasi-experimental research because it includes features that can assist you in designing and analyzing your research study. Here are some ways in which QuestionPro can help in quasi-experimental research:

Design surveys

Randomize participants, collect data over time, analyze data, collaborate with your team.

With QuestionPro, you have access to the most mature market research platform and tool that helps you collect and analyze the insights that matter the most. By leveraging InsightsHub, the unified hub for data management, you can ​​leverage the consolidated platform to organize, explore, search, and discover your  research data  in one organized data repository . 

Optimize Your quasi-experimental research with QuestionPro. Get started now!

LEARN MORE         FREE TRIAL

MORE LIKE THIS

QuestionPro: Leading the Charge in Customer Journey Management and Voice of the Customer Platforms

Sep 17, 2024

What is Driver Analysis? Importance and Best Practices

Was The Experience Memorable? (Part II) — Tuesday CX Thoughts

Data Discovery: What it is, Importance, Process + Use Cases

Sep 16, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Tuesday CX Thoughts (TCXT)
• Uncategorized
• What’s Coming Up
• Workforce Intelligence

8.2 Non-Equivalent Groups Designs

Learning objectives.

• Describe the different types of nonequivalent groups quasi-experimental designs.
• Identify some of the threats to internal validity associated with each of these designs. 

Recall that when participants in a between-subjects experiment are randomly assigned to conditions, the resulting groups are likely to be quite similar. In fact, researchers consider them to be equivalent. When participants are not randomly assigned to conditions, however, the resulting groups are likely to be dissimilar in some ways. For this reason, researchers consider them to be nonequivalent. A  nonequivalent groups design , then, is a between-subjects design in which participants have not been randomly assigned to conditions. There are several types of nonequivalent groups designs we will consider.

Posttest Only Nonequivalent Groups Design

The first nonequivalent groups design we will consider is the posttest only nonequivalent groups design.  In this design, participants in one group are exposed to a treatment, a nonequivalent group is not exposed to the treatment, and then the two groups are compared. Imagine, for example, a researcher who wants to evaluate a new method of teaching fractions to third graders. One way would be to conduct a study with a treatment group consisting of one class of third-grade students and a control group consisting of another class of third-grade students. This design would be a nonequivalent groups design because the students are not randomly assigned to classes by the researcher, which means there could be important differences between them. For example, the parents of higher achieving or more motivated students might have been more likely to request that their children be assigned to Ms. Williams’s class. Or the principal might have assigned the “troublemakers” to Mr. Jones’s class because he is a stronger disciplinarian. Of course, the teachers’ styles, and even the classroom environments might be very different and might cause different levels of achievement or motivation among the students. If at the end of the study there was a difference in the two classes’ knowledge of fractions, it might have been caused by the difference between the teaching methods—but it might have been caused by any of these confounding variables.

Of course, researchers using a posttest only nonequivalent groups design can take steps to ensure that their groups are as similar as possible. In the present example, the researcher could try to select two classes at the same school, where the students in the two classes have similar scores on a standardized math test and the teachers are the same sex, are close in age, and have similar teaching styles. Taking such steps would increase the internal validity of the study because it would eliminate some of the most important confounding variables. But without true random assignment of the students to conditions, there remains the possibility of other important confounding variables that the researcher was not able to control.

Pretest-Posttest Nonequivalent Groups Design

Another way to improve upon the posttest only nonequivalent groups design is to add a pretest. In the  pretest-posttest nonequivalent groups design t here is a treatment group that is given a pretest, receives a treatment, and then is given a posttest. But at the same time there is a nonequivalent control group that is given a pretest, does  not  receive the treatment, and then is given a posttest. The question, then, is not simply whether participants who receive the treatment improve, but whether they improve  more  than participants who do not receive the treatment.

Imagine, for example, that students in one school are given a pretest on their attitudes toward drugs, then are exposed to an anti-drug program, and finally, are given a posttest. Students in a similar school are given the pretest, not exposed to an anti-drug program, and finally, are given a posttest. Again, if students in the treatment condition become more negative toward drugs, this change in attitude could be an effect of the treatment, but it could also be a matter of history or maturation. If it really is an effect of the treatment, then students in the treatment condition should become more negative than students in the control condition. But if it is a matter of history (e.g., news of a celebrity drug overdose) or maturation (e.g., improved reasoning), then students in the two conditions would be likely to show similar amounts of change. This type of design does not completely eliminate the possibility of confounding variables, however. Something could occur at one of the schools but not the other (e.g., a student drug overdose), so students at the first school would be affected by it while students at the other school would not.

Returning to the example of evaluating a new measure of teaching third graders, this study could be improved by adding a pretest of students’ knowledge of fractions. The changes in scores from pretest to posttest would then be evaluated and compared across conditions to determine whether one group demonstrated a bigger improvement in knowledge of fractions than another. Of course, the teachers’ styles, and even the classroom environments might still be very different and might cause different levels of achievement or motivation among the students that are independent of the teaching intervention. Once again, differential history also represents a potential threat to internal validity.  If asbestos is found in one of the schools causing it to be shut down for a month then this interruption in teaching could produce a difference across groups on posttest scores.

If participants in this kind of design are randomly assigned to conditions, it becomes a true between-groups experiment rather than a quasi-experiment. In fact, it is the kind of experiment that Eysenck called for—and that has now been conducted many times—to demonstrate the effectiveness of psychotherapy.

Interrupted Time-Series Design with Nonequivalent Groups

One way to improve upon the interrupted time-series design is to add a control group. The interrupted time-series design with nonequivalent groups involves taking  a set of measurements at intervals over a period of time both before and after an intervention of interest in two or more nonequivalent groups. Once again consider the manufacturing company that measures its workers’ productivity each week for a year before and after reducing work shifts from 10 hours to 8 hours. This design could be improved by locating another manufacturing company who does not plan to change their shift length and using them as a nonequivalent control group. If productivity  increased rather quickly after the shortening of the work shifts in the treatment group but productivity remained consistent in the control group, then this provides better evidence for the effectiveness of the treatment. 

Similarly, in the example of examining the effects of taking attendance on student absences in a research methods course, the design could be improved by using students in another section of the research methods course as a control group. If a consistently higher number of absences was found in the treatment group before the intervention, followed by a sustained drop in absences after the treatment, while the nonequivalent control group showed consistently high absences across the semester then this would provide superior evidence for the effectiveness of the treatment in reducing absences.

Pretest-Posttest Design With Switching Replication

Some of these nonequivalent control group designs can be further improved by adding a switching replication. Using a pretest-posttest design with switching replication design, nonequivalent groups are administered a pretest of the dependent variable, then one group receives a treatment while a nonequivalent control group does not receive a treatment, the dependent variable is assessed again, and then the treatment is added to the control group, and finally the dependent variable is assessed one last time.

As a concrete example, let’s say we wanted to introduce an exercise intervention for the treatment of depression. We recruit one group of patients experiencing depression and a nonequivalent control group of students experiencing depression. We first measure depression levels in both groups, and then we introduce the exercise intervention to the patients experiencing depression, but we hold off on introducing the treatment to the students. We then measure depression levels in both groups. If the treatment is effective we should see a reduction in the depression levels of the patients (who received the treatment) but not in the students (who have not yet received the treatment). Finally, while the group of patients continues to engage in the treatment, we would introduce the treatment to the students with depression. Now and only now should we see the students’ levels of depression decrease.

One of the strengths of this design is that it includes a built in replication. In the example given, we would get evidence for the efficacy of the treatment in two different samples (patients and students). Another strength of this design is that it provides more control over history effects. It becomes rather unlikely that some outside event would perfectly coincide with the introduction of the treatment in the first group and with the delayed introduction of the treatment in the second group. For instance, if a change in the weather occurred when we first introduced the treatment to the patients, and this explained their reductions in depression the second time that depression was measured, then we would see depression levels decrease in both the groups. Similarly, the switching replication helps to control for maturation and instrumentation. Both groups would be expected to show the same rates of spontaneous remission of depression and if the instrument for assessing depression happened to change at some point in the study the change would be consistent across both of the groups. Of course, demand characteristics, placebo effects, and experimenter expectancy effects can still be problems. But they can be controlled for using some of the methods described in Chapter 5.

Switching Replication with Treatment Removal Design

In a basic pretest-posttest design with switching replication, the first group receives a treatment and the second group receives the same treatment a little bit later on (while the initial group continues to receive the treatment). In contrast, in a switching replication with treatment removal design , the treatment is removed from the first group when it is added to the second group. Once again, let’s assume we first measure the depression levels of patients with depression and students with depression. Then we introduce the exercise intervention to only the patients. After they have been exposed to the exercise intervention for a week we assess depression levels again in both groups. If the intervention is effective then we should see depression levels decrease in the patient group but not the student group (because the students haven’t received the treatment yet). Next, we would remove the treatment from the group of patients with depression. So we would tell them to stop exercising. At the same time, we would tell the student group to start exercising. After a week of the students exercising and the patients not exercising, we would reassess depression levels. Now if the intervention is effective we should see that the depression levels have decreased in the student group but that they have increased in the patient group (because they are no longer exercising).

Demonstrating a treatment effect in two groups staggered over time and demonstrating the reversal of the treatment effect after the treatment has been removed can provide strong evidence for the efficacy of the treatment. In addition to providing evidence for the replicability of the findings, this design can also provide evidence for whether the treatment continues to show effects after it has been withdrawn.

Key Takeaways

• Quasi-experimental research involves the manipulation of an independent variable without the random assignment of participants to conditions or counterbalancing of orders of conditions.
• There are three types of quasi-experimental designs that are within-subjects in nature. These are the one-group posttest only design, the one-group pretest-posttest design, and the interrupted time-series design.
• There are five types of quasi-experimental designs that are between-subjects in nature. These are the posttest only design with nonequivalent groups, the pretest-posttest design with nonequivalent groups, the interrupted time-series design with nonequivalent groups, the pretest-posttest design with switching replication, and the switching replication with treatment removal design.
• Quasi-experimental research eliminates the directionality problem because it involves the manipulation of the independent variable. However, it does not eliminate the problem of confounding variables, because it does not involve random assignment to conditions or counterbalancing. For these reasons, quasi-experimental research is generally higher in internal validity than non-experimental studies but lower than true experiments.
• Of all of the quasi-experimental designs, those that include a switching replication are highest in internal validity.
• Practice: Imagine that two professors decide to test the effect of giving daily quizzes on student performance in a statistics course. They decide that Professor A will give quizzes but Professor B will not. They will then compare the performance of students in their two sections on a common final exam. List five other variables that might differ between the two sections that could affect the results.
• regression to the mean
• spontaneous remission

Share This Book

• Increase Font Size

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Am Med Inform Assoc
• v.13(1); Jan-Feb 2006

The Use and Interpretation of Quasi-Experimental Studies in Medical Informatics

Associated data.

Quasi-experimental study designs, often described as nonrandomized, pre-post intervention studies, are common in the medical informatics literature. Yet little has been written about the benefits and limitations of the quasi-experimental approach as applied to informatics studies. This paper outlines a relative hierarchy and nomenclature of quasi-experimental study designs that is applicable to medical informatics intervention studies. In addition, the authors performed a systematic review of two medical informatics journals, the Journal of the American Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics (IJMI), to determine the number of quasi-experimental studies published and how the studies are classified on the above-mentioned relative hierarchy. They hope that future medical informatics studies will implement higher level quasi-experimental study designs that yield more convincing evidence for causal links between medical informatics interventions and outcomes.

Quasi-experimental studies encompass a broad range of nonrandomized intervention studies. These designs are frequently used when it is not logistically feasible or ethical to conduct a randomized controlled trial. Examples of quasi-experimental studies follow. As one example of a quasi-experimental study, a hospital introduces a new order-entry system and wishes to study the impact of this intervention on the number of medication-related adverse events before and after the intervention. As another example, an informatics technology group is introducing a pharmacy order-entry system aimed at decreasing pharmacy costs. The intervention is implemented and pharmacy costs before and after the intervention are measured.

In medical informatics, the quasi-experimental, sometimes called the pre-post intervention, design often is used to evaluate the benefits of specific interventions. The increasing capacity of health care institutions to collect routine clinical data has led to the growing use of quasi-experimental study designs in the field of medical informatics as well as in other medical disciplines. However, little is written about these study designs in the medical literature or in traditional epidemiology textbooks. 1 , 2 , 3 In contrast, the social sciences literature is replete with examples of ways to implement and improve quasi-experimental studies. 4 , 5 , 6

In this paper, we review the different pretest-posttest quasi-experimental study designs, their nomenclature, and the relative hierarchy of these designs with respect to their ability to establish causal associations between an intervention and an outcome. The example of a pharmacy order-entry system aimed at decreasing pharmacy costs will be used throughout this article to illustrate the different quasi-experimental designs. We discuss limitations of quasi-experimental designs and offer methods to improve them. We also perform a systematic review of four years of publications from two informatics journals to determine the number of quasi-experimental studies, classify these studies into their application domains, determine whether the potential limitations of quasi-experimental studies were acknowledged by the authors, and place these studies into the above-mentioned relative hierarchy.

The authors reviewed articles and book chapters on the design of quasi-experimental studies. 4 , 5 , 6 , 7 , 8 , 9 , 10 Most of the reviewed articles referenced two textbooks that were then reviewed in depth. 4 , 6

Key advantages and disadvantages of quasi-experimental studies, as they pertain to the study of medical informatics, were identified. The potential methodological flaws of quasi-experimental medical informatics studies, which have the potential to introduce bias, were also identified. In addition, a summary table outlining a relative hierarchy and nomenclature of quasi-experimental study designs is described. In general, the higher the design is in the hierarchy, the greater the internal validity that the study traditionally possesses because the evidence of the potential causation between the intervention and the outcome is strengthened. 4

We then performed a systematic review of four years of publications from two informatics journals. First, we determined the number of quasi-experimental studies. We then classified these studies on the above-mentioned hierarchy. We also classified the quasi-experimental studies according to their application domain. The categories of application domains employed were based on categorization used by Yearbooks of Medical Informatics 1992–2005 and were similar to the categories of application domains employed by Annual Symposiums of the American Medical Informatics Association. 11 The categories were (1) health and clinical management; (2) patient records; (3) health information systems; (4) medical signal processing and biomedical imaging; (5) decision support, knowledge representation, and management; (6) education and consumer informatics; and (7) bioinformatics. Because the quasi-experimental study design has recognized limitations, we sought to determine whether authors acknowledged the potential limitations of this design. Examples of acknowledgment included mention of lack of randomization, the potential for regression to the mean, the presence of temporal confounders and the mention of another design that would have more internal validity.

All original scientific manuscripts published between January 2000 and December 2003 in the Journal of the American Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics (IJMI) were reviewed. One author (ADH) reviewed all the papers to identify the number of quasi-experimental studies. Other authors (ADH, JCM, JF) then independently reviewed all the studies identified as quasi-experimental. The three authors then convened as a group to resolve any disagreements in study classification, application domain, and acknowledgment of limitations.

Results and Discussion

What is a quasi-experiment.

Quasi-experiments are studies that aim to evaluate interventions but that do not use randomization. Similar to randomized trials, quasi-experiments aim to demonstrate causality between an intervention and an outcome. Quasi-experimental studies can use both preintervention and postintervention measurements as well as nonrandomly selected control groups.

Using this basic definition, it is evident that many published studies in medical informatics utilize the quasi-experimental design. Although the randomized controlled trial is generally considered to have the highest level of credibility with regard to assessing causality, in medical informatics, researchers often choose not to randomize the intervention for one or more reasons: (1) ethical considerations, (2) difficulty of randomizing subjects, (3) difficulty to randomize by locations (e.g., by wards), (4) small available sample size. Each of these reasons is discussed below.

Ethical considerations typically will not allow random withholding of an intervention with known efficacy. Thus, if the efficacy of an intervention has not been established, a randomized controlled trial is the design of choice to determine efficacy. But if the intervention under study incorporates an accepted, well-established therapeutic intervention, or if the intervention has either questionable efficacy or safety based on previously conducted studies, then the ethical issues of randomizing patients are sometimes raised. In the area of medical informatics, it is often believed prior to an implementation that an informatics intervention will likely be beneficial and thus medical informaticians and hospital administrators are often reluctant to randomize medical informatics interventions. In addition, there is often pressure to implement the intervention quickly because of its believed efficacy, thus not allowing researchers sufficient time to plan a randomized trial.

For medical informatics interventions, it is often difficult to randomize the intervention to individual patients or to individual informatics users. So while this randomization is technically possible, it is underused and thus compromises the eventual strength of concluding that an informatics intervention resulted in an outcome. For example, randomly allowing only half of medical residents to use pharmacy order-entry software at a tertiary care hospital is a scenario that hospital administrators and informatics users may not agree to for numerous reasons.

Similarly, informatics interventions often cannot be randomized to individual locations. Using the pharmacy order-entry system example, it may be difficult to randomize use of the system to only certain locations in a hospital or portions of certain locations. For example, if the pharmacy order-entry system involves an educational component, then people may apply the knowledge learned to nonintervention wards, thereby potentially masking the true effect of the intervention. When a design using randomized locations is employed successfully, the locations may be different in other respects (confounding variables), and this further complicates the analysis and interpretation.

In situations where it is known that only a small sample size will be available to test the efficacy of an intervention, randomization may not be a viable option. Randomization is beneficial because on average it tends to evenly distribute both known and unknown confounding variables between the intervention and control group. However, when the sample size is small, randomization may not adequately accomplish this balance. Thus, alternative design and analytical methods are often used in place of randomization when only small sample sizes are available.

What Are the Threats to Establishing Causality When Using Quasi-experimental Designs in Medical Informatics?

The lack of random assignment is the major weakness of the quasi-experimental study design. Associations identified in quasi-experiments meet one important requirement of causality since the intervention precedes the measurement of the outcome. Another requirement is that the outcome can be demonstrated to vary statistically with the intervention. Unfortunately, statistical association does not imply causality, especially if the study is poorly designed. Thus, in many quasi-experiments, one is most often left with the question: “Are there alternative explanations for the apparent causal association?” If these alternative explanations are credible, then the evidence of causation is less convincing. These rival hypotheses, or alternative explanations, arise from principles of epidemiologic study design.

Shadish et al. 4 outline nine threats to internal validity that are outlined in ▶ . Internal validity is defined as the degree to which observed changes in outcomes can be correctly inferred to be caused by an exposure or an intervention. In quasi-experimental studies of medical informatics, we believe that the methodological principles that most often result in alternative explanations for the apparent causal effect include (a) difficulty in measuring or controlling for important confounding variables, particularly unmeasured confounding variables, which can be viewed as a subset of the selection threat in ▶ ; (b) results being explained by the statistical principle of regression to the mean . Each of these latter two principles is discussed in turn.

Threats to Internal Validity

Adapted from Shadish et al. 4

An inability to sufficiently control for important confounding variables arises from the lack of randomization. A variable is a confounding variable if it is associated with the exposure of interest and is also associated with the outcome of interest; the confounding variable leads to a situation where a causal association between a given exposure and an outcome is observed as a result of the influence of the confounding variable. For example, in a study aiming to demonstrate that the introduction of a pharmacy order-entry system led to lower pharmacy costs, there are a number of important potential confounding variables (e.g., severity of illness of the patients, knowledge and experience of the software users, other changes in hospital policy) that may have differed in the preintervention and postintervention time periods ( ▶ ). In a multivariable regression, the first confounding variable could be addressed with severity of illness measures, but the second confounding variable would be difficult if not nearly impossible to measure and control. In addition, potential confounding variables that are unmeasured or immeasurable cannot be controlled for in nonrandomized quasi-experimental study designs and can only be properly controlled by the randomization process in randomized controlled trials.

Example of confounding. To get the true effect of the intervention of interest, we need to control for the confounding variable.

Another important threat to establishing causality is regression to the mean. 12 , 13 , 14 This widespread statistical phenomenon can result in wrongly concluding that an effect is due to the intervention when in reality it is due to chance. The phenomenon was first described in 1886 by Francis Galton who measured the adult height of children and their parents. He noted that when the average height of the parents was greater than the mean of the population, the children tended to be shorter than their parents, and conversely, when the average height of the parents was shorter than the population mean, the children tended to be taller than their parents.

In medical informatics, what often triggers the development and implementation of an intervention is a rise in the rate above the mean or norm. For example, increasing pharmacy costs and adverse events may prompt hospital informatics personnel to design and implement pharmacy order-entry systems. If this rise in costs or adverse events is really just an extreme observation that is still within the normal range of the hospital's pharmaceutical costs (i.e., the mean pharmaceutical cost for the hospital has not shifted), then the statistical principle of regression to the mean predicts that these elevated rates will tend to decline even without intervention. However, often informatics personnel and hospital administrators cannot wait passively for this decline to occur. Therefore, hospital personnel often implement one or more interventions, and if a decline in the rate occurs, they may mistakenly conclude that the decline is causally related to the intervention. In fact, an alternative explanation for the finding could be regression to the mean.

What Are the Different Quasi-experimental Study Designs?

In the social sciences literature, quasi-experimental studies are divided into four study design groups 4 , 6 :

• Quasi-experimental designs without control groups
• Quasi-experimental designs that use control groups but no pretest
• Quasi-experimental designs that use control groups and pretests
• Interrupted time-series designs

There is a relative hierarchy within these categories of study designs, with category D studies being sounder than categories C, B, or A in terms of establishing causality. Thus, if feasible from a design and implementation point of view, investigators should aim to design studies that fall in to the higher rated categories. Shadish et al. 4 discuss 17 possible designs, with seven designs falling into category A, three designs in category B, and six designs in category C, and one major design in category D. In our review, we determined that most medical informatics quasi-experiments could be characterized by 11 of 17 designs, with six study designs in category A, one in category B, three designs in category C, and one design in category D because the other study designs were not used or feasible in the medical informatics literature. Thus, for simplicity, we have summarized the 11 study designs most relevant to medical informatics research in ▶ .

Relative Hierarchy of Quasi-experimental Designs

O = Observational Measurement; X = Intervention Under Study. Time moves from left to right.

The nomenclature and relative hierarchy were used in the systematic review of four years of JAMIA and the IJMI. Similar to the relative hierarchy that exists in the evidence-based literature that assigns a hierarchy to randomized controlled trials, cohort studies, case-control studies, and case series, the hierarchy in ▶ is not absolute in that in some cases, it may be infeasible to perform a higher level study. For example, there may be instances where an A6 design established stronger causality than a B1 design. 15 , 16 , 17

Quasi-experimental Designs without Control Groups

Here, X is the intervention and O is the outcome variable (this notation is continued throughout the article). In this study design, an intervention (X) is implemented and a posttest observation (O1) is taken. For example, X could be the introduction of a pharmacy order-entry intervention and O1 could be the pharmacy costs following the intervention. This design is the weakest of the quasi-experimental designs that are discussed in this article. Without any pretest observations or a control group, there are multiple threats to internal validity. Unfortunately, this study design is often used in medical informatics when new software is introduced since it may be difficult to have pretest measurements due to time, technical, or cost constraints.

This is a commonly used study design. A single pretest measurement is taken (O1), an intervention (X) is implemented, and a posttest measurement is taken (O2). In this instance, period O1 frequently serves as the “control” period. For example, O1 could be pharmacy costs prior to the intervention, X could be the introduction of a pharmacy order-entry system, and O2 could be the pharmacy costs following the intervention. Including a pretest provides some information about what the pharmacy costs would have been had the intervention not occurred.

The advantage of this study design over A2 is that adding a second pretest prior to the intervention helps provide evidence that can be used to refute the phenomenon of regression to the mean and confounding as alternative explanations for any observed association between the intervention and the posttest outcome. For example, in a study where a pharmacy order-entry system led to lower pharmacy costs (O3 < O2 and O1), if one had two preintervention measurements of pharmacy costs (O1 and O2) and they were both elevated, this would suggest that there was a decreased likelihood that O3 is lower due to confounding and regression to the mean. Similarly, extending this study design by increasing the number of measurements postintervention could also help to provide evidence against confounding and regression to the mean as alternate explanations for observed associations.

This design involves the inclusion of a nonequivalent dependent variable ( b ) in addition to the primary dependent variable ( a ). Variables a and b should assess similar constructs; that is, the two measures should be affected by similar factors and confounding variables except for the effect of the intervention. Variable a is expected to change because of the intervention X, whereas variable b is not. Taking our example, variable a could be pharmacy costs and variable b could be the length of stay of patients. If our informatics intervention is aimed at decreasing pharmacy costs, we would expect to observe a decrease in pharmacy costs but not in the average length of stay of patients. However, a number of important confounding variables, such as severity of illness and knowledge of software users, might affect both outcome measures. Thus, if the average length of stay did not change following the intervention but pharmacy costs did, then the data are more convincing than if just pharmacy costs were measured.

The Removed-Treatment Design

This design adds a third posttest measurement (O3) to the one-group pretest-posttest design and then removes the intervention before a final measure (O4) is made. The advantage of this design is that it allows one to test hypotheses about the outcome in the presence of the intervention and in the absence of the intervention. Thus, if one predicts a decrease in the outcome between O1 and O2 (after implementation of the intervention), then one would predict an increase in the outcome between O3 and O4 (after removal of the intervention). One caveat is that if the intervention is thought to have persistent effects, then O4 needs to be measured after these effects are likely to have disappeared. For example, a study would be more convincing if it demonstrated that pharmacy costs decreased after pharmacy order-entry system introduction (O2 and O3 less than O1) and that when the order-entry system was removed or disabled, the costs increased (O4 greater than O2 and O3 and closer to O1). In addition, there are often ethical issues in this design in terms of removing an intervention that may be providing benefit.

The Repeated-Treatment Design

The advantage of this design is that it demonstrates reproducibility of the association between the intervention and the outcome. For example, the association is more likely to be causal if one demonstrates that a pharmacy order-entry system results in decreased pharmacy costs when it is first introduced and again when it is reintroduced following an interruption of the intervention. As for design A5, the assumption must be made that the effect of the intervention is transient, which is most often applicable to medical informatics interventions. Because in this design, subjects may serve as their own controls, this may yield greater statistical efficiency with fewer numbers of subjects.

Quasi-experimental Designs That Use a Control Group but No Pretest

An intervention X is implemented for one group and compared to a second group. The use of a comparison group helps prevent certain threats to validity including the ability to statistically adjust for confounding variables. Because in this study design, the two groups may not be equivalent (assignment to the groups is not by randomization), confounding may exist. For example, suppose that a pharmacy order-entry intervention was instituted in the medical intensive care unit (MICU) and not the surgical intensive care unit (SICU). O1 would be pharmacy costs in the MICU after the intervention and O2 would be pharmacy costs in the SICU after the intervention. The absence of a pretest makes it difficult to know whether a change has occurred in the MICU. Also, the absence of pretest measurements comparing the SICU to the MICU makes it difficult to know whether differences in O1 and O2 are due to the intervention or due to other differences in the two units (confounding variables).

Quasi-experimental Designs That Use Control Groups and Pretests

The reader should note that with all the studies in this category, the intervention is not randomized. The control groups chosen are comparison groups. Obtaining pretest measurements on both the intervention and control groups allows one to assess the initial comparability of the groups. The assumption is that if the intervention and the control groups are similar at the pretest, the smaller the likelihood there is of important confounding variables differing between the two groups.

The use of both a pretest and a comparison group makes it easier to avoid certain threats to validity. However, because the two groups are nonequivalent (assignment to the groups is not by randomization), selection bias may exist. Selection bias exists when selection results in differences in unit characteristics between conditions that may be related to outcome differences. For example, suppose that a pharmacy order-entry intervention was instituted in the MICU and not the SICU. If preintervention pharmacy costs in the MICU (O1a) and SICU (O1b) are similar, it suggests that it is less likely that there are differences in the important confounding variables between the two units. If MICU postintervention costs (O2a) are less than preintervention MICU costs (O1a), but SICU costs (O1b) and (O2b) are similar, this suggests that the observed outcome may be causally related to the intervention.

In this design, the pretests are administered at two different times. The main advantage of this design is that it controls for potentially different time-varying confounding effects in the intervention group and the comparison group. In our example, measuring points O1 and O2 would allow for the assessment of time-dependent changes in pharmacy costs, e.g., due to differences in experience of residents, preintervention between the intervention and control group, and whether these changes were similar or different.

With this study design, the researcher administers an intervention at a later time to a group that initially served as a nonintervention control. The advantage of this design over design C2 is that it demonstrates reproducibility in two different settings. This study design is not limited to two groups; in fact, the study results have greater validity if the intervention effect is replicated in different groups at multiple times. In the example of a pharmacy order-entry system, one could implement or intervene in the MICU and then at a later time, intervene in the SICU. This latter design is often very applicable to medical informatics where new technology and new software is often introduced or made available gradually.

Interrupted Time-Series Designs

An interrupted time-series design is one in which a string of consecutive observations equally spaced in time is interrupted by the imposition of a treatment or intervention. The advantage of this design is that with multiple measurements both pre- and postintervention, it is easier to address and control for confounding and regression to the mean. In addition, statistically, there is a more robust analytic capability, and there is the ability to detect changes in the slope or intercept as a result of the intervention in addition to a change in the mean values. 18 A change in intercept could represent an immediate effect while a change in slope could represent a gradual effect of the intervention on the outcome. In the example of a pharmacy order-entry system, O1 through O5 could represent monthly pharmacy costs preintervention and O6 through O10 monthly pharmacy costs post the introduction of the pharmacy order-entry system. Interrupted time-series designs also can be further strengthened by incorporating many of the design features previously mentioned in other categories (such as removal of the treatment, inclusion of a nondependent outcome variable, or the addition of a control group).

Systematic Review Results

The results of the systematic review are in ▶ . In the four-year period of JAMIA publications that the authors reviewed, 25 quasi-experimental studies among 22 articles were published. Of these 25, 15 studies were of category A, five studies were of category B, two studies were of category C, and no studies were of category D. Although there were no studies of category D (interrupted time-series analyses), three of the studies classified as category A had data collected that could have been analyzed as an interrupted time-series analysis. Nine of the 25 studies (36%) mentioned at least one of the potential limitations of the quasi-experimental study design. In the four-year period of IJMI publications reviewed by the authors, nine quasi-experimental studies among eight manuscripts were published. Of these nine, five studies were of category A, one of category B, one of category C, and two of category D. Two of the nine studies (22%) mentioned at least one of the potential limitations of the quasi-experimental study design.

Systematic Review of Four Years of Quasi-designs in JAMIA

JAMIA = Journal of the American Medical Informatics Association; IJMI = International Journal of Medical Informatics.

In addition, three studies from JAMIA were based on a counterbalanced design. A counterbalanced design is a higher order study design than other studies in category A. The counterbalanced design is sometimes referred to as a Latin-square arrangement. In this design, all subjects receive all the different interventions but the order of intervention assignment is not random. 19 This design can only be used when the intervention is compared against some existing standard, for example, if a new PDA-based order entry system is to be compared to a computer terminal–based order entry system. In this design, all subjects receive the new PDA-based order entry system and the old computer terminal-based order entry system. The counterbalanced design is a within-participants design, where the order of the intervention is varied (e.g., one group is given software A followed by software B and another group is given software B followed by software A). The counterbalanced design is typically used when the available sample size is small, thus preventing the use of randomization. This design also allows investigators to study the potential effect of ordering of the informatics intervention.

Although quasi-experimental study designs are ubiquitous in the medical informatics literature, as evidenced by 34 studies in the past four years of the two informatics journals, little has been written about the benefits and limitations of the quasi-experimental approach. As we have outlined in this paper, a relative hierarchy and nomenclature of quasi-experimental study designs exist, with some designs being more likely than others to permit causal interpretations of observed associations. Strengths and limitations of a particular study design should be discussed when presenting data collected in the setting of a quasi-experimental study. Future medical informatics investigators should choose the strongest design that is feasible given the particular circumstances.

Supplementary Material

Dr. Harris was supported by NIH grants K23 AI01752-01A1 and R01 AI60859-01A1. Dr. Perencevich was supported by a VA Health Services Research and Development Service (HSR&D) Research Career Development Award (RCD-02026-1). Dr. Finkelstein was supported by NIH grant RO1 HL71690.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals

You are here

• Online First
• Effect of Universal Credit on young children’s mental health: quasi-experimental evidence from Understanding Society
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0009-0005-4771-7441 Huihui Song 1 ,
• http://orcid.org/0000-0002-3907-8396 Anwen Zhang 2 ,
• http://orcid.org/0000-0002-4208-9475 Benjamin Barr 1 ,
• Sophie Wickham 1
• 1 Department of Public Health, Policy and Systems , University of Liverpool , Liverpool , UK
• 2 Adam Smith Business School , University of Glasgow , Glasgow , UK
• Correspondence to Dr Huihui Song; hss6c{at}liverpool.ac.uk

Background Child mental health has become an increasingly important issue in the UK, especially in the context of significant welfare reforms. Universal Credit (UC) has introduced substantial changes to the UK’s social security system, significantly impacting low-income families. Our aim was to assess the effects of UC’s introduction on children’s mental health for families eligible for UC versus a comparable non-eligible sample.

Methods Using Understanding Society data from 5806 observations of 4582 children (aged 5 or 8 years) in Great Britain between 2012 and 2018, we created two groups: children whose parents were eligible for UC (intervention group) and children whose parents were ineligible for UC (comparison group). Child mental health was assessed using a parent-reported Strengths and Difficulties Questionnaire. The OR and percentage point change in the prevalence of children experiencing mental health difficulties between the intervention group and the comparison group following the introduction of UC were analysed. We also investigated whether the utilisation of childcare services and changes in household income were mechanisms by which UC impacted children’s mental health.

Results Logistic regression results demonstrated that the prevalence of mental health problems among eligible children whose parents were unemployed increased by an OR of 2.18 (95% CI 1.14 to 4.18), equivalent to an 8-percentage point increase (95% CI 1 to 14 percentage points) following the introduction of UC, relative to the comparison group. Exploring potential mechanisms, we found neither reduced household income nor increased use of childcare services, which served as a proxy for reduced time spent with parents, significantly influenced children’s mental health.

Conclusions UC has led to an increase in mental health problems among recipient children, particularly for children in larger families and those aged 8. Policymakers should carefully evaluate the potential health consequences for specific demographics when introducing new welfare policies.

• HEALTH POLICY
• CHILD HEALTH
• MENTAL HEALTH

Data availability statement

Data are available upon reasonable request.

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See:  https://creativecommons.org/licenses/by/4.0/ .

https://doi.org/10.1136/jech-2024-222293

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Previous research has focused on the health implications of Universal Credit for adult populations; however, a gap remains regarding evaluations of the policy’s influence on the mental health of children post implementation.

WHAT THIS STUDY ADDS

Employing a quasi-experimental study design, this research identified that the implementation of UC was linked to an increase in children’s mental health issues. These findings suggested that benefits policy shocks affected not only adult recipients but also extended to their children, highlighting the broader impact of such policy changes on family well-being.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

In light of the evidence demonstrating the adverse effects of welfare changes on children’s mental health, it is imperative to establish a comprehensive health impact assessment of children’s well-being within any welfare reform evaluation. Furthermore, the more pronounced impact of UC on children in larger families and those aged 8 years underscores the importance of considering household-specific effects in policy implementation. The health outcomes of children should be a central consideration when redesigning welfare systems.

Introduction

Childhood is a critical phase for mental health, characterised by rapid brain growth and development. 1 During this period, children acquire cognitive skills that shape their future mental well-being and are essential for assuming adult roles in society. This underscores the vital importance of providing children with the best possible start in life, particularly in early childhood. In the UK, there has been a concerning trend of worsening mental health among young children. For example, rates of mental health issues among children aged 5–10 years rose from 9% in 2017 to 14% in 2020. 2 Addressing the causes of this increase is a public health priority.

Policy actions can yield unintended consequences, and welfare reform stands out as a potential contributor to such mental health outcomes. 3 Universal Credit (UC) is arguably the biggest overhaul of the welfare system in the UK since the Beveridge reforms of the 1940s. UC has been gradually implemented across the UK for different groups of people. The rollout of UC commenced in April 2013, with eligibility extended to families with children starting in May 2016 ( figure 1 illustrates the timeline of the national expansion of the UC rollout). The Department for Work and Pensions data show that as of February 2022, there were 3.8 million children in over 2 million households who were receiving UC, accounting for 49% of all households under UC. Three-quarters of families with children on UC had a child of primary school age or younger. 4

• Download figure
• Open in new tab
• Download powerpoint

Timeline of UC. Note: Live service started in April 2013 in the North West. It did not involve online applications; only single, childless, unemployed adults without housing costs were eligible initially. In April 2016, full service commenced, accepting new claims from all types of claimants and concluded in December 2018. Natural migration refers to situations when existing claimants of legacy benefits and tax credits experienced a change in circumstances, such as unemployment, and were migrated to UC. Managed migration refers to the process of transferring the remaining claimants of legacy benefits and tax credits to UC. Source: National Audit Office (2018). 4 UC, Universal Credit.

UC has been criticised for its digitalised implementation style, wait for first payment and increased use of conditionality and sanctions. Studies have found negative impacts of UC on employment outcomes, 5 debt, 6 food bank usage, 7 housing insecurity 8 and higher crime rates. 9 10 There are several papers, both quantitative and qualitative, that have explored the impact of UC on the mental health and psychological distress of working-age adults, finding that individuals entering UC experienced a deterioration in their mental health. 11–13 There have not, however, to our knowledge, been any previous studies investigating the impact on children. Therefore, understanding the impact of UC on child mental health is now urgently needed, given the rise in child mental illness we are seeing in the UK. 2

In exploring the mechanisms by which UC impacts children’s mental health, there are multiple factors to consider (see figure 2 ). For example, a reduction in household income under UC may be detrimental to children’s well-being and development. 14 Compared with entitlements under the tax credit system, the majority of working families are worse off under UC, experiencing an average loss of entitlement of £41 per week. 15 In addition, under UC, the introduction of the two-child limit restricts child-related benefits to the first two children in a family with a third or additional baby since April 2017. This means that families lose roughly £237 per child, which reduces the overall income available to households with larger families and has pushed more families into poverty or deeper into poverty. Moreover, the requirement for compulsory intensive job searches for unemployed or low-income claimants may lead parents to rely more on childcare services, reducing the time spent with their children, which could affect children’s mental health.

Possible pathways to poor mental health outcomes of young children under UC. Note: This simplified figure illustrates potential pathways leading to adverse mental health outcomes in young children under UC. For a more detailed illustration of these pathways, please refer to online supplemental appendix 1 . UC, Universal Credit.

Supplemental material

Hence, there is an urgent need to understand both the potential differential effects of UC for different groups of children and the potential mechanisms through which UC impacts children’s mental health to inform future policy implementation. Several important pathways through which UC may impact children’s mental health are depicted in figure 2 .

In this paper, we examined the potential impact of UC on young children’s socioemotional behavioural difficulties, which have been recognised as a critical factor in understanding mental health outcomes, 16 with a growing body of research consistently identifying it as a pivotal marker on the pathway to mental illness. 17 We compared the socioemotional behavioural difficulties of children whose parents were unemployed and therefore eligible for UC to those not eligible for UC before and after UC became available for families with children in 2016. We also explored if there were differential effects of the introduction of UC for younger versus older children and single-child versus multiple-child households. Finally, we explored whether the reduction in household income or changes in childcare service usage were the pathways through which UC impacted children’s socioemotional behavioural difficulties.

Study design and participants

We used data from the UK Household Longitudinal Study (UKHLS). The UKHLS is a large and nationally representative panel survey of approximately 40 000 households. It includes information on households’ social, economic and demographic status, health, employment, and social benefits across the UK from 2009 onwards. 18 The Strengths and Difficulties Questionnaire (SDQ) has been collected in the UKHLS since Wave 3 and is only asked of children aged 5 and 8 years. Therefore, we included data from 2012 to 2018, covering eight waves of data. According to the inclusion and exclusion criteria, data from 5806 observations of 4582 children aged 5 and 8 years were included in the study population. A flowchart of participants and details of the study sample can be found in online supplemental appendix 2 .

Eligibility and policy exposure

From May 2016, households with children became eligible for UC in England, Scotland and Wales. 19 We took a conservative approach to eligibility and classified children’s exposure to UC based on their parents' unemployment status. Children were assigned to the intervention group if at least one of their working-age parents (18–64 years) identified as unemployed and therefore eligible to receive UC. They were assigned to the comparison group in a given wave if their parents identified as anything other than unemployed. Eligibility could vary over time. The interview year was used to determine the period before (<2016) and after policy exposure (≥2016).

The primary outcome of interest was young children’s socioemotional behavioural difficulties using the parent-reported SDQ. This is a short behavioural screening questionnaire for children and was only asked of parents whose young children were aged 5 and 8 years. The composition of the SDQ is detailed in online supplemental appendix 3 . A total difficulties score was created by summing the first four subscales (range 0–40). We used a dichotomised score and constructed a dummy variable indicating mental difficulty, where 0–16 indicated no difficulties and 17–40 indicated socioemotional behavioural difficulties. 20 The dichotomised score better reflects a clinically meaningful effect on child’s mental health, and it is likely that the effect of the policy on social and behavioural difficulties is non-linear, potentially having a greater effect at higher levels of SDQ score. We tested this assumption using quantile regression (see online supplemental appendix 4 ) and repeated the analysis using the continuous score (see online supplemental appendix 7 ).

Following the literature, 21 continuous covariates included the logarithm of household inflation‐adjusted income (household income was measured as the logarithm of the contemporaneous monthly net income from the labour market and all other sources taking away any taxes, deductions and benefits in GB 2010 prices) and the mother’s mental health (measured using the 12‐item General Health Questionnaire). 22

Categorical covariates included the child’s age (either 5 or 8), gender (female=0 and male=1), long-term health condition (‘Excellent’ compared with ‘very good’, ‘good’, ‘fair’ and ‘poor’), the mother’s education level (‘Degree’ compared with ‘other higher’, ‘A levels’, ‘GCSE’ and ‘no/other qualification’), and whether there was only one child in the family (only one child in family=1 and additional children in family=0). Childcare utilisation was measured based on maternal reports, with a value of 1 indicating that the mother reported using childcare services, and 0 otherwise.

Statistical analysis

Main analysis.

To understand if the introduction of UC has had an effect on child socioemotional behavioural difficulties of parents eligible for UC, we first analysed whether the trends in socioemotional behavioural difficulties ran in parallel prior to the intervention. This comparison of trends in the outcome focused on the percentage of children with mental health issues, specifically those with SDQ scores equal to or exceeding 17. This comparison was conducted between the intervention and comparison groups during the preintervention period.

Next, we employed a generalised difference-in-differences framework and logistic models to identify the treatment effect of UC on children’s socioemotional behavioural difficulties between 2012 and 2018. This analysis compared children of parents eligible for UC with those of ineligible parents, adjusting for the covariates described above. Therefore, changes in mental health for the children limit potential biases after controlling for covariates.

We conducted several robustness tests to investigate whether our results were sensitive to model specifications. First, we repeated our main analysis using an alternative approach to eligibility. We classified children’s exposure based on their parent’s reports of working-age benefits. Children were assigned to the intervention group if at least one of their parents reported receiving either UC or one of the six legacy benefits and were therefore eligible to receive (or move onto) UC. They were assigned to the comparison group in a given wave if their parents did not report receiving UC or legacy benefits.

Second, we repeated the main analysis using the continuous measure of SDQ as the outcome. For this model, we used a linear rather than logistic regression model. Third, we constructed a ‘stable treatment’ status for children in order to implement canonical difference-in-differences to overcome potential issues around treatment status staggering. Children were assigned to the intervention group if their parents were unemployed for any period. Once assigned, they were considered to belong to the intervention group for the entire period. The comparison group was defined as children whose parents were always employed, which allowed us to construct a time-invariant comparison group. Fourth, we repeated the analysis, excluding families with more than two children, as UC initially only allowed families with two or fewer children to apply. Fifth, we repeated our main analysis and excluded the top 25% of households with the highest income to improve comparability between the intervention and comparison groups. Sixth, we only included children with two or more observations of the outcome in a linear probability model with individual fixed effects to re-estimate the main findings. Seventh, we used propensity score matching with bootstrapping to overcome demographic variation between the intervention and comparison groups. Eighth, we used the linear ramp model to explore the potential temporal variation in the rollout of UC. Finally, to overcome potential bias in the missing data (ie, structural missingness in the outcome variable and other forms of non-random missingness), we have repeated the main analysis using multiple imputation and inverse probability weighting (IPW).

Exploring heterogeneity effects and mechanisms

We conducted two heterogeneity tests to explore if there were differential effects based on child’s age and household composition, specifically whether the household had only one child or multiple children. We repeated the main analysis using subgroups to explore variability in effects.

To explore the mechanism through which UC potentially affected children’s socioemotional behavioural difficulties, we investigated two policy elements embedded in UC. First, we used the utilisation of childcare as a potential proxy for reduced time spent with parents (and potentially the changes in conditionality under UC). Second, we explored changes in household income as a proxy for changes in benefit income. We repeated the main analysis, substituting the socioemotional behavioural outcome with the mechanism variable. A detailed description of all methodologies is described in online supplemental appendix 4 .

We included 5806 observations from 4582 children (aged 5 or 8 years) in England, Wales and Scotland who participated in the UKHLS between 2012 and 2018. The baseline characteristics of the intervention and comparison groups in the years prior to UC’s introduction are presented in table 1 ( online supplemental appendix 6 outlines the number of observations in the intervention and comparison groups). The socioemotional behavioural difficulty was more prevalent in the intervention group compared with the comparison group. Consistently, the average SDQ scores for the intervention group were 1.8 points higher than those of the comparison group. The comparison group used childcare services more frequently and had a higher household income. There were no large differences between participants in the intervention group and the comparison group in terms of age and gender. Children in the intervention group exhibited worse long-term health conditions and lived in households with more than one child. Additionally, the intervention group had a higher prevalence of mothers experiencing mental health issues and lower levels of educational attainment.

• View inline

Baseline characteristics in the years before Universal Credit was introduced

The trend in the proportion of children with socioemotional behavioural difficulties in both the intervention and comparison groups before and after the introduction of UC is displayed in figure 3 . While the intervention and comparison groups differed in terms of their difficulties prior to UC, this difference, however, should not introduce bias in the analysis, as the difference between the two groups would persist at the same level in the absence of UC (see online supplemental appendix 5 for full regression results of the parallel trend analysis). The parallel trend graph suggested that a greater number of children in the intervention group experienced mental health issues compared with the comparison group following the implementation of UC.

Graphical representation of socioemotional behavioural difficulties in the intervention and comparison groups before and after Universal Credit was introduced. (Observing parallel trends in the preintervention period.)

The difference-in-difference results in table 2 indicated that UC exacerbated children’s socioemotional behavioural difficulties in households with unemployed parents. The effect of UC was to increase the prevalence of difficulties by an OR of 2.18 (95% CI 1.14 to 4.18), equivalent to an 8-percentage point increase (95% CI 1 to 14) among eligible children.

Difference-in-difference estimates of the impact of UC on children’s socioemotional behavioural difficulties

The outcomes of a series of robustness analyses are presented in figure 4 . These included the alternative definition of eligibility criteria, adjustments to the scope of the study population, and the utilisation of different model specifications. Except for the analysis limited to families with fewer than two children, which suggested a weaker effect, the results remained consistent across various checks. More detailed results, including parallel trends and regression outcomes, are provided in online supplemental appendix 7 .

Outcomes of a series of robustness analyses with 95% CIs. Note: When employing continuous SDQ scores to assess children’s mental health conditions, the changes were measured on a different scale rather than in percentage points; thus, this outcome was not included in the graph. However, the results were similar, showing a 1.40 SDQ score increase (95% CI 0.49 to 2.39) for the intervention group after the introduction of Universal Credit. The results are presented in online supplemental appendix 7 . SDQ, Strengths and Difficulties Questionnaire.

The results of the heterogeneity effects showed that following the implementation of UC, the prevalence of socioemotional behavioural difficulties increased by an OR of 2.40 (95% CI 1.20 to 4.83), equivalent to a 9-percentage point increase (95% CI 2 to 16 percentage points) for eligible children in families with two or more children while exhibiting an insignificant effect on eligible children in one-child families ( table 3 ). Additionally, the results suggested that UC negatively impacted children aged 8 (95% CI 5 to 26 percentage points).

Heterogeneity results of the impact of Universal Credit on children’s mental health

Our exploration of the mechanism through which UC potentially affected children’s socioemotional behavioural difficulties found that neither the use of childcare services nor a reduction in household income were the main contributors to children’s experiences of worse socioemotional behavioural difficulties (see online supplemental appendix 9 ).

This paper has demonstrated that the implementation of UC has exacerbated socioemotional behavioural difficulties in children. This corresponded to an 8-percentage point (95% CI 1 to 14 percentage points) increase in the proportion of children with parents eligible for UC based on their employment status experiencing socioemotional behavioural difficulties. This estimation served as a conservative estimate, as some individuals in the comparison group might also have been eligible to apply for UC for reasons other than unemployment, although accounting for only 2% of the comparison group. 11 The findings were strengthened by the robustness tests showing similar effects from different model specifications.

This study emphasised that children in larger families and those aged 8 years may be more susceptible to the impacts of welfare reform, underscoring the importance of intervention strategies. To understand why children were affected by UC, we analysed the treatment effects on two mediators linked to the subelements of UC. The results suggested that neither lower household income nor parents’ use of childcare services were the main factors that caused the observed deteriorating child mental health. This might be because the income measure, based on survey-reported data, does not fully capture the effect, especially at very low-income levels where negative impacts might be concentrated. Additionally, delays in benefit payments, sanctions under UC and the anticipation of moving into employment could be alternative pathways affecting children’s mental health.

Our study adds to the growing body of evidence of the adverse effects of UC on various socioeconomic aspects 5–10 that have focused on the experiences of adults. Regarding the mental health of adults, both quantitative and qualitative studies have explored the impact of UC on working-age adults, consistently finding a decline in mental health among individuals transitioning to UC. 11–13 However, there is a gap in research concerning the impact of UC on children’s mental health.

Our research endeavours to augment the existing body of knowledge by furnishing longitudinal evidence that illuminates the mental health ramifications associated with the transition to UC for children with unemployed parents. By doing so, our study underscores that the unintended consequences of UC extend beyond the recipients themselves, also impacting the mental well-being of children.

This study has some limitations. First, the intervention group had a small sample size, introducing potential uncertainty. The CI suggested that the true impact of UC on children’s mental health may range between 1 and 14 percentage points. Second, the implementation of UC in the UK follows a staggered full-service rollout schedule, leading to variations in the timing of application for eligible children. However, estimating the impact of the UC rollout on children’s mental health is constrained by the limitation of a small sample size within a singular district. Third, we used reported unemployment. However, not all unemployed individuals received UC, and some participants in the comparison group may have become eligible over the course of the analysis, although only a small proportion (<2%) of the comparison group was affected. 11 Alternative definitions, however, indicated similar results. Fourth, the prevalence of missing data and attrition posed common challenges in longitudinal datasets and natural policy methodologies. Lastly, our measure of use of childcare services may not have been an accurate measure of time spent in childcare services and did not reflect the quality of those arrangements, thus hindering our ability to determine the potential mechanisms involved.

Considering the adverse influence of UC on children’s mental health, as outlined in this paper, it is imperative for future government policies in the UK and other countries to consider the well-being of children when reforming the welfare system. The mechanisms for this effect remain unclear. Further research should aim to understand the experience of families with children using UC and the potential pathways for negative and positive effects on child well-being, adapting the service to maximise positive benefits. Furthermore, specific policies related to children, including parental conditionality and the benefit cap, require further research to explore their impact on children and young people. Policymakers should give greater consideration to the health impact of changes to welfare systems on children.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

• Margolis AE
• Ford T , et al
• Kinderman P ,
• Whitehead M
• National Audit Office
• Alvarez-Vilanova J
• Chris Drake
• Loopstra R ,
• Fledderjohann J ,
• Reeves A , et al
• Tiratelli M ,
• Bradford B ,
• Wickham S ,
• Bentley L ,
• Rose T , et al
• Robertson L ,
• Stewart ABR
• Copeland W ,
• Simeonova E
• Ortuño-Sierra J ,
• Aritio-Solana R ,
• Fonseca-Pedrero E
• Ganchimeg T ,
• Naranbaatar N , et al
• Understanding Society
• Child Poverty Action Group

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1

X @benj_barr2

Correction notice This article has been corrcected since it first published. The funding statement has been corrected.

Contributors HS is lead author and guarantor. HS, AZ, SW and BB planned the study and led the drafting and revising of the manuscript. HS and AZ analysed the data. HS, AZ, SW and BB contributed to interpreting the data and drafting and revising the manuscript. All authors approved the submitted version of the manuscript.

Funding SW was funded by a Welcome Trust Society and Ethics Fellowship (200335/Z/15/Z). BB and SW were supported by the UK National Institute of Health Research Public Health Research Programme (NIHR131709).

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

• Open access
• Published: 18 September 2024

Investigating the impact of virtual simulation experiment and massive open online course (MOOC) on medical students’ wound debridement training: a quasi-experimental study

• Wang Zhang 1 ,
• Zhe Xie 1 ,
• Jingfeng Li 2 ,
• Changhuan Liu 1 ,
• Zheng Wang 1 ,
• Yadian Xie 3 ,
• Yuping Liu 4 ,
• Zonghuan Li 1 ,
• Xiaqing Yang 1 ,
• Xue Fang 1 ,
• Xinghuan Wang 5 ,
• Renxiong Wei 2 , 3 &
• Xin Wang 1 , 5 , 6  

BMC Medical Education volume  24 , Article number:  1023 ( 2024 ) Cite this article

Metrics details

This study aims to evaluate the impact of virtual simulation experiment teaching model and Massive Open Online Course (MOOC) teaching model on the teaching effect in debridement teaching.

The study adopted a quasi-experimental design and used virtual simulation technology to construct a virtual simulation experimental teaching platform for debridement. This study was conducted at the Second Clinical College of Wuhan University. The experimental group was composed of 135 third-year clinical medicine students in the 2020 grade, who received the virtual simulation experimental teaching model; the control group was 122 third-year students in the same major in the 2019 grade, who used the MOOC teaching model. The performance of the two groups of students was evaluated through theoretical tests and animal experiment operation. In addition, the effectiveness of the experimental teaching model and student satisfaction were evaluated through questionnaire surveys.

The theoretical test scores and animal experiment report scores of the experimental group were significantly higher than those of the control group, and the debridement animal experiment operation time of the experimental group was shorter than that of the control group, and the difference was statistically significant ( P  < 0.05). The post-class questionnaire survey of the experimental group showed that most students were satisfied with the virtual simulation experimental teaching model and believed that it represented the future teaching trend.

Conclusions

In the teaching of debridement, virtual simulation experiment is an effective t teaching model, which not only helps to improve student performance, but also significantly reduces skill operation time and is recognized by students.

Peer Review reports

Virtual simulation experimental teaching is a pedagogical approach that combines virtual reality technology with experimental teaching methods. In recent years, this approach has seen widespread application in disciplines such as surgery, anatomy, and nursing [ 1 , 2 , 3 ].Numerous studies have demonstrated that simulation-based learning experiences facilitate the integration of theoretical knowledge with practical skills, allowing learners to develop the competencies necessary for independent practice within environments perceived as authentic [ 4 , 5 , 6 ].

General Surgery Experiment is a course on basic surgical operating skills. It is a professional core course for undergraduates in clinical medicine. It is a basic ability that medical students must possess [ 7 ]. Among them, experimental teaching is the connection between theoretical teaching and clinical practice. As one of the basic operations in the general experiment of surgery, debridement is a key part of the compulsory basics of surgery. The main purpose of debridement is to remove foreign matter, necrotic tissue and bacteria in the wound and create good conditions for wound healing. In the medical education of general surgery experiments, how to effectively teach debridement skills is crucial to improving clinical practice capabilities. Massive Open Online Course (MOOC) is a digital course that utilizes digital technology and large databases to store teaching videos, learning content, and online test questions on an Internet platform [ 8 ].However, MOOC teaching method face many challenges in actual operation, such as lack of practical operation and low interaction [ 9 ]. Therefore, how to effectively improve students’ experimental operation skills has become one of the urgent problems to be solved in the current teaching reform of debridement experimental course.

The virtual simulation experiment platform is a computer-based system that replicates real experimental environments and processes using virtual reality, augmented reality, 3D modeling, and data interaction [ 10 , 11 ].At present, surgical virtual simulation experiments mainly focus on virtual endoscopy and orthopedic learning procedures or concepts [ 12 ], virtual simulation experiments have not been fully utilized in clinical medicine [ 13 ], the use of virtual simulation experimental platforms in debridement teaching has not yet been reported. Virtual simulation experimental teaching holds considerable practical significance in debridement experimental courses. It enhances skills and knowledge retention, offers a safe learning environment, boosts learner interest, and conserves medical teaching resources [ 14 , 15 , 16 ]. In addition, by playing the role of virtual doctors, students can cultivate their professionalism early and improve their medical humanistic qualities.

This study compares the virtual simulation experimental teaching model with the MOOC teaching model to analyze the impact of the virtual simulation experimental platform on teaching debridement courses, aiming to explore its potential application in medical education.

Characteristics of debridement virtual simulation experiment

Experimental principle.

The experimental teaching of debridement through virtual simulation adopts an immersive interaction-driven approach, integrating independent learning, human-computer interaction, practice, and assessment. It allows students to learn the treatment process and operational steps of debridement through three-dimensional virtual simulation animations and interactive operations. The platform focuses on clinical scenarios for debridement of moderately injured wounds, specifically those with tendon rupture, this is what our course requires of undergraduates. The scenarios are accurately configured to include the open wound status, ongoing treatment actions, and relevant medical equipment. The platform showcases changes in the wound during the debridement process, aiding in the comprehension of different knowledge points and the relationships between various operational steps. Debridement virtual simulation experiment website ( https://www.ilab-x.com/details/page?id=10474&isView=true ).

Virtual simulation core design

Virtual simulation core design focuses on the debridement scenario of moderately injured wounds with tendon rupture that medical undergraduates should master. The system allows users to enter the practice or assessment module, where they encounter the initial scene with the open wound configuration, treatment actions, and relevant medical equipment. Through three-dimensional interactive teaching, users can engage in dynamic simulation and interactive control, enabling them to view and learn operating steps from various angles, cross-sections, perspectives, rotations, and zooming features. The design also allows for three-dimensional free perspective, where users can rotate, zoom, and pan through the operating parts. Changes in the wound during the debridement process can be displayed in different viewing angle modes, helping users understand the relationship between each knowledge point and operation step. The system emphasizes learning the kessler suture method of tendon as part of basic skills training, providing interactive exercises for mastery. The interface is user-friendly, offering realistic experimental scenarios and operation interactions to enhance the sense of experience and interactivity. The debridement virtual simulation experiment platform is shown in Fig.  1 .

Debridement virtual simulation experiment platform. A : Home page of the debridement virtual simulation experiment platform website; B : Being asked to adjust the order of debridement experiment steps before the virtual simulation experiment; C : Experimental practice and assessment module; D : Debridement The “wound disinfection” operation in the virtual simulation experiment of debridement; E :“Tendon kessler suture method” in the virtual simulation experiment of debridement; F : Schematic diagram of the postoperative treatment steps in the virtual simulation experiment of debridement

Study design and participants

This study was designed as a quasi-experimental study. The experimental group included 135 third-year students majoring in clinical medicine who were enrolled in 2020 and received virtual simulation experimental teaching on debridement. The control group comprised 122 third-year students majoring in clinical medicine who were enrolled in 2019 and received the MOOC teaching method. Although both groups of students did not receive our teaching in the same year, they both received a different teaching model of debridement in the first semester of their junior year. The sample inclusion criteria were: (i) no prior learning experience in debridement experimental courses; (ii) no previous exposure to debridement virtual simulation and debridement MOOC; (iii) obtaining informed consent from all research participants. Exclusion criteria: (i) Students who have received debridement study; (ii) have not completed debridement virtual simulation experiment teaching or MOOC teaching; (iii) have received teaching but have not completed the questionnaire and test paper. This study was approved by the Ethics Committee of Zhongnan Hospital of Wuhan University (2022144 K), and informed consent was obtained from all study participants, confirming their understanding of the study’s purpose, process, potential risks and benefits, and their voluntary participation. Our study did not involve clinical trials, so there was no clinical trial number. We only sent questionnaires to the study population and collected objective data of the exam, which also obtained their informed consent.

Learning program

Experimental group (accepting the virtual simulation experimental teaching model, including virtual simulation experimental platform training, theoretical teaching, and animal experiment classes) and the control group (accepting the MOOC teaching model, including watching the debridement teaching video before the experimental class (no virtual simulation experiment Platform training), theoretical teaching, animal experiment classes). The design of this study is shown in Fig.  2 .

Experimental group teaching process

The students in the experimental group utilized the debridement virtual simulation experimental platform for training. They were tasked with completing the debridement study and assessment module, which consisted of several components. Firstly, knowledge learning, where the system automatically presented information on the definition, purpose, indications, and contraindications of debridement. Subsequently, the students were assessed on the general operating procedures of debridement. Skill training involved working with 3D virtual open wounds of varying degrees, preparing for debridement and suturing, performing intra-operative procedures, and managing post-operative treatment. The system had two main sections: the learning module and the assessment module. Through the assessment module, students could directly evaluate their mastery of debridement and identify areas for improvement in knowledge and skills. Following independent learning, students in the experimental group took part in a debridement animal experiment class. Prior to the experiment, students spent 15 min in the classroom learning the theoretical aspects of debridement.

Control group teaching process

The control group implemented the MOOC learning model and studied on the Wuhan University Luojia online platform ( http://www.mooc.whu.edu.cn/entry/ ). During this process, students were required to complete all course studies. After the students in the control group completed independent learning, they participated in the debridement animal experiment class. Before the animal experiment class, the students were arranged to the classroom and learned the theoretical knowledge of debridement for 15 min in the on-site class. The Wuhan University Luojia online platform is shown in Fig.  3 .

Design of the research study

Luojia Online MOOC Platform of Wuhan University. A - B : Schematic diagram of the website providing a “General Surgery Experiment course” on the Luojia Online MOOC platform of Wuhan University; C - F : The specific operating steps of the debridement animal experiment, which include demonstrating the operation of removing the wound edge skin, rinsing the wound, and suturing the wound

Assessment of teaching effectiveness

Teaching effect evaluation encompasses various components, such as theoretical tests on debridement theory, evaluation of debridement animal experiment reports, and post-teaching questionnaires administered to two groups of students. The theoretical test aims to assess students’ grasp of theoretical knowledge pertaining to debridement. Both groups of students undergo a closed-book theoretical knowledge test before and after the teaching intervention. All students are required to complete this test. Evaluation indicators for the debridement animal experiment report include teacher ratings and operation time. In our study, 2 teachers were responsible for assessing students. Students in each grade were assessed by 2 teachers from the same teaching team and specifically trained to ensure consistency and standardization of grading. Following the teaching session, participant satisfaction is gauged through a questionnaire utilizing a Likert scale. Each item is scored on a scale of 1 to 5, where 5 represents ‘strongly agree’, 4 represents ‘agree’, 3 represents ‘not necessarily’, 2 represents ‘disagree’, and 1 represents ‘strongly disagree’. The questionnaire for the experimental group comprises a survey on learning effect satisfaction, evaluation of virtual simulation experiment projects, and assessment of virtual simulation experiment applications. Conversely, the questionnaire for the control group only focuses on learning effect satisfaction, as they did not utilize the virtual simulation experiment platform for learning purposes. The questionnaires were distributed via the Questionnaire Star online platform ( https://www.wjx.cn/ ). The questionnaire was developed for this study and has been added to the supplementary material. The overall Cronbach’s alpha of the questionnaire is 0.981, indicating strong reliability, while the KMO value of 0.941 suggests good validity.

Statistical analysis

The data obtained were entered into IBM SPSS 23.0 software and data were presented as mean ± standard deviation (SD). Independent t-tests were used for continuous variables that fit a normal distribution, such as age, test scores, and Likert scale scores, nonnormally distributed data were analyzed using non-parametric Mann-Whitney U rank sum test, chi-square test was used for categorical variables such as gender. A significance level of P  < 0.05 was used.

Comparison of general information on the two groups of students

The two groups of students were comparable in terms of age and gender distribution (Table  1 ).

Comparison of theory test scores between the two groups of students

The results of the theory test scores showed that there was no statistically significant difference between the scores of the experimental group (70.59 ± 11.51) and the control group (71.64 ± 11.60) in the pre- theoretical test, whereas the scores of the experimental group (96.00 ± 8.03) in the post-t theoretical test at the end of the course were significantly higher than those of the control group (77.87 ± 11.52) as shown in Table  2 .

Two groups of student’s debridement animal experiment course experimental report results

The results showed that the teacher’s score in the lab report of the animal laboratory class on debridement was higher in the experimental group (93.67 ± 1.58) than in the control group (83.83 ± 5.87). In addition, the operation time of debridement was significantly lower in the experimental group (84.63 ± 9.62) than in the control group of students (96.71 ± 17.28) ( P  < 0.05, Table  3 ).

Two groups of students learning effectiveness satisfaction questionnaire results

The questionnaire participation rate was 89.63% (121/135) in the experimental group and 91.80% (112/122) in the control group. The scores of the student satisfaction questionnaire showed that students in the experimental group were more satisfied with the training compared with the control group, and the difference between the two groups was statistically significant (Table  4 ). The item with the lowest score for the experimental group was “Teacher-student and student-student interaction”, while the item with the lowest score for the control group was “Increase course interest”.

Results of questionnaire survey on virtual simulation experiment of experimental group

Regarding the feedback from the questionnaire, in the evaluation of virtual simulation experiment project, the experimental group had high satisfaction ratings of greater than 4.5 for “Virtual simulation experiment scene realistic”“Comprehensive course knowledge”“Smooth operating system”“Accurate evaluation criteria and analysis”(Figure 4 ). In the evaluation of the application of virtual simulation experiments, more than 90 per cent of the experimental group answered “satisfied” and “very satisfied”. Students believe that the virtual simulation experiment platform is better for learning compared to traditional teaching methods and is the trend for future course teaching. (Fig.  5 ).

Results of the questionnaire survey on the evaluation of virtual simulation experiment project of experimental group. In the figure, the horizontal coordinate is the questionnaire question, and the vertical coordinate is the score value of Likert scale

Results of the questionnaire survey on the evaluation of application of virtual simulation experiment in experimental group. The numbers 1, 2, 3, 4, and 5 after the color squares in the figure represent Likert scale scores respectively; The horizontal coordinate in the figure is the corresponding score, the proportion of students

MOOC construction is a successful model in the reform of theoretical teaching, MOOC advances medical education and practice [ 17 , 18 ]. MOOC teaching method exhibit limitations in terms of authenticity, objectivity, timeliness, and the frequency of feedback from instructors [ 19 ]. Moreover, students frequently experience boredom and distraction during these sessions. To address these challenges, the surgical field has increasingly embraced virtual simulation technology [ 20 ]. Virtual simulation systems not only facilitate repeated practice opportunities without posing risks to patients, but they also offer real-time feedback for both educators and learners.

This study has developed a virtual simulation experimental teaching platform specifically for debridement, thereby addressing a gap in the virtual simulation training for this procedure. The objective of this study is to investigate the effectiveness of virtual simulation experimental technology in debridement education and to leverage advanced technology to enhance the teaching process.

Virtual simulation experimental teaching helps improve students’ debridement experimental learning results

This study compared the effects of virtual simulation experimental teaching and MOOC teaching in debridement education. Prior to the debridement experimental class, there were no statistically significant differences in test scores between the two groups. However, after the experimental class, the students in the experimental group scored significantly higher than those in the control group. These results indicate that virtual simulation experimental teaching can enhance students’ understanding of surgical theories and lead to improved exam performance, aligning with previous research findings [ 21 , 22 ]. The assessment of students’ surgical practical skills is conducted through debridement animal experiment classes. Evaluation of students’ practical operation outcomes is based on teachers’ ratings and operation time documented in the experimental report. Findings indicated that students in the experimental group outperformed those in the control group during debridement experimental teaching, as evidenced by shorter operating times and higher scores in the experimental report. A blend of simulation and deliberate practice has been proven to be more effective in skill acquisition compared to the traditional Halsted method, this is consistent with previous studies [ 23 , 24 , 25 ]. The results of this study show that after virtual simulation experimental teaching, the theoretical performance and operational skills of the experimental group are better than those of the control group. Indicating that the teaching design is effective and should continue to be used in the future. Our findings contrast with previous studies [ 26 , 27 , 28 ] by comparing virtual simulation experimental teaching with MOOC teaching, rather than traditional classroom methods. More importantly, we integrated debridement with virtual simulation experiments, demonstrating that this teaching design is effective. Consequently, the use and promotion of virtual simulation experiments in the continuation of debridement teaching is warranted.

Virtual simulation experimental teaching helps improve students’ learning initiative and is recognized by students

The results of the questionnaire survey revealed that the experimental group exhibited higher satisfaction with the learning outcomes compared to the control group, indicating that virtual simulation experimental teaching effectively enhanced students’ learning motivation. Interestingly, the item receiving the lowest score in the experimental group was ‘Teacher-student and student-student interaction’, while the control group’s lowest scoring item was ‘Increase course interest’. This suggests that, although virtual simulation teaching can enhance interaction in offline experimental classes, there is room for improvement. Teachers can utilize the virtual simulation experiment system to monitor students’ learning progress, activities, and results, identify challenges faced by students, reinforce positive feedback [ 29 ]. In addition, students in the control group believed that MOOC teaching was not very effective in increasing their interest in learning, which was also a shortcoming of the traditional teaching model. However, a similar situation did not occur in the experimental group. The above results can be attributed to the virtual simulation provided by repeated training and student interest in designing experiments. The experimental group showed stronger learning motivation, stronger clinical thinking ability, and the ability to combine theory and practice. These findings are consistent with previous studies [ 30 , 31 ]. By engaging in virtual simulation experimental teaching, students can practice simulated experimental procedures in a virtual laboratory, gaining a deeper understanding of the process and key aspects of experimental operations. This method effectively enhances experimental skills and operational proficiency [ 32 , 33 ]. The questionnaire results from the experimental group’s virtual simulation experiment reveal high satisfaction scores above 4.5 points in areas such as ' Virtual simulation experiment scene realistic’, ‘Comprehensive course knowledge’, ' Smooth operating system’, and ' Accurate evaluation criteria and analysis’. Our research indicates that over 90% of the experimental group expressed satisfaction with the use of virtual simulation experiments. They believe that this approach is more conducive to learning compared to traditional teaching methods and view it as a trend in the future development of courses. This finding aligns with results from similar studies [ 34 , 35 ]. Therefore, the promotion and application of virtual simulation experimental teaching in similar courses is highly recommended [ 36 ].

Limitations

However, this study also has some limitations. For instance, the teaching reform was implemented in only one teaching group, indicating the need for multiple repetitions to gather more data and ensure the reliability of the results. Additionally, the main disadvantage of quasi-experimental studies is the lack of random assignment, and the use of randomized controlled trials is considered in the future.

This study utilized virtual simulation technology for debridement experimental teaching, which resulted in enhanced student performance and notable reduction in skill operation time. The virtual simulation experiment was well-received by students, indicating the effectiveness of this teaching framework and its potential for application in similar courses.

Data availability

The datasets used and analyzed during the current study available from the corresponding author on reasonable request.

Lu J, Yang X, Zhao W, Lin J. Effect analysis of a virtual simulation experimental platform in teaching pulpotomy. BMC Med Educ. 2022;22(1):760.

Article   Google Scholar  

Venkatesan M, Mohan H, Ryan JR, Schürch CM, Nolan GP, Frakes DH, Coskun AF. Virtual and augmented reality for biomedical applications. Cell Rep Med. 2021;2(7):100348.

Plotzky C, Lindwedel U, Sorber M, Loessl B, König P, Kunze C, Kugler C, Meng M. Virtual reality simulations in nurse education: a systematic mapping review. Nurse Educ Today. 2021;101:104868.

Najjuma JN, Bajunirwe F, Twine M, Namata T, Kyakwera CK, Cherop M, Santorino D. Stakeholder perceptions about the establishment of medical simulation-based learning at a university in a low resource setting: a qualitative study in Uganda. BMC Med Educ. 2020;20(1):379.

Jabaay MJ, Marotta DA, Aita SL, Walker DB, Grcevich LO, Camba V, Nolin JR, Lyons J, Giannini J. Jr.: Medical Simulation-based learning outcomes in Pre-clinical Medical Education. Cureus. 2020;12(12):e11875.

Google Scholar  

Theodoulou I, Nicolaides M, Athanasiou T, Papalois A, Sideris M. Simulation-based learning strategies to teach undergraduate students Basic Surgical skills: a systematic review. J Surg Educ. 2018;75(5):1374–88.

Xie ZTM, Wang X, et al. Research on the construction of first-class curriculum of experimental first-class course of general surgery and its effectiveness. Zhejiang Med Educ. 2022;21(6):331–5.

JA R-V JR. The MOOC pivot. Sci (New York NY). 2019;363(6423):130–1.

Leoncio PO. Instructional design of massive Open Online courses. Int J Res Publications. 2023;118(1):0–0.

Shorey S, Ng ED. The use of virtual reality simulation among nursing students and registered nurses: a systematic review. Nurse Educ Today. 2021;98:104662.

Li Y, Ye H, Wu S, Zhao X, Liu Y, Lv L, Zhang P, Zhang X, Zhou Y. Mixed reality and haptic-based Dental Simulator for tooth Preparation: Research, Development, and preliminary evaluation. JMIR Serious Games. 2022;10(1):e30653.

Q W, Y YWLL. Virtual Simulation in Undergraduate Medical Education: a scoping review of recent practice. Front Med. 2022;9:855403.

Zhu H, Xu J, Wang P, Liu H, Chen T, Zhao Z, Ji L. The status of virtual simulation experiments in medical education in China: based on the national virtual simulation experiment teaching Center (iLAB-X). Med Educ Online. 2023;28(1):2272387.

Barsom EZ, Graafland M, Schijven MP. Systematic review on the effectiveness of augmented reality applications in medical training. Surg Endosc. 2016;30(10):4174–83.

Rudolphi-Solero T, Jimenez-Zayas A, Lorenzo-Alvarez R, Domínguez-Pinos D, Ruiz-Gomez MJ, Sendra-Portero F. A team-based competition for undergraduate medical students to learn radiology within the virtual world second life. Insights Imaging. 2021;12(1):89.

HS W, XH F, QJ LM, LJ W. Pharmaceutical comprehensive experiments combining simulated reality and virtual reality. Med Educ. 2022;56(11):1131–2.

Cazellet L, Biondini C. [The MOOC, a democratization of knowledge]. Rev Infirm. 2020;69(265):31–2.

Lynette RG, Leonard C. Advances in medical education and practice: role of massive open online courses. Adv Med Educ Pract 2017, 8 (0):603–9.

Nathalie Caire F, Bruno P, Marie-Claude A, Anne C, Françoise C, Bernard B. Are massive Open Online courses (MOOCs) a useful method in medical education? Answer elements with the example of the MOOC-Clinical reasoning process. Pédagogie médicale. 2017;18(2):47–50.

Li W, Han Z, Hongxiang X. Application of virtual simulation in clinical skills and operation courses. Front Med. 2023;10(0):0–0.

Dolan H, Amidon BJ, Gephart SM. Evidentiary and theoretical foundations for virtual simulation in nursing education. J Prof Nurs. 2021;37(5):810–5.

Borg Sapiano A, Sammut R, Trapani J. The effectiveness of virtual simulation in improving student nurses’ knowledge and performance during patient deterioration: A pre and post test design. Nurse Educ Today. 2018;62:128–33.

VF B. Virtual reality simulation in plastic surgery training. Literature review. J Plast Reconstr Aesthetic Surgery: JPRAS. 2021;74(9):2372–8.

C SK. A virtual simulation-based clinical skills course. Clin Teach. 2024;21(4):e13727.

Greensmith M, Cho J, Hargest R. Changes in surgical training opportunities in Britain and South Africa. Int J Surg. 2016;25:76–81.

Meysam Siyah M, Seyyed Mohsen A, Fakhrosadat M, Danial Y, Hedaiat M. A study to investigate the effectiveness of the application of virtual reality technology in dental education. BMC Med Educ. 2022;22(1):0–0.

Lingyun Z, Xiaojian D, Siyu C. Effect of the case-based learning method combined with virtual reality simulation technology on midwifery laboratory courses: a quasi-experimental study. Int J Nurs Sci. 2024;1(1):76–82.

Panpan W, Shiwen W, Li G, Ning Y, Chi Z, Shaoxia P, Caiyan Z. The effect of virtual simulation technology applied to undergraduate teaching of periodontal probing. Eur J Dent Educ. 2023;0(0):0–0.

Xie H, Wang L, Pang Z, Chen S, Xu G, Wang S. Application of problem-based learning combined with a virtual simulation training platform in clinical biochemistry teaching during the COVID-19 pandemic. Front Med (Lausanne). 2022;9(0):0–0.

X YG. Research on the learning experience of virtual simulation class experimental teaching and learning based on the perspective of nursing students. BMC Nurs. 2023;22(1):367.

SEA AH, AA A, H EB AE. Medical students’ perception of virtual Simulation-based learning in Pharmacology. Cureus. 2023;15(1):e33261.

Zhang B, Li S, Gao S, Hou M, Chen H, He L, Li Y, Guo Y, Wang E, Cao R, et al. Virtual versus jaw simulation in oral implant education: a randomized controlled trial. BMC Med Educ. 2020;20(1):272.

Higgins D, Hayes M, Taylor J, Wallace J. A scoping review of simulation-based dental education. MedEdPublish (2016). 2020;9:36.

Soohyun P, Hyeon Gyeong Y. Effect of virtual-reality Simulation of Indwelling catheterization on nursing students’ skills, confidence, and satisfaction. Clin Simul Nurs. 2023;80(0):46–54.

Jian X, Fei Z. Construction Method and Teaching Satisfaction of Virtual Simulation Experiment. 2022 3rd International Conference on Education, Knowledge and Information Management (ICEKIM) 2022, 0 (0):0–0.

CR MH. Deliberate practice in Simulation-Based Surgical skills Training: a scoping review. J Surg Educ. 2021;78(4):1328–39.

Download references

Acknowledgements

Not applicable.

1. First-class Undergraduate Curriculum Construction Project of Hubei Province (2023044). 2. Key Project of Teaching Construction of Wuhan University School of Medicine(2024ZD20). 3. Virtual Simulation Experimental Teaching Innovation Alliance Research Project Establishment(2024059).

Author information

Authors and affiliations.

Department of Orthopedics Trauma and Microsurgery, Zhongnan Hospital of Wuhan University, Wuhan, 430071, Hubei, China

Wang Zhang, Zhe Xie, Changhuan Liu, Zheng Wang, Zonghuan Li, Xiaqing Yang, Xue Fang & Xin Wang

Department of Spine and Bone Oncology, Zhongnan Hospital of Wuhan University, Wuhan, 430071, Hubei, China

Jingfeng Li & Renxiong Wei

Teaching Affair Office, Zhongnan Hospital of Wuhan University, Wuhan, 430071, Hubei, China

Yadian Xie & Renxiong Wei

Department of anesthesiology, West China Second University Hospital, Sichuan University, Chengdu, 610066, Sichuan, China

Department of Surgery, Second Clinical College, Wuhan University, Wuhan, 430071, Hubei, China

Xinghuan Wang & Xin Wang

Elderly Hip Fracture Diagnosis and Treatment Center, Zhongnan Hospital of Wuhan University, Wuhan, 430071, Hubei, China

You can also search for this author in PubMed   Google Scholar

Contributions

Wang Zhang 、Zhe Xie and Jingfeng Li wrote the main manuscript text. Changhuan Liu、Zheng Wang、Yuping Liu、Zonghuan Li、Xiaqing Yang and Xue Fang reviewed the manuscript. Yadian Xie and Xinghuan Wang provided thesis guidance. Xin Wang and Renxiong Wei were responsible for the design of the entire study, planning and implementation. Each author independently reviewed the content of the manuscript. Wang Zhang, Zhe Xie, Jingfeng Li, Renxiong Wei and Xin Wang were the first batch of review authors, and the remaining authors conducted the second batch review to ensure the logic, consistency and completeness of the research. This includes checking the comprehensiveness of the literature review, the accuracy of the data analysis, the reasonableness of the discussion of results, and the standardization of the references.

Corresponding authors

Correspondence to Renxiong Wei or Xin Wang .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Ethics Committee of Zhongnan Hospital of Wuhan University(2022144 K), and informed consent were obtained from all study participants.

Consent for publication

Consent to publish.

The person in charge of the virtual simulation experimental platform involved in this study agreed to release the relevant photos involving human faces.

Conflict of interest

All authors declare no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Wang Zhang, Zhe Xie and Jingfeng Li are co-first authors.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Zhang, W., Xie, Z., Li, J. et al. Investigating the impact of virtual simulation experiment and massive open online course (MOOC) on medical students’ wound debridement training: a quasi-experimental study. BMC Med Educ 24 , 1023 (2024). https://doi.org/10.1186/s12909-024-05991-1

Download citation

Received : 29 July 2024

Accepted : 04 September 2024

Published : 18 September 2024

DOI : https://doi.org/10.1186/s12909-024-05991-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Virtual simulation experiment
• Massive open online course
• Medical education
• Wound debridement training
• Quasi-experimental study

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

A first-order greedy algorithm for A-optimal experimental design with optimality guarantee

Optimal experimental design (OED) concerns itself with identifying ideal methods of data collection, e.g. via sensor placement. The greedy algorithm , that is, placing one sensor at a time, in an iteratively optimal manner, stands as an extremely robust and easily executed algorithm for this purpose. However, it is a priori unclear whether this algorithm leads to sub-optimal regimes. Taking advantage of the author’s recent work on non-smooth convex optimality criteria for OED, we here present a framework for verifying global optimality for the greedy algorithm, as well as employing gradient-based speed-ups.

MSC2020: 62K05, 62F15, 35R30, 65K10.

0.1 Introduction

Optimal experimental design (OED) can be seen as the field of identifying designs w 𝑤 w italic_w allowing for the best reconstruction of unknown parameters x ∈ X 𝑥 𝑋 x\in X italic_x ∈ italic_X in some ambient space X 𝑋 X italic_X , given that x 𝑥 x italic_x can only be measured indirectly by some w 𝑤 w italic_w -dependent forward map F w subscript 𝐹 𝑤 F_{w} italic_F start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , i.e. one only has access the noisy, design-dependent data g w subscript 𝑔 𝑤 g_{w} italic_g start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT given by

A typical example of the effect of the design on the experiment is the situation where F w = M w ⁢ F subscript 𝐹 𝑤 subscript 𝑀 𝑤 𝐹 F_{w}=M_{w}F italic_F start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT = italic_M start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT italic_F , where F : X → ℝ m : 𝐹 → 𝑋 superscript ℝ 𝑚 F:X\to\mathbb{R}^{m} italic_F : italic_X → blackboard_R start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT and m ∈ ℕ 𝑚 ℕ m\in\mathbb{N} italic_m ∈ blackboard_N are fixed – ubiquitously, F 𝐹 F italic_F is a composition of a finite observation operator and a partial differential equation (PDE) solution operator Alexanderian – and M w ∈ ℝ m × m subscript 𝑀 𝑤 superscript ℝ 𝑚 𝑚 M_{w}\in\mathbb{R}^{m\times m} italic_M start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_m × italic_m end_POSTSUPERSCRIPT is the diagonal matrix with w ∈ ℝ m 𝑤 superscript ℝ 𝑚 w\in\mathbb{R}^{m} italic_w ∈ blackboard_R start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT on the diagonal.

If one imposes w ∈ { 0 , 1 } m 𝑤 superscript 0 1 𝑚 w\in\{0,1\}^{m} italic_w ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT , then w 𝑤 w italic_w acts as a mask on the data. This is usually viewed as sensor placement – the quantity m 𝑚 m italic_m can here be thought to represent the number of candidate locations where the experimenter might elect to place a sensor, while each index k ∈ ℕ 𝑘 ℕ k\in\mathbb{N} italic_k ∈ blackboard_N , k ≤ m 𝑘 𝑚 k\leq m italic_k ≤ italic_m corresponds to the experimenter’s choice of either making a measurement of F ⁢ f 𝐹 𝑓 Ff italic_F italic_f in the k 𝑘 k italic_k -th candidate location, in which case w k := 1 assign subscript 𝑤 𝑘 1 w_{k}:=1 italic_w start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT := 1 , or not to make it, in which case w k := 0 assign subscript 𝑤 𝑘 0 w_{k}:=0 italic_w start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT := 0 .

If, due to e.g. budget or power constraints, one only has the possibility to place m 0 < m subscript 𝑚 0 𝑚 m_{0}<m italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT < italic_m sensors, then this formulation naturally leads to the best sensor placement problem in OED; that of identifying the best selection of m 0 subscript 𝑚 0 m_{0} italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT out of m 𝑚 m italic_m candidate locations to use. In order to determine one design w 𝑤 w italic_w as better than another, one generally fixes as objective a design criterion J : ℝ m → ℝ : 𝐽 → superscript ℝ 𝑚 ℝ J:\mathbb{R}^{m}\to\mathbb{R} italic_J : blackboard_R start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT → blackboard_R , mapping designs w 𝑤 w italic_w to a measure J ⁢ ( w ) 𝐽 𝑤 J(w) italic_J ( italic_w ) of the quality of the reconstruction of f 𝑓 f italic_f . A number of such design criteria exist, including but not limited to A-optimality, D-optimality and expected information gain; we refer to Alexanderian ; Pukelsheim for a comprehensive overview; in this article, we will solely concern ourselves with A-optimality, as detailed in the next section.

0.1.1 State of the art

This article is motivated by the author’s recent contribution Aarset in identifying first-order conditions for global optimality of designs, particularly with respect to the A-optimality condition for infinite-dimensional inverse problems stemming from partial differential equations (PDEs). OED for PDEs is a subject in rapid growth, seeing an influx of novel methods and approaches Alexanderian . As such, this article serves to demonstrate how these conditions fit in with existing methods for optimal experimental design; we will here particularly focus on applicability to the greedy algorithm . While various interesting adaptations thereof have been proposed, e.g.  WuChenGhattas , we will focus on the more elementary default version.

0.2 A-optimal designs

We here briefly introduce the A-optimal criterion; for further details, we refer to Ucinski .

In the Bayesian setting, the linear inverse problem ( 1 ) given prior distribution f ∼ prior 𝒩 ⁢ ( 𝐦 prior , 𝒞 prior ) superscript similar-to prior 𝑓 𝒩 subscript 𝐦 prior subscript 𝒞 prior f\stackrel{{\scriptstyle\text{prior}}}{{\sim}}\mathcal{N}(\mathbf{m}_{\text{% prior}},\mathcal{C}_{\text{prior}}) italic_f start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG prior end_ARG end_RELOP caligraphic_N ( bold_m start_POSTSUBSCRIPT prior end_POSTSUBSCRIPT , caligraphic_C start_POSTSUBSCRIPT prior end_POSTSUBSCRIPT ) has explicit, design-dependent posterior distribution

see Stuart . The A-optimal objective J : ℝ m → ℝ : 𝐽 → superscript ℝ 𝑚 ℝ J:\mathbb{R}^{m}\to\mathbb{R} italic_J : blackboard_R start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT → blackboard_R is given as J ⁢ ( w ) := t ⁢ r ( 𝒞 post ⁢ ( w ) ) assign 𝐽 𝑤 𝑡 𝑟 subscript 𝒞 post 𝑤 J(w):=\mathop{tr}\nolimits(\mathcal{C}_{\text{post}}(w)) italic_J ( italic_w ) := start_BIGOP italic_t italic_r end_BIGOP ( caligraphic_C start_POSTSUBSCRIPT post end_POSTSUBSCRIPT ( italic_w ) ) , which by Mercer’s theorem can be seen as proportional to the pointwise variance in the reconstruction, which an A-optimal design w ∗ superscript 𝑤 w^{*} italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT thus minimises.

0.3 Optimality for sensor placement

Given the above, one can cast the problem of finding the A-optimal design w ∗ m 0 subscript superscript 𝑤 subscript 𝑚 0 \prescript{}{m_{0}}{w}^{*} start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT using exactly m 0 subscript 𝑚 0 m_{0} italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT sensors as the constrained optimisation problem of determining

Assume for what remains that the measurement noise is Gaussian white noise, i.e.  Σ = σ ⁢ I Σ 𝜎 𝐼 \Sigma=\sigma I roman_Σ = italic_σ italic_I , σ > 0 𝜎 0 \sigma>0 italic_σ > 0 in ( 1 ). A key contribution of Aarset is then the following characterisation of the global optima:

Theorem 0.3.1

Given m 0 ≤ m subscript 𝑚 0 𝑚 m_{0}\leq m italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ≤ italic_m and w ∈ K 𝑤 𝐾 w\in K italic_w ∈ italic_K , assume (reordering if necessary) that the indices k 𝑘 k italic_k of w 𝑤 w italic_w are ordered so that

If w k = 1 subscript 𝑤 𝑘 1 w_{k}=1 italic_w start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = 1 for all k ≤ m 0 𝑘 subscript 𝑚 0 k\leq m_{0} italic_k ≤ italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and w k = 0 subscript 𝑤 𝑘 0 w_{k}=0 italic_w start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = 0 for all k > m 0 𝑘 subscript 𝑚 0 k>m_{0} italic_k > italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , then w = w ∗ m 0 𝑤 subscript superscript 𝑤 subscript 𝑚 0 w=\prescript{}{m_{0}}{w}^{*} italic_w = start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT .

Theorem 0.3.1 is remarkable, in that it provides a non-smooth, convex optimality criterion that can be verified explicitly via access to the gradient of the objective functional. Note in particular that Aarset also demonstrates that under mild assumptions, ∇ J ⁢ ( w ) k < 0 ∇ 𝐽 subscript 𝑤 𝑘 0 \nabla J(w)_{k}<0 ∇ italic_J ( italic_w ) start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT < 0 for all k 𝑘 k italic_k , i.e. the standard smooth first-order optimality criterion ∇ J ⁢ ( w ) = 𝟎 ∇ 𝐽 𝑤 0 \nabla J(w)=\mathbf{0} ∇ italic_J ( italic_w ) = bold_0 is never satisfied, that is, the above is the only available optimality criterion. Aarset moreover lays out how the gradient ∇ J ⁢ ( w ) ∇ 𝐽 𝑤 \nabla J(w) ∇ italic_J ( italic_w ) can be computed extremely cheaply for the A-optimal objective J 𝐽 J italic_J , requiring no PDE solves or trace estimation; as such, it will form the basis for our continued analysis.

0.4 Greedy algorithms

In its most naive form, the greedy algorithm can be summarised as follows: Given a number m 𝑚 m italic_m of candidate locations, we wish to find the optimal design w ∗ m 0 subscript superscript 𝑤 subscript 𝑚 0 \prescript{}{m_{0}}{w}^{*} start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT using no more than m 0 subscript 𝑚 0 m_{0} italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT sensors for each m 0 < m subscript 𝑚 0 𝑚 m_{0}<m italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT < italic_m iteratively, by first testing the objective value J ⁢ ( w ) 𝐽 𝑤 J(w) italic_J ( italic_w ) for every configuration using exactly one sensor, i.e. for every candidate location. Explicitly, this leads to 1 , where e k ∈ ℝ m superscript 𝑒 𝑘 superscript ℝ 𝑚 e^{k}\in\mathbb{R}^{m} italic_e start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT denotes the k 𝑘 k italic_k -th unit vector, e l k = δ k = l subscript superscript 𝑒 𝑘 𝑙 subscript 𝛿 𝑘 𝑙 e^{k}_{l}=\delta_{k=l} italic_e start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT = italic_δ start_POSTSUBSCRIPT italic_k = italic_l end_POSTSUBSCRIPT . One then approximates each w ∗ m 0 subscript superscript 𝑤 subscript 𝑚 0 \prescript{}{m_{0}}{w}^{*} start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT by the output w m 0 superscript 𝑤 subscript 𝑚 0 w^{m_{0}} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT .

Several natural questions arise in this context. By construction, Algorithm 1 exactly finds each binary optimal design w m 0 ∈ { 0 , 1 } m superscript 𝑤 subscript 𝑚 0 superscript 0 1 𝑚 w^{m_{0}}\in\{0,1\}^{m} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT in a nested manner, i.e. given that all indices present in w m 0 − 1 superscript 𝑤 subscript 𝑚 0 1 w^{m_{0}-1} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT remain fixed . While this dramatically reduces search complexity compared to e.g. a naive full binary search, going from at most ( m m 0 ) binomial 𝑚 subscript 𝑚 0 \binom{m}{m_{0}} ( FRACOP start_ARG italic_m end_ARG start_ARG italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG ) to scaling at worst quadratically in m 0 subscript 𝑚 0 m_{0} italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , it is not a priori clear whether each w m 0 superscript 𝑤 subscript 𝑚 0 w^{m_{0}} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT is globally optimal, i.e. whether w m 0 = w ∗ m 0 superscript 𝑤 subscript 𝑚 0 subscript superscript 𝑤 subscript 𝑚 0 w^{m_{0}}=\prescript{}{m_{0}}{w}^{*} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT = start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ¸ nor is it clear when such optimality may be lost, whether it can later be regained, or if one permanently enters a sub-optimal regime. Indeed, one may ask whether a binary global optimum even exists, that is, whether w ∗ m 0 subscript superscript 𝑤 subscript 𝑚 0 \prescript{}{m_{0}}{w}^{*} start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT does not satisfy w ∗ m 0 ∈ { 0 , 1 } m subscript superscript 𝑤 subscript 𝑚 0 superscript 0 1 𝑚 \prescript{}{m_{0}}{w}^{*}\in\{0,1\}^{m} start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT , in which case the greedy algorithm cannot hope to identify it.

subscript 𝑚 0 1 w^{m_{0}+1} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + 1 end_POSTSUPERSCRIPT by moving in unit length in the direction of steepest descent, then attempting to verify the optimality criteria again. Put together, this leads to the modified greedy algorithm:

0.5 Numerical experiments

0.5.1 experimental setting.

subscript 𝐵 1 0 \partial B_{1}(0) ∂ italic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( 0 ) and wave number ω := 50 assign 𝜔 50 \omega:=50 italic_ω := 50 , see ColtonKress . Discretisation is carried out via the NGSolve package Schoberl . The source f 𝑓 f italic_f on Ω 0 subscript Ω 0 \Omega_{0} roman_Ω start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT is discretised with a first-order FEM space employing n = 2473 𝑛 2473 n=2473 italic_n = 2473 degrees of freedom, while the Helmholtz solution u 𝑢 u italic_u on Ω 1 subscript Ω 1 \Omega_{1} roman_Ω start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT is discretised with a complex second-order FEM space employing n ′ = 15020 superscript 𝑛 ′ 15020 n^{\prime}=15020 italic_n start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = 15020 degrees of freedom; for detailed treatment on FEM discretisation for the Bayesian setting, see BuithanhGhattasMartinStadler .

With m = 110 𝑚 110 m=110 italic_m = 110 , the measurement points ( x k ) k = 1 m ∈ Ω 1 superscript subscript subscript 𝑥 𝑘 𝑘 1 𝑚 subscript Ω 1 (x_{k})_{k=1}^{m}\in\Omega_{1} ( italic_x start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ) start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT ∈ roman_Ω start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT represent the intersection of a uniform rectangular grid with Ω 1 subscript Ω 1 \Omega_{1} roman_Ω start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT . The prior covariance 𝒞 prior subscript 𝒞 prior \mathcal{C}_{\text{prior}} caligraphic_C start_POSTSUBSCRIPT prior end_POSTSUBSCRIPT was densely defined on L 2 ⁢ ( Ω ) superscript 𝐿 2 Ω L^{2}(\Omega) italic_L start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( roman_Ω ) as two-times application of the solution operator mapping the source f 𝑓 f italic_f to the solution u 𝑢 u italic_u of the Laplacian with Robin boundary condition.

The measurement noise Σ = σ ⁢ I Σ 𝜎 𝐼 \Sigma=\sigma I roman_Σ = italic_σ italic_I in ( 1 ) and ( 0.2 ) is chosen so that σ 𝜎 \sigma italic_σ is proportional to 0.5 % percent 0.5 0.5\% 0.5 % of the highest recorded value ‖ F ⁢ ( f s ) ‖ ∞ subscript norm 𝐹 subscript 𝑓 𝑠 \|F(f_{s})\|_{\infty} ∥ italic_F ( italic_f start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) ∥ start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT over 10 4 superscript 10 4 10^{4} 10 start_POSTSUPERSCRIPT 4 end_POSTSUPERSCRIPT samples f s subscript 𝑓 𝑠 f_{s} italic_f start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT drawn from the prior distribution 𝒩 ⁢ ( 0 , 𝒞 prior ) 𝒩 0 subscript 𝒞 prior \mathcal{N}(0,\mathcal{C}_{\text{prior}}) caligraphic_N ( 0 , caligraphic_C start_POSTSUBSCRIPT prior end_POSTSUBSCRIPT ) ; see BuithanhGhattasMartinStadler .

0.5.2 Experimental results

Over 121 seconds on a 12th Gen Intel(R) Core(TM) i5-12500H (4.50 GHz) processor with 16 cores, Algorithm 2 returned a sequence ( w m 0 ) m 0 = 0 m superscript subscript superscript 𝑤 subscript 𝑚 0 subscript 𝑚 0 0 𝑚 (w^{m_{0}})_{m_{0}=0}^{m} ( italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) start_POSTSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT of approximate optimal designs, as well flags computed via Theorem 0.3.1 , indicating whether each approximate optimal design w m 0 superscript 𝑤 subscript 𝑚 0 w^{m_{0}} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT in fact satisfied the global optimality criterion, in which case w m 0 = w ∗ m 0 superscript 𝑤 subscript 𝑚 0 subscript superscript 𝑤 subscript 𝑚 0 w^{m_{0}}=\prescript{}{m_{0}}{w}^{*} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT = start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT is guaranteed, that is, no loss is incurred from taking the greedy approach. This approach revealed that while globally optimal designs could not be found for m 0 ∈ { 5 , 6 , 8 , 12 , 34 , 52 , 55 , 58 , 87 } subscript 𝑚 0 5 6 8 12 34 52 55 58 87 m_{0}\in\{5,6,8,12,34,52,55,58,87\} italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∈ { 5 , 6 , 8 , 12 , 34 , 52 , 55 , 58 , 87 } , one had w m 0 = w ∗ m 0 superscript 𝑤 subscript 𝑚 0 subscript superscript 𝑤 subscript 𝑚 0 w^{m_{0}}=\prescript{}{m_{0}}{w}^{*} italic_w start_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT = start_FLOATSUBSCRIPT italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_FLOATSUBSCRIPT italic_w start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT for all remaining m 0 ≤ m subscript 𝑚 0 𝑚 m_{0}\leq m italic_m start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ≤ italic_m . In particular, this analysis provides the rather surprising conclusion that in the present setting, global optimality was always immediately recovered, meaning that the greedy approach did not lead the sequence to become permanently stuck in a sub-optimal regime. Figures 1 – 2 demonstrate the output designs, and give some insight into what situations may cause global optimality to become lost, such as symmetry.

0.6 Conclusion and outlook

In this article, we have demonstrated the power of the non-smooth convex optimality criteria Theorem 0.3.1 for greedy designs, showcasing how one can efficiently verify or disprove global optimality of the yielded designs at minimal computational cost. Aarset also addresses the case of multi-frequency inversion, as well as the application of the global optimality for other OED design criteria, which is another fascinating perspective that can be combined with greedy algorithms in a future work. More broadly, this article serves as an illustration precisely of the broad applicability of the results in Aarset .

Acknowledgements.

• (1) Aarset, C.: Redundant-dominant optimal experimental design with applications to trace-free A-optimality. ArXiv (2024)
• (2) Alexanderian, A.: Optimal experimental design for infinite-dimensional Bayesian inverse problems governed by PDEs: a review. Inverse Problems 37 (4), p. 043001 (2021)
• (3) Bui-Thanh, T., Ghattas, O., Martin, J., Stadler, G.: A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems Part I: The Linearized Case, with Application to Global Seismic Inversion. SIAM Journal on Scientific Computing 35 (6) (2013)
• (4) Colton, D., Kress, R.: Inverse Acoustic and Electromagnetic Scattering Theory. Springer New York (2013)
• (5) Pukelsheim, F.: Optimal Design of Experiments. SIAM (2006)
• (6) Schöberl, J.: C++ 11 implementation of finite elements in NGSolve. Institute for analysis and scientific computing, Vienna University of Technology 30 (2014)
• (7) Stuart, A.: Inverse Problems: A Bayesian perspective. Acta Numerica, pp. 451-559 (2010)
• (8) Uciński, D.: Optimal measurement methods for distributed parameter system identification. Systems and control series, CRC Press, Washington, D.C. (2005)
• (9) Wu, K., Chen, P., Ghattas, O.: A Fast and Scalable Computational Framework for Large-Scale High-Dimensional Bayesian Optimal Experimental Design. SIAM/ASA Journal on Uncertainty Quantification 11.1, pp. 236-261 (2023)

Effect of Simulation-Supported Prediction Observation Explanation Activities on Students’ Conception of Learning Physics Related to Solid and Liquid Pressure

• Published: 18 September 2024

Cite this article

• Seyhan Eryılmaz Toksoy   ORCID: orcid.org/0000-0002-8643-1017 1 , 2 &
• Emine Bulut 1  

In this research, it was aimed to determine the effect of Simulation-Supported Prediction Observation Explanation (SSPOE) activities related to solid and liquid pressure on the conceptions of learning physics of 10th grade students. In the research, a quasi-experimental design with pretest-posttest control group, which is one of the quantitative research methods, was used. The sample of the research consisted of 50 students studying in the 10th grade in a technical and vocational high school in Afyonkarahisar province. The students in the Experimental 1 (E1) group carried out the SSPOE activities using a computer, and the students in the Experimental 2 (E2) group performed the SSPOE activities using a smart board in the classroom. The students in the Control group continued the current teaching process without using the SSPOE activities. Lessons were conducted by the same teacher and SSPOE activities lasted for 5 weeks. The data were collected through the Conceptions Of Learning Physics (COLP) scale before and after the application. In the analysis of the data, analysis of covariance (ANCOVA test) was performed. At the end of the analyses it can be said that the SSPOE activities are effective on the high level COLP. Learning environments where students can be active individually are more effective in improving students’ conceptions of learning physics in a positive way.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Explore related subjects

• Artificial Intelligence
• Digital Education and Educational Technology

Data Availability

All data and materials can be accessed when necessary.

Afolabi, A. O., & Yusuf, M. O. (2010). Effects of Computer Assısted Instructıon (CAI) on secondary School Students’ Performance ın Bıology. TOJET: The Turkish Online Journal of Educational Technology , 9 (1), 62–69.

Google Scholar  

Alfiyanti, I. F., & Jatmiko, B. (2020). The effectiveness of Predict observe explain (POE) model with PhET to improve critical thinking skills of Senior High School Students. Studies in Learning and Teaching , 1 (2), 76–85. https://doi.org/10.46627/silet.v1i2.34

Article   Google Scholar  

Alkhateeb, M. A., & Milhem, O. A. Q. B. (2020). Student’s concepts of and approaches to Learnıng and the Relatıonshıps between them. Jurnal Cakrawala Pendidikan , 39 (3), 620–632.

Arı, Z. D. (2017). İşbirlikli Öğrenme Yönteminin Lise Öğrencilerinin Akademik Başarılarına, Öz yeterliklerine ve Öğrenme Anlayışlarına Etkisi. Yüksek Lisans Tezi . Karamanoğlu Mehmetbey Üniversitesi Fen Bilimleri Enstitüsü Biyoloji Ana Bilim Dalı.

Astutik, S., & Prahani, B. K. (2018). The practicality and effectiveness of Collaborative Creativity Learning (CCL) model by using PhET Simulation to increase students’ Scientific Creativity. International Journal of Instruction , 11 (4), 409–424.

Atasoy, B. (2004). Fen öğrenimi ve öğretimi (p. s347). Asil Yayın Dağıtım.

Ayvacı, H. Ş., & Bebek, G. (2018). Fizik öğretimi sürecinde yaşanan sorunların değerlendirilmesine yönelik bir çalışma. Kastamonu Eğitim Dergisi , 26 (1), 125–134.

Baek, Y. (2009). Digital simulation in teaching and learning. İçinde D. Gibson. In Y. K. Baek (Ed.), Digital simulations for improving education: Learning through artificial teaching environments (s.25–51).Hershey . İnformation science reference (IGI Global).

Bahçivan, E., & Kapucu, S. (2014). Adaptation of conceptions of Learning Science Questionnaire into Turkish and science teacher candidates’ conceptions of Learning Science. European Journal of Science and Mathematics Education , 2 (2), 106–118.

Bayram, Y. (2019). Simülasyon (Benzetim) Destekli 5E Öğrenme Döngüsü Modelinin 7. Sınıf Öğrencilerinin Elektrik Konusunu Anlamalarına Ve Elektrik Konusuna Yönelik İlgilerine Etkisinin İncelenmesi. Yüksek Lisans Tezi . Bartın Üniversitesi, Eğitim Bilimleri Enstitüsü.

Bilen, K., & Köse, S. (2013). Kavram Öğretiminde Etkili Bir Strateji TGA (Tahmin- Et- Gözle- Açıkla). Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi , 1 (24), 21–42.

Bilgiç, H. G., & Tüzün, H. (2015). Yükseköğretim kurumları web tabanlı uzaktan eğitim programlarında yaşanan sorunlar. Açıköğretim Uygulamaları ve Araştırmaları Dergisi , 1 (3), 26–50.

Bozkurt, E. (2008). Fizik Eğitiminde Hazırlanan Bir Sanal Laboratuvar Uygulamasının Öğrenci Başarısına Etkisi . Doktora Tezi, Selçuk Üniversitesi Fen Bilimleri Enstitüsü.

Cai, S., Liu, C., Wang, T., Liu, E., & Liang, J. C. (2021). Effects of learning physics using augmented reality on students’ self-efficacy and conceptions of learning. British Journal of Educational Technology , 52 , 1235–1251.

Çelik, H., & Karamustafaoğlu, O. (2016). Science prospective teachers’ self-efficacy and views on the Use of Information Technologies in the teaching of physics concepts. Necatibey Faculty of Education Electronic Journal of Science & Mathematics Education , 10 (1), 182–208.

Çepni, S. (Ed.). (2011). Kuramdan Uygulamaya Fen ve teknoloji öğretimi (9. Baskı) . Pegem A Akademi.

Chen, Y. L., Pan, P. R., Sung, Y. T., & Chang, K. E. (2013). Correcting misconceptions on electronics: Effects of a simulation-based learning environment backed by a conceptual change model. Educational Technology & Society , 16 (2), 212–227.

Chiou, G. L., Lee, M. H., & Tsai, C. C. (2013). High school students’ approaches to learning physics with relationship to epistemic views on physics and conceptions of learning physics. Research in Science & Technological Education , 31 (1), 1–15.

Civelek, T. (2008). Bilgisayar Destekli Fizik Deney Simülasyonlarının öğrenme Üzerindeki Etkileri . Yüksek Lisans Tezi, Bahçeşehir Üniversitesi.

Çıbık, A. S., & Yalçın, N. (2012). Analojilerle Desteklenmiş Proje Tabanlı Öğrenme Yönteminin Fen Bilgisi Öğrencilerinin Fizik Dersine Yönelik Tutumlarına Etkisi. Gazi University Journal of Gazi Educational Faculty (GUJGEF) , 32 (1), 185–203.

Cotton, K. (1997). Computer-assisted instruction. North West Regional Educational Laboratory. Retrieved June 12, 2007, from http://www.borg.com/~rparkany/PromOriginal/ETAP778/CAI.html

Creswell, J. W. (2008). Educational Research: Planning, conducting, and evaluating quantitative and qualitative research . Pearson Education Ltd.

Dart, B. C., Burnett, P. C., Purdie, N., Boulton-Lewis, G., Campbell, J., & Smith, D. (2001). Students’ conceptions of learning, the classroom environment, and approaches to learning. The Journal of Educational Research , 93 (4), 262–270.

de Jong, T., & Van Joolingen, W. R. (1998). Scientific discoveiy learning with computer simulations of conceptual domains. Review of Educational Research, 68 , 179–201.

Demirbilek, M. (2016). Tıp Fakültesi Öğretim Üyelerinin Öğretimde Bilgisayar Tabanlı Simülasyon Kullanımı Hakkında Görüşlerinin Araştırılması. Uludağ Üniversitesi Eğitim Fakültesi Dergisi , 29 (1), 2016, 1–23.

Devlin, M. (2006). Challenging accepted wisdom about conceptions of teaching in academic development (Doctoral dissertation, University of Melbourne).

Eklund-Myrskog, G. (1998). Students’ conceptions of learning in different educational contexts. Higher Education , 35 (3), 299–316.

Entwistle, N. J., & Peterson, E. R. (2004). Conceptions of Learning and Knowledgein Higher Education: Relationships with Study Behaviour and influences of Learning environments. International Journal of Educational Research , 41 , 407–428. https://doi.org/10.1016/j.ijer.2005.08.009

Evans, C., & Kozhevnikova, M. (2011). Styles of practice: How learning is affected by students’ and teachers’ perceptions and beliefs, conceptions and approaches to learning. Research Papers in Education , 26 (2), 133–148.

Field, A. P. (2013). Discovering statistics using SPSS: and sex and drugs and rock ‘n’roll (4th Edition). London: Sage.

Furqani, D., Feranie, S., & Winarno, N. (2018). The Effect of Predict-observe-explain (POE) strategy on students’ conceptual mastery and critical thinking in Learning Vibration and Wave. Journal of Science Learning , 2 (1), 1–8.

Gökhale, A. (1996). Effectiveness of computer simulation for enhancing higher order thinking. [Electronic version]. Journal of Industrial Teacher Education , 33 , 36–46.

Güven, E. (2011). Çevre Eğitiminde Tahmin-Gözlem-Açıklama Destekli Proje Tabanlı Öğrenme Yönteminin Farklı Değişkenler Üzerine Etkisi Ve Yönteme İlişkin Öğrenci Görüşleri . Doktora Tezi. Gazi Üniversitesi.

Güvercin, Z. (2010). Fizik Dersinde Simülasyon Destekli Yazılımın Öğrencilerin Akademik Başarısına, Tutumlarına Ve Kalıcılığa Etkisi . Yüksek Lisans Tezi. Çukurova Üniversitesi.

Ho, H. N. J., Liang, J. C., & Tsai, C. C. (2021). The Interrelationship among High School Students’ conceptions of Learning Science, Self-regulated Learning Science, and Science Learning Self-Efficacy. International Journal of Science and Mathematics Education , 1–20.

Kabigting, L. D. C. (2021). Computer Simulation on Teaching and Learning of selected topics in physics. European Journal of Interactive Multimedia and Education , 2 (2), e02108. https://doi.org/10.30935/ejimed/10909

Kağnıcı, A. (2019). STEM etkinlikleriyle zenginleştirilmiş öğrenme modelinin 11.sınıf öğrencilerinin akademik başarısına ve öğrenme anlayışlarına etkisi . Yüksek Lisans Tezi. Karamanoğlu Mehmetbey Üniversitesi.

Kapucu, S. (2017). Lise Öğrencilerinin Fizik Öğrenme Anlayışlarının, Fizik Öğrenme Yaklaşımlarını, Fizik Öğrenme Öz-Yeterliliklerini Ve Fiziğe Yönelik İlgilerini Yordama Gücü. Adıyaman Üniversitesi Sosyal Bilimler Enstitüsü Dergisi , 25 , 133–158.

Kapucu, S., & Bahçivan, E. (2016). Lise öğrencilerinin fizik öğrenme anlayışlarının cinsiyet, sosyo-ekonomik durum ve fizik başarıları açısından incelenmesi. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi , 16 (2), 494–511.

Kearney, M., Treagust, D., Yeo, S., & Zadnik, M. (2001). Student and teacher perceptions of the use of multimedia supported understanding. Research in Science Education , 31 (4), 589–615.

Kırpık, M. A., & Engin, A. O. (2009). Fen bilimlerinin öğretiminde laboratuvarın yeri önemi ve biyoloji öğretimi ile ilgili temel sorunlar. Kafkas Üniversitesi Fen Bilimleri Enstitüsü Dergisi , 2 (2), 61–72.

Küçük, T. (2014). Işık Ünitesinde Simülasyon Yönteminin Kullanılmasının Öğrencilerin Fen Başarısına Ve Fen Tutumlarına Etkisi. Onsekiz Mart Üniversitesi, Eğitim Bilimleri Enstitüsü . Çanakkale.

Lai, P-Y., & Chan, K. W. (2005). A structural model of conceptions of learning, Achievement motivation and learning strategies of Hong Kong teacher education students. Paper presented at the AARE Conference, 28 Nov–2 Dec,University of Western Sydney, Australia.

Lee, M. H., Johanson, R. E., & Tsai, C. C. (2008). Exploring Taiwanese high schoolstudents’ conceptions of and approaches to learning science through a structural equation modeling analysis. Science Education , 92 (2), 191–220.

Li, J. (2003). US and Chinese cultural beliefs about learning. Journal of Educational Psychology , 95 (2), 258.

Li, M., Zheng, C., Liang, J. C., Zhang, Y., & Tsai, C. C. (2018). Conceptions, selfregulation, and strategies of learning science among Chinese high school students. International Journal of Science and Mathematics Education , 16 (1), 69–87.

Liang, J. C., & Tsai, C. C. (2010). Relational Analysis of College Science‐Major Students’ epistemological beliefs toward Science and conceptions of Learning Science. International Journal of Science Education , 32 (17), 2273–2289.

Lin, Y. H., Liang, J. C., & Tsai, C. C. (2012). Effects of different forms of physiology instruction on the development of students’ conceptions of and approaches to science learning. Advances in Physiology Education , 36 (1), 42–47.

Marouchou, D. V. (2011). Can students’ concept of learning influence their learning outcomes? Higher Learning Research Communications , 2 (2), 18–33.

McLean, M. (2001). Can we relate conceptions of learning to student academic achievement? Teaching in Higher Education , 6 (3), 399–413.

Mesci, G., & Uzoglu, M. (2021). Examining of preservice science teachers‟ conceptions of learning science: A Q method study. Journal of Education in Science Environment and Health (JESEH) , 7 (1), 44–55. https://doi.org/10.21891/jeseh.806100

Ministry of National Education [MNE]. (2018). Secondary education physics course (9th, 10th, 11th and 12th grades) curriculum . Council of Education.

Mısır, N. (2009). Elektrostatik ve elektrik akımı ünitelerinde TGA yöntemine dayalı olarak geliştirilen etkinliklerin uygulanması ve etkililiğinin incelenmesi [Yayımlanmamış yüksek Lisans Tezi] . Karadeniz Teknik Üniversitesi.

Nawaz, S., Srivastava, N., Yu, J. H., Khan, A. A., Kennedy, G., Bailey, J., & Baker, R. S. (2021). How difficult is the Task for you? Modelling and Analysis of Students’ Task Difficulty sequences in a Simulation-based POE Environment. International Journal of Artificial Intelligence in Education , 32 (2), 233–262.

Özçelik, M. A. (2019). Lise Öğrencilerinin Bilimsel Epistemolojik İnanç, Fen Öğrenme Anlayışı Ve Genetik Konusundaki Kavramsal Başarıları Arasındaki İlişkilerin İncelenmesi . Yüksek Lisans Tezi, Abant İzzet Baysal Üniversitesi, Eğitim Bilimleri Enstitüsü.

Özgür, H., & Tosun, N. (2012). Öğretmen Adaylarının Derin Ve Yüzeysel Öğrenme Yaklaşımlarının Çeşitli Değişkenler Açısından İncelenmesi. Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi , 12 (4), 113–125.

Pekdağ, B. (2010). Alternative methods in learning chemistry: Learning with animation, simulation, video and multimedia. Journal of Turkish Science Education , 7 (2), 79–110.

Price, A., Wieman, C., & Perkins, K. (2019). Teaching with simulations: Teachers use simulations for student motivation, content learning, and engagement in science practices. The Science Teacher , 86 (7), 46–52. https://www.jstor.org/stable/26899147

Purdie, N., Hattie, J., & Douglas, G. (1996). Student conceptions of learning and their use of self-regulated learning strategies: A cross-cultural comparison. Journal of Educational Psychology , 88 , 87–100.

Ramasundarm, V., Grunwald, S., Mangeot, A., Comerford, N. B., & Bliss, C. M. (2005). Development of an environmental virtual field laboratory. Computers , 45 , 21–34.

Sachs, J., & Chan, C. (2003). Dual scaling analysis of Chinese students’ conceptions of learning. Educational Psychology , 23 (2), 181–193.

Sadi, Ö. (2015). The Analysis of High School Students’ Conceptions of Learning in Different Domains. International Journal of Environmental & Science Education , 2015, 10(6), 813–827.

Sadi, Ö., & Çevik, M. (2016). Investigating of conceptions of learning biology with respect to gender, grade level and school type. In SHS Web of Conferences (Vol. 26, p. 01025). EDP Sciences.

Sadi, O., & Lee, M. H. (2015). The conceptions of learning science for sciencemathematics groups and literature-mathematics groups in Turkey. Research in Science & Technological Education , 33 (2), 182–196. https://doi.org/10.1080/02635143.2014.996543

Sadi, Ö., & Uyar, M. (2014). The Turkish Adaptation of the conceptions of learning science questionnaire: The study of validity and reliability. Journal Of Educatıonal And Instructıonal Studıes In The World , 4 (2). Issn: 2146–7463.

Sağlam, H. (2017). Öğrencilerin Fizik Başarılarının Başarı Yönelimleri, Fizik Öz Yeterlik İnançları ve Fizik Öğrenme Anlayışları Açısından İncelenmesi . Yüksek Lisans Tezi. Boğaziçi Üniversitesi.

Sağlam, H., & Toğrol, A. Y. (2018). High school students’ physics achievement in terms of their achievement goal orientations, self-efficacy beliefs and learning conceptions of physics. Boğaziçi Üniversitesi Eğitim Dergisi , 35 (1), 31–50.

Schunk, D. H. (2009). Yapılandırmacı Teori. Öğrenme Teorileri Eğitimsel Bir Bakış. (Çev. Mahmut Yasin Demir) (pp. 234–277). Nobel Yayın Dağıtım sf.

Stratus, S. H. (2005). New, improved, comprehensive, automated driver’s license test and vision screening system. FHWA-AZ-04-559 . Arizona Department of Transportation.

Suprapto, N., Chang, T. S., & Ku, C. H. (2017). Conception of learning physics and self-efficacy among Indonesian university students. Journal of Baltic Science Education , 16 (1), 7.

Tao, P. K., & Gunstone, R. F. (1999). The process of conceptual change in force and motion during computer-supported physics instruction. Journal of Research in Science Teaching , 36 , 859–882.

Taşkın, N. R. (2012). Ortaöğretim 10.Sınıf Öğrencilerinin Biyoloji Öğrenme Anlayışları Ile Biyoloji Öğrenme Yaklaşımlarının Çeşitli Değişkenler Açısından İncelenmesi. Yüksek Lisans Tezi, Balıkesir Üniversitesi Fen Bilimleri Enstitüsü Ortaöğretim Fen Ve Matematik Alanları Eğitimi . Anabilim Dalı Biyoloji Eğitimi Bilim Dalı.

Tekin, S. (2006). Tahmin-Gözlem-Açıklama Stratejisine Dayalı Fen Bilgisi Laboratuar Deneyleri Tasarlanması ve Bunların Öğrenci Kazanımlarına Katkılarının İrdelenmesi. VII. Fen Bilimleri Ve Matematik Eğitimi Kongresi Bildiriler Kitabı. Gazi Üniversitesi . 07–09 Eylül 2006 Ankara.

Tsai, C. (2004). Conceptions of learning science among high school students in Taiwan: A phenomenographic analysis. International Journal of Science Education , 26 (14), 1733–1750.

Tüfekçioğlu, M. B. (2021). Secondary School Students’ Learnıng Conceptıons and Attıtudes towards Learning Foreign Language . Yeditepe University.

Tyler, R. W. (1949). Basic principles of curriculum and instruction . Universty of Chicago.

Ulukök, Ş., Çelik, H., & Sarı, U. (2013). Basit elektrik devreleriyle ilgili bilgisayar destekli uygulamaların deneysel süreç becerilerinin gelişimine etkisi. Kuramsal Eğitim Bilim Dergisi , 6 (1), 77–101.

Uyanık, G. (2017). Fen Bilimleri Öğretiminde Tahmin-Gözlem-Açıklama Yönteminin Akademik Başarı Ve Kalıcılığa Etkisi. Uluslararası Sosyal Bilimler Eğitimi Dergisi , 3 (1), 1–13.

Van Rossum, E. J., Deijkers, R., & Hamer, R. (1985). Students’ learning conceptions and their interpretation of significant educational concepts. Higher Education , 14 (6), 617–641.

White, R., & Gunstone, R. (1992). Probing understanding (1st ed., p. 196). The Falmer Pres.

Yaman, F. (2012). Bilgisayara Dayalı Tahmin-Gözlem-Açıklama (TGA) Etkinliklerinin Öğrencilerin Asit-Baz Kimyasına Yönelik Kavramsal Anlamalarına Etkisi: Türkiye ve ABD Örneği . Doktora Tezi, Karadeniz Teknik Üniversitesi.

Yaman, F., & Ayas, A. (2015). Assessing changes in high school students’ conceptual understanding through concept maps before and after the computer-based predict–observe–explain (CB-POE) tasks on acid–base chemistry at the secondary level. Chemistry Education Research and Practice , 16 , 843–855.

Yaşar, Ş., & Baran, M. (2020). Oyunlarla desteklenmiş TGA (tahmin et-gözleaçıkla) yöntemine dayalı etkinliklerin 10. sınıf öğrencilerinin fizik başarısına etkisi. Marmara Üniversitesi Atatürk Eğitim Fakültesi Eğitim Bilimleri Dergisi , 52 (52), 420–441.

Yenice, N., Alpak Tunç, G., & Candarlı, F. (2019). Fen Eğitiminde TGA Uygulamasının 6. Sınıf Öğrencilerinin Problem Çözme Becerileri Üzerindeki Etkisinin İncelenmesi. İnönü Üniversitesi Eğitim Bilimleri Enstitüsü Dergisi , 6 (11), 16–27.

Yolaş Kolçak, D. (2010). Lise öğrencilerine fizik konularının öğretilmesinde klasik ve bilgisayar destekli deney metotlarının etkilerinin karşılaştırılması . Yüksek Lisans Tezi, Gazi Üniversitesi, Ankara.

Download references

Not applicable.

Author information

Authors and affiliations.

Department of Computer and Instructional Technologies, Faculty of Education, Recep Tayyip Erdoğan University, Rize, Turkey

Seyhan Eryılmaz Toksoy & Emine Bulut

Department of Mathematics and Science Education, Faculty of Education, Recep Tayyip Erdoğan University, Rize, Turkey

Seyhan Eryılmaz Toksoy

You can also search for this author in PubMed   Google Scholar

Contributions

This research is the product of a master’s thesis completed by Emine Bulut under the supervision of Seyhan Eryılmaz Toksoy. Emine Bulut was more active in the data collection process. Other processes were completed equally.

Corresponding author

Correspondence to Seyhan Eryılmaz Toksoy .

Ethics declarations

Ethical approval.

Before starting the research, ethics committee approval was obtained from Recep Tayyip University social and human sciences ethics committee. The approval was taken with the decision numbered 2021/263 at the meeting held on 20.12.2021.

Informed Consent

People in the research group were informed about the research, and participation in the research was voluntary.

Consent to Participate

It was explained to the participants that the data collected would be used only in scientific research.

Consent to Publish

Competing interests, statement regarding research involving human participants and/or animals.

In this research, data was collected from students.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Eryılmaz Toksoy, S., Bulut, E. Effect of Simulation-Supported Prediction Observation Explanation Activities on Students’ Conception of Learning Physics Related to Solid and Liquid Pressure. J Sci Educ Technol (2024). https://doi.org/10.1007/s10956-024-10158-0

Download citation

Accepted : 12 September 2024

Published : 18 September 2024

DOI : https://doi.org/10.1007/s10956-024-10158-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Conceptions of Learning
• Prediction-Observation-Explanation
• Find a journal
• Publish with us
• Track your research

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Journal Proposal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Fatigue behaviour of high-performance green epoxy biocomposite laminates reinforced by optimized long sisal fibers.

1. Introduction

2. biocomposite materials, 2.1. green epoxy matrix, 2.2. natural reinforcement, 2.3. manufacturing of the biocomposites, 3. experimental test methods, 3.1. static tensile tests, 3.2. fatigue tests, 4. analysis of results and damage evaluation, 4.1. static mechanical properties, 4.2. fatigue life, 5. comparison with other natural and synthetic fiber composites, 6. conclusions.

• The laminates exhibit a good fatigue performance, with fatigue ratios close to 0.5 for unidirectional and angle-ply (±7.5°) laminates and close to 0.4 for cross-ply and quasi-isotropic laminates. Interestingly, the absolute fatigue strength values at 10 6 cycles are equal to about 220 MPa, 150 MPa, 115 MPa and 65 MPa, respectively, for unidirectional, braided-ply, cross-ply and quasi-isotropic lay-up.
• Such fatigue performances are comparable to those of ordinary structural steels and are better than those of different aluminum alloys; consequently, the UD laminates can be used to advantageously replace steel and aluminum in structural applications related to components subject to both static and fatigue loading.
• Unlike described in the literature in relation to synthetic fiber composites, although the braided-ply layup exhibits the best relative fatigue behavior (negligible damage until 85% of the fatigue life), it does not provide the best absolute fatigue strength due to the significant relative reduction in the static strength.
• Interestingly, the fatigue strength of 150 MPa of cross-ply laminate indicates that such lay-up can be widely exploited in the design of structural and semi-structural mechanical components subject to biaxial fatigue loading, as does the fatigue strength of 65 MPa of the quasi-isotropic laminate, which is about 4–5 times the fatigue strength of the matrix alone, indicating that such laminates (or equivalent random discontinuous fiber configurations, the so-called MAT) can be advantageously used to replace plastics in the presence of generic fatigue loading.
• Appropriate models for predicting fatigue behavior at high and low numbers of cycles have been proposed.
• The comparison with both natural fiber and synthetic fiber composites reported in the literature has highlighted that at “low cycles number” fatigue the analyzed biocomposites exhibit better performance than all the comparable composites reported in the literature, also including high-cost and high-environmental impact carbon composites. Such an advantage is preserved up to about 3 10 5 cycles for all other composites, and it is lost for higher fatigue cycles with respect to high-cost and high-stiffness natural fiber biocomposites.
• The fatigue performances of the analyzed biocomposites are always superior to those of the common fiberglass; such a result confirms previous results that have already been reported in the literature by the same authors, which show in detail that the proposed high-performance biocomposites can be advantageously used to replace common fiberglass in order to increase the use of green materials.

Author Contributions

Data availability statement, acknowledgments, conflicts of interest.

• Pickering, K.L.; Aruan Efendy, M.G.; Le, T.M. A review of recent developments in natural fibre composites and their mechanical performance. Compos. Part A Appl. Sci. Manuf. 2016 , 83 , 98–112. [ Google Scholar ] [ CrossRef ]
• Ahmad, F.; Choi, H.S.; Park, M.K. A review: Natural fiber composites selection in view of mechanical, light weight, and economic properties. Macromol. Mater. Eng. 2015 , 300 , 10–24. [ Google Scholar ] [ CrossRef ]
• Koronis, G.; Silva, A.; Fontul, M. Green composites: A review of adequate materials for automotive applications. Compos. Part B Eng. 2013 , 44 , 120–127. [ Google Scholar ] [ CrossRef ]
• Campilho, R. Natural Fiber Composites , 1st ed.; CRC Press: Boca Raton, FL, USA, 2015. [ Google Scholar ] [ CrossRef ]
• Mahboob, Z.; Bougherara, H. Fatigue of flax-epoxy and other plant fibre composites: Critical review and analysis. Compos. Part A Appl. Sci. Manuf. 2018 , 109 , 440–462. [ Google Scholar ] [ CrossRef ]
• Liang, S.; Gning, P.B.; Guillaumat, L. Properties evolution of flax/epoxy composites under fatigue loading. Int. J. Fatigue 2014 , 63 , 36–45. [ Google Scholar ] [ CrossRef ]
• Liang, S.; Gning, P.B.; Guillaumat, L. A comparative study of fatigue behaviour of flax/epoxy and glass/epoxy composites. Compos. Sci. Technol. 2012 , 72 , 535–543. [ Google Scholar ] [ CrossRef ]
• El Sawi, I.; Fawaz, Z.; Zitoune, R.; Bougherara, H. An investigation of the damage mechanisms and fatigue life diagrams of flax fiber-reinforced polymer laminates. J. Mater. Sci. 2014 , 49 , 2338–2346. [ Google Scholar ] [ CrossRef ]
• Bensadoun, F.; Vallons, K.A.M.; Lessard, L.B.; Verpoest, I.; Van Vuure, A.W. Fatigue behaviour assessment of flax-epoxy composites. Compos. Part A Appl. Sci. Manuf. 2016 , 82 , 253–266. [ Google Scholar ] [ CrossRef ]
• Gassan, J. A study of fibre and interface parameters affecting the fatigue behaviour of natural fibre composites. Compos.–Part A Appl. Sci. Manuf. 2002 , 33 , 369–374. [ Google Scholar ] [ CrossRef ]
• De Vasconcellos, D.S.; Touchard, F.; Chocinski-Arnault, L. Tension-tension fatigue behaviour of woven hemp fibre reinforced epoxy composite: A multi-instrumented damage analysis. Int. J. Fatigue 2014 , 59 , 159–169. [ Google Scholar ] [ CrossRef ]
• Yuanjian, T.; Isaac, D.H. Impact and fatigue behaviour of hemp fibre composites. Compos. Sci. Technol. 2007 , 67 , 3300–3307. [ Google Scholar ] [ CrossRef ]
• Shahzad, A.; Isaac, D.H. Fatigue Properties of Hemp and Glass Fiber Composites. Polym. Compos. 2014 , 35 , 1926–1934. [ Google Scholar ] [ CrossRef ]
• Fotouh, A.; Wolodko, J.D.; Lipsett, M.G. Fatigue of natural fiber thermoplastic composites. Compos. Part B Eng. 2014 , 62 , 175–182. [ Google Scholar ] [ CrossRef ]
• Towo, A.N.; Ansell, M.P. Fatigue of sisal fibre reinforced composites: Constant-life diagrams and hysteresis loop capture. Compos. Sci. Technol. 2008 , 68 , 915–924. [ Google Scholar ] [ CrossRef ]
• Shah, D.U. Damage in biocomposites: Stiffness evolution of aligned plant fibre composites during monotonic and cyclic fatigue loading. Compos. Part A Appl. Sci. Manuf. 2016 , 83 , 160–168. [ Google Scholar ] [ CrossRef ]
• Shah, D.U.; Schubel, P.J.; Clifford, M.J.; Licence, P. Fatigue life evaluation of aligned plant fibre composites through S-N curves and constant-life diagrams. Compos. Sci. Technol. 2013 , 74 , 139–149. [ Google Scholar ] [ CrossRef ]
• Katunin, A.; Wachla, D.; Santos, P.; Reis, P.N.B. Fatigue life assessment of hybrid bio-composites based on self-heating temperature. Compos. Struct. 2023 , 304 , 116456. [ Google Scholar ] [ CrossRef ]
• Guo, R.; Xian, G.; Li, C.; Hong, B. Effect of fiber hybrid mode on the tension–tension fatigue performance for the pultruded carbon/glass fiber reinforced polymer composite rod. Eng. Fract. Mech. 2022 , 260 , 108208. [ Google Scholar ] [ CrossRef ]
• Xian, G.; Zhou, P.; Bai, Y.; Wang, J.; Li, C.; Dong, S.; Guo, R.; Li, J.; Du, H.; Zhong, J. Design, preparation and mechanical properties of novel glass fiber reinforced polypropylene bending bars. Constr. Build. Mater. 2024 , 429 , 136455. [ Google Scholar ] [ CrossRef ]
• Wilt, J.; GangaRao, H.; Liang, R.; Mostoller, J. Structural responses of FRP sheet piles under cantilever loading. Sustain. Struct. 2023 , 3 , 8–9. [ Google Scholar ] [ CrossRef ]
• Zuccarello, B.; Militello, C.; Bongiorno, F. Influence of the anisotropy of sisal fibers on the mechanical properties of high performance unidirectional biocomposite lamina and micromechanical models. Compos. Part A Appl. Sci. Manuf. 2021 , 143 , 106320. [ Google Scholar ] [ CrossRef ]
• Zuccarello, B.; Bongiorno, F.; Militello, C. Basalt Fiber Hybridization Effects on High-Performance Sisal-Reinforced Biocomposites. Polymer 2022 , 14 , 1457. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zuccarello, B.; Zingales, M. Toward high performance renewable agave reinforced biocomposites: Optimization of fiber performance and fiber-matrix adhesion analysis. Compos. Part B Eng. 2017 , 122 , 109–120. [ Google Scholar ] [ CrossRef ]
• Bongiorno, F.; Militello, C.; Zuccarello, B. Mode I translaminar fracture toughness of high performance laminated biocomposites reinforced by sisal fibers: Accurate measurement approach and lay-up effects. Compos. Sci. Technol. 2022 , 217 , 109089. [ Google Scholar ] [ CrossRef ]
• Militello, C.; Bongiorno, F.; Epasto, G.; Zuccarello, B. Low-velocity impact behaviour of green epoxy biocomposite laminates reinforced by sisal fibers. Compos. Struct. 2020 , 253 , 112744. [ Google Scholar ] [ CrossRef ]
• Zuccarello, B.; Marannano, G.; Mancino, A. Optimal manufacturing and mechanical characterization of high performance biocomposites reinforced by sisal fibers. Compos. Struct. 2018 , 194 , 575–583. [ Google Scholar ] [ CrossRef ]
• Zuccarello, B.; Militello, C.; Bongiorno, F. Environmental aging effects on high-performance biocomposites reinforced by sisal fibers. Polym. Degrad. Stab. 2023 , 211 , 110319. [ Google Scholar ] [ CrossRef ]
• Zuccarello, B.; Bartoli, M.; Bongiorno, F.; Militello, C.; Tagliaferro, A.; Pantano, A. New concept in bioderived composites: Biochar as toughening agent for improving performances and durability of agave-based epoxy biocomposites. Polymers 2021 , 13 , 198. [ Google Scholar ] [ CrossRef ]
• Pantano, A.; Militello, C.; Bongiorno, F.; Zuccarello, B. Analysis of the parameters affecting the stiffness of short sisal fiber biocomposites manufactured by compression-molding. Polymers 2022 , 14 , 154. [ Google Scholar ] [ CrossRef ]
• Pantano, A.; Bongiorno, F.; Marannano, G.; Zuccarello, B. Enhancement of Static and Fatigue Strength of Short Sisal Fiber Biocomposites by Low Fraction Nanotubes. Appl. Compos. Mater. 2021 , 28 , 91–112. [ Google Scholar ] [ CrossRef ]
• ASTM D 3822 ; Standard Test Method for Tensile Properties of Single Textile Fibers. ASTM International: West Conshohocken, PA, USA, 2001.
• Agarwal, B.D.; Broutman, L.J.; Chandrashekhara, K. Analysis and Performance of Fiber Composites ; John Wiley & Sons: New York, NY, USA; New Delhi, India, 1998. [ Google Scholar ] [ CrossRef ]
• Barbero, E.J. Introduction to Composite Materials Design ; Taylo & Francis Group: New York, NY, USA, 1999. [ Google Scholar ]
• ASTM D 3039 ; Standard Test Method for Tensile Properties of Polymer Matrix Composite Materials. ASTM International: West Conshohocken, PA, USA, 2014.
• Plumtree, A.; Melo, M.; Dahl, J. Damage evolution in a [±45]2S CFRP laminate under block loading conditions. Int. J. Fatigue 2010 , 32 , 139–145. [ Google Scholar ] [ CrossRef ]
• Ye, L. On fatigue damage accumulation and material degradation in composite materials. Compos. Sci. Technol. 1989 , 36 , 339–350. [ Google Scholar ] [ CrossRef ]
• Kim, H.S.; Huang, S. S-n curve characterisation for composite materials and prediction of remaining fatigue life using damage function. J. Compos. Sci. 2021 , 5 , 76. [ Google Scholar ] [ CrossRef ]
• D’Amore, A.; Caprino, G.; Stupak, P.; Zhou, J.; Nicolais, L. Effect of stress ratio on the flexural fatigue behaviour of continuous strand mat reinforced plastics. Sci. Eng. Compos. Mater. 1996 , 5 , 1–8. [ Google Scholar ] [ CrossRef ]
• Mandell, J.F.; Reed, R.M.; Samborsky, D.D. Fatigue of Fiberglass Wind Turbine Blade Materials. American Society of Mechanical Engineers, Solar Energy Division. 1992. Available online: https://www.osti.gov/biblio/10105798 (accessed on 1 August 1992).
• Kawai, M.; Yajima, S.; Hachinohr, E.A.; Takano, Y. Off-Axis Fatigue Behavior of Unidirectional Carbon Fiber Reinforced Composites at Room and High Temperatures. J. Compos. Mater. 2001 , 35 , 545–576. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Zuccarello, B.; Militello, C.; Bongiorno, F. Fatigue Behaviour of High-Performance Green Epoxy Biocomposite Laminates Reinforced by Optimized Long Sisal Fibers. Polymers 2024 , 16 , 2630. https://doi.org/10.3390/polym16182630

Zuccarello B, Militello C, Bongiorno F. Fatigue Behaviour of High-Performance Green Epoxy Biocomposite Laminates Reinforced by Optimized Long Sisal Fibers. Polymers . 2024; 16(18):2630. https://doi.org/10.3390/polym16182630

Zuccarello, B., C. Militello, and F. Bongiorno. 2024. "Fatigue Behaviour of High-Performance Green Epoxy Biocomposite Laminates Reinforced by Optimized Long Sisal Fibers" Polymers 16, no. 18: 2630. https://doi.org/10.3390/polym16182630

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals