the scientific method example experiments

• Most Recent
• Free Silly Handwriting
• Easy Sub Plans Template
• Sprinkle Topped Shop
• My TpT Shop
• Amazon Favorites
• Free Video Series

The Sprinkle Topped Teacher

7 Easy Scientific Method Experiments

Kids’ natural curiosity never fails to amaze me. Their imaginations and observation skills run wild, especially at the elementary level. And the classroom is the perfect place to explore and exercise their curious minds!

When it comes to introducing younger students to scientific concepts like drawing observations and conclusions, the scientific method is a great place to start. It doesn’t have to be anything crazy. I’ve seen some pretty intense resources that teach the scientific method for kids, and they’ve been anything but kid friendly!

My preferred way to teach science is to boil the scientific method down to these 5 steps:

• Asking a research question
• Making a hypothesis
• Doing the experiment
• Taking observations
• Writing a conclusion

Keeping the scientific method for kids simple lets them explore their world without confusing them too much. When it comes to science concepts, we want to ease younger students in — not overwhelm them. This helps kids build a love of science that will last their whole lives!

With all that being said, I’ve gathered my favorite easy scientific method experiments for younger students into one bundle for you! These 7 Easy Science Experiments to Teach the Scientific Method are amazing because they all follow the same framework. This helps students know what to expect when it’s time to experiment and keeps your curriculum cohesive. Once we do one or two, my class gets into a nice groove and doesn’t need much direction on my part.

Plus, each of these experiments are available in a digital format, so they’re perfect for in-person or distance learning! And since they are so easy for students to follow, students will have no problem completing them at home.

What are the 7 easy scientific method experiments?

I’m glad you asked! Here is everything that is included in the scientific method for kids bundle:

1. Rainbow Milk Experiment

In the Rainbow Milk Magic Experiment, students will combine milk, dish soap, and food coloring to learn all about why the colors begin to swirl and look as if they are exploding into a rainbow. This is such a simple science experiment that works great with students of any age!

2. Tornado in a Bottle Experiment

This Tornado in a Bottle Experiment is the perfect way to teach the scientific method to kids. Students will practice measuring to fill a water bottle, then add dish soap and of course some glitter! They will then create a vortex to simulate a tornado and learn all about tornadoes.

3. Fingerprint Science Experiment

In the Fingerprint Science Experiment, students will become detectives and investigate their fingerprints while learning about the scientific method! This STEM fingerprint science experiment will cover the three types of fingerprints and super fun facts about fingerprints in humans and animals.

4. Marshmallow Toothpick Tower Science Experiment

The Marshmallow Toothpick Tower Science Experiment teaches students about building structures. They get to build their own masterpieces with marshmallows and toothpicks. As a bonus, this one ends in a tasty snack that students can enjoy!

5. Coffee Filter Digital Science Experiment

Students will learn about pigment and chromatography through this engaging experiment. They will get to draw a picture on a coffee filter using markers and observe what happens when it is sprayed with water. This is a fantastic way to introduce students to the concept of chemistry!

6. Slime Experiment

What kid doesn’t love slime?! This fun experiment lets them make their own with just a few household supplies. I love using this one during Halloween — it’s got the perfect spooky vibe!

7. Clean a Dirty Penny Science Experiment

Students love to collect and bring in a dirty penny for this science experiment. Students discover which cleaning solution works best to clean it and why using the scientific method! All you need are pennies, water, dish soap, salt, and vinegar. It’s a great option for Presidents’ Day, too!

What’s included in each scientific method for kids experiment?

I recently edited this bundle of experiments to include a table of contents, digital versions on Google Slides, and some great teacher tips to help your experiments run smoothly and make life easier for you. Each experiment includes…

●       Explanation of the experiment, great for parents to follow at home!

●       Guiding Question and Hypothesis

●       Experiment (Picture and written)

●       Observations (Picture and written)

●       Conclusion

●       The science behind the experiment explained (includes fill in the blank option as well)

There you have it: everything you need to teach the scientific method to your students or a child at home!

Teaching the scientific method to kids doesn’t have to be complicated. It’s best to stick to 5 steps and use the same experimental format to keep science lessons cohesive. My 7 Easy Science Experiments to Teach the Scientific Method are an amazing option for anyone looking to introduce students to key STEM concepts!

How often do you experiment in your classroom? What’s your favorite experiment to do? I’d love to hear your thoughts!

Share this:

You may also like, fall activities for 2nd grade, free 2nd grade math warm-up, how to teach students to restate the question, 2nd grade fractions activity, number search puzzle for fun multiplication and division practice.

15 Scientific Method Examples

Viktoriya Sus (MA)

Viktoriya Sus is an academic writer specializing mainly in economics and business from Ukraine. She holds a Master’s degree in International Business from Lviv National University and has more than 6 years of experience writing for different clients. Viktoriya is passionate about researching the latest trends in economics and business. However, she also loves to explore different topics such as psychology, philosophy, and more.

Learn about our Editorial Process

Chris Drew (PhD)

This article was peer-reviewed and edited by Chris Drew (PhD). The review process on Helpful Professor involves having a PhD level expert fact check, edit, and contribute to articles. Reviewers ensure all content reflects expert academic consensus and is backed up with reference to academic studies. Dr. Drew has published over 20 academic articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education and holds a PhD in Education from ACU.

The scientific method is a structured and systematic approach to investigating natural phenomena using empirical evidence . 

The scientific method has been a lynchpin for rapid improvements in human development. It has been an invaluable procedure for testing and improving upon human ingenuity. It’s led to amazing scientific, technological, and medical breakthroughs.

Some common steps in a scientific approach would include:

• Observation
• Question formulation
• Hypothesis development
• Experimentation and collecting data
• Analyzing results
• Drawing conclusions

Definition of Scientific Method

The scientific method is a structured and systematic approach to investigating natural phenomena or events through empirical evidence. 

Empirical evidence can be gathered from experimentation, observation, analysis, and interpretation of data that allows one to create generalizations about probable reasons behind those happenings. 

As mentioned in the article published in the journal  Nature,

“ As schoolchildren, we are taught that the scientific method involves a question and suggested explanation (hypothesis) based on observation, followed by the careful design and execution of controlled experiments, and finally validation, refinement or rejection of this hypothesis” (p. 237).

The use of scientific methods permits replication and validation of other people’s scientific analyses, leading toward improvement upon previous results, and solid empirical conclusions. 

Voit (2019) adds that:

“…it not only prescribes the order and types of activities that give a scientific study validity and a stamp of approval but also has substantially shaped how we collectively think about the endeavor of investigating nature” (p. 1).

This method aims to minimize subjective biases while maximizing objectivity helping researchers gather factual data. 

It follows set procedures and guidelines for testing hypotheses using controlled conditions, assuring optimum accuracy and relevance in concluding by assessing a range of aspects (Blystone & Blodgett, 2006).

Overall, the scientific method provides researchers with a structured way of inquiry that seeks insightful explanations regarding evidence-based investigation grounded in facts acquired from an array of fields.

15 Examples of Scientific Method

• Medicine Delivery : Scientists use scientific method to determine the most effective way of delivering a medicine to its target location in the body. They perform experiments and gather data on the different methods of medicine delivery, monitoring factors such as dosage and time release.
• Agricultural Research : Scientific method is frequently used in agricultural research to determine the most effective way to grow crops or raise livestock. This may involve testing different fertilizers, irrigation methods, or animal feed, measuring yield, and analyzing data.
• Food Science and Nutrition : Nutritionists and food scientists use the scientific method to study the effects of different food types and diet on health. They design experiments to understand the impact of dietary changes on weight, disease risk, and overall health outcomes.
• Environmental Studies : Researchers use scientific method to study natural ecosystems and how human activities impact them. They collect data on things like biodiversity, water quality, and pollution levels, analyzing changes over time.
• Psychological Studies : Psychologists use the scientific method to understand human behavior and cognition. They conduct experiments under controlled conditions to test theories about learning, memory, social interaction, and more.
• Climate Change Research : Climate scientists use the scientific method to study the Earth’s changing climate. They collect and analyze data on temperature, CO2 levels, and ice coverage to understand trends and make predictions about future changes.
• Geology Exploration : Geologists use scientific method to analyze rock samples from deep in the earth’s crust and gather information about geological processes over millions of years. They evaluate data by studying patterns left behind by these processes.
• Space Exploration : Scientists use scientific methods in designing space missions so that they can explore other planets or learn more about our solar system. They employ experiments like landing craft exploration missions as well as remote sensing techniques that allow them to examine far-off planets without having physically land on their surfaces.
• Archaeology : Archaeologists use the scientific method to understand past human cultures. They formulate hypotheses about a site or artifact, conduct excavations or analyses, and then interpret the data to test their hypotheses.
• Clinical Trials : Medical researchers use scientific method to test new treatments and therapies for various diseases. They design controlled studies that track patients’ outcomes while varying variables like dosage or treatment frequency.
• Industrial Research & Development : Many companies use scientific methods in their R&D departments. For example, automakers may assess the effectiveness of anti-lock brakes before releasing them into the marketplace through tests with dummy targets.
• Material Science Experiments : Engineers have extensively used scientific method experimentation efforts when designing new materials and testing which options could be flexible enough for certain applications. These experiments might include casting molten material into molds and then subjecting it to high heat to expose vulnerabilities
• Chemical Engineering Investigations : Chemical engineers also abide by scientific method principles to create new chemical compounds & technologies designed to be valuable in the industry. They may experiment with different substances, changing materials’ concentration and heating conditions to ensure the final end-product safety and reliability of the material.
• Biotechnology : Biotechnologists use the scientific method to develop new products or processes. For instance, they may experiment with genetic modification techniques to enhance crop resistance to pests or disease.
• Physics Research : Scientists use scientific method in their work to study fundamental principles of the universe. They seek answers for how atoms and molecules are breaking down and related events that unfold naturally by running many simulations using computer models or designing sophisticated experiments to test hypotheses.

Origins of the Scientific Method

The scientific method can be traced back to ancient times when philosophers like Aristotle used observation and logic to understand the natural world. 

These early philosophers were focused on understanding the world around them and sought explanations for natural phenomena through direct observation (Betz, 2010).

In the Middle Ages, Muslim scholars played a key role in developing scientific inquiry by emphasizing empirical observations. 

Alhazen (a.k.a Ibn al-Haytham), for example, introduced experimental methods that helped establish optics as a modern science. He emphasized investigation through experimentation with controlled conditions (De Brouwer, 2021).

During the Scientific Revolution of the 17th century in Europe, scientists such as Francis Bacon and René Descartes began to develop what we now know as the scientific method observation (Betz, 2010).

Bacon argued that knowledge must be based on empirical evidence obtained through observation and experimentation rather than relying solely upon tradition or authority. 

Descartes emphasized mathematical methods as tools in experimentation and rigorous thinking processes (Fukuyama, 2021).

These ideas later developed into systematic research designs , including hypothesis testing, controlled experiments, and statistical analysis – all of which are still fundamental aspects of modern-day scientific research.

Since then, technological advancements have allowed for more sophisticated instruments and measurements, yielding far more precise data sets scientists use today in fields ranging from Medicine & Chemistry to Astrophysics or Genetics.

So, while early Greek philosophers laid much groundwork toward an observational-based approach to explaining nature, Islam scholars furthered our understanding of logical reasoning techniques and gave rise to a more formalized methodology.

Steps in the Scientific Method

While there may be variations in the specific steps scientists follow, the general process has six key steps (Blystone & Blodgett, 2006).

Here is a brief overview of each of these steps:

1. Observation

The first step in the scientific method is to identify and observe a phenomenon that requires explanation. 

This can involve asking open-ended questions, making detailed observations using our senses or tools, or exploring natural patterns, which are sources to develop hypotheses. 

2. Formulation of a Hypothesis

A hypothesis is an educated guess or proposed explanation for the observed phenomenon based on previous observations & experiences or working assumptions derived from a valid literature review . 

The hypothesis should be testable and falsifiable through experimentation and subsequent analysis.

3. Testing of the Hypothesis

In this step, scientists perform experiments to test their hypothesis while ensuring that all variables are controlled besides the one being observed.

The data collected in these experiments must be measurable, repeatable, and consistent.

4. Data Analysis

Researchers carefully scrutinize data gathered from experiments – typically using inferential statistics techniques to analyze whether results support their hypotheses or not.

This helps them gain important insights into what previously unknown mechanisms might exist based on statistical evidence gained about their system.

See: 15 Examples of Data Analysis

5. Drawing Conclusions 

Based on their data analyses, scientists reach conclusions about whether their original hypotheses were supported by evidence obtained from testing.

If there is insufficient supporting evidence for their ideas – trying again with modified iterations of the initial idea sometimes happens.

6. Communicating Results

Once results have been analyzed and interpreted under accepted principles within the scientific community, scientists publish findings in respected peer-reviewed journals.

These publications help knowledge-driven communities establish trends within respective fields while indirectly subjecting papers reviews requests boosting research quality across the scientific discipline.

Importance of the Scientific Method

The scientific method is important because it helps us to collect reliable data and develop testable hypotheses that can be used to explain natural phenomena (Haig, 2018).

Here are some reasons why the scientific method is so essential:

• Objectivity : The scientific method requires researchers to conduct unbiased experiments and analyses, which leads to more impartial conclusions. In this way, replication of findings by peers also ensures results can be relied upon as founded on sound principles allowing others confidence in building further knowledge on top of existing research.
• Precision & Predictive Power : Scientific methods usually include techniques for obtaining highly precise measurements, ensuring that data collected is more meaningful with fewer uncertainties caused by limited measuring errors leading to statistically significant results having firm logical foundations. If predictions develop scientifically tested generalized defined conditions factored into the analysis, it helps in delivering realistic expectations
• Validation : By following established scientific principles defined within the community – independent scholars can replicate observation data without being influenced by subjective biases or prejudices. It assures general acceptance among scientific communities who follow similar protocols when researching within respective fields.
• Application & Innovation : Scientific concept advancements that occur based on correct hypothesis testing commonly lead scientists toward new discoveries, identifying potential breakthroughs in research. They pave the way for technological innovations often seen as game changers, like mapping human genome DNA onto creating novel therapies against genetic diseases or unlocking secrets of today’s universe through discoveries at LHC.
• Impactful Decision-Making : Policymakers can draw from these scientific findings investing resources into informed decisions leading us toward a sustainable future. For example, research gathered about carbon pollution’s impact on climate change informs debate making policy action decisions about our planet’s environment, providing valuable knowledge-useful information benefiting societies (Haig, 2018).

The scientific method is an essential tool that has revolutionized our understanding of the natural world.

By emphasizing rigorous experimentation, objective measurement, and logical analysis- scientists can obtain more unbiased evidence with empirical validity . 

Utilizing this methodology has led to groundbreaking discoveries & knowledge expansion that have shaped our modern world from medicine to technology. 

The scientific method plays a crucial role in advancing research and our overall societal consensus on reliable information by providing reliable results, ensuring we can make more informed decisions toward a sustainable future. 

As scientific advancements continue rapidly, ensuring we’re applying core principles of this process enables objectives to progress, paving new ways for interdisciplinary research across all fields, thereby fuelling ever-driving human curiosity.

Betz, F. (2010). Origin of scientific method.  Managing Science , 21–41. https://doi.org/10.1007/978-1-4419-7488-4_2

Blystone, R. V., & Blodgett, K. (2006). WWW: The scientific method.  CBE—Life Sciences Education ,  5 (1), 7–11. https://doi.org/10.1187/cbe.05-12-0134

De Brouwer , P. J. S. (2021).  The big r-book: From data science to learning machines and big data . John Wiley & Sons, Inc.

Defining the scientific method. (2009).  Nature Methods ,  6 (4), 237–237. https://doi.org/10.1038/nmeth0409-237

Fukuyama, F. (2012).  The end of history and the last man . New York: Penguin.

Haig, B. D. (2018). The importance of scientific method for psychological science.  Psychology, Crime & Law ,  25 (6), 527–541. https://doi.org/10.1080/1068316x.2018.1557181

Voit, E. O. (2019). Perspective: Dimensions of the scientific method.  PLOS Computational Biology ,  15 (9), e1007279. https://doi.org/10.1371/journal.pcbi.1007279

• Viktoriya Sus (MA) #molongui-disabled-link Cognitive Dissonance Theory: Examples and Definition
• Viktoriya Sus (MA) #molongui-disabled-link 15 Free Enterprise Examples
• Viktoriya Sus (MA) #molongui-disabled-link 21 Sunk Costs Examples (The Fallacy Explained)
• Viktoriya Sus (MA) #molongui-disabled-link Price Floor: 15 Examples & Definition

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 23 Achieved Status Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 15 Ableism Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 25 Defense Mechanisms Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 15 Theory of Planned Behavior Examples

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

Scientific Method Experiments for Kids: Easy & Fun Setup

Krystal DeVille

December 25, 2022

The Scientific Method is a method and process used to conduct research. It is an easy way to provide a road map that kids can use to quench their hunger for curiosity.

They are constantly trying to figure out how the world works around them, and by providing them with the Scientific Method as an outline, they can navigate their way to success.

It can help students develop traditional cognitive skills, including logic, rationalization, and problem-solving through the application of the Scientific Method.

Table of Contents

What is the Scientific Method

It may seem boring to experiment with the Scientific Method, but it provides kids with the tools needed to solve any problem or question!

It doesn’t necessarily mean filling out a worksheet as done in school or, in many cases, even writing anything down. The Scientific Method for kids is a proven method and process of research. It works as a step-by-step guideline to teach people, including kids, how to experiment properly and come to an evidence-based solution.

It is how people study and learn things! If you think your child can study and learn things, then they can most certainly use the scientific method.

Steps of the Scientific Method

• Ask a Question (Channel your inner curiosity and pose a question)
• Make a Hypothesis (Create a prediction: What do they think will happen?)
• Research (Gather information and observe what it is you are really studying)
• Experiment (This step may need to be repeated, A LOT. Keep on trying!)
• Make Observations (Yup, more observations. Examine what is happening.)
• Draw Conclusions (What happened compared to what you thought would happen? What would happen if you change a variable? This could be the part where you revisit step 4 and try all over again)
• Share and Discuss Results (Share them with everyone! The more you collaborate, the better your study will be)

Scientific Method Experiments for Preschoolers

For preschoolers, we will use a more toned-down version of the scientific method to not inundate them with a long, scary list.

• Ask a Question
• Ask them what they think will happen ( Hypothesis )
• Test it out ( Experiment )
• What actually happened? ( Conclusions )

Scientific Experiments:

The Fastest Fizz

• Sink or Swim

Blow the Biggest Bubble

Materials Needed:

• Two identical glasses or jars (Any type of container for the water)
• Ice cubes to fill the glass up to halfway
• Water (Cold and hot tap water will do)
• Two Alka-Seltzer Tablets
• Thermometer (Optional: To measure the temperature of the water before dropping the Alka-Seltzer)

Scientific Method Steps :

• Ask a Question: How does the temperature of water affect the fizzing of Alka-Seltzer?
• Hypothesis : Will the Alka-Seltzer fizz faster in warm or cold water?
• Experiment:

Step 1: Fill one glass halfway with ice cubes

Step 2: Add cold water to that glass up to an inch from the rim

Step 3: Stir the ice cubes with the spoon to even out the temperature

Step 4: Remove the ice cubes with the spoon right before you will drop the Alka-Seltzer in the glasses

Step 5: Add hot tap water up to an inch from the rim in the second glass

Step 6: Drop the Alka-Seltzer into the glass with warm water and time how long it takes for the tablet to disappear

Step 7: Drop the Alka-Seltzer into the glass with cold water and time how long it takes for the tablet to disappear

• Conclusion: Which glass made the tablet disappear faster? Why might that be? What else dissolves in water?

If you like to find more activities you can do at home, please check out our article, Baking Soda and Vinegar Experiments .

Sink or Swim?

• Big bowl (Any large container that holds water)
• Recommended items: Twig

Scientific Method Steps:

• Ask a Question: Which types of materials will float or sink in the water?
• Hypothesis : Which items do they think will float or sink?

Step 1: Have them explore the backyard or a playground for items

Step 2: Encourage them to retrieve a variety of items including a leaf, a pebble, and/or twigs

Step 3: Place each item separately in the container of water

Step 4: Examine what happens!

• Conclusion: Which items floated and which didn’t? Why might that be!?

• At least 3 different varieties
• I suggest Bubblicious, Bubbaloo, Bazooka, or even a balloon.
• Ask a Question: What type of gum blows the biggest bubbles?
• Hypothesis : Which type of gum do they think will be bigger?

Step 1: Start with one brand of bubblegum

Step 2: Each person chews the gum for 5 minutes

Step 3: Begin blowing bubbles with your piece of gum

Step 4: Blow 5 bubbles with that brand of gum

Step 5: Measure each bubble with the ruler and write down the results

Step 6: Repeat steps 1-5 using a different brand of gum

• Conclusion: Which gum blew the biggest bubble?

Kids love Microscopes as they can explore the unseen world around them. Please take a look at our article for more information, Microscope Activities for Kids – Fun Experiments Kids Will Love!

Scientific Method Experiments for Elementary-Aged Kids

Elephant toothpaste.

• Empty Plastic Bottle
• A wide washable surface area (A large tub or tray would work well – preferable if you are in the kitchen, bathroom, or outdoors; it will get messy)
• 3% Hydrogen Peroxide (It could be a higher percentage for a bigger reaction)
• Measuring Spoons (At least 2)
• Measuring Cups (At least 2)
• Safety goggles
• Liquid food coloring (Optional)
• Question: What happens when you mix yeast with hydrogen peroxide? (This may need some background information) After the first experiment: engage their curiosity by leading them to questions like: How would the experiment change if you add more yeast?
• Hypothesis : What do they think will happen? What might happen if you add more or less yeast?
• Research: Look into the chemical breakdown of what Hydrogen Peroxide is (H₂O₂) and how that can be broken down. If mixed with a catalyst, such as yeast, then the breakdown reaction can occur very quickly!
• Experiment :

Step 1: Put on your safety goggles (It may be called elephant toothpaste but don’t put it in your mouth!)

Step 2: Measure ½ cup of hydrogen peroxide and carefully pour it into the bottle.

Step 3: Add food coloring at this point if you opt to use that.

Step 4: In a separate measuring cup, mix one tablespoon of yeast and three tablespoons of warm water.

Step 5: Mix the yeast and water for about 30 seconds.

Step 6: Pour the yeast mixture into the bottle of peroxide and watch what happens.

• Observations: What happened? How fast did it happen? How much of a reaction did you get?
• Conclusions : What happens when you add yeast to hydrogen peroxide? What would happen if you added more or less yeast?
• Share and Discuss Results! Feel free to try again using different variables!

Rubber Egg Experiment

• Household Vinegar
• Glass/Jar/Container
• Food coloring (Optional)
• Plate or tray
• Question: Can you bounce an egg?
• Hypothesis: What do they think will happen? Will an egg really be able to bounce? How high will it be able to bounce?
• Research: An eggshell is made of calcium carbonate and vinegar is an acid. When the two are combined, a chemical reaction occurs that once again mixes to form a carbon dioxide gas. The vinegar can cross with the selectively permeable membrane of the eggshell through osmosis. Then, the vinegar proceeds to thicken the outer layer of the remaining eggshell to make it “bouncy.”

Step 1: Carefully place an egg in the glass cup or jar.

Step 2: Pour white vinegar in the glass until the egg is completely submerged (At this point you can add the food coloring if you’d like).

Step 3: Leave the egg in the glass for at least 48 hours.

Step 4: After 48 hours, or when the egg is translucent, you can remove it from the glass and run it through water.

Step 5: Gently rub off the exterior of the egg until it is completely translucent (Or the white part is gone).

Step 6: Examine the egg and begin bouncing, lightly at first and then test how far it can go (Have a plate or tray handy as this part could get messy).

• Observations: What happened? You should also encourage them to observe the changes during the 48-hour transition process. What did it look like? What happened to the size of the egg? How did the egg feel? How high can it bounce without breaking apart?
• Conclusions : What happens when you cover an egg in vinegar? How did that happen? What other liquids will have a similar reaction?

For more experiments with only a few simple items, check out our article, Easy Experiments in Fluids .

Air Cannon Experiment

Materials Needed: (This should be done with an older elementary-aged child or make sure they are mature enough to do this)

• Plastic Bottle
• Rubber Band
• Pieces of paper/plastic objects to knock over. (Dominoes, paper towel roll, plastic cups).
• Stickers (Optional)
• Question: Can you control air? Why does air move? How powerful can air be?
• Hypothesis : What do they think will happen? How strong can you make air when it is controlled? What objects can be pushed over by air?
• Research: Air is a gas making it difficult to contain or hold its shape. Air pressure and density can be felt by temperature changes or changes in elevation. Even tiny air molecules have weight, and when billions of those molecules come together, they weigh down on other objects. Look into the changes of air pressures at different levels of elevation even into the atmosphere.

Step 1: Cut the bottom of the end of the plastic bottle.

Step 2: Attach the balloon to the open part that you cut off. (If you need to, you can cut the end of the balloon to make it fit).

Step 3: Attach a rubber band to the outside of the balloon to make sure it is securely attached.

Step 4: Decorate the bottle with stickers to make it look cool! (Optional step of course)

Step 5: Stretch the end of the balloon back and release.

Step 6: See how many controlled objects you can knock down. (Again, think dominoes, paper towel rolls, plastic cups, etc.).

• Observations: What happened? How does this happen? How many objects were successfully knocked down? Which objects stood strong? Why might that be?
• Conclusions : How can air be controlled? What other objects can you knock down? What is needed to make the air vortex stronger and more powerful?

While it’s always fun to build an air cannon, I understand this can be complex, for younger kids. You do have the option to buy one to play around with. You can find the current price of one right here .

Oil vs. Water

• A glass cup/jar
• Vegetable Oil
• Food Coloring
• Measuring Cup
• Toothpick (Optional)
• Question: What happens when you mix oil and water?
• Hypothesis : What do they think will happen? Will they mix? How will they mix? What will happen when you add in food coloring?
• Research: Researching density will make this experiment more valuable to your kids. Water molecules are polarized and more dense than oil. Oil is made up of non-polar molecules and is a lot less dense. Polar molecules only dissolve in polar solvents while non-polar molecules only dissolve in non-polar solvents making them not compatible, therefore, not allowing them to mix.

Step 1: Pour ½ cup of oil into a glass cup.

Step 2: Pour ½ cup of water into the glass cup.

Step 3: Allow it to settle and watch what happens.

Step 4: Add a drop or two of food coloring.

Step 5: Allow it to settle and watch what happens. (If the food coloring drop needs a little bit of a push, you can do so with the toothpick)

Step 6: Examine the glass and see what happens.

• Observations: What happened? Why didn’t they mix? What happened to the food coloring?
• Conclusions : What happens when oil and water are in the same container? Why does that happen? What happens if you add different food coloring to the container? Why doesn’t the food coloring mix with the oil? What does food coloring mix with? What else doesn’t oil mix with?

Magic Inflating Balloon

• Water bottle (Soda bottle will work just fine too)
• Baking Soda
• Funnel (Two preferably: If you have one, then make sure you wash it completely before using it again)
• Question: Can you fill a balloon without air?
• Hypothesis : What do they think will happen? Will a balloon fill with baking soda and vinegar?
• Research: You need carbon dioxide gas to fill a balloon. When baking soda (a base) and vinegar (an acid) are combined, they create carbon dioxide gas. Since the gas doesn’t have a shape, it will quickly expand, filling in all of the space that it can, resulting in an inflated balloon.

Step 1: Put ⅓ of a cup of vinegar into a water bottle.

Step 2: Attach the other funnel (or the washed funnel) to the open end of the balloon.

Step 3: Place two teaspoons of baking soda into the balloon then remove the funnel from the balloon.

Step 4: Attach the open end of the balloon to the top of the water bottle. (Try not to let the baking soda mix with the vinegar when attaching)

Step 5: When the balloon is securely attached, lift the top of it so the baking soda does drop into the water bottle.

Step 6: Watch the balloon magically inflate!

• Observations: What happened? How fast did it happen? How did that happen? How long until it deflates?
• Conclusions : What happens when you mix a base and an acid? Will another acid work just as well? Is the size of the balloon dependent on the amount of acid?

Is soda really that bad for your teeth?

• Baby teeth (This one may take a little while to get, depending on their age. Egg shells work similarly, but the curiosity may not fully be there for them. Just make sure you hard-boil them).
• Different types of soda (At least 3 – can include a sugary drink such as Gatorade too)
• Multiple glass jars (Enough for each separate
• Question: How bad is soda for your teeth?
• Hypothesis : What do they think will happen? How will the soda/drink affect the teeth?
• Research: Research the acidity of different drinks and the effect they have on tooth enamel. Erosion will be the biggest effect as it occurs when the acid and carbonation of the soda touches the outer layer of the tooth, or the tooth enamel.

Step 1: Pour each soda or drink into its own separate glass jar up to ¼ of the way full (Up to ½ if you are using an egg).

Step 2: Place a tooth in each of the glass jars.

Step 4: Take out the tooth every day and record the changes that you’ve observed.

Step 5: After five days, take them out for good and record your final observations.

• Observations: What happened? What do they feel like? What do they look like?
• Conclusions : How did different types of soda affect the teeth? What will happen if you brush that tooth? How much do you have to brush to clean it?

Magic Cloud

• Water Bottle (The bigger, the better for this experiment, get a 2L bottle at minimum)
• Rubbing Alcohol
• A tablespoon
• A cork or a rubber stopper (Clay will work well too)
• Ball pump or Bicycle tire pump
• Safety Goggles
• Question: What is a cloud made of? How does a cloud form?
• Hypothesis : What do they think will happen? Do they think they’ll be able to simulate a cloud? How can you create a cloud?
• Research: Look into the changes in the atmosphere and how clouds emerge. The pump simulates the mass junction of air molecules to create clouds. Take this as an opportunity to research different types of clouds too.

Step 1: Pour 1 tablespoon of rubbing alcohol into the bottle.

Step 2: Shake the bottle around to ensure the rubbing alcohol touches all parts of the bottle.

Step 3: Use the pushpin to poke a hole in the middle of the bottle cap. (You may have to move it around a bit to make sure it is big enough for the bicycle pump)

Step 4: Add the cork or rubber stopper to the inside of the bottle cap.

Step 5: Put on your safety goggles. Slide the needle of the pump into the hole and start pumping. (The more you pump, the bigger the cloud).

Step 6: Keep your safety goggles on. Hold the bottle firmly and open the cap to release the pressure. (Be aware that a loud noise will occur here, so be prepared).

• Observations: What happened? How much of a reaction did you get?
• Conclusions : What would happen if you were to close the cap again in the middle of releasing the cloud? What would happen if you added more rubbing alcohol? How big would the cloud be if pumped more air into it?

There are a few ways you can do this, and I go into more in detail in this article .

Wrapping Up

A lot of these experiments are great but do require a lot of preparation and guidance from the parents/adults. If you really want to engage their curiosity, put every question they have to the test.

Do double-stuffed Oreos actually have twice the amount of stuffing? Test it. Are footlong hoagies really 1 foot long? Measure it. Can you revive a dead marker? Try it. Why do volcanoes explode? Research it. There is no need to keep them wondering, and it’ll pique their curiosity for future experiences.

The Scientific Method gives them the appropriate avenue to test some of these ideas, even if it is just verbally referencing them. You might learn some pretty cool things on the way too. Have fun experimenting!

Do your kids like to get messy? You can find messy STEM projects in our article, Messy STEM Science Experiments for Kids!

Leave a Comment Cancel reply

Save my name, email, and website in this browser for the next time I comment.

most recent

Activities and Games , Toy Gift Guides

Best stem subscription boxes for kids: hands-on reviews.

Science , Engineering , Math , Technology

Kiwico vs mel science: a complete breakdown.

Best Chemistry Kits for Kids (for Every Age!)

Teach Kids to Code , Activities and Games , Technology , Toy Gift Guides

11 best coding robots to teach kids to code (for all ages).

Activities and Games , Math

20 stem projects that are great for middle school.

Engineering

6 projects for learning about simple machines.

Science and the scientific method: Definitions and examples

Here's a look at the foundation of doing science — the scientific method.

The scientific method

Hypothesis, theory and law, a brief history of science, additional resources, bibliography.

Science is a systematic and logical approach to discovering how things in the universe work. It is also the body of knowledge accumulated through the discoveries about all the things in the universe. 

The word "science" is derived from the Latin word "scientia," which means knowledge based on demonstrable and reproducible data, according to the Merriam-Webster dictionary . True to this definition, science aims for measurable results through testing and analysis, a process known as the scientific method. Science is based on fact, not opinion or preferences. The process of science is designed to challenge ideas through research. One important aspect of the scientific process is that it focuses only on the natural world, according to the University of California, Berkeley . Anything that is considered supernatural, or beyond physical reality, does not fit into the definition of science.

When conducting research, scientists use the scientific method to collect measurable, empirical evidence in an experiment related to a hypothesis (often in the form of an if/then statement) that is designed to support or contradict a scientific theory .

"As a field biologist, my favorite part of the scientific method is being in the field collecting the data," Jaime Tanner, a professor of biology at Marlboro College, told Live Science. "But what really makes that fun is knowing that you are trying to answer an interesting question. So the first step in identifying questions and generating possible answers (hypotheses) is also very important and is a creative process. Then once you collect the data you analyze it to see if your hypothesis is supported or not."

The steps of the scientific method go something like this, according to Highline College :

• Make an observation or observations.
• Form a hypothesis — a tentative description of what's been observed, and make predictions based on that hypothesis.
• Test the hypothesis and predictions in an experiment that can be reproduced.
• Analyze the data and draw conclusions; accept or reject the hypothesis or modify the hypothesis if necessary.
• Reproduce the experiment until there are no discrepancies between observations and theory. "Replication of methods and results is my favorite step in the scientific method," Moshe Pritsker, a former post-doctoral researcher at Harvard Medical School and CEO of JoVE, told Live Science. "The reproducibility of published experiments is the foundation of science. No reproducibility — no science."

Some key underpinnings to the scientific method:

• The hypothesis must be testable and falsifiable, according to North Carolina State University . Falsifiable means that there must be a possible negative answer to the hypothesis.
• Research must involve deductive reasoning and inductive reasoning . Deductive reasoning is the process of using true premises to reach a logical true conclusion while inductive reasoning uses observations to infer an explanation for those observations.
• An experiment should include a dependent variable (which does not change) and an independent variable (which does change), according to the University of California, Santa Barbara .
• An experiment should include an experimental group and a control group. The control group is what the experimental group is compared against, according to Britannica .

The process of generating and testing a hypothesis forms the backbone of the scientific method. When an idea has been confirmed over many experiments, it can be called a scientific theory. While a theory provides an explanation for a phenomenon, a scientific law provides a description of a phenomenon, according to The University of Waikato . One example would be the law of conservation of energy, which is the first law of thermodynamics that says that energy can neither be created nor destroyed. 

A law describes an observed phenomenon, but it doesn't explain why the phenomenon exists or what causes it. "In science, laws are a starting place," said Peter Coppinger, an associate professor of biology and biomedical engineering at the Rose-Hulman Institute of Technology. "From there, scientists can then ask the questions, 'Why and how?'"

Laws are generally considered to be without exception, though some laws have been modified over time after further testing found discrepancies. For instance, Newton's laws of motion describe everything we've observed in the macroscopic world, but they break down at the subatomic level.

This does not mean theories are not meaningful. For a hypothesis to become a theory, scientists must conduct rigorous testing, typically across multiple disciplines by separate groups of scientists. Saying something is "just a theory" confuses the scientific definition of "theory" with the layperson's definition. To most people a theory is a hunch. In science, a theory is the framework for observations and facts, Tanner told Live Science.

The earliest evidence of science can be found as far back as records exist. Early tablets contain numerals and information about the solar system , which were derived by using careful observation, prediction and testing of those predictions. Science became decidedly more "scientific" over time, however.

1200s: Robert Grosseteste developed the framework for the proper methods of modern scientific experimentation, according to the Stanford Encyclopedia of Philosophy. His works included the principle that an inquiry must be based on measurable evidence that is confirmed through testing.

1400s: Leonardo da Vinci began his notebooks in pursuit of evidence that the human body is microcosmic. The artist, scientist and mathematician also gathered information about optics and hydrodynamics.

1500s: Nicolaus Copernicus advanced the understanding of the solar system with his discovery of heliocentrism. This is a model in which Earth and the other planets revolve around the sun, which is the center of the solar system.

1600s: Johannes Kepler built upon those observations with his laws of planetary motion. Galileo Galilei improved on a new invention, the telescope, and used it to study the sun and planets. The 1600s also saw advancements in the study of physics as Isaac Newton developed his laws of motion.

1700s: Benjamin Franklin discovered that lightning is electrical. He also contributed to the study of oceanography and meteorology. The understanding of chemistry also evolved during this century as Antoine Lavoisier, dubbed the father of modern chemistry , developed the law of conservation of mass.

1800s: Milestones included Alessandro Volta's discoveries regarding electrochemical series, which led to the invention of the battery. John Dalton also introduced atomic theory, which stated that all matter is composed of atoms that combine to form molecules. The basis of modern study of genetics advanced as Gregor Mendel unveiled his laws of inheritance. Later in the century, Wilhelm Conrad Röntgen discovered X-rays , while George Ohm's law provided the basis for understanding how to harness electrical charges.

1900s: The discoveries of Albert Einstein , who is best known for his theory of relativity, dominated the beginning of the 20th century. Einstein's theory of relativity is actually two separate theories. His special theory of relativity, which he outlined in a 1905 paper, " The Electrodynamics of Moving Bodies ," concluded that time must change according to the speed of a moving object relative to the frame of reference of an observer. His second theory of general relativity, which he published as " The Foundation of the General Theory of Relativity ," advanced the idea that matter causes space to curve.

In 1952, Jonas Salk developed the polio vaccine , which reduced the incidence of polio in the United States by nearly 90%, according to Britannica . The following year, James D. Watson and Francis Crick discovered the structure of DNA , which is a double helix formed by base pairs attached to a sugar-phosphate backbone, according to the National Human Genome Research Institute .

2000s: The 21st century saw the first draft of the human genome completed, leading to a greater understanding of DNA. This advanced the study of genetics, its role in human biology and its use as a predictor of diseases and other disorders, according to the National Human Genome Research Institute .

• This video from City University of New York delves into the basics of what defines science.
• Learn about what makes science science in this book excerpt from Washington State University .
• This resource from the University of Michigan — Flint explains how to design your own scientific study.

Merriam-Webster Dictionary, Scientia. 2022. https://www.merriam-webster.com/dictionary/scientia

University of California, Berkeley, "Understanding Science: An Overview." 2022. ​​ https://undsci.berkeley.edu/article/0_0_0/intro_01  

Highline College, "Scientific method." July 12, 2015. https://people.highline.edu/iglozman/classes/astronotes/scimeth.htm  

North Carolina State University, "Science Scripts." https://projects.ncsu.edu/project/bio183de/Black/science/science_scripts.html  

University of California, Santa Barbara. "What is an Independent variable?" October 31,2017. http://scienceline.ucsb.edu/getkey.php?key=6045  

Encyclopedia Britannica, "Control group." May 14, 2020. https://www.britannica.com/science/control-group  

The University of Waikato, "Scientific Hypothesis, Theories and Laws." https://sci.waikato.ac.nz/evolution/Theories.shtml  

Stanford Encyclopedia of Philosophy, Robert Grosseteste. May 3, 2019. https://plato.stanford.edu/entries/grosseteste/  

Encyclopedia Britannica, "Jonas Salk." October 21, 2021. https://www.britannica.com/ biography /Jonas-Salk

National Human Genome Research Institute, "​Phosphate Backbone." https://www.genome.gov/genetics-glossary/Phosphate-Backbone  

National Human Genome Research Institute, "What is the Human Genome Project?" https://www.genome.gov/human-genome-project/What  

‌ Live Science contributor Ashley Hamer updated this article on Jan. 16, 2022.

Sign up for the Live Science daily newsletter now

Get the world’s most fascinating discoveries delivered straight to your inbox.

'I have never written of a stranger organ': The rise of the placenta and how it helped make us human

Supercooling of Earth's inner core may finally reveal how old it is

Color-blind people may be less picky eaters. Here's why.

Most Popular

• 2 Angular roughshark: The pig-faced shark that grunts when captured
• 3 The moon might still have active volcanoes, China's Chang'e 5 sample-return probe reveals
• 4 How did people clean themselves before soap was invented?
• 5 'I have never written of a stranger organ': The rise of the placenta and how it helped make us human

• Grades 6-12
• School Leaders

Get 50% off your first box of Home Chef! 🥙

What Are the Scientific Method Steps?

Explore with a well-organized and curious approach.

The scientific method not only teaches students how to conduct experiments, but it also enables them to think critically about processes that extend beyond science and into all aspects of their academic lives. Just like detectives, scientists, and explorers, students can use this scientific method structured-steps approach to explore, question, and discover. 

What is the scientific method?

What are the steps of the scientific method, how does the scientific method encourage critical thinking, how are the scientific method steps used in the classroom.

• Free printable scientific method steps worksheet
• Free printable scientific method steps posters

The scientific method is like a structured adventure for exploring the world that encourages discovery by finding answers and solving puzzles. With the scientific method steps, students get to ask questions, observe, make educated guesses (called hypotheses), run experiments, collect and organize data, draw sensible conclusions, and share what they’ve learned. Students can explore the natural world with a well-organized and curious approach. 

The scientific method steps can vary by name, but the process as a whole is the same across grade levels. There are as many as seven steps, but sometimes they are combined. Below are six steps that make the process accessible to younger learners.

1. Question

Encourage students to ask why, what, when, where, or how about a particular phenomenon or topic. Get them wondering about something that they find interesting or have a passion for. 

2. Research

Teach them to use their senses to gather information and make notes—for example, what are they seeing, hearing, etc.

3. Hypothesize

Based on observations, students will then make a hypothesis, which is an educated guess—it’s what they think will happen in an experiment. 

4. Experiment

To test their hypothesis, students can conduct an investigation or experiment and collect data. Data collection can involve charts, graphs, and observations.

Students can then look at the results of their experiment and interpret what that means in the grand scheme of their original question. From the data collected, students can then apply the new knowledge to their original question. 

Just like real scientists, students can communicate their findings with their classmates in a presentation, lab write-up, and many other ways.  ADVERTISEMENT

Be sure to check out our free printable scientific method posters and free scientific method steps printable .

The scientific method fosters critical thinking in students by promoting curiosity, observation, hypothesis formation, problem-solving, data analysis, logical reasoning, and effective communication. This structured approach equips students with vital skills for science and everyday life, while also promoting open-mindedness, adaptability, and reflective thinking, enhancing their critical thinking abilities across various situations.

The scientific method isn’t just about experiments, it’s a valuable tool that helps students become critical thinkers in all areas of their studies. From forming hypotheses to conducting experiments and sharing findings, it equips them with important skills. Plus, it encourages open-mindedness and adaptability. By using the scientific method, students start a lifelong adventure of learning and solving problems.

Even students as young as kindergarten can begin learning and exploring the scientific method steps. Plus, the scientific method is used all the way through high school and beyond, so it’s not a one-and-done skill. If you’re looking for hands-on ways for students to practice the scientific method, we compiled science experiments, labs, and demonstrations for elementary through middle school teachers to share with their students:

• Kindergarten Experiments and Projects
• 1st Grade Experiments and Projects
• 2nd Grade Experiments and Projects
• 3rd Grade Experiments and Projects
• 4th Grade Experiments and Projects
• 5th Grade Experiments and Projects
• 6th Grade Experiments and Projects
• 7th Grade Experiments and Projects
• 8th Grade Experiments and Projects
• High School Experiments and Projects

Free Printable Scientific Method Worksheet

This worksheet includes space for students to fill in every step of the scientific inquiry process along with prompts to ensure they stay on track. 

Free Printable Scientific Method Posters

Looking for a visual aid to help your students remember the steps to the scientific method? Get our free printable scientific method posters.

Copyright © 2024. All rights reserved. 5335 Gate Parkway, Jacksonville, FL 32256

for the Answer!

Kristi Harjo

Adventures in ISTEM

8 Cool Scientific Method Experiments for Middle Schoolers to Try

August 10, 2023 by Kristi

It’s back to school time, and that means it’s time to reintroduce students to key science skills like the scientific method.  There are some simple and engaging scientific method experiments you can have your students do where they practice the key steps of the scientific method process.

The scientific method is a process that allows students to figure things out about the world.  They can use it to solve problems and learn more about what’s around them. 

It starts with making observations. Then, they ask questions about what they are seeing.  From there, they make an educated guess as to why the observing the phenomenon and what is causing it.  They create a controlled experiment to test their hypothesis where they gather data and more observations.  Next, they analyze the data to look for patterns and answers. Finally, they draw their conclusions and share their findings.  

Scientific Method Experiment 1: Paper Towel Test Strength

Growing up, for us, it was commercials that were designed to tell us which brand is best and why.  Now with Tik Tok and other social media, it is important for students to fact check what they are seeing. I like to show students old paper towel commercials and then have students create scientific method experiments to test the claims.

• First, I will show them one or two paper towel commercials that claim their brand is the best for strength and not tearing while it cleans the mess.
• Then, the students turn the commercials claim into a hypothesis.
• From there, they design a test to test the claim.  I usually provide four to five different towels, including the brown school paper towel and a generic store brand towel. I also provide different types of weights like marbles, washers, or pennies.
• The students perform their paper towel experiments and collect data.
• They then analyze the data and then use the evidence to determine if the commercial claim is true or false.

Scientific Method Experiment 2: Paper Towel Test Absorbancy

This is just like the strength test, but it focuses on the paper towels ability to absorb a spill. The steps are the same as the other test.  You could have half the class doing one test and the other half of the class doing the other test.  This way, you are using a lot of the same materials, and students can compare their results. In the end, you could see if they can create a test to determine which is the best paper towel for strength and absorbency.

• First, I will show them one or two paper towel commercials that claim their brand is the best for absorbing a mess over the other.
• From there, they design a test to test the claim.  I usually provide four to five different towels, including the brown school paper towel and a generic store brand towel.
• The students perform their experiments and collect data.

Scientific Method Experiment 3: Grow that gummy

I find that students love doing labs that involve food.  They especially like eating the leftover candies that weren’t used when the experiment is over. For this scientific method experiment , students use gummy candy and different liquids to determine which one will make the gummy candy grow the largest.  

• Start off by having the students make a hypothesis as to which type of liquid will make the gummy candy grow the largest. For liquids, you could use water, salt water, vinegar, milk, soda, juice, and vegetable oil.
• Then have students design an experiment that will test their hypothesis.  Have them share ideas for how they will measure the gummy and what factors will need to be controlled.
• Students will conduct the experiment and record their observations.
• They will then analyze the results and draw conclusions as to which liquid made the gummy candy grow the largest.
• Students will then share their results and compare their results and tests with other groups.  This step is important because if they created a controlled experiment, they should draw the same conclusions even though the actual data numbers might be different or the way they designed the test might be different.

Scientific Method Experiment 4: Candy Letter Lab

My students are always fascinated by this lab. Students act like magicians as they have the letters on the candy levitate to the top of the liquid without touching the candy.  

• Start by using hard candy that has a letter or word stamped on it.  If find that Skittles and m&m’s work best for this. You might want to have some groups test Skittles while others test m&m’s to see if they get different results.
• Have students create a hypothesis for which liquid they think will be the best at removing the letter from the candy.
• Next, students will design an experiment that will test their hypothesis.  
• They will then analyze the results and draw conclusions as to which liquid was the fastest at removing the letter from the candy.
• Students will then share their results and compare their results and tests with other groups.  

Scientific Method Experiment 5: Where did the stripes go?

This is a great lab to do during the winter holidays when candy canes are in more abundance.  In this lab, the students design scientific method experiments to see which liquid will remove the stripes from the candy cane the fastest.  

• Start by using a regular candy cane or red and white peppermint candy.  I find the mini candy canes work the best and are not that expensive when you buy them in bulk.
• Have students create a hypothesis for which liquid they think will be the best at removing the red stripes from the candy cane. 
• They will then analyze the results and draw conclusions as to which liquid was the fastest at removing the stripes from the candy cane.

Scientific Method Experiment 6: Growing plants

One scientific method experiment that is good to do at the beginning of a long unit is growing plants from seeds.  This takes a while to see results, so it’s one that you will want to start and then check on periodically over a few weeks. 

The best seeds to use for this would be green beans, spinach, lettuce, or radish.  They have short germination periods.  To start, students can discuss what plants need to grow and thrive.  They can come up with a variety of different questions about how different factors might affect plant growth.

• Have students choose one question they want to test.
• Then, students create a hypothesis for their question. 
• They will then analyze the results and draw conclusions to determine if their hypothesis was supported or not supported.

Scientific Method Experiment 7: Pendulum swing

This lab is great for students to determine not only the hypothesis but also the question. You might want to start off with a demonstration of a single pendulum.  You can then start an “I Wonder” session.  I wonder how adding more weight affects the number of swings? I wonder how adding more weight affects the time it takes a pendulum to swing back and forth 10 times. Have students come up with their own I Wonder questions.  Once you have a good list, they can then choose one of them that they would like to test and investigate.

• Have students create a hypothesis for their question. 
• Students will conduct the pendulum experiment and record their observations.

Scientific Method Experiment 8: Crystal Growing

This is another lab I like to do before winter break.  Students can not only practice the steps of the scientific method, but they can also create ornaments or sun catchers that they can then take home.  In this scientific method experiment , students will be given different questions about crystals and decide which question they would like to test.

• Does the type of solution affect the amount of crystal growth?
• Does the type of solution affect the size of the crystals?
• Does the level of saturation affect the amount of crystal growth?
• Does the level of saturation affect the size of the crystals?
• Does the temperature of the solution affect the amount of crystal growth?
• Does the temperature of the solution affect the size of the crystals?

Why Teach the Scientific Method

Having students practice using the steps of the scientific method helps them to develop the soft skills that they will need outside of school and when they enter adulthood.

• Critical thinking skills- Being able to analyze data, draw conclusions, and make evidence-based decisions.
• Problem-solving abilities- approaching challenges using a systematic approach by identifying the problem, forming a hypothesis, and finding solutions.
• Communication skills- communicating effectively their findings and using evidence to support their conclusions.

Practicing the steps of the scientific method provides middle school students with a host of valuable benefits that extend beyond the classroom. Engaging in scientific method experiments such as testing paper towel strength and absorbency, growing crystals, and investigating candy properties helps students develop critical thinking skills, problem-solving abilities, and effective communication.

These skills are vital for their future endeavors, enabling them to make evidence-based decisions, tackle real-life challenges, and express their findings clearly. Embracing the scientific method empowers students to explore the world around them and equips them with essential skills for success in adulthood.

Looking for something?

• Privacy Policy
• Terms of Use
• Flipped Lessons
• Claim Evidence Reasoning
• Science Readings
• Social & Emotional Learning
• Digital Lessons

Join the Newsletter

Get tips & freebies right in your inbox!

• Science Notes Posts
• Contact Science Notes
• Todd Helmenstine Biography
• Anne Helmenstine Biography
• Free Printable Periodic Tables (PDF and PNG)
• Periodic Table Wallpapers
• Interactive Periodic Table
• Periodic Table Posters
• Science Experiments for Kids
• How to Grow Crystals
• Chemistry Projects
• Fire and Flames Projects
• Holiday Science
• Chemistry Problems With Answers
• Physics Problems
• Unit Conversion Example Problems
• Chemistry Worksheets
• Biology Worksheets
• Periodic Table Worksheets
• Physical Science Worksheets
• Science Lab Worksheets
• My Amazon Books

Steps of the Scientific Method 2

The scientific method is a system scientists and other people use to ask and answer questions about the natural world. In a nutshell, the scientific method works by making observations, asking a question or identifying a problem, and then designing and analyzing an experiment to test a prediction of what you expect will happen. It’s a powerful analytical tool because once you draw conclusions, you may be able to answer a question and make predictions about future events.

These are the steps of the scientific method:

• Make observations.

Sometimes this step is omitted in the list, but you always make observations before asking a question, whether you recognize it or not. You always have some background information about a topic. However, it’s a good idea to be systematic about your observations and to record them in a lab book or another way. Often, these initial observations can help you identify a question. Later on, this information may help you decide on another area of investigation of a topic.

• Ask a question, identify a problem, or state an objective.

There are various forms of this step. Sometimes you may want to state an objective and a problem and then phrase it in the form of a question. The reason it’s good to state a question is because it’s easiest to design an experiment to answer a question. A question helps you form a hypothesis, which focuses your study.

• Research the topic.

You should conduct background research on your topic to learn as much as you can about it. This can occur both before and after you state an objective and form a hypothesis. In fact, you may find yourself researching the topic throughout the entire process.

• Formulate a hypothesis.

A hypothesis is a formal prediction. There are two forms of a hypothesis that are particularly easy to test. One is to state the hypothesis as an “if, then” statement. An example of an if-then hypothesis is: “If plants are grown under red light, then they will be taller than plants grown under white light.” Another good type of hypothesis is what is called a “ null hypothesis ” or “no difference” hypothesis. An example of a null hypothesis is: “There is no difference in the rate of growth of plants grown under red light compared with plants grown under white light.”

• Design and perform an experiment to test the hypothesis.

Once you have a hypothesis, you need to find a way to test it. This involves an experiment . There are many ways to set up an experiment. A basic experiment contains variables, which are factors you can measure. The two main variables are the independent variable (the one you control or change) and the dependent variable (the one you measure to see if it is affected when you change the independent variable).

• Record and analyze the data you obtain from the experiment.

It’s a good idea to record notes alongside your data, stating anything unusual or unexpected. Once you have the data, draw a chart, table, or graph to present your results. Next, analyze the results to understand what it all means.

• Determine whether you accept or reject the hypothesis.

Do the results support the hypothesis or not? Keep in mind, it’s okay if the hypothesis is not supported, especially if you are testing a null hypothesis. Sometimes excluding an explanation answers your question! There is no “right” or “wrong” here. However, if you obtain an unexpected result, you might want to perform another experiment.

• Draw a conclusion and report the results of the experiment.

What good is knowing something if you keep it to yourself? You should report the outcome of the experiment, even if it’s just in a notebook. What did you learn from the experiment?

How Many Steps Are There?

You may be asked to list the 5 steps of the scientific method or the 6 steps of the method or some other number. There are different ways of grouping together the steps outlined here, so it’s a good idea to learn the way an instructor wants you to list the steps. No matter how many steps there are, the order is always the same.

Related Posts

2 thoughts on “ steps of the scientific method ”.

You raise a valid point, but peer review has its limitations. Consider the case of Galileo, for example.

That’s a good point too. But that was a rare limitation due to religion, and scientific consensus prevailed in the end. It’s nowhere near a reason to doubt scientific consensus in general. I’m thinking about issues such as climate change where so many people are skeptical despite 97% consensus among climate scientists. I was just surprised to see that this is not included as an important part of the process.

Comments are closed.

Six Steps of the Scientific Method

Learn What Makes Each Stage Important

ThoughtCo. / Hugo Lin 

• Scientific Method
• Chemical Laws
• Periodic Table
• Projects & Experiments
• Biochemistry
• Physical Chemistry
• Medical Chemistry
• Chemistry In Everyday Life
• Famous Chemists
• Activities for Kids
• Abbreviations & Acronyms
• Weather & Climate
• Ph.D., Biomedical Sciences, University of Tennessee at Knoxville
• B.A., Physics and Mathematics, Hastings College

The scientific method is a systematic way of learning about the world around us. The key difference between the scientific method and other ways of acquiring knowledge is that, when using the scientific method, we make hypotheses and then test them with an experiment.

Anyone can use the scientific method to acquire knowledge by asking questions and then working to find the answers to those questions. Below are the six steps involved in the scientific method and variables you may encounter when working with this method.

The Six Steps

The number of steps in the scientific method can vary from one description to another (which mainly happens when data and analysis are separated into separate steps), however, below is a fairly standard list of the six steps you'll likely be expected to know for any science class:

• Purpose/Question Ask a question.
• Research Conduct background research. Write down your sources so you can cite your references. In the modern era, you might conduct much of your research online. As you read articles and papers online, ensure you scroll to the bottom of the text to check the author's references. Even if you can't access the full text of a published article, you can usually view the abstract to see the summary of other experiments . Interview experts on a topic. The more you know about a subject, the easier it'll be to conduct your investigation.
• Hypothesis Propose a hypothesis . This is a sort of educated guess about what you expect your research to reveal. A hypothesis is a statement used to predict the outcome of an experiment. Usually, a hypothesis is written in terms of cause and effect. Alternatively, it may describe the relationship between two phenomena. The null hypothesis or the no-difference hypothesis is one type of hypothesis that's easy to test because it assumes changing a variable will not affect the outcome. In reality, you probably expect a change, but rejecting a hypothesis may be more useful than accepting one.
• Experiment Design and experiment to test your hypothesis. An experiment has an independent and dependent variable. You change or control the independent variable and record the effect it has on the dependent variable . It's important to change only one variable for an experiment rather than try to combine the effects of variables in an experiment. For example, if you want to test the effects of light intensity and fertilizer concentration on the growth rate of a plant, you're looking at two separate experiments.
• Data/Analysis Record observations and analyze the meaning of the data. Often, you'll prepare a table or graph of the data. Don't throw out data points you think are bad or that don't support your predictions. Some of the most incredible discoveries in science were made because the data looked wrong! Once you have the data, you may need to perform a mathematical analysis to support or refute your hypothesis.
• Conclusion Conclude whether to accept or reject your hypothesis. There's no right or wrong outcome to an experiment, so either result is fine. Accepting a hypothesis doesn't necessarily mean it's correct! Sometimes repeating an experiment may give a different result. In other cases, a hypothesis may predict an outcome, yet you might draw an incorrect conclusion. Communicate your results. You can compile your results into a lab report or formally submit them as a paper . Whether you accept or reject the hypothesis, you likely learned something about the subject and may wish to revise the original hypothesis or form a new one for a future experiment.

When Are There Seven Steps?

Some teach the scientific method with seven steps instead of six. In the seven-step model, the first step is to make observations. Even if you don't make observations formally, you should think about prior experiences with a subject to ask a question or solve a problem.

Formal observations are a type of brainstorming that can help you find an idea and form a hypothesis. Observe your subject and record everything about it. Include colors, timing, sounds, temperatures, changes, behavior, and anything that strikes you as interesting or significant.

When you design an experiment, you're controlling and measuring variables. There are three types of variables:

• Controlled Variables:  You can have as many  controlled variables  as you like. These are parts of the experiment that you try to keep constant throughout an experiment so they won't interfere with your test. Writing down controlled variables is a good idea because it helps make your experiment  reproducible , which is important in science! If you have trouble duplicating results from one experiment to another, there may be a controlled variable you missed.
• Independent Variable:  This is the variable you control.
• Dependent Variable:  This is the variable you measure. It's called the dependent variable because it  depends  on the independent variable.
• Null Hypothesis Examples
• Scientific Method Flow Chart
• Random Error vs. Systematic Error
• What Is an Experimental Constant?
• Scientific Variable
• What Is a Hypothesis? (Science)
• What Are the Elements of a Good Hypothesis?
• What Are Examples of a Hypothesis?
• What Is a Testable Hypothesis?
• Scientific Hypothesis Examples
• Scientific Method Vocabulary Terms
• Understanding Simple vs Controlled Experiments
• The Role of a Controlled Variable in an Experiment
• What Is the Difference Between a Control Variable and Control Group?
• What Is a Controlled Experiment?
• DRY MIX Experiment Variables Acronym

• History & Society
• Science & Tech
• Biographies
• Animals & Nature
• Geography & Travel
• Arts & Culture
• Games & Quizzes
• On This Day
• One Good Fact
• New Articles
• Lifestyles & Social Issues
• Philosophy & Religion
• Politics, Law & Government
• World History
• Health & Medicine
• Browse Biographies
• Birds, Reptiles & Other Vertebrates
• Bugs, Mollusks & Other Invertebrates
• Environment
• Fossils & Geologic Time
• Entertainment & Pop Culture
• Sports & Recreation
• Visual Arts
• Demystified
• Image Galleries
• Infographics
• Top Questions
• Britannica Kids
• Saving Earth
• Space Next 50
• Student Center

scientific method

Our editors will review what you’ve submitted and determine whether to revise the article.

• University of Nevada, Reno - College of Agriculture, Biotechnology and Natural Resources Extension - The Scientific Method
• World History Encyclopedia - Scientific Method
• LiveScience - What Is Science?
• Verywell Mind - Scientific Method Steps in Psychology Research
• WebMD - What is the Scientific Method?
• Chemistry LibreTexts - The Scientific Method
• National Center for Biotechnology Information - PubMed Central - Redefining the scientific method: as the use of sophisticated scientific methods that extend our mind
• Khan Academy - The scientific method
• Simply Psychology - What are the steps in the Scientific Method?
• Stanford Encyclopedia of Philosophy - Scientific Method

scientific method , mathematical and experimental technique employed in the sciences . More specifically, it is the technique used in the construction and testing of a scientific hypothesis .

The process of observing, asking questions, and seeking answers through tests and experiments is not unique to any one field of science. In fact, the scientific method is applied broadly in science, across many different fields. Many empirical sciences, especially the social sciences , use mathematical tools borrowed from probability theory and statistics , together with outgrowths of these, such as decision theory , game theory , utility theory, and operations research . Philosophers of science have addressed general methodological problems, such as the nature of scientific explanation and the justification of induction .

The scientific method is critical to the development of scientific theories , which explain empirical (experiential) laws in a scientifically rational manner. In a typical application of the scientific method, a researcher develops a hypothesis , tests it through various means, and then modifies the hypothesis on the basis of the outcome of the tests and experiments. The modified hypothesis is then retested, further modified, and tested again, until it becomes consistent with observed phenomena and testing outcomes. In this way, hypotheses serve as tools by which scientists gather data. From that data and the many different scientific investigations undertaken to explore hypotheses, scientists are able to develop broad general explanations, or scientific theories.

See also Mill’s methods ; hypothetico-deductive method .

What is the Scientific Method: Steps, Definition, and Examples

Table of Contents

What is the Scientific Method?

The scientific method is an empirical process used to acquire scientific knowledge. It is broadly applied to various sciences and enables the testing and validation of a scientific hypothesis. The problem is defined based on various observations. For example, a question can arise from the observation of a natural phenomenon. This question can lead to the formulation of a hypothesis and predictions. These can be tested by collecting data using the appropriate methodology. The final steps of the scientific method include data analysis and validation of the hypothesis. Altogether, the conclusions drawn from the scientific method will lead to new questions. This will ultimately improve our knowledge towards a better comprehension of the world surrounding us.

When was the Scientific Method Invented? Who Invented the Scientific Method?

Even though various scientific methodologies were elaborated in ancient Egypt and Babylonia, the inventor of the scientific method is usually considered to be Aristotle 1 . This antique Greek philosopher introduced empiricism to science in his text Posterior Analytics 2 . In other words, empiricism means that our scientific knowledge must be based on observations and empirical evidence. This is a key concept of the scientific method. The term “scientific method” became popular much later during the 19 th and 20 th centuries when it was broadly introduced into dictionaries and encyclopedias 3 . 

What are the Steps of the Scientific Method?

What is the first step of the scientific method, step 1- what is a scientific question and how to use the scientific method.

First of all, the scientific method begins with a question, something that needs to be answered. This problem can arise from initial observations leading to a specific question, which would ideally be something that you can measure or quantify. This initial question will later lead to the formulation of the working hypothesis.

What is the Second Step of the Scientific Method?

Step 2- literature search.

Before performing scientific experiments in a laboratory, every scientist will begin his research by doing an extensive literature search. This is a crucial step of the scientific method because it will reveal what is already known about the problem. The idea is to see if anything relevant to the question is already known. In addition, the literature search can be used to determine the appropriate methodology to address the question.

What is the Third Step of the Scientific Method?

Step 3- formulation of the hypothesis and predictions.

Following extensive background research, the scientist can then formulate the hypothesis. It is a plausible assumption based on the scientific knowledge and the methodology available. The scientist can then predict the possible outcome before performing any experiments. For example, a scientist will formulate the hypothesis that if he changes the parameter or variable X, it could result in different effects (A, B, or C).

What is the Fourth Step of the Scientific Method?

Step 4- experimental design, scientific experiment, and data collection.

Obviously, experiments are an important part of the scientific method. Every rigorous scientific experiment needs to be performed using the appropriate methodology. For instance, the instrument used to test the hypothesis must be accurate and efficient. In order to be valid, the experiment must be performed along with appropriate control groups and in controlled conditions to assess the effect of a single parameter at a time. Furthermore, the scientist must take into account all the factors that can introduce a bias during data collection. The experiment also needs to be reproduced a few times to make sure that the results are reproducible and are not obtained randomly. Finally, different methodologies can be used to test the same hypothesis, therefore strengthening the validity of the scientific findings.

What is the Fifth Step of the Scientific Method?

Step 5- data analysis.

Once data collection is over, the scientist can proceed to its analysis. The collected data can be presented in different ways such as pictures, schemas, videos, etc. If numerical data was obtained, it can be presented in a chart. The type of chart selected for graphical representations depends on the type of question. For example, proportions are easily represented in a pie chart whereas a bar chart will be better suited to show the evolution of monthly sales of a company through the years. In addition, the scientist can perform various mathematical equations and statistical analyses to further characterize his dataset.

What is the Sixth Step of the Scientific Method?

Step 6- hypothesis validation or invalidation, and formulation of new related questions.

It is now time to draw conclusions about the initial question. The data collected and analyzed can either validate or invalidate the hypothesis. When drawing conclusions, the scientist must be critical regarding the quality of the data obtained and he should also consider the limitations of the methodology used for testing. Often, the conclusions will lead to additional questions and the formulation of new hypotheses.

What is the Seventh Step of the Scientific Method?

Step 7- sharing the scientific discoveries: publication and peer review.

Someone could easily become an improvised scientist and apply the scientific method to validate or invalidate his own hypothesis. However, what makes the strength of the scientific method is to share the knowledge gained from a scientific experiment that was performed. This way, the scientific community can benefit from the work of others before establishing their own hypotheses. Every research project published therefore contribute to broader scientific advances, even when the initial hypothesis was proven wrong.  In addition, our comprehension of a specific scientific topic is constantly evolving as it can be either validated or even sometimes challenged by the completion of more advanced research projects.

The scientific method is a cornerstone of science and this is why it is important to teach it to kids. This concept is generally taught to children during the 4 th , 5 th, or 6 th grade. The scientific method can help these kids to develop critical thinking and to give them the tools required to solve complex problems.

How to Use the Scientific Method and How to Design an Experiment Using the Scientific Method? An Example Applied to Drug Discovery

The scientific method can be applied to answer various questions related to biology, psychology, sociology, etc. Here, we have already explained all the steps constituting the scientific method and their respective order. Let’s now see a fictional example to show how the scientific method can be applied to solve complex problems in the pharmaceutical industry.

Step1: What is a Scientific Question?

Let’s say that a chemist is looking for new drugs that could be used in the pharmaceutical industry. The initial question could be something like “Is there a better treatment to control the blood pressure of patients?”. This is a good example showing how the rigorous application of the scientific method can answer a complex question.

Step 2: Literature research

The scientist will then proceed to an extensive literature search and gather all the information available for the active molecules already used as treatments. During his research, the chemist noticed a molecule that could be chemically transformed to alter its structure. In addition, the structure of the original molecule is available, and bio-informatics analysis indicates that the modification would occur in the active site of the molecule.

Step 3: What is an Example of a Hypothesis, How to Write a Hypothesis, and What is a Prediction in Science?

The scientist, therefore, emits the hypothesis that this modification could increase the efficiency of the treatment. He then predicts that the modification of the molecule will increase its binding to receptors located on the surface of blood vessels and that it will reduce blood pressure and side effects.

Step 4: Experiment and data collection

In vitro experiments.

The scientist decides to first test his hypothesis by measuring how the alteration of the active molecule can affect its capacity to bind the receptor. He will use purified molecules from either the original formula or the altered version of the molecule. Then, he will measure the binding capacity of the molecules towards their target receptor in a test tube.

In Vivo Experiments

To assess the biological properties of the newly identified molecule, the scientist will next use animals to analyze how the molecule can affect a complex organism such as rats. This is a complex experiment that needs to be designed properly in order to draw the right conclusions. The scientist decides to use obese rats that are prone to high blood pressure to test the efficiency of his new drug. Three groups will be monitored. The first group will be obese rats receiving no treatment at all. The second will contain animals receiving the original form of the molecule whereas the third will be administered the new molecule.

The experiment must be performed in controlled conditions

In order to be valid, the experiment needs to be performed in controlled conditions. To consider additional factors that might introduce a bias during data analysis, the groups compared must be homogeneous. Many factors can influence data interpretation and to make sure to draw the right conclusions, the scientist decides to use only male rats of approximately the same age. The blood pressure of these animals will then be monitored over the weeks and blood samples will be taken to reveal changes in its content.

Step 5: Data analysis

The results obtained during data collection can be presented in various graphical representations. For instance, the strength of the binding exhibited by these different molecules can be easily compared in a simple bar chart. The blood pressure measurements for each group can be presented as a function of time since the beginning of the treatment in a scatter plot. In addition, a trend line or regression line can be drawn on the graph to emphasize the various trends exhibited by each group of animals.

Step 6: Validation of the hypothesis

Once the different scientific experiments are performed, the scientist will be able to re-examine the initial hypothesis. If the methodology was appropriate and the influence of external factors was reduced to a minimum, the scientist will then be able to use his data and analysis to validate or invalidate his initial hypothesis.

In this example, the scientist will conclude that the modification of an existing molecule used to regulate blood pressure can increase its efficiency in comparison with the original drug. However, a major limitation of this study is that it was performed on an animal model. One could therefore ask if this newly identified molecule would be equally efficient on human patients. As you can see, the application of the scientific method for this research raised another important question, which can then be addressed by other scientists.

Step 7: Publication and peer review

In order to benefit the entire scientific community, a scientist must publish his findings. First, the scientist will first write an article summarizing his research project. He can then submit his article to a scientific journal where it will be reviewed by peers to ensure the quality of the results before their publication. Once the results are published, they can be accessible to the whole scientific community and can be cited in the work of other scientists. Altogether, this process allows the expansion of knowledge in a particular scientific field.

The Scientific Method – A Short Quiz

Question 1: classify these steps of the scientific method in the right order.

• Literature search
• Ask a question
• Publication
• Data analysis
• Validation of the hypothesis
• Formulation of the hypothesis and predictions

A) 2-3-7-1-5-6-4

B) 3-2-7-1-5-6-4

C) 3-2-7-1-5-4-6

D) 2-3-7-1-5-4-6

Question 2: To be able to draw valid conclusions, a scientist must use a methodology that…

• Generate reproducible data
• Can appropriately test the hypothesis
• Is precise enough to distinguish between conditions
• Is performed in a controlled environment

B) 1, 2 and 3

C) 2, 3 and 4

D) 1, 2, 3 and 4

Question 3: True or false. A scientific study is invalid and cannot be published if the hypothesis was wrong.

B) False   

Now that you know the different steps of the scientific method, what do you think about this reasoning process? Don’t be shy and share your thoughts with us in the comment section below!

Check my previous post to see how to experiment with light refraction through a prism!

1- Wikipedia – The history of the scientific method

2- Aristotle, considered the inventor of the scientific methods – Posterior Analytics

3- Wikipedia – Scientific method

Images created using logomakr.com

• Scientific method

Be the first to comment

Leave a reply cancel reply.

Your email address will not be published.

Save my name, email, and website in this browser for the next time I comment.

Copyright © 2024 | WordPress Theme by MH Themes

Scientific Method for Kids: Steps and FREE Printable Template

Just so you know, this post contains affiliate links. That means if you use them to make a purchase, I may earn a commission. You can read my full affiliate disclosure  HERE .

Are you looking for an easy way to teach the scientific method for kids? I’m sharing each step in this post, along with a free template you can print to help your kids walk through the scientific process.

Being inquisitive is in a kid’s nature. They LOVE asking questions (and come up with some of the best ones!). And although I love to answer them, as they get older I know that part of my job as a homeschool mom is to not answer each and every question for them.

Sooner or later, they need to have the tools to gather information and draw conclusions on their own , whether that happens inside science class or not. For kids, learning the scientific method steps can be a great way to build on skills that will be valuable in many areas of their lives.

An Englishman named Sir Francis Bacon helped create what we know as the scientific method. I thought it was interesting to learn, though, that there were multiple professional scientists and scholars who developed the scientific method over many years , and that many disagree on different ways to implement it. However, there are basic steps that have come to be accepted and taught as the scientific method (and that we get to explore with our kids!).

Scientific Method Steps for Kids

These steps to the scientific method that help us guide our research involve observing and asking questions, formulating a scientific hypothesis (or educated guess), planning and running an experiment, evaluating data and drawing conclusions.

Why Should Kids Learn the Scientific Method?

Why is it important that we teach our kids the scientific method? Isn’t it enough to just have fun doing experiments when we want to? We learn so much from them without formally teaching all of these steps!

Well, yes – we certainly learn a ton from fun experiments! But as my kids are getting older, I am finding they are needing a little bit more structure in thinking through their thoughts, making guesses based on thoughtful observations, and drawing valid conclusions after careful study. These are all important skills to learn and practice. (And don’t worry – I created a FREE printable that will help you walk through this entire process. It’s not as scary as it sounds!) .

Here are some other reasons you’ll want to use the scientific method for kids in your homeschool:

• The scientific process helps us dig in and learn more about God and His creation (and there is so much to observe and learn!).
• Scientists actually use the scientific method to research, study, learn, and solve problems. And your kids will have fun being scientists too!
• Following the scientific method steps allows us to conduct experiments correctly. It also teaches young kids the importance of documenting their steps and research so those that come behind them can replicate results, or build on their research. Carefully recording information is an important skill they will be able to translate to so many other areas.

Steps of the Scientific Method for Kids

Here are the basic steps in the scientific method you’ll want to teach your kids:

• Observe and ask a question.
• Form a hypothesis.
• Do an experiment.
• Evaluate your data.
• Draw a conclusion.

Let’s go over each of the steps in a little bit more detail with an easy example, so you can see how you might teach it in your homeschool. You’ll then be able to use this method with any of your science experiments in your homeschool; my free printable will definitely help walk you through the steps as well , and is a great addition to your kids homeschool scientific journal.

How to Teach the Scientific Method for Kids

Let’s use an easy example to walk through how the scientific method might look.

1. Ask a Question.

You have a couple of plants sitting on your windowsill, and your son asks the simple question about why they are sitting on the ledge. He wonders how the amount of light affects how a plant grows, and if it will grow without light.

And so begins the process of scientific investigation!

The first step for your kids is to take note of what questions they have, or what problems they might want to solve. What is something they are unsure about? Have them brainstorm some ideas and then do some observing and initial research to lay the groundwork for their experiment and help with the next step.

2. Make a Hypothesis.

Next, they’ll form a hypothesis, or an educated guess (a guess made from good reasoning and observing!), about what they think will happen. Before they do so, it might be helpful to make sure the question is clarified and is written in “testable” form, which will help them be clear in their experiment.

For example, the question from above might be, “I wonder if plants can grow in the dark?” You can help guide your child in re-forming this question to make it testable by using phrasing like:

• “Does changing __________ affect ___________ ?”
• “If I change __________, will it affect __________ ?”
• How does changing __________ affect __________ ?”

So – “How does the amount of light affect plant growth?”, or, “If I change the amount of sunlight a plant gets, will it affect it’s growth?”

Your child can then make a guess about what they think will happen, a possible answer, and begin planning on how to conduct the experiment.

3. Do an Experiment.

Now comes the fun part – finding out if you are right or wrong! Your child gets to map out their procedure (what they will actually do, how and what they will record) and make a list of materials they will need to conduct the experiment.

An important part of this step is noting what the independent and dependent variables will be. An independent variable is something that you will change; a dependent variable is what you will measure. A good experiment will only have ONE independent variable at a time – you don’t want to change too many things at once, or it will be hard to measure what actually produced the results.

In our example, our independent variable will be the light source – we’ll put one plant in sunlight and one in a dark place. We’ll try to keep them in rooms of the same temperature, and we’ll use the same amount of soil, same amount of water, and the same number and type of seed in each cup.

The dependent variable is what we are going to measure – how much the plants grow.

4. Evaluate Your Data.

This is the step where they record what is happening, what they observe, and then evaluate the results.

Drawing pictures and/or making graphs can definitely be helpful to display the data collected. Let them be creative with how they record this, but remind them that accuracy and being able to share data with others is important!

5. Draw a Conclusion.

Finally, they get to make a final conclusion. Did the experiment answer your question, and was your hypothesis correct? Thinking about what they learned and other new questions that may have arose as a result of the experiment are worth noting.

In some cases, it might be worthwhile to repeat the experiment with a new hypothesis, or try it again in a slightly different way to help draw conclusions. For example, maybe we could repeat the plant experiment with a different type of plant to see if we got the same results.

Have Fun With It!

Remember that learning about God’s creation and natural world through science experiments should be fun and enjoyable! Choose activities your kids will love, or let them help you decide what fun experiments to try. The skills that they will learn as they work through the scientific method will be valuable for them in years to come.

Can I Use the Scientific Process with my youngest kids?

Absolutely! Even young children will love following along as you perform experiments with your older kids. Although they won’t be able to journal or record data like the bigger kids, simply walking through and talking about the series of steps is a great way to introduce them to the scientific process.

When your little ones ask a good question that might make for a simple science experiment, ask them what they think will happen! Then recruit any older siblings to join in on the fun.

Science Experiment Ideas

You might have kids who are very scientifically-minded, and ask questions about everything! If not, sometimes coming up with a fun experiment can be a stumbling block.

Here are a few scientific questions your kids might have fun exploring, to get you rolling on the first step of practicing the scientific process for kids. These are also perfect to investigate for a science fair project at your school or homeschool co-op!

• Does music affect animal behavior?
• Where are the most germs in your home?
• Which paper towel brand is the strongest?
• What is the best way to keep cut flowers fresh the longest?
• What plant fertilizer works best?
• What brand of battery lasts the longest?
• How much weight can the surface tension of water hold?
• Which soda decays fallen out teeth the most?
• Can hamsters learn tricks?
• What is the effect of salt on the boiling temperature of water? On it’s density ?
• What type of grass seed grows the fastest?

Once they get started brainstorming, your kids can probably come up with lots of ideas to explore for their own experiments!

Scientific Method for Kids Books

There are a few great books you might want to check out as you introduce the scientific method steps. They are also wonderful for experiment ideas!

What is the Scientific Method? Science Book for Kids Mad Margaret Experiments with the Scientific Method (In the Science Lab) Awesome Science Experiments for Kids: 100+ Fun STEM / STEAM Projects and Why They Work Awesome Kitchen Science Experiments for Kids: 50 STEAM Projects You Can Eat! Steve Spangler’s Super-Cool Science Experiments for Kids: 50 mind-blowing STEM projects you can do at home Smithsonian 10-Minute Science Experiments: 50+ quick, easy and awesome projects for kids 101 Great Science Experiments: A Step-by-Step Guide The 101 Coolest Simple Science Experiments

Step-by-Step Scientific Method Printable

My older kids are in 2nd and 3rd grade right now, and I created this printable for them (so it works well for mid-upper elementary-aged students). However, you can absolutely adapt it if you have younger students , or students who aren’t quite ready to do so much writing just yet. You can still teach the process!

This FREE printable comes with one sheet that lists out all the scientific method steps; this is perfect to print out and hang in your homeschool room to refer to. And if your kids are younger, you can verbally talk through the process while doing simple experiments with them, which will help prepare them for a more in-depth process when they get older.

There is also one-page-per-step that walks them through the scientific method, and a basic list of some fun science experiment ideas to try. I also have a one-page sheet your kids can use to write out their process, if you want a simpler, more compact template.

Grab yours for FREE today!

Grab your FREE Scientific Method Printables!

Subscribe to my list and join thousands of other homeschool mamas looking for homeschool help, inspiration, and fun.

Thank you for subscribing!

Please check your email for your Scientific Method Printables.

I hope you have so much fun making memories with your kids learning about science and the scientific method for kids.

Drop a comment below: What has been your favorite science experiment you have done with your kids? What would they say is their favorite?

Related Posts

• Solar System Print-and-Go Unit Study
• Road Trip Ideas For Kids + Free Printable Activity Pack
• How to Create and Organize a Homeschool Portfolio

Blessed Homeschool is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com. You can read my full affiliate disclosure HERE.

More homeschool inspiration...

Hello! Can the Scientific Method freebie be used with students I am teaching on Outschool? I am using a slide show to teach the steps, but it would be nice to have some resources for them to actually carry out the process. If this does not follow your copyright rules, I totally understand. Thanks for your consideration.

Thank you for asking, Christina, and I would love for you to use it! If possible, please give them the link (or send it to their parents) so they can download it for free from my website. I hope you get a lot of good use out of it!

Sara, I really enjoyed your post and will look into Science Shepherd. I’ve tried to use the “subscribe” button to access the free scientific method packet, but can’t seem to connect. I’m hoping this will suffice to subscribe to your channel. Thanks and have a blessed coming school year, Bonnie

Thank you so much Bonnie!! I was having an error on my site for downloading the packet, but I think it’s fixed. Please let me know if you still have issues downloading it!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

• COVID-19 Tracker
• Biochemistry
• Anatomy & Physiology
• Microbiology
• Neuroscience
• Animal Kingdom
• NGSS High School
• Latest News
• Editors’ Picks
• Weekly Digest
• Quotes about Biology

Scientific Method

Reviewed by: BD Editors

The scientific method is a series of processes that people can use to gather knowledge about the world around them, improve that knowledge, and attempt to explain why and/or how things occur. This method involves making observations, forming questions, making hypotheses, doing an experiment, analyzing the data, and forming a conclusion. Every scientific experiment performed is an example of the scientific method in action, but it is also used by non-scientists in everyday situations.

Scientific Method Overview

The scientific method is a process of trying to get as close as possible to the  objective truth . However, part of the process is to constantly refine your conclusions, ask new questions, and continue the search for the rules of the universe. Through the scientific method, scientists are trying to uncover how the world works and discover the laws that make it function in that way. You can use the scientific method to find answers for almost any question, though the scientific method can yield conflicting evidence based on the method of experimentation. In other words, the scientific method is a very useful way to figure things out – though it must be used with caution and care!

Scientific Method Steps

The exact steps of the scientific method vary from source to source , but the general procedure is the same: acquiring knowledge through observation and testing.

Making an Observation

The first step of the scientific method is to make an observation about the world around you. Before hypotheses can be made or experiments can be done, one must first notice and think about some sort of phenomena occurring. The scientific method is used when one does not know why or how something is occurring and wants to uncover the answer. But, before you can form a question you must notice something puzzling in the first place.

Asking a Question

Next, one must ask a question based on their observations. Here are some examples of good questions:

• Why is this thing occurring?
• How is this thing occurring?
• Why or how does it happen this way?

Sometimes this step is listed first in the scientific method, with making an observation (and researching the phenomena in question) listed as second. In reality, both making observations and asking questions tend to happen around the same time.

One can see a confusing occurrence and immediately think, “why is it occurring?” When observations are being made and questions are being formed, it is important to do research to see if others have already answered the question or uncovered information that may help you shape your question. For example, if you find an answer to why something is occurring, you may want to go a step further and figure out how it occurs.

Forming a Hypothesis

A hypothesis is an educated guess to explain the phenomena occurring based on prior observations. It answers the question posed in the previous step. Hypotheses can be specific or more general depending on the question being asked, but all hypotheses must be testable by gathering evidence that can be measured. If a hypothesis is not testable, then it is impossible to perform an experiment to determine whether the hypothesis is supported by evidence.

Performing an Experiment

After forming a hypothesis, an experiment must be set up and performed to test the hypothesis. An experiment must have an independent variable (something that is manipulated by the person doing the experiment), and a dependent variable (the thing being measured which may be affected by the independent variable). All other variables must be controlled so that they do not affect the outcome. During an experiment, data is collected. Data is a set of values; it may be quantitative (e.g. measured in numbers) or qualitative (a description or generalization of the results).

For example, if you were to test the effect of sunlight on plant growth, the amount of light would be the independent variable (the thing you manipulate) and the height of the plants would be the dependent variable (the thing affected by the independent variable). Other factors such as air temperature, amount of water in the soil, and species of plant would have to be kept the same between all of the plants used in the experiment so that you could truly collect data on whether sunlight affects plant growth. The data that you would collect would be quantitative – since you would measure the height of the plant in numbers.

Analyzing Data

After performing an experiment and collecting data, one must analyze the data. Research experiments are usually analyzed with statistical software in order to determine relationships among the data. In the case of a simpler experiment, one could simply look at the data and see how they correlate with the change in the independent variable.

Forming a Conclusion

The last step of the scientific method is to form a conclusion. If the data support the hypothesis, then the hypothesis may be the explanation for the phenomena. However, multiple trials must be done to confirm the results, and it is also important to make sure that the sample size—the number of observations made—is big enough so that the data is not skewed by just a few observations.

If the data do not support the hypothesis, then more observations must be made, a new hypothesis is formed, and the scientific method is used all over again. When a conclusion is drawn, the research can be presented to others to inform them of the findings and receive input about the validity of the conclusion drawn from the research.

Scientific Method Examples

There are very many examples of the use of the scientific method throughout history because it is the basis for all scientific experiments. Scientists have been conducting experiments using the scientific method for hundreds of years.

One such example is Francesco Redi’s experiment on spontaneous generation. In the 17 th Century, when Redi lived, people commonly believed that living things could spontaneously arise from organic material. For example, people believed that maggots were created from meat that was left out to sit. Redi had an alternate hypothesis: that maggots were actually part of the fly life cycle!

He conducted an experiment by leaving four jars of meat out: some uncovered, some covered with muslin, and some sealed completely. Flies got into the uncovered jars and maggots appeared a short time later. The jars that were covered had maggots on the outer surface of the muslin, but not inside the jars. Sealed jars had absolutely no maggots whatsoever.

Redi was able to conclude that maggots did not spontaneously arise in meat. He further confirmed the results by collecting captured maggots and growing them into adult flies. This may seem like common sense today, but back then, people did not know as much about the world, and it is through experiments like these that people uncovered what is now common knowledge.

Scientists use the scientific method in their research, but it is also used by people who aren’t scientists in everyday life. Even if you were not consciously aware of it, you have used the scientific method many times when solving problems around you.

For example, say you are at home and a lightbulb goes out. Noticing that the lightbulb is out is an observation. You would then naturally question, “Why is the lightbulb out?” and come up with possible guesses, or hypotheses. For example, you may hypothesize that the bulb has burned out. Then you would perform a very small experiment in order to test your hypothesis; namely, you would replace the bulb and analyze the data (“Did the light come back on?”).

If the light turned back on, you would conclude that the lightbulb had, in fact, burned out. But if the light still did not work, you would come up with other hypotheses (“The socket doesn’t work”, “Part of the lamp is broken,” “The fuse went out”, etc.) and test those.

1. Which step of the scientific method comes immediately after making observations and asking a question?

2. A scientist is performing an experiment to determine if the amount of light that rodents are exposed to affects their sleep cycle. She places some rodents in a room with 12 hours of light and 12 hours of darkness, some in a room with 24-hour light, and some in 24-hour darkness. What is the independent variable in this experiment?

3. What is the last step of the scientific method?

Enter your email to receive results:

Cite This Article

Subscribe to our newsletter, privacy policy, terms of service, scholarship, latest posts, white blood cell, t cell immunity, satellite cells, embryonic stem cells, popular topics, hydrochloric acid, cellular respiration, horticulture, adenosine triphosphate (atp).

3 Science Experiments Using the Scientific Method

July 1, 2018

If you’ve ever conducted a scientific experiment with your child, you’ve probably noticed that most kid-friendly experiments rely heavily upon observation. For most of your child’s life, he or she has been asked to observe a reaction or result of an experiment to learn about the underlying scientific principal. 

What Exactly is Scientific Method? 

You’ve probably heard of the scientific method, and used it yourself when you were in school. The scientific method is used by scientists to ensure that the results of their experiments are reliable and valid. When kids use the scientific method, they learn more and think critically, asking questions and making predictions about their experiments. 

Scientists start with a question they want to answer, which serves as a goal and sets a purpose for the experiment. This is the most important part! Every experiment should start with a big question that guides the research being conducted. Next, participants form a hypothesis , or a prediction based on prior knowledge. After gathering materials needed for the experiment, the procedure is conducted and scientists make observations and record data and results. Finally, a conclusion is reached and published. 

Your child can emulate this process at home by simply modifying each experiment to include a big question, and a hypothesis that will drive their experiment and process. Let’s explore ideas for 3rd grade science experiments using the scientific method! Learn more how to incorporate science into your child's routine . 

  Experiment 1. Which Liquids Melt the Fastest? 

Simple scientific method experiments should be easy and fun and include everyday supplies you can find in your own home! This experiment will help your child understand how various household liquids melt at different rates.

For detailed instructions to complete this experiment, and for even more information on using the scientific method with your child, check out the full video guided by Kids Academy teacher!

Materials Needed: 

• Different liquids, like milk, water, iced tea, and orange juice
• Ice cube trays

Experiement 2. How Does Water Travel from Roots to Leaves? 

Has your child ever wondered why a plant’s leaves are supple and moist? When we water plants, how does the moisture travel from the roots to the leaves? This experiment teaches kids about the xylem tubes that transport water through plants through the process of capillary action! 

• 3 clear, glass jars
• 3 different colors of food coloring
• 3 celery stalks

Step 1: Help your child form a big question before getting started. 

Step 2: Encourage your child to make predictions based on their prior knowledge. For instance, plants are alive, just like people, and may have cells or structures that transport the water from roots to tip.

Step 3: Gather the above materials.

Step 4: Cut the bottom off the celery stalks, about one inch from the base.

Step 5: Fill each jar about half way with water. Add a few drops of food coloring to each, ensuring that each jar is a different color. 

Step 6: Place a stalk of celery in each jar and let them sit for about 20 minutes to an hour. 

Step 7: Observe the results! Rip apart the stalks to see how the coloring travels through each stalk. Notice how the color reaches the leaves at the very tip of the stalk! 

Step 8: Record results and help your child draw a conclusion. 

Explain to your child that xylem tubes are structures in plants that carry water from the roots up through the tip of the plant. This process is called capillary action, and it works a lot like a straw sucking the water up through the plant! 

Play & Learn Science

  Experiment 3. Does it Dissolve? 

Do all substances dissolve in water? Kids explore the varying levels of solubility of common household substances in this fun-filled experiment! 

• 4 clear, glass jars filled with plain tap water
• Talcum or baby powder
• Granulated sugar

Step 1: Help your child form a big question before starting the experiment. 

Step 2: Make a hypothesis for each substance. Perhaps the salt will dissolve because your child has watched you dissolve salt or sugar in water when cooking. Maybe the baby powder will not dissolve because of its powdery texture. Help your child write down his or her predictions. 

Step 3: Scoop a teaspoon of each substance in the jars, only adding one substance per jar. Stir it up! 

Step 4: Observe whether or not each substance dissolves and record the findings! 

Your child will likely note that that sugar and salt dissolve, while the flour will partially dissolve, and the baby powder will remain intact. The grainy crystals of the sugar and salt are easily dissolved in water, but the dry, powdery substances are likely to clump up or remain at the bottom of the jar.  

As you can see, the scientific method is easy to work into your child’s scientific experiments. Not only does it increase your child’s scientific learning and critical thinking skills, but it sparks curiosity and motivates kids as they learn to ask questions and prove their ideas! Get started today with the above ideas, and bring the scientific method home to your child during your next exciting science experiment! 

  Play & Learn Science

Related Articles

Cancel anytime

You'll be able to manage the favorite spreadsheets list.

You’ll be able to hide/mark the accomplished tasks.

• School / District Account
• Family Account
• 2 PDF worksheets per day
• Interactive worksheets
• Targeted ads
• KidsAcademy ads

$ 1.99 / month

• Printable and interactive worksheets
• Learning videos
• Ad-free browsing

• Engage students and save time with ready-to-use premium educational activities.
• Unlimited Learning Library access

$9.99 / month

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 17 September 2024

Improving rigor and reproducibility in western blot experiments with the blotRig analysis

• Cleopa Omondi 1 ,
• Austin Chou 1 ,
• Kenneth A. Fond 1 ,
• Kazuhito Morioka 1 ,
• Nadine R. Joseph 1 ,
• Jeffrey A. Sacramento 1 ,
• Emma Iorio 1 ,
• Abel Torres-Espin 1 , 3 , 4 ,
• Hannah L. Radabaugh 1 ,
• Jacob A. Davis 1 ,
• Jason H. Gumbel 1 ,
• J. Russell Huie 1 , 2 &
• Adam R. Ferguson 1 , 2  

Scientific Reports volume  14 , Article number:  21644 ( 2024 ) Cite this article

Metrics details

• Biochemistry
• Biological techniques
• Computational biology and bioinformatics
• Neuroscience

Western blot is a popular biomolecular analysis method for measuring the relative quantities of independent proteins in complex biological samples. However, variability in quantitative western blot data analysis poses a challenge in designing reproducible experiments. The lack of rigorous quantitative approaches in current western blot statistical methodology may result in irreproducible inferences. Here we describe best practices for the design and analysis of western blot experiments, with examples and demonstrations of how different analytical approaches can lead to widely varying outcomes. To facilitate best practices, we have developed the blotRig tool for designing and analyzing western blot experiments to improve their rigor and reproducibility. The blotRig application includes functions for counterbalancing experimental design by lane position, batch management across gels, and analytics with covariates and random effects.

Similar content being viewed by others

A critical path to producing high quality, reproducible data from quantitative western blot experiments

A data analysis framework for combining multiple batches increases the power of isobaric proteomics experiments

OmicScope unravels systems-level insights from quantitative proteomics data

Introduction.

Proteomic technologies such as protein measurement with folin phenol reagent were first introduced by Lowry et al. in 1951 1 . The resulting qualitative data are typically confirmed by a second, independent method such as western blot (WB) 2 , 3 . The WB method, first described by Towbin et al. 4 and Burnette 5 in 1979 and 1981, respectively, uses specific antibody-antigen interactions to confirm the protein present in the sample mixture. Quantitative WB (qWB assay) is a technique to measure protein concentrations in biological samples with four main steps: (1) protein separation by size, (2) protein transfer to a solid support, (3) marking a target protein using proper primary and secondary antibodies for visualization, and (4) semi -quantitative analysis 6 . Importantly, qWB data is considered semi -quantitative because methods to control for experimental variability ultimately yield relative comparisons of protein levels rather than absolute protein concentrations 2 , 3 , 7 , 8 . Similarly, western blotting applying ECL (enhanced chemiluminescence) is considered a semi-quantitative method because it lacks cumulative luminescence linearity and offers limited quantitative reproducibility 9 . However, the emergence of highly sensitive fluorescent labeling techniques, which exhibit a wider quantifiable linear range, greater sensitivity, and improved stability when compared to the conventional ECL detection method, now permits the legitimate characterization of protein expression as linearly quantitative 10 . Current methodologies do not sufficiently account for diverse sources of variability, producing highly variable results between different laboratories and even within the same lab 11 , 12 , 13 . Indeed, qWB data exhibits more variability compared to other experimental techniques such as enzyme linked immunosorbent assay (ELISA) 14 . For example, results have shown that qWB can produce significant variability in detecting host cell proteins and lead to researchers missing or overestimating true biological effects 15 . This in turn results in publication of irreproducible qWB interpretations, which leads to loss of its credibility 13 . In the serious cases, qWB results may even provide clinical misdiagnosis 16 that could impact on a larger public health concern due to the prevalence of WB in biomedical research, such as diagnosis of SARS-CoV2 infection 17 .

The process of recognizing and accounting for variability in WB analyses will ultimately improve reproducibility between experiments. A growing body of studies has shown that this requires a fundamental shift in the experimental methodology across data acquisition, analysis, and interpretation to achieve precise and accurate results 2 , 3 , 11 , 12 , 13 .

Here we highlight experimental design practices that enable a statistics-driven approach to improve the reproducibility of qWBs. Specifically, we discuss major sources of variability in qWB including the non-linearity in antibody signal 2 , 3 ; imbalanced experimental design 13 ; lack of standardization in the treatment of technical replicates 3 , 18 ; and variability between protein loading, lanes, and blots 2 , 7 , 19 . To address these issues, we provide new comprehensive suggestions for quantitative evaluation of protein expression by combining linear range characterization for antibodies, appropriate counterbalancing during gel loading, running technical replicates across multiple gels, and by taking careful consideration of the analysis method. By applying these experimental practices, we can then account for more sources of variability by running analysis of covariance (ANCOVA) or generalized linear mixed models (LMM). Such approaches have been shown to successfully improve reproducibility compared to other methods 13 .

Good options for qWB protein bands analysis using free, downloadable tools are available for researchers. Amongst others, LI-COR Image Studio Lite can be used to measure the intensity of protein bands in western blots and calculate their relative abundance . Likewise, ThermoFisher ImageQuant Lite offers features such as the ability to perform background subtraction and normalization. However, to date, no specific tools are freely available to provide a map to counterbalance samples, which overcome imperfect uniform protein electrophoresis/transfer and perform statistical analysis. Here, we present blotRig, a tool for researchers with functionalities to counterbalance samples and perform statistical analysis.

To help improve WB rigor we developed the blotRig protocol and application harnessing a database of 6000 + western blots from N = 281 subjects (rats and mice) collected by multiple UCSF labs on core equipment. To demonstrate blotRig best practices in a real-world experiment, we carried out prospective multiplexed WB analysis of protein lysate from lumbar cord in rodent models of spinal cord injury (SCI) (N = 29 rats) in 2 groups (experimental group & control group). In order to show that these experimental suggestions could improve qWB reproducibility, we compared different statistical approaches to handling loading controls and technical replicates. Specifically, we applied two strategies to integrate loading controls: (i) normalizing the target protein levels by dividing by the loading control or (ii) treating the loading control as a covariate in a LMM. Additionally, we analyzed technical replicates in four ways: (1) assume each sample was only run once without replication, (2) treat each technical replicate as an independent sample, (3) use the mean of the three technical replicate values, and 4) treat the replicate as a random effect in a LMM. Altogether, we found that the statistical power of the experiment was significantly increased when we used loading control as a covariate with technical replicates as a random effect during analysis. In addition, the effect size was increased, and the p-value of our analysis decreased when using this LMM, suggesting the potential for greater sensitivity in our WB experiment when using this approach 20 . Through rigorous experimental design and statistical analysis we show that we can account for greater variability in the data and more clearly identify underlying biological effects.

Materials and methods

All experiments protocol were approved by the University Laboratory Animal Care Committee at University of California, San Francisco (UCSF, CA, USA) and followed the animal guidelines of the National Institutes of Health Guide for the Care and Use of Laboratory animals (National Research Council (US) Committee for the Update of the Guide for the Care and Use of Laboratory Animals, 2011). We followed The ARRIVE guidelines (Animal Research: Reporting In Vivo Experiments) to describe our in vivo experiments.

Male Simonsen Long Evans rats (188–385 g; Gilroy (Santa Clara, CA, USA), (N = 29) aged 3 weeks were housed under standard conditions with a 12-h light–dark cycle (6:30 am to 6:30 pm) and were given food and water ad libitum. The animals were housed mostly in pairs in 30 × 30 × 19-cm isolator cages with solid floors covered with a 3 cm layer of wood chip bedding. The experimenters were blind to the identity of treatments and experimental conditions, and all experiments were designed to minimize suffering and limit the number of animals required.

Anesthesia and surgery

We performed non-survival spinal cord injury and spared nerve injury surgeries on animals. Specifically, 3 week old female rats were anesthetized with continuous inhalation of isoflurane (1–5% mg/kg) while on oxygen (0.6–1 mg/kg) in accordance with the IACUC surgical and anesthesia guidelines. Preoperative 0.5% lidocaine local infiltration was applied once at surgical site, avoiding injection into muscle. Fur over the T7–T9 thoracic level was shaved. The dorsal skin was aseptically prepared with surgical iodine or chlorhexidine and 70% ethanol. A small longitudinal incision was made along the spine through the skin, fascia, and muscle to expose the T7-T9 vertebrae. Animals undergoing sham procedure did not undergo laminectomy and immediately proceeded to wound closure. Overlying muscle and subcutaneous tissue was sutured closed using an absorbable suture in a layered fashion. External skin was reinforced using monofilament suture or tissue glue as needed. Animals were euthanized after 30 min to extract spinal cord tissue through fluid expulsion.

Experimental methodology

In accordance with established quality standards for preclinical neurological research 21 , experimenters were kept blind to experimental group conditions throughout the entire study. Western blot loading order was determined a priori by a third-party coder, who ensured that a representative sample from each condition was included on each gel in a randomized block design. The number of subjects per condition was kept consistent across groups for each experiment to ensure that proper counterbalancing could be achieved across independent western runs. All representative western images presented in the figures represent lanes from the same gel. Sometimes, the analytical comparisons of interest were not available on adjacent lanes even though they come from the same gel because of our randomized counterbalancing procedure.

• Western blot

The example western blot data used in this paper are taken from a model of spared nerve injury in animals with spinal cord injury. The nerve injury model used is based on models from pain literature 22 , where two of the three branches of the sciatic nerve are transected, sparing the sural nerve (SNI) 23 . Two surgeons perform the procedure simultaneously, with injuries occurring 5 min apart. The spinal cord of animals was obtained based on fluid expulsion model 24 and a 1 cm section of the lumbar region was excised at the lumbar enlargement section. The tissue was then preserved in a -80 degree freezer until it was needed for an experiment, at which point it was thawed and used to run a Western blot. We conducted a Western blot analysis on 29 samples from animals using standard biochemical methods. We measured the protein levels of the AMPA receptor subunit GluA2 and used beta-actin as a loading control. The data from these experiments was then aggregated and used for statistical analysis.

Protein assay

We assayed sample protein concentration using a bicinchoninic acid (BCA assay (Pierce) for reliable quantification of total protein using a plate reader (Tecan; GeNios) with triplicate samples (technical replicates) detected against a Bradford Assay (BSA) standard curve. Technical replicates are multiple measurements that are performed under the same conditions in order to quantify and correct for technical variability and improve the accuracy and precision of the results (48). We ran the same WB loading scheme three times (technical replicates of the entire gel) and measured the protein levels of AMPA receptors.

Polyacrylamide gel electrophoresis and multiplexed near-infrared immunoblotting

The approach involved performing serial 1:2 dilutions with cold Laemmli sample buffer in room temperature; 15 μg of total protein per sample was loaded into separate lanes on a precast 10–20% electrophoresis gel (Tris–HCl polyacrylamide, BioRad) to establish linear range (Fig.  1 ). The blotRig software helps counterbalance sample positions across the gel by treatment condition. (Fig.  2 ). A kaleidoscope ladder was loaded on the first lane of each gel to confirm molecular weight (Fig.  2 ). The gel was electrophoresed for 30 min at 200 V in SDS buffer (25 mm Tris, 192 mm glycine, 0.1% SDS, pH 8.3; BioRad). Protein was transferred to a nitrocellulose membrane in cold transfer buffer (25 mm Tris, 192 mm glycine, 20% ethanol, pH 8.3). Membrane transfer was confirmed using Ponceau S stain (67) followed by a quick rinse and blocking in Odyssey blocking buffer (Li-Cor) containing Tween-20.

Determining linear range of antibodies to optimize parametric analysis of Western blot data. When small or large protein concentrations are loaded, there is often a possibility that their representation on western blot band density may become non-linear. If there is a disconnect between the observed and expected protein concentrations, results may be inaccurate. Thus determining the linear range wherein, a one-unit increase in protein is reflected in a linear increase in band density for each western blot antibody is a crucial initial step to ensure confidence in reproducibility of the linear models commonly applied to western blot data analysis.

Counterbalancing to reduce bias. ( A) Experimental design. A simple hypothetical experimental design for illustrating counterbalancing. Two experimental groups (Wild Type vs Transgenic), with two treatments (Drug vs Vehicle) analyzed within each individual. This 2 (Experimental Condition) by 2 (Tissue Area) design yields four groups. ( B) Counter-balanced Gel Loading. The goal of appropriate counterbalancing is to optimize the sequence in which samples are loaded such that groups are represented equally across the gel. Those with red X have with the experimental groups and treatment condition grouped in the same area of the gel, and thus variability across the gel may be conflated with group differences. In contrast, those with the green check are organized so that experimental condition and treatment condition are better placed to reduce the possibility of any single group being over-represented in a particular area of the gel.

The membrane was blocked for 1 h in Odyssey Blocking Buffer (Li-Cor) containing 0.1% Tween-20, followed by an overnight incubation in primary antibody solution at 4 °C. Membrane incubation was done in a primary antibody solution containing Odyssey blocking buffer, Tween-20, appropriate primary antibody receptor targeting1:2000 mouse PSD-95 (cat # MA1-046,Thermofisher), 1:200 rabbit GluA1 (cat # AB1504, Millipore), 1:200 rabbit GluA2 (cat # AB1766, Millipore), 1:200 rabbit pS831(cat # 04–823, Millipore), 1:200 p880 (cat#07–294, Millipore) or 1:1,500 mouse actin loading control (cat # 612,857, BD Transduction)]. Following incubation, the membrane was washed 4 × 5 min with Tris-buffered saline containing 0.1% Tween 20 (TTBS) and incubated in fluorescent-labeled secondary antibody (1:30 K LiCor IRdye appropriate goat anti-rabbit in Odyssey blocking buffer plus 0.2% Tween 20) for 1 h in the dark. Subsequent to 4 × 5 min washes in TTBS, followed by a 5 min wash in TBS.

Membrane incubation was used to detect the presence of a specific protein or antigen on a membrane. In this case, the membrane was incubated with a fluorescently labeled secondary antibody solution that was specifically tuned to the emission spectra of the laser lines used by the Li-Cor Odyssey quantitative near-infrared molecular imaging system instrument. This allows for specific detection of the protein of interest on the membrane. The sample is then imaged using an infrared imaging system that is optimized for detecting the specific wavelengths of light emitted by the fluorescent label. Additional rounds of incubation and imaging are performed to detect additional proteins using the multiplexing functionality of the Li-Cor instrument, with each round adding new bands at different molecular weight ranges. This allows for the detection of multiple proteins in the same sample, maximizing the proteomic detection.

Quantitative near-IR densitometric analysis

Using techniques optimized in the our lab 25 , 26 , we established near-infrared labeling and detection techniques (Odyssey Infrared Imaging System, Li-Cor) to quantify linear intensity detection of fluorescently labeled protein bands. The biochemistry is performed in a blinded, counterbalanced fashion, and three independent replications of the assay are run on different days 27 . Fluorescent Western blotting utilizes fluorescent-labeled secondary antibodies to detect the target protein, which allows for more sensitive and specific detection compared to chemiluminescence 11 , 28 , 29 . Additionally, fluorescence imaging allows multiple detection of a target protein and internal loading control in the same blot, which enables more accurate correction of sample-to-sample and lane-to-lane variation 11 , 30 , 31 . This provides a more accurate and reliable quantification of the target protein, making it a popular choice for quantitative analysis of WB data.

It is good practice for the pipetting experimenter to remain blind to experimental conditions during gel loading, transfer, and densitometric quantification. We achieved this using de-identified tube codes and a priori gel loading sequences that were developed by an outside experimenter using the method implemented in the blotRig software.

Statistical analyses

Statistical analyses were performed using the R statistical software. Our WB data was analyzed using parametric statistics. The WB was run using three independent replications and covariance corrected by beta-actin loading control, with replication statistically controlled as a random factor. Significance was assessed at p < 0.05 25 , 26 , 32 , 33 , 34 . We report estimated statistical power and standardized regression coefficient effect sizes in the results section.

All ANOVAs were run using the stats R package; standardized effect size was calculated using the parameters R package 35 . Linear mixed models were run using the lme4 R package. Observed power was calculated by Monte Carlo simulation (1000x) run on the fitted model (either ANOVA or LMM) using the simR package 36 . For the development of the blotRig interface, the R packages used included: shiny, tidyverse, DT, shinythemes, shinyjs, and sortable ) 37 , 38 , 39 , 40 , 41 , 42 . You can access the blotRig analysis software, which includes code for inputting experimental parameters for all Western blot analysis, through the following link: https://atpspin.shinyapps.io/BlotRig/.

Designing reproducible western blot experiments

Determining linear range for each primary antibody.

Most WB analyses assume semi -quantitatively that the relationship between qWB assay optical density data (i.e. western band signal) and protein abundance is linear 2 , 3 , 11 , 18 . Accordingly, most qWB analyses use statistical tests (t-test; ANOVA) that assume a linear effect. However, recent studies have shown that the relationship can potentially be highly non-linear 19 As Fig.  1 illustrates, the WB band signal can become non-linearly correlated with protein concentrations at low and high values. This may result in inaccurate quantification of relative target protein amount in the experiment and violates the assumptions for linear model which can lead to false inferences. To address the assumption of linearity, it is important to first determine the optimal linear range for each protein of interest so that one can be confident that a unit change in band density reflects a linear change in protein concentration. This enables an experimenter to accurately quantify the protein of interest and apply linear statistical methods appropriately for hypothesis testing.

Counterbalancing during experimental design

Counterbalancing is the practice of having each experimental condition represented on each gel and evenly distributing them to prevent overrepresentation of the same experimental groups in consecutive lanes. For example, imagine an experimental design in which we are studying two experimental groups (wild type and transgenic animals) and are also looking at two treatment conditions (Drug and Vehicle). The best way to determine the effects and interactions between our experimental and treatment groups would be to create a balanced factorial design. A factorial design is one in which all combinations of levels across factors are represented. For the current example, a balanced factorial design would produce four groups, covering each possible combination (Drug-treated Wild Type, Vehicle-treated Wild Type, Drug-Treated Transgenic and Vehicle-treated Transgenic) (Fig.  2 A). During WB gel loading, experimenters often distribute their samples unevenly such that certain experimental conditions may be missing on some gels or samples from the same experimental condition are loaded adjacently on a gel. This is problematic because we know that polyacrylamide gel electrophoresis (PAGE) gels are not perfectly uniform, reflecting a source of technical variability 43 ; in the worst case, if we have only loaded a single experimental group on a gel and found a significant effect of the group, we cannot conclude if the effect is due to the experimental condition or a technical problem of the gel. At minimum, experimenters should ensure that every group in a factorial design is represented on each gel to avoid confounding technical gel effects with experimental differences. If the number of combinations is too large to represent on a single gel because of the number of factors or the number of levels of the factors, then a smaller "fractional factorial" design will provide maximal counterbalancing to ensure unbiased estimates of all factor effects and the most important interactions.

In addition, experimenters can further counter technical variability by arranging experimental groups on each gel to ensure adequately counterbalanced design assuming the uniformed protein concentration and fluid volume of all samples. This importantly addresses the variability due to physical effects within an individual gel. In our example, this means alternating the tissue areas and experimental conditions as much as possible to minimize similar samples from being loaded next to one another (Fig.  2 B). By spreading the possibility of technical variability across all samples by counterbalancing across and within gels, we can mitigate potential technical effects that can bias our results. Proper counterbalancing also enables us to implement more rigorous statistical analysis to account for and remove more technical variability 25 , 26 , 32 , 33 . Overall, this will help to ensure that experimenters can find the same result in the future and improve reproducibility.

Technical replication

Technical replicates are used to measure the precision of an assay or method by repeating the measurement of the same sample multiple times. The results of these replicates can then be used to calculate the variability and error of the assay or method 13 . This is important to establish the reliability and accuracy of the results. Most experimenters acknowledge the importance of running technical replicates to avoid false positives and negatives due to technical error 13 . Even beyond extreme results, technical replicates can account for the differences in gel makeup, human variability in gel loading, and potential procedural discrepancies. In fact, most studies run at least duplicates; however, the experimental implementation of replicates (e.g., running replicates on the same gel or separate gels) as well as the statistical analysis of replicates (e.g., dropping “odd-man-out” or taking the mean or standard deviation) can differ greatly 44 , 45 . This experimental variability ultimately impedes our ability to meaningfully compare results. For experimenters to establish accuracy and advance reproducibility in WB experiments, it is important to implement standardized and rigorous protocols to handle technical replicates 11 , 13 . In doing so, we can further reduce the technical variability with statistical methods during analysis.

As underscored previously, we recommend that technical replicates are counterbalanced on separate gels to mitigate any possible gel effect. Additionally, by running triplicates, we can treat replicates as a random effect in a LMM during statistical analysis. Importantly, triplicates provide more values to measure the distribution of technical variance to ensure the robustness of the LMM than only running duplicates. This approach isolates and removes technical variance from biological variation which ultimately improves our sensitivity for true experimental effects 46 .

In the following demonstration of statistical methods, we replicated all WB analyses in triplicate with a randomized counterbalanced design. We then explore how the way in which technical replicates and loading controls are incorporated into analysis can have a significant impact on both the sensitivity of our results and the interpretation of the findings. An example mockup of a dataset illustrating the various ways in which western blot data are typically prepared for analysis can be found in Fig.  3 .

Western Blot Gel and Replication Strategies. ( A) Illustration of Western Blot Gel. This depiction of a typical multiplexed western blot gel highlights the antibody-labeled target protein bands of interest (green/yellow) and housekeeping protein loading control that is always run and quantified in the same sample and lane as the target of interest. Total protein stain (fluorescent ponceau stain) is shown in red can can be used as an alternative loading control. Specific, quantification is typically executed on a single antibody-labeled channel for the target protein and housekeeping protein loading control (gray scale image). ( B) Balanced Factorial Technical Replicate Strategy. Here we show the western blot data for the first 3 subjects from an example dataset. In a balanced factorial design, an equal number of samples from all possible experimental groups are represented on each gel. This table shows the subject number, the technical replicate, experimental group, and the band quantifications for both the target protein and the loading control. A ratio of target protein and loading control is also calculated. ( C) Other Common Technical Replicate Strategies. In this example table are two of the other ways western blot data are typically formatted. Some experimenters choose to not include technical replicates, with only one sample from each subject quantified. In another replication strategy, technical replicates are averaged. Averaging may bias or skew the data. We recommend running technical replicates on separate gels or batches, and using gel/batch as a random factor when analyzing western blot data.

Statistical methodology to improve western blot analysis

Loading control as a covariate.

Most qWB assay studies use loading controls (either a housekeeping protein or total protein within lane) to ensure that there are no biases in total protein loaded in a particular lane 2 , 11 , 27 . The most common way that loading controls are used to account for variability between lanes is by normalizing the target protein expression values by dividing it by the loading control values (Fig.  3 ) , resulting in a ratio between target protein to loading control 2 , 47 , 48 . However, ratios may violate assumptions of common statistical test used to analyze qWB (e.g., t-test, ANOVA, etc.) 49 This ultimately hinders the ability to statistically account for the variance in qWB outcomes and have a reliable estimate of the statistics. An alternative approach to improve the parametric properties would be to include loading control values as a covariate—a variable that is not our experimental factors but that may affect the outcome of interest and presents a source of variance that we may account for 50 . For instance, we know the amount of protein loaded is a source of variability in WB quantification, so we can use the loading control as a covariate to adjust for that variance. In doing so, we extend the method of ANOVA into that of ANCOVA 51 . This approach accounts for the technical variability present between lanes while meeting the necessary assumptions for parametric statistics which helps curb bias and averts false discoveries.

Replication and subject as a random effect

Most WB studies use ANOVA, a test that allows comparison of the means of three or more independent samples, for quantitative analysis of WB data 49 . One of the assumptions in ANOVA is the independence of observations 49 . This is problematic because we often collect multiple observations from the same analytical unit, for example different tissue samples from a single subject, or technical replicates. As a result, those observations don’t qualify as independent and should be analyzed using models controlling for variability within units of observations (e.g., the animal) to mitigate inferential errors (false positives and negatives) 52 caused by what is known as pseudoreplication. This arises when the quantity of measured values or data points surpasses the number of actual replicates, and the statistical analysis treats all data points as independent, resulting in their full contribution to the final result 53 .

In addition, when conducting experiments, it is important to consider the randomness of the conditions being observed. Treating both subjects and conditions as fixed effects can lead to inaccurate p-values. Instead, subjects/ animals should be treated as random effects and the conditions should be considered as a sample from a larger population 54 . This is especially important when collecting data from different replicates or gels, as the separate technical replicate runs should be considered as random.

In Fig.  4 we use a simple experimental design comparing the difference in a target protein between two experimental groups to demonstrate four of the most common ways researchers tend to analyze western blot data: (1) running each sample once without replication, (2) treating each technical replicate as an independent sample, (3) taking the mean of technical replicate values, and (4) treating subject and replication as a random effect (Fig.  4 ). We then tested how effect size, power, and p value are affected by each of these strategies to get a sense of how much these estimates vary between analyses. For each of these strategies, we also tested the difference between using the ratio of target protein to loading controls versus using loading control as a statistical covariate. For further exploration of the way these data are prepared and analyzed, see the data workup in Supplementary Figs.  1 and 2 .

Effect of different replication and loading control strategies on statistical outcomes. Eight possible strategies are shown, representing the most common ways in which replication and loading controls are treated in a typical Western blot analysis. Four replication strategies: either no replication at all, 3 technical replicate gels treated as independent, mean of three replicates, or replicate treated as a random effect in a linear mixed model. These are crossed with two loading control strategies: either target protein is divided by loading control, or loading control is treated as a covariate in a linear mixed model. ( A) Effect Size: Standardized effect size coefficient is generally improved when loading control is treated as a covariate, compared to using a ratio of the target protein and loading control values. ( B) Power: By treating each replication as independent the statistical power is increased (due to the inaccurate assumption that technical replicates are not related, thus artificially tripling the n). Conversely, including the variability inherent in technical replicates as a part of the statistical model, we work to identify and account for a major source of variability, thus improving power in a more appropriate way. ( C) P value: As expected, when each replication is inaccurately treated as independent the p value is low (due to artificially inflated n). We found that using the mean of replications and loading controls as covariates also resulted in a p value below 0.05. The smallest p value was found when including replication as a random factor. Across each of these statistical measures, only when replication is included as a random factor and loading control as a covariate do we see a strong effect size, high power, and low p value.

In the first scenario, we imagined that no technical replication was run at all (by using only the first replication). With this strategy, we found that standardized effect size is weak, power is low, and the p value was high (Fig.  4 ). Second, we demonstrate how analytical output would be different if we did run three technical replicates, but treated each as independent. As discussed above, this strategy does not take into account the fact that each sample is being run three times, and consequently the overall n of your experiment is artificially tripled! As one might expect, observed power is quite high, and our p value is low (< 0.05). Power is increased by an increase in sample size, so it is not surprising that the power is much higher if we erroneously report that we have a 3X larger sample size (i.e., pseudoreplication) 53 . In this case, the observed power is inflated and an artifact of inappropriate statistics, and the probability of a false positive is considerably increased with respect to the expected 5%.

So, what would be a more appropriate way to handle technical replicates? One method that researchers often use is to take the mean of their technical replicates. This does ensure that we are not artificially inflating our sample size, which is certainly an improvement over the previous strategy. With this strategy, we do find that our p value is less than 0.05 (when loading control is treated as a covariate). But we also see that our power is still low. We have effectively taken our replicates into account by collapsing across them within each sample, but this can be dangerous. If there is wide variation across replicates of a particular sample, then taking the mean of three replicates could produce an inaccurate estimate of the ‘true’ sample value. Ideally, we want to find a solution where instead of collapsing this variation, we add it to our statistical model so that we can better understand what amount of variation is randomly coming from within technical replicates, and in turn what amount of variation is actually due to potential differences in our experimental groups.

To achieve this, we need to model both the fixed effect of all groups in a full factorial design, and the random effect of replication across western blot gels. When we use both fixed and random effects, this is referred to as a linear mixed model (LMM). When using this strategy, we find that our effect size remains strong, and our p value is low. But importantly, we now have strong observed power (Fig.  4 ). This suggests that we can achieve greater sensitivity in our WB experiment when using this approach . Specifically, if we implement careful counterbalancing while designing our experiments, then we can use the variability between gels to our advantage during analysis using linear mixed effects model 55 .

LMM is recommended because it takes into account both the multiple observations within a single subject/animal in a given condition and differences across subjects observed in multiple conditions. This reduces chances of inaccurate p-values and improves reliability 56 . Further, treating both subjects and replication as random effects generalizes the results to the population of subjects and also to the population of conditions 57 .

Real world application of blotRig software for western blot experimental design, technical replication, and statistical analysis

We have designed a user interface that is designed to facilitate appropriate counterbalancing and technical replication for western blot experimental design. The ‘blotRig’ application is run through RStudio, and can be found here: https://atpspin.shinyapps.io/BlotRig/ Upon starting the blotRig application, the user is prompted to upload a comma separated values (CSV) spreadsheet. This spreadsheet should include separate columns for subject ID and experimental group. The user is then prompted to enter the total number of lanes that are available on their particular western blot gel apparatus. The blotRig software will first run a quality check to confirm that each subject ID (unique sample or subject) is only found in one experimental group. If duplicates are found, a warning will be shown that specifies which subjects are repeated across groups. If no errors are found, a centered gel map will be generated that illustrates the western blot gel lanes into which each subject should be loaded (Fig.  5 A). The decision for each lane loading is based on two main principles outlined above: (1) each western blot gel should hold a representative sample of each experimental group (2) samples from the same experimental group are not loaded in adjacent lanes whenever possible. This ensures that proper counterbalancing is achieved so that we can limit the chances that the inherent variability within and across western blot gels is confounded with the experimental groups that we are interested in experimentally testing.

Example of the blotRig Gel Creator interface. ( A ) Illustration of the blotRig interface. User has entered their sample IDs, experimental groups, and the number of lanes per western blot gel. ( B) The blotRig system then creates a counterbalanced gel map that ensures each gel contains a representative from each experimental group. This illustration shows the exact lane for each gel in which each sample should be run.

Once the gel map has been generated, the user can then select to export this gel map to a CSV spreadsheet. This sheet is designed to clearly show which gel each sample is on, which lane on each gel a sample is found, what experimental group each sample belongs to, and importantly, a repetition of each of these values for three technical replicates (Fig.  5 B). User will also see columns for Target Protein and Loading Control. These are the cells where the user can then input their densitometry values upon completing their western blot runs. Once this spreadsheet is filled out, it is then ready to go for blotRig analysis.

To analyze western blot data, users can upload the completed template that was exported in the blotRig experimental design phase or their own CSV file under the ‘Analysis’ tab (Fig.  6 ). The blotRig software will first ask the user to identify which columns from the spreadsheet represent Subject/SampleID, Experimental Group, Protein Target, Loading Control, and Replication. The blotRig software will again run a quality check to confirm that there are no subject/sample IDs that are duplicated across experimental groups. If no errors are found, the data will then be ready to analyze. The blotRig analysis will then be run, using the principles discussed above. Specifically, a linear mixed-model runs using the lmer R package, with Experimental Group as a fixed effect, Loading Control as a covariate, and Replication (nested within Subject/Sample ID) as a random factor. Analytical output is then displayed, giving a variety of statistical results from the linear mixed model output table, including fixed and random effects and associated p values (Fig.  6 ). A bar graph of group means and 95% confidence interval error bars will also be generated, along with a summary of the group means, standard error of the mean, and upper/lower 95% confidence intervals. These outputs can be directly reported in the results sections of papers, improve the statistical rigor of published WB reports. In addition, since the entire pipeline is opensource, the blotRig code itself can be reported to support transparency and reproducibility.

Workflow for running statistical analysis of replicate western blot data using blotRig. First, fill out spreadsheet with subject ID, experimental group assignment, number of technical replication, the densitometry values for your target proteins and loading controls. After saving this spreadsheet as a.csv file, the file can be uploaded to blotRig. Tell blotRig the exact names of each of your variables, then click ‘Run Analysis’. This will produce a statistical output using linear mixed model testing for group differences using loading control as a covariate and replication as a random effect. Bar graph with error bars and summary statistics can then be exported.

Although the western blot technique has proven to be a workhorse for biological research, the need to enhance its reproducibility is critical 13 , 19 , 27 . Current qWB assay methods are still lacking for reproducibly identifying true biological effects 13 . We provide a systematic approach to generate quantitative data from western blot experiments that incorporates key technical and statistical recommendations which minimize sources of error and variability throughout the western blot process. First, our study shows that experimenters can improve the reproducibility of western blots by applying the experimental recommendations of determining the linear range for each primary antibody, counterbalancing during experimental design, and running technical triplicates 13 , 27 . Furthermore, these experimental implementations allow for application of the statistical recommendations of incorporating loading controls as covariates and analyzing gel and subject as random effects 58 , 59 . Altogether, these enable more rigorous statistical analysis that accounts for more technical variability which can improve the effect size, observed power, and p-value of our experiments and ultimately better identify true biological effects.

Biomedical research has continued to rely on p-values for determining and reporting differences between experimental groups, despite calls to retire the p-value 60 . Power (sensitivity) calculations have also become increasingly common. In brief, p-values and the related alpha value are associated with Type I error rate—the probability of rejecting the null hypothesis (i.e., claiming there is an effect) when there is no true effect 61 . On the other hand, power effectively measures the probability of rejecting the null hypothesis (i.e. stating there is not effect) when there is indeed a true underlying effect—a concept that is closely related to reducing the Type II error rate 59 , 62 . Critically, empirical evidence estimates that the median statistical power of studies in neuroscience is between ∼ 8% and ∼ 31%, yet best practices suggest that an experimenter should aim to achieve a power of 80% with an alpha of 0.05 20 . By being underpowered, experiments are at higher likelihood of producing a false inference. If an underpowered experiment is seeking to reproduce a previous observation, the resulting false negative may throw into question the original findings and directly exacerbate the reproducibility crisis 59 . Even more alarmingly, a low power also increases the likelihood that a statistically significant result is actually a false positive due to small sample size problems 61 . In our analyses, we show that our technical and statistical recommendations lower the p-value (indicating that the observed relationship between variables is less likely to be due to chance) as well as observed power of our experiments. This translates into the ability to better avoid false negatives when there is a true effect as well as reduce the likelihood of false positives when there is not a true experimental effect, both of which will ultimately improve the reproducibility of qWB assay experiments.

Another useful component of statistical analyses that is not as commonly reported but is critically related to p-value and power is effect size. Effect size is a statistical measure that describes the magnitude of the difference between two groups in an experiment 63 . It is used to quantify the strength of the relationship between the variables being studied 63 . The estimated effect size is important because it answers the most frequent question that researchers ask: how big is the difference between experimental groups, or how strong is the relationship or association? 63 . The combination of standardized effect size, p-value and power reflect crucial experimental results that can be broadly understood and compared with findings from other studies 62 , thus improving comparability of qWB experiments 49 , 63 . In particular, studies with large effect sizes have more power: we are more likely to detect a true positive experimental effect and avoid the false negative if the underlying difference between experimental groups is large 46 . In some cases, the calculated effect size is greatly influenced by how sources of variance are handled during analysis 13 . Our results demonstrate that by reducing the residual variance (by modeling the random effect of replication) the estimated effect size of our experiment increases. This could mean that the magnitude of the difference between the groups in our experiment is larger than it was originally thought to be. This could be due to a variety of factors such as improving the experimental design, sample size, or the measurement of the variables 13 . Likewise, conducting a power analysis is an essential step in experimental design that should be done before collecting data to ensure that the study is adequately powered to detect an effect of a certain size 64 .

Increasingly, power analysis is becoming a requirement for publications and grant proposals 65 . This is because a study with low statistical power is more likely to produce false negative results, which means that the study may fail to detect a real effect that actually exists. This can lead to the rejection of true hypotheses, wasted resources, and potentially harmful conclusions. In brief, given an experimental effect size and variance, we can calculate the sample size needed to achieve an alpha of 0.05 and power of 0.8; an increased sample size reduces the standard error of mean (SEM), which is the measured spread of sample means and consequently increases the power of the experiment 66 . We have demonstrated that our experimental and statistical recommendations lead to a lower p value (Fig.  3 C) and effect size (Fig.  3 B) without changing the sample size. This may be of greatest interest to researchers: more rigorous analytics ultimately improves experimental sensitivity without relying solely on increasing the sample size.

Reducing the sample size of an experiment can be beneficial for several reasons, one of which is cost-effectiveness. A smaller sample size can lead to a reduction in the number of animals or other resources that are needed for the study, which can result in lower costs. Additionally, it can also save time and reduce the duration of the experiment, as fewer subjects need to be recruited, and the data collection process can be completed more quickly. However, it is important to note that reducing the sample size can also lead to decreased statistical power. As a result, reducing sample size too much can increase the risk of a type II error, failing to detect significance when there is a true effect 62 .Therefore, it is important to consider the trade-off between sample size and power when designing an experiment, and to use statistical techniques like power analysis to ensure that the sample size is sufficient to detect an effect of a certain size. Moreover, when using animals in research, it's always important to consider the ethical aspect and the 3Rs principles of reduction, refinement, and replacement 55 .

Despite our best efforts in creating a balanced, full factorial experimental design, there will always be random variation in biological experiments. Fixed effects such as experimental group differences are expected to be generalizable if the experiment is replicated. Random effects (such as gel variation) on the other hand are unpredictable across experiments. Western blot analyses are particularly susceptible to this random gel variation, as different values may be observed for technical replicates run on different gels. By using a linear mixed model paired with rigorous full factorial design, we can ensure that we account for as much of that random variation as possible. When we acknowledge, identify, and model random effects we enhance the possibility of discovering our fixed effect of experimental treatment, if one exists.

The linear mixed model framework discussed above assumes that our western blot outcome measures are on a linear scale. As described above, parametric work to identify the linear range of a protein of interest is critical for ensuring that the results of a LMM (or ANOVA and t-test) are accurate and interpretable. While we recommend using loading control (or total protein control) as a covariate in a linear mixed model, many bench researchers may prefer to use the within-lane loading control (or total protein) to normalize target protein values. It is important to consider that in doing so, one creates a ratio value that is multiplicative instead of linear. This property has the side effect of artificially distorting the variance. To account for this non-linearity, we recommend that one uses semi-parametric mixed models such as generalized estimating equations with a gamma distribution link function that appropriately represents ratio data.

There has been recent recognition that an appropriate study design can be achieved by balancing sample size (n), effect size, and power 31 . The experimental and statistical approach presented in this study provide insight into how more rigorous planning for western blot experimental design and corresponding statistical analysis without depending on p-values only can acquire precise data resulting in true biological effects. Using blotRig as a standardized, integrated western blot methodology, quantitative western blot may become highly reproducible, reliable, and a less controversial protein measurement technique 18 , 28 , 67 .

Study reporting

This study is reported in accordance with ARRIVE guidelines.

Supporting information

This article contains supporting information. You can access the blotRig analysis software, which includes code for inputting experimental parameters for all Western blot analysis, through the following link: https://atpspin.shinyapps.io/BlotRig/

Data availability

The datasets and computer code generated or used in this study are accessible in a public, open-access repository at https://doi.org/10.34945/F51C7B and https://github.com/ucsf-ferguson-lab/blotRig/ respectively.

Abbreviations

American association for accreditation of laboratory animal care

Animal research reporting of in vivo experiments

American veterinary medical association

Institutional animal care and use committee

Quantitative western blot

Enzyme linked immunosorbent assay

Severe acute respiratory syndrome coronavirus 2

Analysis of covariance

Analysis of variance

Spinal cord injury

Spared nerve injury

α-Amino-3-hydroxy-5-methyl-4-isoxazoleproprionic acid

Glutamate receptor 1

Glutamate receptor 2

Linear mixed models

Tris-buffered saline containing 0.1% Tween 20

Polyacrylamide gel electrophoresis

Standard error of mean

Lowry, O., Rosebrough, N., Farr, A. L. & Randall, R. Protein measurement with the Folin phenol reagent. J. Biol. Chem. 193 , 265–275. https://doi.org/10.1016/S0021-9258(19)52451-6 (1951).

PubMed   Google Scholar  

Aldridge, G. M., Podrebarac, D. M., Greenough, W. T. & Weiler, I. J. The use of total protein stains as loading controls: An alternative to high-abundance single protein controls in semi-quantitative immunoblotting. J. Neurosci. Methods 172 , 250–254. https://doi.org/10.1016/j.jneumeth.2008.05.00 (2008).

PubMed   PubMed Central   Google Scholar  

McDonough, A. A., Veiras, L. C., Minas, J. N. & Ralph, D. L. Considerations when quantitating protein abundance by immunoblot. Am. J. Physiol. Cell Physiol. 308 , C426-433. https://doi.org/10.1152/ajpcell.00400.2014 (2015).

Towbin, H., Staehelin, T. & Gordon, J. Electrophoretic transfer of proteins from polyacrylamide gels to nitrocellulose sheets: Procedure and some applications. PNAS 76 , 4350–4354. https://doi.org/10.1073/pnas.76.9.4350 (1979).

ADS   PubMed   PubMed Central   Google Scholar  

Burnette, W. N. “Western blotting”: Electrophoretic transfer of proteins from sodium dodecyl sulfate-polyacrylamide gels to unmodified nitrocellulose and radiographic detection with antibody and radioiodinated protein A. Anal. Biochem. 112 , 195–203. https://doi.org/10.1016/0003-2697(81)90281-5 (1981).

Mahmood, T. & Yang, P.-C. Western blot: Technique, theory, and trouble shooting. N. Am. J. Med. Sci. 4 , 429–434. https://doi.org/10.4103/1947-2714.100998 (2012).

Alegria-Schaffer, A., Lodge, A. & Vattem, K. Performing and optimizing Western blots with an emphasis on chemiluminescent detection. Methods Enzymol. 463 , 573–599. https://doi.org/10.1016/S0076-6879(09)63033-0 (2009).

Khoury, M. K., Parker, I. & Aswad, D. W. Acquisition of chemiluminescent signals from immunoblots with a digital SLR camera. Anal. Biochem. 397 , 129–131. https://doi.org/10.1016/j.ab.2009.09.041 (2010).

Zellner, M. et al. Fluorescence-based western blotting for quantitation of protein biomarkers in clinical samples. Electrophoresis 29 , 3621–3627. https://doi.org/10.1002/elps.200700935 (2008).

Gingrich, J. C., Davis, D. R. & Nguyen, Q. Multiplex detection and quantitation of proteins on western blots using fluorescent probes. Biotechniques 29 , 636–642. https://doi.org/10.2144/00293pf02 (2000).

Janes, K. A. An analysis of critical factors for quantitative immunoblotting. Sci. Signal 8 , rs2. https://doi.org/10.1126/scisignal.2005966 (2015).

Mollica, J. P., Oakhill, J. S., Lamb, G. D. & Murphy, R. M. Are genuine changes in protein expression being overlooked? Reassessing western blotting. Anal. Biochem. 386 , 270–275. https://doi.org/10.1016/j.ab.2008.12.029 (2009).

Pillai-Kastoori, L., Schutz-Geschwender, A. R. & Harford, J. A. A systematic approach to quantitative western blot analysis. Anal. Biochem. 593 , 113608. https://doi.org/10.1016/j.ab.2020.113608 (2020).

Aydin, S. A short history, principles, and types of ELISA, and our laboratory experience with peptide/protein analyses using ELISA. Peptides 72 , 4–15. https://doi.org/10.1016/j.peptides.2015.04.012 (2015).

Seisenberger, C. et al. Questioning coverage values determined by 2D western blots: A critical study on the characterization of anti-HCP ELISA reagents. Biotechnol. Bioeng. 118 , 1116–1126. https://doi.org/10.1002/bit.27635 (2021).

Edwards, V. M. & Mosley, J. W. Reproducibility in quality control of protein (western) immunoblot assay for antibodies to human immunodeficiency virus. Am. J. Clin. Pathol. 91 , 75–78. https://doi.org/10.1093/ajcp/91.1.75 (1989).

Matschke, J. et al. Neuropathology of patients with COVID-19 in Germany: A post-mortem case series. Lancet Neurol. 19 , 919–929. https://doi.org/10.1016/S1474-4422(20)30308-2 (2020).

Murphy, R. M. & Lamb, G. D. Important considerations for protein analyses using antibody based techniques: Down-sizing western blotting up-sizes outcomes. J. Physiol. 591 , 5823–5831. https://doi.org/10.1113/jphysiol.2013.263251 (2013).

Butler, T. A. J., Paul, J. W., Chan, E.-C., Smith, R. & Tolosa, J. M. Misleading westerns: Common quantification mistakes in western blot densitometry and proposed corrective measures. Biomed. Res. Int. 2019 , 5214821. https://doi.org/10.1155/2019/5214821 (2019).

Button, K. S. et al. Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 , 365–376. https://doi.org/10.1038/nrn3475 (2013).

Landis, S. C. et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490 , 187–191. https://doi.org/10.1038/nature11556 (2012).

Shields, S. D., Eckert, W. A. & Basbaum, A. I. Spared nerve injury model of neuropathic pain in the mouse: A behavioral and anatomic analysis. J. Pain 4 , 465–470. https://doi.org/10.1067/s1526-5900(03)00781-8 (2003).

Decosterd, I. & Woolf, C. Spared nerve injury: An animal model of persistent peripheral neuropathic pain. Pain 87 , 149–158. https://doi.org/10.1016/S0304-3959(00)00276-1 (2000).

Richner, M., Jager, S. B., Siupka, P. & Vaegter, C. B. Hydraulic extrusion of the spinal cord and isolation of dorsal root ganglia in rodents. J. Vis. Exp. https://doi.org/10.3791/55226 (2017).

Ferguson, A. R. et al. Cell death after spinal cord injury is exacerbated by rapid TNFα-induced trafficking of GluR2-lacking AMPARS to the plasma membrane. J Neurosci 28 , 11391–11400. https://doi.org/10.1523/JNEUROSCI.3708-08.2008 (2008).

Ferguson, A. R., Huie, J. R., Crown, E. D. & Grau, J. W. Central nociceptive sensitization vs. spinal cord training: Opposing forms of plasticity that dictate function after complete spinal cord injury. Front. Physiol. 3 , 1. https://doi.org/10.3389/fphys.2012.00396 (2012).

Google Scholar  

Taylor, S. C., Berkelman, T., Yadav, G. & Hammond, M. A defined methodology for reliable quantification of western blot data. Mol. Biotechnol. 55 , 217–226. https://doi.org/10.1007/s12033-013-9672-6 (2013).

Bakkenist, C. J. et al. A quasi-quantitative dual multiplexed immunoblot method to simultaneously analyze ATM and H2AX phosphorylation in human peripheral blood mononuclear cells. Oncoscience 2 , 542–554. https://doi.org/10.18632/oncoscience.162 (2015).

Wang, Y. V. et al. Quantitative analyses reveal the importance of regulated Hdmx degradation for p53 activation. Proc. Natl. Acad. Sci. USA 104 , 12365–12370. https://doi.org/10.1073/pnas.0701497104 (2007).

Bass, J. et al. An overview of technical considerations for western blotting applications to physiological research. Scand. J. Med. Sci. Sports 27 , 4–25. https://doi.org/10.1111/sms.12702 (2017).

Lazzeroni, L. C. & Ray, A. The cost of large numbers of hypothesis tests on power, effect size and sample size. Mol. Psychiatry 17 , 108–114. https://doi.org/10.1038/mp.2010.117 (2012).

Huie, J. R. et al. AMPA receptor phosphorylation and synaptic colocalization on motor neurons drive maladaptive plasticity below complete spinal cord injury. eNeuro https://doi.org/10.1523/ENEURO.0091-15.2015 (2015).

Stück, E. D. et al. Tumor necrosis factor alpha mediates GABAA receptor trafficking to the plasma membrane of spinal cord neurons in vivo. Neural Plast https://doi.org/10.1155/2012/261345 (2012).

Krzywinski, M. & Altman, N. Points of significance: Power and sample size. Nat. Method. 10 , 1139–1140. https://doi.org/10.1038/nmeth.2738 (2013).

R Core Team R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/ (2021).

Green, P. & MacLeod C. J. “simr: An R package for power analysis of generalised linear mixed models by simulation.” Meth. Ecol. Evolut. 7 (4), 493–498. https://doi.org/10.1111/2041-210X.12504 , https://CRAN.R-project.org/package=simr (2016).

Attali, D. shinyjs: Easily Improve the User Experience of Your Shiny Apps in Seconds. R package version 2.1.0, https://deanattali.com/shinyjs/ (2022).

Chang, W. et al. shiny: Web Application Framework for R. R package version 1.9.1.9000, https://github.com/rstudio/shiny , https://shiny.posit.co/ (2024).

Chang, W. shinythemes: Themes for Shiny. R package version 1.2.0, https://github.com/rstudio/shinythemes (2024).

de Vries, A., Schloerke, B., Russell, K. sortable: Drag-and-Drop in ‘shiny’ Apps with ‘SortableJS’. R package version 0.5.0, https://github.com/rstudio/sortable (2024).

Wickham, H. et al. Welcome to the tidyverse. JOSS 4 (43), 1686. https://doi.org/10.21105/joss.01686 (2019).

ADS   Google Scholar  

Xie, Y., Cheng, J., Tan, X. DT: A Wrapper of the JavaScript Library ‘DataTables’. R package version 0.33.1, dt. https://github.com/rstudio/ (2024).

Krzywinski, M. & Altman, N. Points of significance: Analysis of variance and blocking. Nat Methods 11 , 699–700. https://doi.org/10.1038/nmeth.3005 (2014).

Heidebrecht, F., Heidebrecht, A., Schulz, I., Behrens, S.-E. & Bader, A. Improved semiquantitative western blot technique with increased quantification range. J. Immunol. Methods 345 , 40–48. https://doi.org/10.1016/j.jim.2009.03.018 (2009).

Huang, Y.-T. et al. Robust comparison of protein levels across tissues and throughout development using standardized quantitative western blotting. J. Vis. Exp. https://doi.org/10.3791/59438 (2019).

Krzywinski, M. & Altman, N. Points of view: Designing comparative experiments. Nat. Methods 11 , 597–598. https://doi.org/10.1038/nmeth.2974 (2014).

Thacker, J. S., Yeung, D. H., Staines, W. R. & Mielke, J. G. Total protein or high-abundance protein: Which offers the best loading control for western blotting?. Anal. Biochem. 496 , 76–78. https://doi.org/10.1016/j.ab.2015.11.022 (2016).

Zeng, L. et al. Direct blue 71 staining as a destaining-free alternative loading control method for western blotting. Electrophoresis 34 , 2234–2239. https://doi.org/10.1002/elps.201300140 (2013).

Jaeger, T. F. Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. J. Mem. Lang. 59 , 434–446. https://doi.org/10.1016/j.jml.2007.11.007 (2008).

Mefford, J. & Witte, J. S. The covariate’s dilemma. PLoS Genet. 8 , e1003096. https://doi.org/10.1371/journal.pgen.1003096 (2012).

Schneider, B. A., Avivi-Reich, M. & Mozuraitis, M. A cautionary note on the use of the analysis of covariance (ANCOVA) in classification designs with and without within-subject factors. Front. Psychol. 6 , 474. https://doi.org/10.3389/fpsyg.2015.00474 (2015).

Nieuwenhuis, S., Forstmann, B. U. & Wagenmakers, E.-J. Erroneous analyses of interactions in neuroscience: A problem of significance. Nat. Neurosci. 14 , 1105–1107. https://doi.org/10.1038/nn.2886 (2011).

Freeberg, T. M. & Lucas, J. R. Pseudoreplication is (still) a problem. J. Com. Psychol. 123 , 450–451. https://doi.org/10.1037/a0017031 (2009).

Judd, C. M., Westfall, J. & Kenny, D. A. Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. J. Pers. Soc. Psychol. 103 , 54–69. https://doi.org/10.1037/a0028347 (2012).

Lee, O. E. & Braun, T. M. Permutation tests for random effects in linear mixed models. Biometrics 68 , 486–493. https://doi.org/10.1111/j.1541-0420.2011.01675.x (2012).

MathSciNet   PubMed   Google Scholar  

Baayen, R. H., Davidson, D. J. & Bates, D. M. Mixed-effects modeling with crossed random effects for subjects and items. J. Mem. Lang 59 , 390–412. https://doi.org/10.1016/j.jml.2007.12.005 (2008).

Barr, D. J., Levy, R., Scheepers, C. & Tily, H. J. Random effects structure for confirmatory hypothesis testing: Keep it maximal. J. Mem. Lang. https://doi.org/10.1016/j.jml.2012.11.001 (2013).

Blainey, P., Krzywinski, M. & Altman, N. Points of significance: Replication. Nat. Methods 11 , 879–880. https://doi.org/10.1038/nmeth.3091 (2014).

Drubin, D. G. Great science inspires us to tackle the issue of data reproducibility. Mol. Biol. Cell 26 , 3679–3680. https://doi.org/10.1091/mbc.E15-09-0643 (2015).

Amrhein, V., Greenland, S. & McShane, B. Scientists rise up against statistical significance. Nature 567 , 305–307. https://doi.org/10.1038/d41586-019-00857-9 (2019).

ADS   PubMed   Google Scholar  

Cohen, J. The earth is round (p <.05). Am. Psychol. 49 , 997–1003. https://doi.org/10.1037/0003-066X.49.12.997 (1994).

Ioannidis, J. P. A., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22 , 450–456. https://doi.org/10.1097/EDE.0b013e31821b506e (2011).

Sullivan, G. M. & Feinn, R. Using effect size-or why the P value Is not enough. J. Grad. Med. Educ. 4 , 279–282. https://doi.org/10.4300/JGME-D-12-00156.1 (2012).

Brysbaert, M. & Stevens, M. Power analysis and effect size in mixed effects models: A tutorial. J. Cogn. 1 , 9. https://doi.org/10.5334/joc.10 (2018).

Kline, R. B. Beyond significance testing: Reforming data analysis methods in behavioral research. Am. Psychol. Associat . https://doi.org/10.1037/10693-000 (2024).

Rosner, Bernard (Bernard A.). Fundamentals of biostatistics. (Boston, Brooks/Cole, Cengage Learning, 2011).

Bromage, E., Carpenter, L., Kaattari, S. & Patterson, M. Quantification of coral heat shock proteins from individual coral polyps. Mar. Ecol. Progress Ser. 376 , 123–132 (2009).

Download references

Acknowledgements

The authors would like to thank Alexys Maliga Davis for data librarian services.

This work was supported by a National Institutes of Health/National Institute of Neurological Disorders and Stroke grant (R01NS088475) to A. R. F. NIH NINDS: R01NS122888, UH3NS106899, U24NS122732, US Veterans Affairs (VA): I01RX002245, I01RX002787, I50BX005878, Wings for Life Foundation, Craig H. Neilsen Foundation. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Correspondence and requests for materials should be addressed to A.R.F.

Author information

Authors and affiliations.

Weill Institute for Neurosciences, University of California, San Francisco, CA, USA

Cleopa Omondi, Austin Chou, Kenneth A. Fond, Kazuhito Morioka, Nadine R. Joseph, Jeffrey A. Sacramento, Emma Iorio, Abel Torres-Espin, Hannah L. Radabaugh, Jacob A. Davis, Jason H. Gumbel, J. Russell Huie & Adam R. Ferguson

San Francisco Veterans Affairs Medical Center, San Francisco, CA, USA

J. Russell Huie & Adam R. Ferguson

School of Public Health Sciences, Faculty of Health Sciences, University of Waterloo, Waterloo, ON, Canada

Abel Torres-Espin

Department of Physical Therapy, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, AB, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

C.O: Writing-original draft preparation, Investigation, Validation, Data Curation, Visualization, Formal Analysis, Writing-Review & Editing; A. C: Formal analysis, Writing-Review & Editing; K. A. F: Software, Writing-Review & Editing; K. M: Methodology, Writing-Review & Editing; N. R. J: Writing—Review & Editing; J. A. S: Investigation, Project Administration, Writing—Review & Editing; E. I: Resources, Writing-Review & Editing; A.T.E: Software, Writing—Review & Editing; H. L. R: Software, Writing—Review & Editing; J. A. D: Investigation, Writing—Review & Editing; J. H. G: Investigation, Writing – Review & Editing; J. R. H: Conceptualization, Methodology, Validation, Formal Analysis, Investigation, Data Curation, Writing- Review & Editing, Visualization, Supervision; A. R. F:Conceptualization, Methodology, Validation, Formal Analysis, Resources, Investigation, Data Curation, Writing-Review & Editing, Visualization, Supervision, Project Administration, Funding Acquisition.

Corresponding authors

Correspondence to J. Russell Huie or Adam R. Ferguson .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary figure 1., supplementary figure 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Omondi, C., Chou, A., Fond, K.A. et al. Improving rigor and reproducibility in western blot experiments with the blotRig analysis. Sci Rep 14 , 21644 (2024). https://doi.org/10.1038/s41598-024-70096-0

Download citation

Received : 19 December 2023

Accepted : 13 August 2024

Published : 17 September 2024

DOI : https://doi.org/10.1038/s41598-024-70096-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Analytical chemistry
• Biostatistics
• Computational biology
• Computational chemistry

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

A standardized framework to test event-based experiments

• Original Manuscript
• Open access
• Published: 16 September 2024

Cite this article

You have full access to this open access article

• Alex Lepauvre   ORCID: orcid.org/0000-0002-4191-1578 1 , 2 ,
• Rony Hirschhorn   ORCID: orcid.org/0000-0001-7710-5159 3 ,
• Katarina Bendtz   ORCID: orcid.org/0000-0002-8262-3652 4 ,
• Liad Mudrik   ORCID: orcid.org/0000-0003-3564-6445 3 , 5 , 7 &
• Lucia Melloni   ORCID: orcid.org/0000-0001-8743-5071 1 , 6 , 7  

The replication crisis in experimental psychology and neuroscience has received much attention recently. This has led to wide acceptance of measures to improve scientific practices, such as preregistration and registered reports. Less effort has been devoted to performing and reporting the results of systematic tests of the functioning of the experimental setup itself. Yet, inaccuracies in the performance of the experimental setup may affect the results of a study, lead to replication failures, and importantly, impede the ability to integrate results across studies. Prompted by challenges we experienced when deploying studies across six laboratories collecting electroencephalography (EEG)/magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and intracranial EEG (iEEG), here we describe a framework for both testing and reporting the performance of the experimental setup. In addition, 100 researchers were surveyed to provide a snapshot of current common practices and community standards concerning testing in published experiments’ setups. Most researchers reported testing their experimental setups. Almost none, however, published the tests performed or their results. Tests were diverse, targeting different aspects of the setup. Through simulations, we clearly demonstrate how even slight inaccuracies can impact the final results. We end with a standardized, open-source, step-by-step protocol for testing (visual) event-related experiments, shared via protocols.io. The protocol aims to provide researchers with a benchmark for future replications and insights into the research quality to help improve the reproducibility of results, accelerate multicenter studies, increase robustness, and enable integration across studies.

Similar content being viewed by others

Issues and recommendations from the OHBM COBIDAS MEEG committee for reproducible EEG and MEG research

Reporting checklists in neuroimaging: promoting transparency, replicability, and reproducibility

Variability in the analysis of a single neuroimaging dataset by many teams

Avoid common mistakes on your manuscript.

Introduction

The remarkable progress of experimental human neuroscience in recent decades, fueled by the development of technologies to survey the brain non-invasively, has been partly overshadowed by the many examples of replication failures (e.g., Hirschhorn & Schonberg, 2024 ). Replication failures may stem from a number of factors (Open Science Collaboration, 2015 ), for instance, low standards of power calculations (Button et al., 2013 ; Ioannidis, 2005 ), the use of questionable statistical methods (e.g., Cumming, 2014 ; Wicherts et al., 2016 ), publication biases in favor of positive findings (Fanelli, 2012 ), or scarce description of the methods (Poldrack et al., 2008 ; Simmons et al., 2011 ).

The community has responded to those challenges by promoting better scientific practices that address those issues (Munafò et al., 2017 ). By now, determining sample sizes based on power analysis has become a common practice (e.g., Mumford & Nichols, 2008 ), journals also more routinely publish negative or null results (e.g., Baxter & Burwell, 2017 ), and preregistering the planned design and analyses is on the rise (e.g., AsPredicted, Open Science Framework). These procedures have had a large impact on the scientific community (Logg & Dorison, 2021 ; Protzko et al., 2023 ). One aspect that has received less attention in the empirical sciences, however, is the functioning of the experiment itself: does the experiment run as expected? While at first glance the reader may assume that experiments always run as planned and that if errors occur, they do not significantly impact the results, here we show a large variance among researchers when it comes to testing the experimental framework. Furthermore, we demonstrate that errors in the functioning of the experiment (e.g., the timing of the stimuli) can impact the results and their interpretation. Despite its importance, no standardized procedure for testing and reporting the quality of the experimental setups currently exists.

Standardized procedure s are becoming more relevant given the increased number of multi-lab studies (e.g., Frank et al., 2017 ; Melloni et al., 2021 ; Pavlov et al., 2021 ) and the increased availability of openly shared data. The large diversity of software and hardware poses a major challenge when integrating data across different laboratories, which often acquire data using different setup specifications. A similar issue arises when reusing openly shared data collected in various neuroscientific paradigms (Sejnowski et al., 2014 ). Without metadata describing the functioning of the experimental setup itself (variability in the presentation duration of the stimuli, reliability of the timestamps, etc.), integrating multiple datasets can pose problems, as it necessitates determining a priori how comparable the data are (Carp, 2012 ). For that, information about if and how the experiment was tested is important.

To demonstrate the need for a standardized testing framework, we first surveyed current practices in the field when it comes to testing and reporting the functioning of experimental setups in neuroscience. Testing practices varied among researchers, and many acknowledged discovering malfunctioning upon data collection. We then used simulations to demonstrate that even minor inaccuracies in hardware and software can alter results. Finally, we propose a standardized framework for testing and reporting the functioning of experimental setups for event-based designs. We provide an easy-to-use protocol, openly available in protocols.io .

Common testing practices: A survey

To investigate current practices of testing experimental setups in behavioral and neural science, 100 psychologists and neuroscientists reported on studies they had recently carried out. The majority of respondents studied human participants (94/100), collected neural data (67/100), and were early-career researchers (40/100 graduate students, 36/100 postdoctoral/senior researchers, 16/100 principal investigators).

Almost all respondents reported testing the experimental setup prior to data acquisition in a large majority of experiments (91/100), while a few (5/100) tested only some experiments or never tested the setup before data collection (4/100). The aspects of the experimental setup tested varied greatly among researchers (Fig. 1 ). Most tested the overall duration of the experiment (84/96), while a smaller proportion tested the accuracy of the event timings (60/96; see Box 1). There was also considerable diversity in the methods used to test the experimental setup, both between researchers (e.g., manual checks [48/96], scripted checks ([1/96], both types ([47/96]) and within the same lab between experiments (when asked whether the tests were based on a protocol, 53/96 responded that each experiment was tested differently; see Supplementary Material 1 for survey and the full set of results).

The type and frequency of pre-experimental tests, declared by researchers to have been conducted prior to their last published experiment ( N  = 96). Y -axis: percentage of respondents. X -axis: aspect of the experimental environment tested (selected from a list of options; see Supplementary Material 1 ). The terms "event timing" and "event content" are defined in Box 1

Strikingly, a large proportion of researchers (64/100) reported noticing an issue after data collection that could have been avoided through prior testing. This reinforces the need for a streamlined procedure to benchmark the experimental setup (or experimental environment , as defined in Box 1), which could prevent the collection of unusable data and facilitate replication.

Box 1 Definitions of terms

The image illustrates the connections between the various components and devices used in an example experiment. Top left : A participant sits in an experimental environment (see Box 1). They face a computer screen displaying a green star. Underneath is an eye-tracking device (black) and a response box. An EEG cap and amplifier are displayed as an example of neural measurement. Top right : The screen connects to the experimental computer which runs the experimental software (see Box 1). Bottom right : An example peripheral device recording neural data, connected to both the experimental computer (top right) and the EEG amplifier (top left). Bottom left : An example peripheral device recording eye-tracking data connected to both the eye-tracking camera (top left) and the experimental computer (top right). Clocks in each box indicate the device’s internal clock. Dashed arrows depict connections between components of independent devices. Solid black arrows represent connections sending triggers from the experimental computer to peripherals used for synchronization between recording devices.

Despite performing tests, results in publications (80/96) either because they considered reporting the results irrelevant (43/96) or because they did not know where to present them (38/96), or both (15/96; Figs. 3 ). Reporting practices, from reporting in the methods section, in the preregistration, in their lab book, or supplementary materials . A small proportion of researchers who never reported results (10/80) assumed that all published work had been thoroughly checked. However, as we have seen, the assumption that experimental tests and methods are consistent and error-free is not warranted, given the variations in testing procedures and the prevalence of errors identified retrospectively, during or after data collection. Thus, reporting of test results is crucial. If widely adopted, this practice would encourage more thorough testing, reduce errors, and enhance data accuracy across experiments and datasets.

Reporting practices of researchers who declared testing their last published study ( N  = 96, since four respondents declared not testing their experiments at all). Outer circle: responses to “Did you report about the checks you performed and their results?” Red: respondents declaring not reporting the results of the tests ( N  = 80). Light blue: respondents declaring reporting the results of some tests ( N  = 15). Blue: one respondent who reported the results of all the performed tests. Inner circle: responses to “If you did not report the checks, why not?” Orange: because it was irrelevant ( N  = 3 out of the “some” category, N  = 40 out of the “no” category). Grayish-blue: because it wasn’t known where to report the results ( N  = 11 [“some” category], N  = 26 [“no” category], and 1 reporting the test). Tan: because it was irrelevant and not known where to report the results ( N  = 1 [“some” category], N  = 14 [“no” category])

Simulations of experimental environment malfunction

The survey demonstrated that research practices for testing the experimental setup vary widely, and that if tests are conducted, they are often not reported. Yet, researchers acknowledged discovering inaccuracies after data acquisition which could have been prevented. How severe are those inaccuracies to warrant extra testing and reporting? We simulated inaccuracies in event content and event timing to demonstrate whether and how they can affect experimental results (Supplementary Material 2 for full simulation). For this demonstration, we focused on reaction time data and P1 event-related potential (ERP), as both are widely used measurements and highly susceptible to timing inaccuracies. Both behavioral and neural simulation results are generalizable to other ERP components and uncontrolled events (events evoked by participants’ responses; see Box 1). We simulated two experimental conditions, representing two stimulus groups (see Supplementary Material 2 for the complete procedure). The simulation followed a common experimental hypothesis of a difference between two conditions, in both the P1 average amplitude and mean reaction times. The difference between the two conditions was referred to as \(\theta\) . To simulate inaccuracies in event contents, we shuffled stimulus labels on some trials (between 2% and 40%); for inaccuracies in event timings, we introduced a jitter of a predefined duration (from 2 to 40 ms) on some trials (between 2% and 40%). Simulations show that content and timing inaccuracies considerably diminish and sometimes even obliterate statistical differences between experimental conditions.

Figure 4 A shows that at small effect sizes ( \(\theta \hspace{0.17em}\) = 0.2), a small proportion of shuffled-label trials (20%) was enough to abolish a statistical effect in P1 amplitude. For the reaction time, the same was observed when 5% or more labels were shuffled across trials (Fig. 4 B). Jitter in stimulus timing also significantly affected the measured statistic: Figure 5 B shows that a jitter of 16 ms (a frame on a 60 Hz monitor) on 15% of the trials was enough to render P1 effects insignificant. For a 32 ms jitter, the same outcome was observed when 5% of the trials were affected. The effect of jitter was comparable for the reaction time: at \(\theta\) = 0.2, a 16 ms jitter affecting 15% of trials was sufficient to abolish the effect, and the same was observed at larger effect sizes ( \(\theta\) = 0.5) with a 32 ms jitter (two frames) in 5% of trials (Fig. 5 D).

Effect of label shuffle on the P1 (A) and reaction time (B) t -statistic. The heatmap represents the observed t -statistic as a function of the simulated effect size ( x -axis) and proportion of trials for which the labels were shuffled ( y -axis). The color bar is centered on 1.96. Values below significance are colored in shades of blue, while values above significance are colored in shades of orange.

Effects of timing inaccuracies on t -statistic for ERPs (A, B) and reaction time (C, D). A, C 3D plot relating the proportion of jittered trials ( x -axis), jitter duration ( y -axis), and t -statistic ( z -axis) as a function of effect size ( \(\uptheta\) , color bar) for P1 amplitude and reaction times, respectively. Gray hyperplane depicts significance threshold of t  = 1.96. An example at \(\uptheta\)  = 0.2 for P1 amplitude ( B ) and reaction time ( D ). The color bar centered on 1.96, values below significance are colored in shades of blue, while values above significance are colored in shades of orange

Taken together, the simulation results reinforce the importance of testing the recorded stimulus contents, showcasing the effect that timing inaccuracies and imprecision of the hardware (experimental computer [ EC ]; the computer on which the experiment runs; Box 1) can have. Recent studies suggest that inaccuracies in modern experimental software ( ES ; e.g., PsychoPy: Brainard, 1997 ; Psychtoolbox: Peirce, 2007 ) are minimal (on the sub-millisecond level, Bridges et al., 2020 ). This is only the case when the experiment is run in an ideal experimental environment. Accordingly, proper testing is required to ensure that this is indeed the case. This is even more important given the variations in the interaction between the EC and the ES (for example, in Psychtoolbox, a screen flip can be missed if too many textures are open; see definition in Box 1). Thus, our simulation results highlight the need for standardized testing of the experimental environment. Here, we argue that performing a few basic tests can increase event-based experiment reproducibility, improve data integration across datasets, and ultimately minimize errors, increasing efficiency. Next, we describe a standardized framework of tests.

We describe a standardized framework to benchmark the experimental environment in event-based designs including a standardized reporting protocol (protocols.io). The framework is aimed at helping researchers without imposing additional burdens on their standard experimental procedures. As such, it strikes a balance between exhaustiveness and ease of use. As a starting point, the framework is best suited for studies involving visual stimuli while collecting responses from participants (neural responses as well as behavior and eye-tracking data). Extension to other modalities (e.g., auditory, tactile) and response devices (e.g., microphone) will be needed. To illustrate the framework, we programmed a simple experiment in which we conducted all the tests, described step by step in a Jupyter notebook (see Supplemental Material 3 ). The notebook aims to help researchers in understanding the implementation of the framework, serving as an accessible resource that can be adapted for testing future experiments.

Each section starts with the motivation for testing a given aspect of the experimental environment, followed by the testing guidelines and the standardized reporting protocol (see protocols.io ). A successful visual event-based experiment necessitates thorough testing and validation of four key aspects: (1) the completeness and accuracy of the log file regarding event content , (2) the same for event timing , (3) the alignment between actual events and the planned experimental design, and (4) the reliability of peripheral triggers. This ensures experimental integrity and comparability across different studies and laboratories. Testing typically involves running the entire experiment at least once in the final experimental setup, as the hardware significantly influences the precision and accuracy of the experiment. While these aspects are discussed separately, they can generally be tested in a single experiment run, unless specified otherwise. We provide a visual representation of the implementation of all steps of the framework (see Figs.  6 ).

Flowchart of the implementation of the testing framework. (1) First, the experiment should be adjusted to present a photodiode square and record the sounds from a contact microphone measuring the sound produced by keyboard presses. In addition, a response sequence should be planned. (2) Then, a set of pre-run tests should be performed. The first consists of measuring the size of the stimuli in centimeters to compute the size of the observed stimuli in degrees of visual angles. In addition, several trials of the experiment should be run, manually annotating what was presented on the screen in each trial. These manual annotations should then be compared to the log file entries to ensure that the log file accurately records what was actually presented. (3) The experiment should then be run in full while recording the photodiode and microphone signals as well as the log file for offline analysis. (4) Then, the recorded photodiode signal should be compared to the log file to estimate event timing inaccuracies. The microphone signal should be compared to the log file response timestamping to assess the responses’ timestamping inaccuracies. In addition, the logged responses should be compared to the planned response sequence defined in step (1) to ensure that the log file accurately records the pressed buttons. Finally, the experimental design can be tested for correctness based on the log file and photodiode timing information

Validating the features of the controlled events

Why test it.

In most visual experiments, standardizing the visual angle and eccentricity across participants is crucial. The initial step involves correctly setting up the visual angle and eccentricity in the experimental environment. This standardization is foundational for all subsequent tests and ensures that experimental events are presented under consistent conditions. This step is critical for both multi-lab studies and single-lab experiments, facilitating accurate replication by ensuring visual equivalence between original and replication setups.

How to test it?

First, three measurements should be obtained: the screen’s height and width in pixels (i.e., the screen resolution), the screen’s height and width in metric units, and the distance between the eye and the screen (in metric units). From the screen dimensions in centimeters and pixels, a conversion factor between pixels and centimeters should be computed as follows:

Using either the right or the left equation (i.e., the height or the width) should yield the same results, except for displays in which the pixel aspect ratio is not 1:1. In all other cases, inequality indicates measurement issues. After obtaining the conversion factor c , event sizes and offsets can be calculated in degrees of visual angles from screen pixels and size in metric units.

The next step is to measure the size and eccentricity of a specific experimental event. Once presented, the size and eccentricity of the visual event can be measured in metric units. Then, the inverse tangent can be used to calculate the vertical and horizontal visual angle of a stimulus using right-angle trigonometry:

where both size and distance are in metric units.

Similarly, the eccentricity should be calculated in visual angles by measuring the distance between the center of the screen and the center of the stimulus of interest (assuming fixation is in the middle of the screen) and applying the same formula ( 2 ).

Note that in the case that stimuli are off-centered, the measurement of the stimulus size in visual angles needs to be adjusted to account for the tilt between the screen and the eye. SR Research offers a free tool for visual angle calculation, for all types of stimuli (centered, one-sided, off-center; see www.sr-research.com/visual-angle-calculator/ ). When experimental events differ in size and eccentricity (e.g., stimuli displayed in different locations or differing in size), each such event should be measured at least once (per location, per size).

How to report it?

To report the relevant visual features in degrees of visual angle, the measured distance of the screen should be reported in centimeters (to describe the conditions under which the size of the stimuli was tested). For the stimuli, the measured height and width should be reported in degrees of visual angle. If different stimuli sizes are relevant to the experimental design, those should be measured and reported too. Similarly, if the stimuli are presented at a given eccentricity, both the expected and measured horizontal and vertical distance from the center of the screen should be reported (also in visual angles). Reporting both the horizontal and vertical visual angles from the expected center of the participant’s gaze provides the unique position of the stimuli. Thus, it is preferable to report the distance between the center of the stimulus and where participants are supposed to fixate (usually, the center of the screen).

Testing the reliability of the log file event content

Controlled events.

For accurate analysis, researchers must ensure that the content of events presented to participants is correctly recorded in the log files (see Box 1). This involves verifying that logged events match the actual events presented. Errors in the experimental software (ES) or hardware (EC) can lead to incorrect logging, such as mislabeling stimulus categories or identities, which can significantly alter experimental results. Any discrepancies require rectifying and retesting until the log file accurately reflects the presented content. While systematically checking log files, especially for experiments with complex stimuli like videos, can be challenging and time-consuming, it is crucial for ensuring the validity, interpretability, and reproducibility of the results.

Comparing the on-screen content with that of the log file where the content is documented requires running the experiment, ideally, from start to end (without participants), noting the event content presented on the screen (e.g., the stimulus identifier and relevant features such as orientation, location, color). Manually noting the content of each stimulus throughout the entire experiment might not be feasible, especially for experiments containing hundreds of trials. The compromise recommended here is to minimally check the content of each event condition at least once (though exhaustive testing is, of course, preferred). For example, suppose an experiment presents two stimulus groups (e.g., faces and objects) at four possible locations. In that case, the manual recording of event content during the experiment should at least cover a stimulus from each group appearing at each location once. By event condition, we refer to any feature relevant to the experimental design (e.g., category, location, task relevance, color, congruence). For designs with nested conditions (e.g., stimuli of different groups presented in different task-relevance conditions), a condition is understood as a combination of conditions (a task-relevant face constitutes a condition, and task-irrelevant faces another).

As events can be fast-paced, it might be impossible to mark them manually in real time. Therefore, we recommend following one of two options: One possibility is using external recording devices and software (cameras, microphones, or other recording software) to record the presented events, tagging them based on the recording’s playback. Importantly, the recording device needs to be external to the experimental environment (e.g., not a recording software running on the EC), as otherwise, it might interfere with the functioning of the experimental environment (as a screen-recording software is not expected to run in the data collection phase). Alternatively, the pace of the experiment could be slowed down for testing purposes in the ES such that the event content can be noted. The downside is that then two separate tests are required: one for testing the event content and a separate one for testing the reliability of the event timings (see next section).

The logged on-screen contents are then compared to the saved log files of the test run, expecting complete consistency between the two. Discrepancies point to malfunction of the ES, requiring correction prior to data collection.

The report on logging content inaccuracies should briefly describe the test method, detailing (i) the number of different conditions tested (considering unique combinations of nested conditions), (ii) the number of individual events tested within each condition, and (iii) the count of events that were incorrectly logged out of the total presented. Ideally, in a fully functioning experiment during final data collection, this count of inaccurately logged events should be zero.

Uncontrolled events

Uncontrolled events are those that depend on the participants’ behavior without any control from the experimenter (see Box 1), which are made on devices recorded by the EC (and not on peripherals). This test validates the assumption that the EC correctly registers EC responses into the log files. This is done by comparing the actual responses (made by the experimenter during the test) with the logged ones (recorded in the log files).

To validate the fidelity of the logged responses, the experiment should be run from start to end, recording the actual and logged responses made under a systematic, preplanned response sequence. This response plan provides a “recipe” to evaluate whether responses are properly logged and correspond to the executed responses. The purpose of the plan is to test the correct assignment of response buttons, counterbalancing, handling multiple or erroneous responses per single event, and logging responses that occur at unexpected moments during the experiment. To test the correct assignment of response buttons, the response plan should include at least one response of each type the participant is expected to make. When response mapping is counterbalanced within an experiment (e.g., a key is mapped to “Yes” in one block and “No” in another), the response plan should also include responses of the same buttons before and after such changes to determine whether the mappings are reflected in the log files. Handling multiple or erroneous responses is crucial, as participants’ behavior might deviate from that expected by the researchers. Therefore, the response plan should include responses using unexpected keys, cases where a key is pressed more or fewer times than expected (e.g., multiple presses when a single press is expected), or when more than one button is pressed simultaneously.

The final step is to compare the response plan with those recorded on the log file. If executed responses followed the response plan, any incompatibility found points to errors in the ES which require correction prior to data collection.

The count of uncontrolled events' content inaccuracies should include a description of the test procedure and response plan, along with three key metrics: (i) the number of different response types (various buttons pressed), (ii) the number of responses for each type, and (iii) the count of responses inaccurately logged out of the total responses. Ideally, in the final data collection phase, the number of inaccurately logged responses should be zero.

Testing the reliability of the log file event timing

Malfunctions in hardware or software can lead to inaccuracies or a lack of synchrony among three crucial timestamps: the time when a request to display an event is made, the actual occurrence of the event in the experimental hardware, and the time of the event as recorded in the log file. Discrepancies between these timings often reflect EC and ES limitations, rather than human error. Once a request to present an event occurs, the EC processes it along with other requests received at a given time leading to potential delays in the execution. In addition, the computer presents stimuli at a given refresh rate, limiting the display update to a certain number of times per second. As such, the stimulus presentation request has a narrow window to be processed, and when missed, the presentation only occurs in the next frame, unintentionally prolonging the previous event. Various factors (e.g., the EC’s graphics processing unit and CPU, parallel programs running in the background other than the ES) affect processing times and latencies caused by them. Furthermore, logging the timing of an event in the log files is inferred rather than logged in real time, as modern computers do not operate in real time. Thus, discrepancies can arise between the actual event timing and the inferred time recorded in the log file. As such, jitters are to be expected. Yet, large timing deviations require intervention in either the EC or ES. For example, the EC might have insufficient system resources available to run the experiment (solution: free up EC memory, stop unnecessary programs running in the background, e.g., antivirus), or the ES code might be written inefficiently (solution: improve the ES based on the specifics of the software being used). Our simulations showed that inaccuracies in event timing can affect results, which can be detrimental in studies requiring high timing precision (e.g., visual masking paradigms). The significance of precise timing in experiments has been acknowledged before (Plant, 2016 ), and various solutions to reduce inaccuracies have been proposed (Calcagnotto et al., 2021 ; Kothe et al., 2024 ). Experimental environments vary greatly, and so do the patterns of temporal discrepancies across setups (Bridges et al., 2020 ). Thus, characterizing and reporting the differences between environments is critical for comparing results across studies.

To evaluate event timing, a "ground truth" measurement, representing the actual physical timing of events, is essential. This requires an external device like a photodiode, which detects light changes, to record visual events on the screen. By attaching a photodiode to the screen and displaying luminance changes (like black versus white) in sync with the stimulus presentation, one can measure the start and end of each event and calculate its duration. Therefore, to accurately track event timings in an experiment, the experimental software (ES) should be modified to include extreme luminance changes alongside experimental events. The testing process involves the following steps:

Modify the ES to allow photodiode-based testing

While a photodiode sensor can be placed in any location where events appear on the screen, we advocate for a systematic approach. In this step, the researchers integrate into the ES the simultaneous presentations of the evaluated event and a square at one screen corner (or a location not overlapping with the stimulus, where the photodiode is easily attached). The square should “turn on” (e.g., white; RGB 255, 255, 255) at each event onset and offset, and should be “turned off” (e.g., black; RGB 0, 0, 0) otherwise (or vice versa; Fig. 7 ). This can be achieved by drawing both the test square and the visual event to a back buffer before querying the EC to display a new frame such that both stimulus and test square overlap in time.

Depiction of visual presentation adjustment to enable testing of timing with a photodiode. A square at the corner of the screen should be flashed to white simultaneously with the onset of each event of interest and return to black thereafter. In this example, the square flashes to white at the onset of each visual stimulus (a colored star) as well as at the offset of each visual stimulus. Critically, the square should not remain white for the entire duration of the stimulus but only for a brief duration at the onset to enable the detection of transitions between each event of interest

Attach the photodiode and run the experiment

Once the ES displays the test square, place the photodiode in the location displaying the square, and run the experiment in its entirety. This step will create two files to be compared in the following steps: the log file and the photodiode output file, where the luminance level was recorded.

Extract event onsets and offsets from the recorded signal

The next step is to parse the recorded photodiode signal, which is done by setting a threshold discriminating between the two photodiode states, i.e., “on” and “off” (see Fig. 8 .1.). The threshold binarizes the signal such that values of “1” indicate samples above threshold (“on”), and “0” sample below threshold (“off”). Then, the onset of each event can be retrieved by finding the transition from off to on samples. This is achieved by computing the discrete difference (i.e., the difference between sample n +1 and sample n in the signal, see Fig. 8 .2 and 7.3) and locating the time points where this difference is equal to 1 (see Fig. 8 .4). Importantly, this step should yield the timestamp of the event in temporal units (seconds or milliseconds) by indexing the continuous time vector of the recording. Alternatively, sample units can be converted to seconds by multiplying the sample by the inverse of the sampling frequency.

Pipeline to compute the log file timestamping delays using the photodiode. The recorded signal must first be parsed (left panel) to extract the photodiode timestamps. Then, the extracted timestamps can be compared to the log file timestamping by investigating the difference between intervals of successive events as recorded by the photodiode and log file (right panel)

Compare photodiode event timings to log files

The initial test ensures that the count of events detected by the photodiode aligns with those logged in the file. If the log file content has already been confirmed, any mismatch in event numbers could indicate issues with the photodiode signal quality. A reliable measuring device (photodiode) is required in order to be considered “ground truth” and is a prerequisite for testing the experiment event timing.

Assuming a reliable photodiode recording, then the photodiode measurements are compared to the log file timings. Two values need to be computed: the discrete difference between successive events’ timestamps in the (1) photodiode ( \({\Delta }_{photo i}\) ), and those logged by the (2) EC ( \({\Delta }_{log i}\) ):

where \({t}_{photo i}\) and \({t}_{log i}\) are the timestamps for a given \(i\) event in the photodiode recording and the experimental output, respectively (see Fig. 8 .5.a and b). The logging timestamping inaccuracy is then computed as the difference between \({\Delta }_{photo i}\) and \({\Delta }_{log i}\) as

\({\Delta }_{i}\) constitutes the log file timestamping inaccuracy for every single event (see Fig. 8 .6), assuming a reliable photodiode.

For a well-calibrated setup, the expected difference between \({\Delta }_{photo i}\) and \({\Delta }_{log i}\) is on the order of milliseconds or lower (Bridges et al., 2020 ). The average in \({\Delta }_{i}\) should approximate zero, with a small standard deviation in the order of a few milliseconds. Large discrepancies indicate a problem, either with the timing at which events are displayed or at which they are logged. Inspecting the difference between both timestamp vectors might reveal the cause of those discrepancies (missing events, events that are systematically displaced in time, etc.).

The average and standard deviation values of \({\Delta }_{i}\) across events should be reported.

Uncontrolled event timing (e.g., reaction times) can also be affected by hardware and software limitations. Reaction time effects are often in the range of tens of milliseconds (Schlossmacher et al., 2020 ; van Gaal et al., 2010 ), making it necessary to determine the recording precision. Timestamping of uncontrolled events can show delays as well as jitters . It is therefore crucial to test both and report the results.

A method advocated by Psychtoolbox (and their KeyboardLatencyTest method) is to concurrently record the sound associated with the actual press of a button. This is done by placing a microphone close to the response device used in the experiment. This requires a modification of the ES to log the microphone-recorded sound into a file. Then, the button press onset can be extracted from the audio file and compared to the timestamps of responses recorded in the log file. The steps are as follows:

Modify the ES to record sound

This can be done by adding a statement at the beginning of the code to continuously record the sound throughout the experiment.

Attach the microphone and run the experiment

A contact microphone should be attached close to the keys being pressed on the response device. Sharply pressing the keys during the test run ensures easy processing in the next step. External recording devices (photodiode, microphone) are for testing purposes only and can be removed for data collection.

After obtaining the microphone recordings and log files, the analysis steps to extract and compare the triggers are the same as the procedure described in the sections “ Extract event onsets and offsets from the recorded signal ” and “ Compare photodiode event timings to log files ,” to compute the \({\Delta }_{r}\) (Eq. 4 , with r denoting responses), respectively.

Like the test for the precision of controlled event timing, the average and standard deviation of \({\Delta }_{r}\) should be documented. In an optimally calibrated experimental setting, the average \({\Delta }_{r}\) is expected to be near zero, with the standard deviation within the range of milliseconds.

Validating the experimental design parameters

After the content and timing of events has been validated, specific aspects of the experimental design can be evaluated (e.g., the duration and balancing of event groups, conditions, their sequential presentation). As there are countless choices of experimental designs, our aim is not to cover all potential designs, but instead to provide the community with a systematic method for testing and reporting experimental design aspects.

Below, we focus on two examples validating the experimental design with respect to content and timing.

Concerning adhering to the experimental design content, the aim is to confirm that those rules that researchers wish to enforce in the controlled events (e.g., presented stimuli) are indeed implemented (e.g., order of presentation, constraints on sequential trials). To test the content requirements, the following steps are proposed:

Know the experiment content requirements

Document as explicitly as possible what is expected to be enforced, e.g., number of stimulus repetitions, randomization scheme, event order, stimulus locations, and balancing of event groups. Every requirement pertaining to event content in the experimental design should be specified and checked.

Ensure that the relevant information is recorded in the log file

Ensure the information required to test the previous step is stored in the log file.

Prepare checks to make sure that the log file meets each requirement

Prepare, ideally, a programmatic script that reads the log file and ensures that its content adheres to each requirement listed in the first step following the completion of the test run. In cases where researchers have pre-made sequences including all the information about the flow of the events within an experiment, tests can be conducted in those pre-made files.

With respect to validating the timings of the experimental design, the goal is to assess how closely the actual durations of events align with their intended durations. Here, after comparing the photodiode output with the log file, researchers compare the actual duration of the observed events with their planned duration, as specified in the experimental design timing scheme.

The first step is to obtain the observed event durations from the photodiode recordings:

where OEO stands for the onset and offset of each event i. Then, one can compute how much these durations deviate from the plan:

where \(Planned duration\) is the planned duration of each event \(i\) according to the experimental design.

The report for experiments with varying experimental design content-based rules should include a comprehensive list of design choices. Evaluating how well the design meets these requirements involves counting the number of events, out of the total tested, that comply with each requirement. For nested designs, report the number of events for each combination. If relevant, include the count of events per condition in each block. Notably, there should be no deviations, as any would suggest noncompliance with the study's plan. If content errors are discovered, a reassessment of the experimental software (ES) is needed, followed by retesting after corrections are made and inaccuracies are corrected.

For event timing, the precision of the planned timings should be reported. We advocate reporting the mean and standard deviation \({\Delta }_{i}\) . The mean is expected to approach zero, and the standard deviation to be within the few milliseconds range. Larger values are suggestive of errors or hardware issues that might require attention.

Testing peripheral triggers

So far, we have described tests to benchmark the EC and ES. Yet, when the experimental environment contains peripherals (a typical case for neuroscience experiments), tests to assess the interaction of the peripherals with the EC and ES (i.e., triggers) are also necessary. Triggers serve a dual purpose here: (1) they provide temporal markers for events of interest on which to focus offline data analysis, and (2) they play a pivotal role in maintaining synchrony between the EC and the recording system (e.g., electroencephalography [EEG], eye-tracking device). This is useful for addressing issues related to clock drift, which can be detected and corrected when sending triggers marking events of interest to multiple devices (Niso et al., 2022 ).

The interpretation of the signal recorded by these devices depends on the integrity of the trigger transmissions (Boudewyn et al., 2023 ; Luck, 2014 ), which is the focus of the current test. Akin to previous tests, both the content and the timing of the triggers representing the controlled and uncontrolled events are evaluated. These tests should be performed for each peripheral device used in the experiment.

Peripheral trigger content

Assuming the log file event records are accurate, the congruence between each logged event and its corresponding trigger content is assessed. Any deviations point to problems with trigger logging or the peripheral device, necessitating review and correction.

Peripheral trigger timing

Compute the discrete difference between successive peripheral trigger timestamps, and compare it against the discrete differences between consecutive observed events, as recorded by the photodiode. The difference between these two arrays provides an estimate of the peripheral trigger temporal jitter ( \({\Delta }_{i}\) ):

Note that this method evaluates temporal jitter but is not suited to evaluate delays between events and peripheral triggers. As described by Farzan et al. ( 2017 ), delays can be measured by recording the photodiode signal on the same computer clock as the peripheral of interest. The delay of the trigger onset (marked below as \({\Delta }_{i}\) too) can then be directly compared to the detected photodiode onset as:

The count of events where the trigger content does not align with the log file content should be documented, and reported as the count of mismatched events over the total count of events.

The report should contain the average and standard deviation of trigger timings. For systematic delay tests, both the mean and standard deviation should be reported.

Standardized report

The framework is summarized in a checklist available on protocols.io, alongside a standardized format for reporting these results. Both the checklist and the report are in the protocols.io platform, under this link . Testing the experimental environment is crucial, and so is the accompanying detailed report of the test results to altogether enhance transparency and foster reproducibility and replicability.

We present a framework to systematically test and report the performance of experimental environments. We aimed to minimize financial burden by relying on hardware that either most labs already have, or is inexpensive to acquire or build (e.g., photodiode and microphone). Our protocol enables all tests to be conducted in a single test run, making the framework more efficient for researchers and preventing additional work due to errors or limitations discovered during the data collection stage. Furthermore, the framework also includes guidelines for detailed reporting of the parameters and results of each test, as means to increase scientific transparency. Such transparency allows the scientific community to better evaluate the conclusions of the study, as they rely heavily on the proper functioning of the experiment. As indicated by the survey results, more than half of researchers did not share their test results because they did not know where or how to write them; the current framework would hopefully help overcome this hurdle.

The framework presented herein is designed to be applicable to most recording modalities used in human neurosciences. To test the content and timing of the experimental events, the only requirement is that the peripherals are capable of receiving triggers (which is a standard feature in experimental environments containing peripherals in event-related designs). To assess event timing accuracy, jitters are estimated by comparing the intervals between events recorded by the system with those obtained from physical measurements: jitter-free peripherals should show identical intervals. Although the clocks of different devices may drift apart over long periods, computing the intervals between events occurring in relatively short succession overcomes this problem. Notably, this method does not account for systematic delays between two systems (i.e., if a peripheral device receives the triggers with a systematic delay of 30 ms with respect to the physical event). To address this, systematic delays should be quantified by recording the physical signals on the same computer as the peripheral being used, as described in section " Testing peripheral triggers " (Eq. 8 ). As such, our framework offers a comprehensive set of tests for timing issues applicable to a wide range of technologies used in cognitive neuroscience.

Investing extra time and resources in testing and reporting the experimental environment is worthwhile, as simulations show that malfunctions in recording event timing and content can significantly impact results. Researchers should aim to reduce controllable errors and characterize noise in their setups to increase the likelihood of detecting real effects and reduce false negatives. The benefits of this framework extend beyond individual experiments, enhancing replication efforts and scientific reliability. Recent replication challenges in neuroscience and psychology (e.g., Kristal et al., 2020 ) highlight the need for quality assurance in experimental environments, especially when multiple labs collaborate (e.g., COGITATE: COGITATE Consortium et al., 2023 ; eegManyLabs: Pavlov et al., 2021 ; The International Brain Laboratory et al., 2021 ). Without strict quality controls, the potential benefits of multi-site data collection risk being overshadowed by inter-site variability, masking real effects observable in single-site datasets (e.g., de Vries et al., 2022 ; Farzan et al., 2017 ). The step-by-step process of conducting the framework on protocols.io enables researchers to thoroughly test their experiments in an effective and relatively non-time-consuming manner. A detailed demonstration of the application of our framework to an experiment is provided as a Jupyter notebook (see Supplemental Material 3 ).

We believe the short time invested is outweighed by the benefits in the long run. We acknowledge that for some, the proposed framework and reporting approach might seem excessive and may also be met with skepticism, as it may increase the burden on the researchers. Yet, as our survey shows, most researchers do encounter issues only after data collection begins. Addressing errors retrospectively is time-consuming, and in extreme cases, undetected issues can lead to retractions (e.g., Grave et al., 2021 ) or flawed results. Standardized testing and reporting can identify problems early, aiding replication and consistency across studies, including multi-lab projects. We hope this practice will gradually become integral to good scientific conduct, similar to the adoption of preregistration, which was initially met with skepticism (Paret et al., 2022 ), as it required more effort and resources—but over time proved to be highly beneficial (Gentili et al., 2021 ; Protzko et al., 2023 ).

Finally, we argue that our proposed framework may not be costlier than current testing practices in the field. Our survey indicates that researchers already invest time in testing their experimental setups and also recognize the need for testing before data collection. Yet, these efforts often go unreported. Without a standardized test protocol, each researcher and lab must devise their own methods, with many creating unique tests for each experiment. Our framework outlines four key aspects to test in event-based experiments: reliability of log file (1) event content and (2) event timing, (3) fulfillment of the experimental design, and (4) reliability of the peripheral device. These tests are broad enough to cover most visual presentation designs, needing only minor adjustments for specific cases. With careful planning, a single full experimental run, including all peripherals, can suffice for comprehensive testing. Most steps, except for logging event content, can be automated with minimal execution time. While script development might initially take time, these scripts are generally reusable across studies, offering long-term efficiency.

Thus, we believe that the research community can benefit from these resources. This framework can enhance the credibility of research findings, improve research efficiency and cost-effectiveness, and, by reporting test results, increase the transparency and reproducibility of research methods.

Code and data availability

All code and data used and generated for this paper are openly available at https://github.com/Cogitate-consortium/ExperimentTestingFramework

Baxter, M. G., & Burwell, R. D. (2017). Promoting transparency and reproducibility in Behavioral Neuroscience: Publishing replications, registered reports, and null results. Behavioral Neuroscience, 131 (4), 275–276. https://doi.org/10.1037/bne0000207

Article   PubMed   PubMed Central   Google Scholar  

Boudewyn, M. A., Erickson, M. A., Winsler, K., Ragland, J. D., Yonelinas, A., Frank, M., Silverstein, S. M., Gold, J., MacDonald III, A. W., Carter, C. S., Barch, D. M., & Luck, S. J. (2023). Managing EEG studies: How to prepare and what to do once data collection has begun. Psychophysiology , n/a (n/a), e14365. https://doi.org/10.1111/psyp.14365

Brainard, D. H. (1997). The psychophysics toolbox. Spatial Vision, 10 (4), 433–436.

Article   PubMed   Google Scholar  

Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: Comparing a range of experiment generators, both lab-based and online. PeerJ, 8 , e9414. https://doi.org/10.7717/peerj.9414

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14 (5), 365–376.

Calcagnotto, L., Huskey, R., & Kosicki, G. M. (2021). The accuracy and precision of measurement: Tools for validating reaction time stimuli. Computational Communication Research, 3 (2), 1–20.

Article   Google Scholar  

Carp, J. (2012). On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments. Frontiers in Neuroscience, 6 , 149. https://doi.org/10.3389/fnins.2012.00149

Consortium, C., Ferrante, O., Gorska-Klimowska, U., Henin, S., Hirschhorn, R., Khalaf, A., Lepauvre, A., Liu, L., Richter, D., Vidal, Y., Bonacchi, N., Brown, T., Sripad, P., Armendariz, M., Bendtz, K., Ghafari, T., Hetenyi, D., Jeschke, J., Kozma, C., …, & Melloni, L. (2023). An adversarial collaboration to critically evaluate theories of consciousness (p. 2023.06.23.546249). bioRxiv. https://doi.org/10.1101/2023.06.23.546249

Cumming, G. (2014). The new statistics: why and how. Psychological Science, 25 (1), 7–29.

de Vries, S. E. J., Siegle, J. H., & Koch, C. (2022). Sharing Neurophysiology Data from the Allen Brain Observatory: Lessons Learned (arXiv:2212.08638). arXiv. https://doi.org/10.48550/arXiv.2212.08638

Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90 (3), 891–904.

Farzan, F., Atluri, S., Frehlich, M., Dhami, P., Kleffner, K., Price, R., Lam, R. W., Frey, B. N., Milev, R., Ravindran, A., McAndrews, M. P., Wong, W., Blumberger, D., Daskalakis, Z. J., Vila-Rodriguez, F., Alonso, E., Brenner, C. A., Liotti, M., Dharsee, M., & Kennedy, S. H. (2017). Standardization of electroencephalography for multi-site, multi-platform and multi-investigator studies: Insights from the Canadian biomarker integration network in depression. Scientific Reports, 7 (1), 1. https://doi.org/10.1038/s41598-017-07613-x

Frank, M. C., Bergelson, E., Bergmann, C., Cristia, A., Floccia, C., Gervain, J., Hamlin, J. K., Hannon, E. E., Kline, M., Levelt, C., Lew-Williams, C., Nazzi, T., Panneton, R., Rabagliati, H., Soderstrom, M., Sullivan, J., Waxman, S., & Yurovsky, D. (2017). A Collaborative approach to infant research: Promoting reproducibility, best practices, and theory-building. Infancy, 22 (4), 421–435. https://doi.org/10.1111/infa.12182

Gentili, C., Cecchetti, L., Handjaras, G., Lettieri, G., & Cristea, I. A. (2021). The case for preregistering all region of interest (ROI) analyses in neuroimaging research. European Journal of Neuroscience, 53 (2), 357–361. https://doi.org/10.1111/ejn.14954

Grave, J., Soares, S. C., Morais, S., Rodrigues, P., & Madeira, N. (2021). Retraction notice to “The effects of perceptual load in processing emotional facial expression in psychotic disorders” [Psychiatry Research Volume 250C April 2017, pages 121—128]. Psychiatry Research, 303 , 114077. https://doi.org/10.1016/j.psychres.2021.114077

Hirschhorn, R., & Schonberg, T. (2024). Replication. In  Encyclopedia of the Human Brain (2nd ed.). Elsevier.

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2 (8), e124.

Kothe, C., Shirazi, S. Y., Stenner, T., Medine, D., Boulay, C., Crivich, M. I., ... & Makeig, S. (2024). The lab streaming layer for synchronized multimodal recording. bioRxiv , 2024-02. https://doi.org/10.1101/2024.02.13.580071

Kristal, A. S., Whillans, A. V., Bazerman, M. H., Gino, F., Shu, L. L., Mazar, N., & Ariely, D. (2020). Signing at the beginning versus at the end does not decrease dishonesty. Proceedings of the National Academy of Sciences, 117 (13), 7103–7107. https://doi.org/10.1073/pnas.1911695117

Logg, J. M., & Dorison, C. A. (2021). Pre-registration: Weighing costs and benefits for researchers. Organizational Behavior and Human Decision Processes, 167 , 18–27. https://doi.org/10.1016/j.obhdp.2021.05.006

Luck, S. J. (2014). An Introduction to the Event-Related Potential Technique (2nd ed.). MIT Press.

Google Scholar  

Melloni, L., Mudrik, L., Pitts, M., & Koch, C. (2021). Making the hard problem of consciousness easier. Science, 372 (6545), 911–912. https://doi.org/10.1126/science.abj3259

Mumford, J. A., & Nichols, T. E. (2008). Power calculation for group fMRI studies accounting for arbitrary design and temporal autocorrelation. NeuroImage, 39 (1), 261–268. https://doi.org/10.1016/j.neuroimage.2007.07.061

Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1 (1), 1–9. https://doi.org/10.1038/s41562-016-0021

Niso, G., Krol, L. R., Combrisson, E., Dubarry, A. S., Elliott, M. A., François, C., Héjja-Brichard, Y., Herbst, S. K., Jerbi, K., Kovic, V., Lehongre, K., Luck, S. J., Mercier, M., Mosher, J. C., Pavlov, Y. G., Puce, A., Schettino, A., Schön, D., Sinnott-Armstrong, W., …, Chaumon, M. (2022). Good scientific practice in EEG and MEG research: Progress and perspectives. NeuroImage , 257 , 119056. https://doi.org/10.1016/j.neuroimage.2022.119056

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349 (6251), aac4716.

Paret, C., Unverhau, N., Feingold, F., Poldrack, R. A., Stirner, M., Schmahl, C., & Sicorello, M. (2022). Survey on open science practices in functional neuroimaging. NeuroImage, 257 , 119306. https://doi.org/10.1016/j.neuroimage.2022.119306

Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C. S. Y., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., Clayson, P. E., Cruse, D., Czeszumski, A., Dreber, A., Dumas, G., Ehinger, B., Ganis, G., He, X., Hinojosa, J. A., …, & Mushtaq, F. (2021). #EEGManyLabs: Investigating the replicability of influential EEG experiments. Cortex , 144 , 213–229. https://doi.org/10.1016/j.cortex.2021.03.013

Peirce, J. W. (2007). PsychoPy—Psychophysics software in Python. Journal of Neuroscience Methods, 162 (1), 8–13. https://doi.org/10.1016/j.jneumeth.2006.11.017

Pernet, C., Garrido, M., Gramfort, A., Maurits, N., Michel, C. M., Pang, E., Salmelin, R., Schoffelen, J. M., Valdes-Sosa, P. A., & Puce, A. (2018). Best practices in data analysis and sharing in neuroimaging using MEEG . https://doi.org/10.31219/osf.io/a8dhx

Plant, R. R. (2016). A reminder on millisecond timing accuracy and potential replication failure in computer-based psychology experiments: An open letter. Behavior Research Methods, 48 (1), 408–411.

Poldrack, R. A., Fletcher, P. C., Henson, R. N., Worsley, K. J., Brett, M., & Nichols, T. E. (2008). Guidelines for reporting an fMRI study. Neuroimage, 40 (2), 409–414.

Protzko, J., Krosnick, J., Nelson, L., Nosek, B. A., Axt, J., Berent, M., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., MacInnis, B., O’Donnell, M., Perfecto, H., Pustejovsky, J. E., Roeder, S. S., Walleczek, J., & Schooler, J. W. (2023). High replicability of newly discovered social-behavioural findings is achievable. Nature Human Behaviour, 8 (2), 311–319. https://doi.org/10.1038/s41562-023-01749-9

Schlossmacher, I., Dellert, T., Pitts, M., Bruchmann, M., & Straube, T. (2020). Differential Effects of Awareness and Task Relevance on Early and Late ERPs in a No-Report Visual Oddball Paradigm. Journal of Neuroscience, 40 (14), 2906–2913. https://doi.org/10.1523/JNEUROSCI.2077-19.2020

Sejnowski, T. J., Churchland, P. S., & Movshon, J. A. (2014). Putting big data to good use in neuroscience. Nature Neuroscience, 17 (11), 11. https://doi.org/10.1038/nn.3839

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22 (11), 1359–1366.

The International Brain Laboratory, Aguillon-Rodriguez, V., Angelaki, D., Bayer, H., Bonacchi, N., Carandini, M., Cazettes, F., Chapuis, G., Churchland, A. K., Dan, Y., Dewitt, E., Faulkner, M., Forrest, H., Haetzel, L., Häusser, M., Hofer, S. B., Hu, F., Khanal, A., Krasniak, C., …, & Zador, A. M. (2021). Standardized and reproducible measurement of decision-making in mice. eLife , 10 , e63711. https://doi.org/10.7554/eLife.63711

van Gaal, S., Ridderinkhof, K. R., Scholte, H. S., & Lamme, V. A. F. (2010). Unconscious Activation of the Prefrontal No-Go Network. Journal of Neuroscience, 30 (11), 4143–4150. https://doi.org/10.1523/JNEUROSCI.2992-09.2010

Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in psychology, 7 , 1832. https://doi.org/10.3389/fpsyg.2016.01832

Download references

Acknowledgments

We would like to thank the members of the COGITATE consortium for the inspiration for and support of this manuscript. Furthermore, we want to thank Sarah Brendecke and Felix Bernoulli from the graphics department of the Max Planck Institute for empirical aesthetics for their support in generating the graphics of this paper as well as Ryszard Auksztulewicz for his support with the ERP simulation. A special thanks to Tanya Brown for help with the formatting of the materials and to the members of the NCClab for their input on the testing framework.

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Neural Circuits, Consciousness and Cognition Research Group, Max Planck Institute of Empirical Aesthetics, Frankfurt am Main, Germany

Alex Lepauvre & Lucia Melloni

Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, 6500 HB, the Netherlands

Alex Lepauvre

Sagol School of Neuroscience, Tel-Aviv University, Tel Aviv, Israel

Rony Hirschhorn & Liad Mudrik

Boston Children’s Hospital, Harvard Medical School, Boston, USA

Katarina Bendtz

School of Psychological Sciences, Tel-Aviv University, Tel Aviv, Israel

Liad Mudrik

Department of Neurology, NYU Grossman School of Medicine, New York, USA

Lucia Melloni

Canadian Institute for Advanced Research (CIFAR), Brain, Mind, and Consciousness Program, Toronto, ON, Canada

Liad Mudrik & Lucia Melloni

You can also search for this author in PubMed   Google Scholar

Contributions

The authors confirm their contribution to the paper as follows: Conceptualization : A. Lepauvre, R. Hirschhorn, K. Bendtz, L. Mudrik, L. Melloni; Data curation : A. Lepauvre, R. Hirschhorn; Formal Analysis : A. Lepauvre, R. Hirschhorn; Funding acquisition : L. Melloni, L. Mudrik; Investigation : A. Lepauvre, R. Hirschhorn, K. Bendtz; Methodology : A. Lepauvre, R. Hirschhorn, K. Bendtz, L. Melloni; Project administration : A. Lepauvre; Resources : L. Melloni, L. Mudrik; Software : A. Lepauvre, R. Hirschhorn; Supervision : L. Melloni, L. Mudrik; Visualization : A. Lepauvre, R. Hirschhorn; Writing—original draft : A. Lepauvre, R. Hirschhorn, K. Bendtz; Writing—review & editing : A. Lepauvre, R. Hirschhorn, L. Melloni, L. Mudrik

Corresponding author

Correspondence to Alex Lepauvre .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 1588 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Lepauvre, A., Hirschhorn, R., Bendtz, K. et al. A standardized framework to test event-based experiments. Behav Res (2024). https://doi.org/10.3758/s13428-024-02508-y

Download citation

Accepted : 15 August 2024

Published : 16 September 2024

DOI : https://doi.org/10.3758/s13428-024-02508-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Replication
• Experimental methods
• Pre-acquisition tests
• Find a journal
• Publish with us
• Track your research