program evaluation research

Program Evaluation Research

Cite this chapter.

• Jerald Jay Thomas  

1468 Accesses

According to the Joint Committee on Standards for Educational Evaluation (2011), program evaluation as a method of research is a means of systematically evaluating an object or educational program. As straightforward and succinct as that definition is, you will find that evaluation research borrows heavily from other methods of research. Evaluation research has at its root the assumption that the value, quality, and effectiveness of an educational program can be appraised through a variety of data sources. As educators, we find ourselves making evaluations daily, and in a variety of contexts. The evaluations we make according to Fitzpatrick, Sanders, and Worthen (2011) fall along a continuum between formal evaluation and informal evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Similar content being viewed by others

Evaluating Educational Programs

Evaluation Paradigms

A Case Study for the 10-Step Approach to Program Evaluation

American Educational Research Association. (2011). Codeof ethics of the American Educational Research Association . Retrieved from http://www.aera.net/uploadedFiles/About_AERA/Ethical_Standards/CodeOfEthics%281%29.pdf.

Google Scholar  

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines . Upper Saddle River, NJ: Pearson Education.

Greene, J. C. (2005). Mixed methods. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 397–298). Thousand Oaks, CA: Sage.

Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards . Newbury Park, CA: Sage.

McNeil, K., Newman, I., & Steinhauser, J. (2005). How to be involved in program evaluation . Lanham, MD: Scarecrow Education.

Sanders, J. R. (2000). Evaluating school programs: An educator’s guide. Thousand Oaks, CA: Corwin Press.

Scriven, M. (1991). Beyond formative and summative evaluation. In G. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: A quarter century (pp. 19–64). Chicago, IL: University of Chicago Press.

Download references

You can also search for this author in PubMed   Google Scholar

Editor information

Copyright information.

© 2012 Sheri R. Klein

About this chapter

Thomas, J.J. (2012). Program Evaluation Research. In: Klein, S.R. (eds) Action Research Methods. Palgrave Macmillan, New York. https://doi.org/10.1057/9781137046635_9

Download citation

DOI : https://doi.org/10.1057/9781137046635_9

Publisher Name : Palgrave Macmillan, New York

Print ISBN : 978-1-349-29560-9

Online ISBN : 978-1-137-04663-5

eBook Packages : Palgrave Education Collection Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

A Framework for Program Evaluation

Effective program evaluation is a systematic way to improve and account for public health actions by involving procedures that are useful, feasible, ethical, and accurate. The Framework for Evaluation in Public Health  guides public health professionals in their use of program evaluation. It is a practical, nonprescriptive tool, designed to summarize and organize essential elements of program evaluation.

Adhering to the steps and standards of this framework will allow an understanding of each program’s context and will improve how program evaluations are conceived and conducted. Furthermore, the framework encourages an approach to evaluation that is integrated with routine program operations.

The emphasis is on practical, ongoing evaluation strategies that involve all program stakeholders, not just evaluation experts.

Understanding and applying the elements of this framework can be a driving force for planning effective public health strategies, improving existing programs, and demonstrating the results of resource investments.

Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48 (No. RR-11)

• summarize the essential elements of program evaluation,
• provide a framework for conducting effective program evaluations,
• clarify steps in program evaluation,
• review standards for effective program evaluation, and
• address misconceptions regarding the purposes and methods of program evaluation.
• Framework for Program Evaluation in Public Health – MMWR 1999;48(No. RR-11)
• Overview of the Framework – Provides 3-page key points from framework
• Framework Summary – Provides summary of the framework with details
• CDC Evaluation Framework – YouTube video describing the evaluation framework

E-mail: [email protected]

To receive email updates about this page, enter your email address:

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case AskWhy Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Evaluation Research: Definition, Methods and Examples

Content Index

• What is evaluation research
• Why do evaluation research

Quantitative methods

Qualitative methods.

• Process evaluation research question examples
• Outcome evaluation research question examples

What is evaluation research?

Evaluation research, also known as program evaluation, refers to research purpose instead of a specific method. Evaluation research is the systematic assessment of the worth or merit of time, money, effort and resources spent in order to achieve a goal.

Evaluation research is closely related to but slightly different from more conventional social research . It uses many of the same methods used in traditional social research, but because it takes place within an organizational context, it requires team skills, interpersonal skills, management skills, political smartness, and other research skills that social research does not need much. Evaluation research also requires one to keep in mind the interests of the stakeholders.

Evaluation research is a type of applied research, and so it is intended to have some real-world effect.  Many methods like surveys and experiments can be used to do evaluation research. The process of evaluation research consisting of data analysis and reporting is a rigorous, systematic process that involves collecting data about organizations, processes, projects, services, and/or resources. Evaluation research enhances knowledge and decision-making, and leads to practical applications.

LEARN ABOUT: Action Research

Why do evaluation research?

The common goal of most evaluations is to extract meaningful information from the audience and provide valuable insights to evaluators such as sponsors, donors, client-groups, administrators, staff, and other relevant constituencies. Most often, feedback is perceived value as useful if it helps in decision-making. However, evaluation research does not always create an impact that can be applied anywhere else, sometimes they fail to influence short-term decisions. It is also equally true that initially, it might seem to not have any influence, but can have a delayed impact when the situation is more favorable. In spite of this, there is a general agreement that the major goal of evaluation research should be to improve decision-making through the systematic utilization of measurable feedback.

Below are some of the benefits of evaluation research

• Gain insights about a project or program and its operations

Evaluation Research lets you understand what works and what doesn’t, where we were, where we are and where we are headed towards. You can find out the areas of improvement and identify strengths. So, it will help you to figure out what do you need to focus more on and if there are any threats to your business. You can also find out if there are currently hidden sectors in the market that are yet untapped.

• Improve practice

It is essential to gauge your past performance and understand what went wrong in order to deliver better services to your customers. Unless it is a two-way communication, there is no way to improve on what you have to offer. Evaluation research gives an opportunity to your employees and customers to express how they feel and if there’s anything they would like to change. It also lets you modify or adopt a practice such that it increases the chances of success.

• Assess the effects

After evaluating the efforts, you can see how well you are meeting objectives and targets. Evaluations let you measure if the intended benefits are really reaching the targeted audience and if yes, then how effectively.

• Build capacity

Evaluations help you to analyze the demand pattern and predict if you will need more funds, upgrade skills and improve the efficiency of operations. It lets you find the gaps in the production to delivery chain and possible ways to fill them.

Methods of evaluation research

All market research methods involve collecting and analyzing the data, making decisions about the validity of the information and deriving relevant inferences from it. Evaluation research comprises of planning, conducting and analyzing the results which include the use of data collection techniques and applying statistical methods.

Some of the evaluation methods which are quite popular are input measurement, output or performance measurement, impact or outcomes assessment, quality assessment, process evaluation, benchmarking, standards, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. There are also a few types of evaluations that do not always result in a meaningful assessment such as descriptive studies, formative evaluations, and implementation analysis. Evaluation research is more about information-processing and feedback functions of evaluation.

These methods can be broadly classified as quantitative and qualitative methods.

The outcome of the quantitative research methods is an answer to the questions below and is used to measure anything tangible.

• Who was involved?
• What were the outcomes?
• What was the price?

The best way to collect quantitative data is through surveys , questionnaires , and polls . You can also create pre-tests and post-tests, review existing documents and databases or gather clinical data.

Surveys are used to gather opinions, feedback or ideas of your employees or customers and consist of various question types . They can be conducted by a person face-to-face or by telephone, by mail, or online. Online surveys do not require the intervention of any human and are far more efficient and practical. You can see the survey results on dashboard of research tools and dig deeper using filter criteria based on various factors such as age, gender, location, etc. You can also keep survey logic such as branching, quotas, chain survey, looping, etc in the survey questions and reduce the time to both create and respond to the donor survey . You can also generate a number of reports that involve statistical formulae and present data that can be readily absorbed in the meetings. To learn more about how research tool works and whether it is suitable for you, sign up for a free account now.

Create a free account!

Quantitative data measure the depth and breadth of an initiative, for instance, the number of people who participated in the non-profit event, the number of people who enrolled for a new course at the university. Quantitative data collected before and after a program can show its results and impact.

The accuracy of quantitative data to be used for evaluation research depends on how well the sample represents the population, the ease of analysis, and their consistency. Quantitative methods can fail if the questions are not framed correctly and not distributed to the right audience. Also, quantitative data do not provide an understanding of the context and may not be apt for complex issues.

Learn more: Quantitative Market Research: The Complete Guide

Qualitative research methods are used where quantitative methods cannot solve the research problem , i.e. they are used to measure intangible values. They answer questions such as

• What is the value added?
• How satisfied are you with our service?
• How likely are you to recommend us to your friends?
• What will improve your experience?

LEARN ABOUT: Qualitative Interview

Qualitative data is collected through observation, interviews, case studies, and focus groups. The steps for creating a qualitative study involve examining, comparing and contrasting, and understanding patterns. Analysts conclude after identification of themes, clustering similar data, and finally reducing to points that make sense.

Observations may help explain behaviors as well as the social context that is generally not discovered by quantitative methods. Observations of behavior and body language can be done by watching a participant, recording audio or video. Structured interviews can be conducted with people alone or in a group under controlled conditions, or they may be asked open-ended qualitative research questions . Qualitative research methods are also used to understand a person’s perceptions and motivations.

LEARN ABOUT:  Social Communication Questionnaire

The strength of this method is that group discussion can provide ideas and stimulate memories with topics cascading as discussion occurs. The accuracy of qualitative data depends on how well contextual data explains complex issues and complements quantitative data. It helps get the answer of “why” and “how”, after getting an answer to “what”. The limitations of qualitative data for evaluation research are that they are subjective, time-consuming, costly and difficult to analyze and interpret.

Learn more: Qualitative Market Research: The Complete Guide

Survey software can be used for both the evaluation research methods. You can use above sample questions for evaluation research and send a survey in minutes using research software. Using a tool for research simplifies the process right from creating a survey, importing contacts, distributing the survey and generating reports that aid in research.

Examples of evaluation research

Evaluation research questions lay the foundation of a successful evaluation. They define the topics that will be evaluated. Keeping evaluation questions ready not only saves time and money, but also makes it easier to decide what data to collect, how to analyze it, and how to report it.

Evaluation research questions must be developed and agreed on in the planning stage, however, ready-made research templates can also be used.

Process evaluation research question examples:

• How often do you use our product in a day?
• Were approvals taken from all stakeholders?
• Can you report the issue from the system?
• Can you submit the feedback from the system?
• Was each task done as per the standard operating procedure?
• What were the barriers to the implementation of each task?
• Were any improvement areas discovered?

Outcome evaluation research question examples:

• How satisfied are you with our product?
• Did the program produce intended outcomes?
• What were the unintended outcomes?
• Has the program increased the knowledge of participants?
• Were the participants of the program employable before the course started?
• Do participants of the program have the skills to find a job after the course ended?
• Is the knowledge of participants better compared to those who did not participate in the program?

MORE LIKE THIS

Interactive Forms: Key Features, Benefits, Uses + Design Tips

Sep 4, 2024

Closed-Loop Management: The Key to Customer Centricity

Sep 3, 2024

Net Trust Score: Tool for Measuring Trust in Organization

Sep 2, 2024

Why You Should Attend XDAY 2024

Aug 30, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Tuesday CX Thoughts (TCXT)
• Uncategorized
• What’s Coming Up
• Workforce Intelligence

• Evaluation Research Design: Examples, Methods & Types

As you engage in tasks, you will need to take intermittent breaks to determine how much progress has been made and if any changes need to be effected along the way. This is very similar to what organizations do when they carry out  evaluation research.  

The evaluation research methodology has become one of the most important approaches for organizations as they strive to create products, services, and processes that speak to the needs of target users. In this article, we will show you how your organization can conduct successful evaluation research using Formplus .

What is Evaluation Research?

Also known as program evaluation, evaluation research is a common research design that entails carrying out a structured assessment of the value of resources committed to a project or specific goal. It often adopts social research methods to gather and analyze useful information about organizational processes and products.  

As a type of applied research , evaluation research typically associated  with real-life scenarios within organizational contexts. This means that the researcher will need to leverage common workplace skills including interpersonal skills and team play to arrive at objective research findings that will be useful to stakeholders. 

Characteristics of Evaluation Research

• Research Environment: Evaluation research is conducted in the real world; that is, within the context of an organization. 
• Research Focus: Evaluation research is primarily concerned with measuring the outcomes of a process rather than the process itself. 
• Research Outcome: Evaluation research is employed for strategic decision making in organizations. 
• Research Goal: The goal of program evaluation is to determine whether a process has yielded the desired result(s). 
• This type of research protects the interests of stakeholders in the organization. 
• It often represents a middle-ground between pure and applied research. 
• Evaluation research is both detailed and continuous. It pays attention to performative processes rather than descriptions. 
• Research Process: This research design utilizes qualitative and quantitative research methods to gather relevant data about a product or action-based strategy. These methods include observation, tests, and surveys.

Types of Evaluation Research

The Encyclopedia of Evaluation (Mathison, 2004) treats forty-two different evaluation approaches and models ranging from “appreciative inquiry” to “connoisseurship” to “transformative evaluation”. Common types of evaluation research include the following: 

• Formative Evaluation

Formative evaluation or baseline survey is a type of evaluation research that involves assessing the needs of the users or target market before embarking on a project.  Formative evaluation is the starting point of evaluation research because it sets the tone of the organization’s project and provides useful insights for other types of evaluation.  

• Mid-term Evaluation

Mid-term evaluation entails assessing how far a project has come and determining if it is in line with the set goals and objectives. Mid-term reviews allow the organization to determine if a change or modification of the implementation strategy is necessary, and it also serves for tracking the project. 

• Summative Evaluation

This type of evaluation is also known as end-term evaluation of project-completion evaluation and it is conducted immediately after the completion of a project. Here, the researcher examines the value and outputs of the program within the context of the projected results. 

Summative evaluation allows the organization to measure the degree of success of a project. Such results can be shared with stakeholders, target markets, and prospective investors. 

• Outcome Evaluation

Outcome evaluation is primarily target-audience oriented because it measures the effects of the project, program, or product on the users. This type of evaluation views the outcomes of the project through the lens of the target audience and it often measures changes such as knowledge-improvement, skill acquisition, and increased job efficiency. 

• Appreciative Enquiry

Appreciative inquiry is a type of evaluation research that pays attention to result-producing approaches. It is predicated on the belief that an organization will grow in whatever direction its stakeholders pay primary attention to such that if all the attention is focused on problems, identifying them would be easy. 

In carrying out appreciative inquiry, the research identifies the factors directly responsible for the positive results realized in the course of a project, analyses the reasons for these results, and intensifies the utilization of these factors. 

Evaluation Research Methodology 

There are four major evaluation research methods, namely; output measurement, input measurement, impact assessment and service quality

• Output/Performance Measurement

Output measurement is a method employed in evaluative research that shows the results of an activity undertaking by an organization. In other words, performance measurement pays attention to the results achieved by the resources invested in a specific activity or organizational process. 

More than investing resources in a project, organizations must be able to track the extent to which these resources have yielded results, and this is where performance measurement comes in. Output measurement allows organizations to pay attention to the effectiveness and impact of a process rather than just the process itself. 

Other key indicators of performance measurement include user-satisfaction, organizational capacity, market penetration, and facility utilization. In carrying out performance measurement, organizations must identify the parameters that are relevant to the process in question, their industry, and the target markets. 

5 Performance Evaluation Research Questions Examples

• What is the cost-effectiveness of this project?
• What is the overall reach of this project?
• How would you rate the market penetration of this project?
• How accessible is the project? 
• Is this project time-efficient? 

• Input Measurement

In evaluation research, input measurement entails assessing the number of resources committed to a project or goal in any organization. This is one of the most common indicators in evaluation research because it allows organizations to track their investments. 

The most common indicator of inputs measurement is the budget which allows organizations to evaluate and limit expenditure for a project. It is also important to measure non-monetary investments like human capital; that is the number of persons needed for successful project execution and production capital. 

5 Input Evaluation Research Questions Examples

• What is the budget for this project?
• What is the timeline of this process?
• How many employees have been assigned to this project? 
• Do we need to purchase new machinery for this project? 
• How many third-parties are collaborators in this project? 

• Impact/Outcomes Assessment

In impact assessment, the evaluation researcher focuses on how the product or project affects target markets, both directly and indirectly. Outcomes assessment is somewhat challenging because many times, it is difficult to measure the real-time value and benefits of a project for the users. 

In assessing the impact of a process, the evaluation researcher must pay attention to the improvement recorded by the users as a result of the process or project in question. Hence, it makes sense to focus on cognitive and affective changes, expectation-satisfaction, and similar accomplishments of the users. 

5 Impact Evaluation Research Questions Examples

• How has this project affected you? 
• Has this process affected you positively or negatively?
• What role did this project play in improving your earning power? 
• On a scale of 1-10, how excited are you about this project?
• How has this project improved your mental health? 

• Service Quality

Service quality is the evaluation research method that accounts for any differences between the expectations of the target markets and their impression of the undertaken project. Hence, it pays attention to the overall service quality assessment carried out by the users. 

It is not uncommon for organizations to build the expectations of target markets as they embark on specific projects. Service quality evaluation allows these organizations to track the extent to which the actual product or service delivery fulfils the expectations. 

5 Service Quality Evaluation Questions

• On a scale of 1-10, how satisfied are you with the product?
• How helpful was our customer service representative?
• How satisfied are you with the quality of service?
• How long did it take to resolve the issue at hand?
• How likely are you to recommend us to your network?

Uses of Evaluation Research 

• Evaluation research is used by organizations to measure the effectiveness of activities and identify areas needing improvement. Findings from evaluation research are key to project and product advancements and are very influential in helping organizations realize their goals efficiently.     
• The findings arrived at from evaluation research serve as evidence of the impact of the project embarked on by an organization. This information can be presented to stakeholders, customers, and can also help your organization secure investments for future projects. 
• Evaluation research helps organizations to justify their use of limited resources and choose the best alternatives. 
•  It is also useful in pragmatic goal setting and realization. 
• Evaluation research provides detailed insights into projects embarked on by an organization. Essentially, it allows all stakeholders to understand multiple dimensions of a process, and to determine strengths and weaknesses. 
• Evaluation research also plays a major role in helping organizations to improve their overall practice and service delivery. This research design allows organizations to weigh existing processes through feedback provided by stakeholders, and this informs better decision making. 
• Evaluation research is also instrumental to sustainable capacity building. It helps you to analyze demand patterns and determine whether your organization requires more funds, upskilling or improved operations.

Data Collection Techniques Used in Evaluation Research

In gathering useful data for evaluation research, the researcher often combines quantitative and qualitative research methods . Qualitative research methods allow the researcher to gather information relating to intangible values such as market satisfaction and perception. 

On the other hand, quantitative methods are used by the evaluation researcher to assess numerical patterns, that is, quantifiable data. These methods help you measure impact and results; although they may not serve for understanding the context of the process. 

Quantitative Methods for Evaluation Research

A survey is a quantitative method that allows you to gather information about a project from a specific group of people. Surveys are largely context-based and limited to target groups who are asked a set of structured questions in line with the predetermined context.

Surveys usually consist of close-ended questions that allow the evaluative researcher to gain insight into several  variables including market coverage and customer preferences. Surveys can be carried out physically using paper forms or online through data-gathering platforms like Formplus . 

• Questionnaires

A questionnaire is a common quantitative research instrument deployed in evaluation research. Typically, it is an aggregation of different types of questions or prompts which help the researcher to obtain valuable information from respondents. 

A poll is a common method of opinion-sampling that allows you to weigh the perception of the public about issues that affect them. The best way to achieve accuracy in polling is by conducting them online using platforms like Formplus. 

Polls are often structured as Likert questions and the options provided always account for neutrality or indecision. Conducting a poll allows the evaluation researcher to understand the extent to which the product or service satisfies the needs of the users. 

Qualitative Methods for Evaluation Research

• One-on-One Interview

An interview is a structured conversation involving two participants; usually the researcher and the user or a member of the target market. One-on-One interviews can be conducted physically, via the telephone and through video conferencing apps like Zoom and Google Meet. 

• Focus Groups

A focus group is a research method that involves interacting with a limited number of persons within your target market, who can provide insights on market perceptions and new products. 

• Qualitative Observation

Qualitative observation is a research method that allows the evaluation researcher to gather useful information from the target audience through a variety of subjective approaches. This method is more extensive than quantitative observation because it deals with a smaller sample size, and it also utilizes inductive analysis. 

• Case Studies

A case study is a research method that helps the researcher to gain a better understanding of a subject or process. Case studies involve in-depth research into a given subject, to understand its functionalities and successes. 

How to Formplus Online Form Builder for Evaluation Survey 

• Sign into Formplus

In the Formplus builder, you can easily create your evaluation survey by dragging and dropping preferred fields into your form. To access the Formplus builder, you will need to create an account on Formplus. 

Once you do this, sign in to your account and click on “Create Form ” to begin. 

• Edit Form Title

Click on the field provided to input your form title, for example, “Evaluation Research Survey”.

Click on the edit button to edit the form.

Add Fields: Drag and drop preferred form fields into your form in the Formplus builder inputs column. There are several field input options for surveys in the Formplus builder. 

Edit fields

Click on “Save”

Preview form.

• Form Customization

With the form customization options in the form builder, you can easily change the outlook of your form and make it more unique and personalized. Formplus allows you to change your form theme, add background images, and even change the font according to your needs. 

• Multiple Sharing Options

Formplus offers multiple form sharing options which enables you to easily share your evaluation survey with survey respondents. You can use the direct social media sharing buttons to share your form link to your organization’s social media pages. 

You can send out your survey form as email invitations to your research subjects too. If you wish, you can share your form’s QR code or embed it on your organization’s website for easy access. 

Conclusion  

Conducting evaluation research allows organizations to determine the effectiveness of their activities at different phases. This type of research can be carried out using qualitative and quantitative data collection methods including focus groups, observation, telephone and one-on-one interviews, and surveys. 

Online surveys created and administered via data collection platforms like Formplus make it easier for you to gather and process information during evaluation research. With Formplus multiple form sharing options, it is even easier for you to gather useful data from target markets.

Connect to Formplus, Get Started Now - It's Free!

• characteristics of evaluation research
• evaluation research methods
• types of evaluation research
• what is evaluation research
• busayo.longe

You may also like:

Formal Assessment: Definition, Types Examples & Benefits

In this article, we will discuss different types and examples of formal evaluation, and show you how to use Formplus for online assessments.

What is Pure or Basic Research? + [Examples & Method]

Simple guide on pure or basic research, its methods, characteristics, advantages, and examples in science, medicine, education and psychology

Assessment vs Evaluation: 11 Key Differences

This article will discuss what constitutes evaluations and assessments along with the key differences between these two research methods.

Recall Bias: Definition, Types, Examples & Mitigation

This article will discuss the impact of recall bias in studies and the best ways to avoid them during research.

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

What is Program Evaluation? Generating Knowledge for Improvement

• Archives and Museum Informatics 4(1):5-16

• University of Massachusetts Amherst

• University of Alabama

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Hüseyin Şeyhanlıoğlu
• Hasan Bardakçi

• Kasih Prihantoro
• Estera Bašić

• Željka Barić
• Tahan Samuel Lumban Toruan
• TahanSamuel LumbanToruan
• Aiah As Mbayo
• New Dir Eval

• Clifford B Geertz
• J A Simpson

• Michael S. Scriven
• G B Rossman
• S F And Rallis
• L S F Brown
• Sharon F Rallis
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

10 23. Program evaluation

Chapter outline.

• What is program evaluation? (5 minute read time)
• Planning your program evaluation (20 minute read time, including video)
• Process evaluations and implementation science (7 minute read time)
• Outcome and impact evaluations (5 minute read time)
• Ethics and culture in program evaluation (10 minute read time)

Content warning: discussions of BMI/weight/obesity, genocide, and residential schools for indigenous children.

Imagine you are working for a nonprofit focused on children’s health and wellness in school. One of the grants you received this year funds a full-time position at a local elementary school for a teacher who will be integrating kinesthetic learning into their lesson plans for math classes for third graders. Kinesthetic learning is learning that occurs when the students do something physical to help learn and reinforce information, instead of listening to a lecture or other verbal teaching activity. You have read research suggesting that students retain information better using kinesthetic teaching methods and that it can reduce student behavior issues. You want to know if it might benefit your community.

When you applied for the grant, you had to come up with some outcome measures that would tell the foundation if your program was worth continuing to fund – if it’s having an effect on your target population (the kids at the school). You told the foundation you would look at three outcomes:

• How did using kinesthetic learning affect student behavior in classes?
• How did using kinesthetic learning affect student scores on end-of-year standardized tests?
• How did the students feel about kinesthetic teaching methods?

But, you say, this sounds like research! However, we have to take a look at the purpose, origin, effect , and execution of the project to understand the difference, which we do in section 23.1 in this chapter. Those domains are where we can find the similarities and differences between program evaluation and research.

Realistically, as a practitioner, you’re far more likely to engage in program evaluation than you are in research. So, you might ask why you are learning research methods and not program evaluation methods, and the answer is that you will use research methods in evaluating programs. Program evaluation tends to focus less on generalizability, experimental design, and replicability, and instead focuses on the practical application of research methods to a specific context in practice.

23.1 What is program evaluation?

Learning objectives.

Learners will be able to…

• Define program evaluation
• Discuss similarities and differences between program evaluation and research
• Determine situations in which program evaluation is more appropriate than research

Program evaluation can be defined as the systematic process by which we determine if social programs are meeting their goals, how well the program runs, whether the program had the desired effect, and whether the program has merit according to stakeholders (including in terms of the monetary costs and benefits). It’s important to know what we mean when we say “evaluation.” Pruett (2000) [1]  provides a useful definition: “Evaluation is the systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program” (para. 1). That nod to scientific methods is what ties program evaluation back to research, as we discussed above. Program evaluation is action-oriented, which makes it fit well into social work research (as we discussed in Chapter 1 ).

Often, program evaluation will consist of mixed methods because its focus of is so heavily on the effect of the program in your specific context . Not that research doesn’t care about the effects of programs – of course it does! But with program evaluation, we seek to ensure the way that we are applying our program works in our agency, with our communities and clients. Thinking back to the example at the beginning of the chapter, consider the following: Does kinesthetic learning make sense for your school? What if your classroom spaces are too small? Are the activities appropriate for children with differing physical abilities who attend your school? What if school administrators are on board, but some parents are skeptical?

The project we talked about in the introductions – a real project, by the way – was funded by a grant from a foundation. The reality of the grant funding environment is that funders want to see that their money is not only being used wisely, but is having a material effect on the target population. This is a good thing, because we want to know our programs have a positive effect on clients and communities. We don’t want to just keep running a program because it’s what we’ve always done. (Consider the ethical implications of continuing to run an ineffective program.) It also forces us as practitioners to plan grant-funded programs with an eye toward evaluation. It’s much easier to evaluate your program when you can gather data at the beginning of the program than when you have to work backwards at the middle or end of the program.

How do program evaluation and research relate to each other?

As we talked about above, program evaluation and research are similar, particularly in that they both rely on scientific methods. Both use quantitative and qualitative methods, like data analysis and interviews. Effective program evaluation necessarily involves the research methods we’ve talked about in this book. Without understanding research methods, your program evaluation won’t be very rigorous and probably won’t give you much useful information.

However, there are some key differences between the two that render them distinct activities that are appropriate in different circumstances. Research is often exploratory and not evaluative at all, and instead looks for relationships between variables to build knowledge on a subject. It’s important to note at the outset that what we’re discussing below is not universally true of all projects. Instead, the framework we’re providing is a broad way to think about the differences between program evaluation and research. Scholars and practitioners disagree on whether program evaluation is a subset of research or something else entirely (and everything in between). The important thing to know about that debate is that it’s not settled, and what we’re presenting below is just one way to think about the relationship between the two.

According to Mathison (2008) [2] , the differences between program evaluation and research have to do with the domains of purpose, origins, effect and execution. 

Let’s think back to our example from the start of the chapter – kinesthetic teaching methods for 3rd grade math – to talk more about these four domains.

To understand this domain, we have to ask a few questions: why do we want to research or evaluate this program? What do we hope to gain? This is the  why  of our project (Mathison). Another way to think about it is as the  aim  of your research, which is a concept you hopefully remember from Chapter 2.

Through the lens of program evaluation, we’re evaluating this program because we want to know its effects, but also because our funder probably only wants to give money to programs that do what they’re supposed to do. We want to gather information to determine if it’s worth it for our funder – or for  us  – to invest resources in the program.

If this were a research project instead, our purpose would be congruent, but different. We would be seeking to add to the body of knowledge and evidence about kinesthetic learning, most likely hoping to provide information that can be generalized beyond 3rd grade math students. We’re trying to inform further development of the body of knowledge around kinesthetic learning and children. We’d also like to know if and how we can apply this program in contexts other than one specific school’s 3rd grade math classes. These are not the only research considerations, but just a few examples.

Purpose and origins can feel very similar and be a little hard to distinguish. The main difference is that origins are about the  who , whereas purpose is about the  why  (Mathison). So, to understand this domain, we have to ask about the source of our project – who wanted to get the project started? What do they hope this project will contribute?

For a program evaluation, the project usually arises from the priorities of funders, agencies, practitioners and (hopefully) consumers of our services. They are the ones who define the purpose we discussed above and the questions we will ask.

In research, the project arises from a researcher’s intellectual curiosity and desire to add to a body of knowledge around something they think is important and interesting. Researchers define the purpose and the questions asked in the project.

The effect of program evaluation and research is essentially what we’re going to use our results for. For program evaluation, we will use them to make a decision about whether a program is worth continuing, what changes we might make to the program in the future or how we might change the resources we devote going forward. The results are often also used by our funders to make decisions about whether they want to keep funding our program or not. (Outcome evaluations aren’t the only thing that funders will look at – they also sometimes want to know whether our processes in the program were faithful to what we described when we requested funding. We’ll discuss outcome and process evaluations in section 23.4.)

The effect of research – again, what we’re going to use our results for – is typically to add to the knowledge and evidence base surrounding our topic. Research can certainly be used for decision-making about programs, especially to decide which program to implement in the first place. But that’s not what results are primarily used for, especially by other researchers.

Execution is fundamentally the  how  of our project. What are the circumstances under which we’re running the project?

Program evaluation projects that most of us will ever work on are frequently based in a nonprofit or government agency. Context is extremely important in program evaluation (and program implementation). As most of us will know, these are environments with lots of moving parts. As a result, running controlled experiments is usually not possible, and we sometimes have to be more flexible with our evaluations to work with the resources we actually have and the unique challenges and needs of our agencies. This doesn’t mean that program evaluations can’t be rigorous or use strong research methods. We just have to be realistic about our environments and plan for that when we’re planning our evaluation.

Research is typically a lot more controlled. We do everything we can to minimize outside influences on our variables of interest, which is expected of rigorous research. Of course, some research is  extremely  controlled, especially experimental research and randomized controlled trials. this all ties back to the purpose, origins, and effects of research versus those of program evaluation – we’re primarily building knowledge and evidence.

In the end, it’s important to remember that these are guidelines, and you will no doubt encounter program evaluation projects that cross the lines of research, and vice versa. Understanding how the two differ will help you decide how to move forward when you encounter the need to assess the effect of a program in practice.

Key Takeaways

• Program evaluation is a systematic process that uses the scientific research method to determine the effects of social programs.
• Program evaluation and research are similar, but they differ in purpose, origins, effect and execution.
• The purpose of program evaluation is to judge the merit or worth of a program, whereas the purpose of research is primarily to contribute to the body of knowledge around a topic.
• The origins of program evaluation are usually funders and people working in agencies, whereas research originates primarily with scholars and their scientific interests.
• Program evaluations are typically used to make decisions about programs, whereas research is used to add to the knowledge and evidence base around a topic.
• Executing a program evaluation project requires a strong understanding of your setting and context in order to adapt your evaluation to meet your goals in a realistic way. The execution of research is much more controlled and seeks to minimize the influence of context.
• If you were conducting a research project on the kinesthetic teaching methods that we talked about in this chapter, what is one research question you could study that aligns with the purpose, origins, and effects of research?
• Consider the research project you’ve been building throughout this book. What is one program evaluation question you could study that aligns with the purpose, origins, and effects of program evaluation? How might its execution look different than what you’ve envisioned so far?

23.2 Planning your program evaluation

• Discuss how planning a program evaluation is similar and different from planning a research project
• Identify program stakeholders
• Identify the basics of logic models and how they inform evaluation
• Produce evaluation questions based on a logic model

Planning a program evaluation project requires just as much care and thought as planning a research project. But as we discussed in section 23.1, there are some significant differences between program evaluation and research that mean your planning process is also going to look a little different. You have to involve the program stakeholders at a greater level than that found with most types of research, which will sometimes focus your program evaluation project on areas you wouldn’t have necessarily chosen (for better or worse). Your program evaluation questions are far less likely to be exploratory; they are typically evaluative and sometimes explanatory.

For instance, I worked on a project designed to increase physical activity for elementary school students at recess. The school had noticed a lot of kids would just sit around at recess instead of playing. As an intervention, the organization I was working with hired recess coaches to engage the kids with new games and activities to get them moving. Our plan to measure the effect of recess coaching was to give the kids pedometers at a couple of different points during the year, and see if there was any change in their activity level as measured by the number of steps they took during recess. However, the school was also concerned with the rate of obesity among students, and asked us to also measure the height and weight of the students to calculate BMI at the beginning and end of the year. I balked at this because kids are still growing and BMI isn’t a great measure to use for kids and some kids were uncomfortable with us weighing them (with parental consent), even though no other kids would be in the room. However, the school was insistent that we take those measurements, and so we did that for all kids whose parents consented and who themselves assented to have their weight measured. We didn’t think BMI was an important measure, but the school did, so this changed an element of our evaluation.

In an ideal world, your program evaluation is going to be part of your overall program plan. This very often doesn’t happen in practice, but for the purposes of this section, we’re going to assume you’re starting from scratch with a program and really internalized the first sentence of this paragraph. (It’s important to note that no one  intentionally leaves evaluation out of their program planning; instead, it’s just not something many people running programs think about. They’re too busy… well, running programs. That’s why this chapter is so important!)

In this section, we’re going to learn about how to plan your program evaluation, including the importance of logic models. You may have heard people groan about logic models (or you may have groaned when you read those words), and the truth is, they’re a lot of work and a little complicated. Teaching you how to make one from start to finish is a little bit outside the scope of this section, but what I am going to try to do is teach you how to interpret them and build some evaluation questions from them. (Pro-tip: logic models are a heck of a lot easier to make in Excel than Word.)

It has three primary steps: engaging stakeholders, describing the program and focusing the evaluation.

Step 1: Engaging stakeholders

Stakeholders are the people and organizations that have some interest in or will be impacted by our program. Including as many stakeholders as possible when you plan your evaluation will help to make it as useful as possible for as many people as possible. The key to this step is to listen. However, a note of caution: sometimes stakeholders have competing priorities, and as the program evaluator, you’re going to have to help navigate that. For example, in our kinesthetic learning program, the teachers at your school might be interested in decreasing classroom disruptions or enhancing subject matter learning, while the administration is solely focused on test scores, while the administration is solely focused on test scores. Here is where it’s a great idea to use your social work ethics and research knowledge to guide conversations and planning. Improved test scores are great, but how much does that actually  benefit the students?

Step 2: Describe the program

Once you’ve got stakeholder input on evaluation priorities, it’s time to describe what’s going into the program and what you hope your participants and stakeholders will get out of it. Here is where a logic model becomes an essential piece of program evaluation. A logic model “ is a graphic depiction (road map) that presents the shared relationships among the resources, activities, outputs, outcomes, and impact for your program” ( Centers for Disease Control , 2018, para. 1). Basically, it’s a way to show how what you’re doing is going to lead to an intended outcome and/or impact. (We’ll discuss the difference between outcomes and impacts in section 23.4.)

Logic models have several key components, which I describe in the list below (CDC, 2018). The components are numbered because of where they come in the “logic” of your program – basically, where they come in time order.

• Inputs: resources (e.g. people and material resources) that you have to execute your program.
• Activities: what you’re actually doing with your program resources.
• Outputs: the direct products and results of your program.
• Outcomes: the changes that happen because of your program inputs and activities.
• Impacts: the long-term effects of your program.

The CDC also talks about moderators – what they call “contextual factors” – that affect the execution of your program evaluation. This is an important component of the execution of your project, which we talked about in 23.1. Context will also become important when we talk about implementation science in section 23.3.

Let’s think about our kinesthetic learning project. While you obviously don’t have full information about what the project looks like, you’ve got a good enough idea for a little exercise below.

Step 3: Focus the evaluation

So now you know what your stakeholder priorities are and you have described your program. It’s time to figure out what questions you want to ask that will reflect stakeholder priorities and are actually possible given your program inputs, activities and outputs.

Why do inputs, activities and outputs matter for your question?

• Inputs are your resources for the evaluation – do you have to do it with existing staff, or can you hire an expert consultant? Realistically, what you ask is going to be affected by the resources you can dedicate to your evaluation project, just like in a research project.
• Activities are what you can actually evaluate – for instance, what effect does using hopscotch to teach multiplication have?
• And finally, outputs are most likely your indicators of change – student engagement with administrators for behavioral issues, end-of-grade math test scores, for example.
• Program evaluation planning should be rigorous like research planning, but will most likely focus more on stakeholder input and evaluative questions
• The three primary steps in planning a program evaluation project are engaging stakeholders, describing your program, and focusing your evaluation.
• Logic models are a key piece of information in planning program evaluation because they describe how a program is designed to work and what you are investing in it, which are important factors in formulating evaluation questions.
• Who would the key stakeholders be? What is each stakeholder’s interest in the project?
• What are the activities (the action(s) you’re evaluating) and outputs (data/indicators) for your program? Can you turn them into an evaluation question?

23.3 Process evaluation and implementation science

• Define process evaluation
• Explain why process evaluation is important for programs
• Distinguish between process and outcome measures
• Explain the purpose of implementation science and how it relates to program evaluation

Something we often don’t have time for in practice is evaluating how things are going internally with our programs. How’s it going with all the documentation our agency asks us to complete? Is the space we’re using for our group sessions facilitating client engagement? Is the way we communicate with volunteers effective? All of these things can be evaluated using a process evaluation , which is an analysis of how well your program ended up running, and sometimes how well it’s going in real time.  If you have the resources and ability to complete one of these analyses, I highly recommend it – even if it stretches your staff, it will often result in a greater degree of efficiency in the long run. (Evaluation should, at least in part, be about the long game.)

From a research perspective, process evaluations can also help you find irregularities in how you collect data that might be affecting your outcome or impact evaluations. Like other evaluations, ideally, you’re going to plan your process evaluation before you start the project. Take an iterative approach, though, because sometimes you’re going to run into problems you need to analyze in real time.

The RAND corporation is an excellent resource for guidance on program evaluation, and they describe process evaluations this way: “Process evaluations typically track attendance of participants, program adherence, and how well you followed your work plan. They may also involve asking about satisfaction of program participants or about staff’s perception of how well the program was delivered. A process evaluation should be planned before the program begins and should continue while the program is running” (RAND Corporation, 2019, para. 1) [3] .

There are several key data sources for process evaluations (RAND Corporation, 2019) [4] , some of which are listed below.

• Participant data: can help you determine if you are actually reaching the people you intend to.
• Focus groups: how did people experience the program? How could you improve it from the participant perspective?
• Satisfaction surveys: did participants get what they wanted from the program?
• Staff perception data: How did the program go for staff? Were expectations realistic? What did they see in terms of qualitative changes for participants?
• Program adherence monitoring: how well did you follow your program plans?

Using these data sources, you can learn lessons about your program and make any necessary adjustments if you run the program again. It can also give you insights about your staff’s needs (like training, for instance) and enable you to identify gaps in your programs or services.

Implementation science: The basics

A further development of process evaluations, i mplementation science is “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services.” (Bauer, Damschroder, Hagerdorn, Smith & Kilbourne, 2015) [5]

Put more plainly, implementation science studies how we put evidence-based interventions (EBIs) into practice. It’s e ssentially a form of process evaluation, just at a more macro level. Implementation science is a r elatively new field of study that focuses on how to best put interventions into practice, and it’s i mportant because it helps us analyze on a macro level those factors that might affect our ability to implement a program. Implementation science focuses on the context of program implementation, which has significant implications for program evaluation.

A useful framework for implementation science is the EPIS (Exploration, Preparation, Implementation and Sustainment) framework. It’s not the only one out there, but I like it because to me, it sort of mirrors the linear nature of a logic model.

The EPIS framework was developed by Aarons, Hurlburt and Horwitz (first published 2011). (The linked article is behind a paywall, the abstract is still pretty useful, and if you’re affiliated with a college or university, you can probably get access through your library.) This framework emphasizes the importance of the context in which your program is being implemented – inner, organizational, context and outer, or the political, public policy and social contexts. What’s happening in your organization and in the larger political and social sphere that might affect how your program gets implemented?

There are a few key questions in each phase, according to Aarons, Hurlburt and Horwitz (2011) [6] :

• Exploration phase: what is the problem or issue we want to address? What are our options for programs and interventions? What is the best way to put them into practice? What is the organizational and societal context that we need to consider when choosing our option?
• Preparation: which option do we want to adopt? What resources will we need to put that option into practice? What are our organizational or sociopolitical assets and challenges in putting this option into practice?
• Implementation: what is actually happening now that we’re putting our option into practice? How is the course of things being affected by contexts?
• Sustainment: what can we do to ensure our option remains viable, given competing priorities with funding and public attention?

Implementation is a new and rapidly advancing field, and realistically, it’s beyond what a lot of us are going to be able to evaluate in our agencies at this point. But even taking pieces of it – especially the pieces about the importance of context for our programs and evaluations – is useful. Even if you don’t use it as an evaluative framework, the questions outlined above are good ones to ask when you’re planning your program in the first place.

• A  process evaluation is an analysis of how your program actually ran, and sometimes how it’s running in real time.
• Process evaluations are useful because they can help programs run more efficiently and effectively and reveal agency and program needs.
• The EPIS model is a way to analyze the implementation of a program that emphasizes distinct phases of implementation and the context in which the phases happen.
• The EPIS model is also useful in program planning, as it mirrors the linear process of a logic model .
• Consider your research project or, if you have been able to adapt it, your program evaluation project. What are some inner/organizational context factors that might affect how the program gets implemented and what you can evaluate?
• What are some things you would want to evaluate about your program’s process? What would you gain from that information?

23.4 Outcome and impact evaluations

• Define outcome
• Explain the principles of conducting an outcome evaluation
• Define impact
• Explain the principles of conducting an impact evaluation
• Explain the difference between outcomes and impacts

A lot of us will use “outcome” and “impact” interchangeably, but the truth is, they are different. An o utcome is the final condition that occurs at the end of an intervention or program. It is the short-term effect – for our kinesthetic learning example, perhaps an improvement over last year’s end-of-grade math test scores. An i mpact is the long-term condition that occurs at the end of a defined time period after an intervention. It is the longer-term effect – for our kinesthetic learning example, perhaps better retention of math skills as students advance through school. Because of this distinction, outcome and impact evaluations are going to look a little different.

But first, let’s talk about how these types of evaluations are the same. Outcome and impact evaluations are all about change. As a result, we have to know what circumstance, characteristic or condition we are hoping will change because of our program.  We also need to figure out what we think the causal link between our intervention or program and the change is, especially if we are using a new type of intervention that doesn’t yet have a strong evidence base.

For both of these types of evaluations, you have to consider what type of research design you can actually use in your circumstances – are you coming in when a program is already in progress, so you have no baseline data? Or can you collect baseline data to compare to a post-test? For impact evaluations, how are you going to track participants over time?

The main difference between outcome and impact evaluation is the timing and, consequently, the difficulty and level of investment. You can pretty easily collect outcome data from program participants at the end of the program. But tracking people over time, especially for populations social workers serve, can be extremely difficult. It can also be difficult or impossible to control for whatever happened in your participant’s life between the end of the program and the end of your long-term measurement period.

Impact evaluations require careful planning to determine how your follow-up is going to happen. It’s a good practice to try to keep intermittent contact with participants, even if you aren’t taking a measurement at that time, so that you’re less likely to lose track of them.

• Outcomes are short-term effects that can be measured at the end of a program.
• Outcome evaluations apply research methods to the analysis of change during a program and try to establish a logical link between program participation and the short-term change.
• Impacts are long-term effects that are measured after a period of time has passed since the end of a program.
• Impact evaluations apply research methods to the analysis of change after a defined period of time has passed after the end of a program and try to establish a logical link between program participation and long-term change.
• Is each of the following examples an outcome or an impact? Choose the correct answer.

23.5 Ethics and culture in program evaluation

• Discuss cultural and ethical issues to consider when planning and conducting program evaluation
• Explain the importance of stakeholder and participant involvement to address these issues

In a now decades-old paper, Stake and Mabry (1998) [7] point out, “The theory and practice of evaluation are of little value unless we can count on vigorous ethical behavior by evaluators” (p. 99). I know we always say to use the most recent scholarship available, but this point is as relevant now as it was over 20 years ago. One thing they point out that rings particularly true for me as an experienced program evaluator is the idea that we evaluators are also supposed to be “program advocates” (p. 99). We have to work through competing political and ideological differences from our stakeholders, especially funders, that, while sometimes present in research, are especially salient for program evaluation given its origins.

There’s not a rote answer for these ethical questions, just as there are none for the practice-based ethical dilemmas your instructors hammer home with you in classes. You need to use your research and social work ethics to solve these problems. Ultimately, do your best to focus on rigor while meeting stakeholder needs.

One of the most important ethical issues in program evaluation is the implication of not evaluating your program. Providing an ineffective intervention to people can be extremely harmful. And what happens if our intervention actually causes harm? It’s our duty as social workers to explore these issues and not just keep doing what we’ve always done because it’s expedient or guarantees continued funding. I’ve evaluated programs before that turned out to be ineffective, but were required by state law to be delivered to a certain population. It’s not just potentially harmful to clients; it’s also a waste of precious resources that could be devoted to other, more effective programs.

We’ve talked throughout this book about ethical issues and research. All of that is applicable to program evaluation too. Federal law governing IRB practice does not require that program evaluation go through IRB if it is not seeking to gather generalizable knowledge, so IRB approval isn’t a given for these projects. As a result, you’re even more responsible for ensuring that your project is ethical.

Cultural considerations

Ultimately, social workers should start from a place of humility in the face of cultures or groups of which we are not a part. Cultural considerations in program evaluation look similar to those in research. Something to consider about program evaluation, though: is it your duty to point out potential cultural humility issues as part of your evaluation, even if you’re not asked to? I’d argue that it is.

It is also important we make sure that our definition of success is not oppressive. For example, in Australia, the government undertook a program to remove Aboriginal children from their families and assimilated them into white culture.  The program was viewed as successful, but the measures of success were based on oppressive beliefs and stereotypes. This is why stakeholder input is essential – especially if you’re not a member of the group you’re evaluating, stakeholders are going to be the ones to tell you that you may need to reconsider what “success” means.

Unrau , Gabor, and Grinnell (2007) [8] identified several important factors to consider when designing and executing a culturally sensitive program evaluation. First, evaluators need “a clear understanding of the impact of culture on human and social processes generally and on evaluation processes specifically and… skills in cross-cultural communications to ensure that they can effectively interact with people from diverse backgrounds” (p. 419). These are also essential skills in social work practice that you are hopefully learning in your other classes! We should strive to learn as much as possible about the cultures of our clients when they differ from ours.

The authors also point out that evaluators need to be culturally aware and make sure the way they plan and execute their evaluations isn’t centered on their own ethnic experience and that they aren’t basing their plans on stereotypes about other cultures. In addition, when executing our evaluations, we have to be mindful of how our cultural background affects our communication and behavior, because we may need to adjust these to communicate (both verbally and non-verbally) with our participants in a culturally sensitive and appropriate way.

Consider also that the type of information on which  you  place the most value may not match that of people from other cultures. Unrau , Gabor, and Grinnell (2007) [9] point out that mainstream North American cultures place a lot of value on hard data and rigorous processes like clinical trials. (You might notice that we spend a lot of time on this type of information in this textbook.) According to the authors, though, cultures from other parts of the world value relationships and storytelling as evidence and important information. This kind of information is as important and valid as what we are teaching you to collect and analyze in most of this book.

Being the squeaky wheel about evaluating programs can be uncomfortable. But as you go into practice (or grow in your current practice), I strongly believe it’s your ethical obligation to push for evaluation. It honors the dignity and worth of our clients. My hope is that this chapter has given you the tools to talk about it and, ultimately, execute it in practice.

• Ethical considerations in program evaluation are very similar to those in research.
• Culturally sensitive program evaluation requires evaluators to learn as much as they can about cultures different from their own and develop as much cultural awareness as possible.
• Stakeholder input is always important, but it’s essential when planning evaluations for programs serving people from diverse backgrounds.
• Consider the research project you’ve been working on throughout this book. Are there cultural considerations in your planning that you need to think about?
• If you adapted your research project into a program evaluation, what might some ethical considerations be? What ethical dilemmas could you encounter?
• Pruett, R. (2000). Program evaluation 101. Retrieved from https://mainweb-v.musc.edu/vawprevention/research/programeval.shtml ↵
• Mathison, S. (2007). What is the difference between research and evaluation—and why do we care? In N. L. Smith & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 183-196). New York: Guilford. ↵
• RAND Corporation. (2020). Step 07: Process evaluation. Retrieved from https://www.rand.org/pubs/tools/TL259/step-07.html. ↵
• Bauer, M., Damschroder, L., Hagedorn, H., Smith, J. & Kilbourne, A. (2015). An introduction to implementation science for the non-specialist. BMC Psychology, 3 (32). ↵
• Aarons, G., Hurlburt, M. & Horwitz, S. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38 (1), pp. 4-23. ↵
• Stake, R. & Mabry, L. (2007). Ethics in program evaluation. Scandinavian Journal of Social Welfare, 7 (2). ↵
• Unrau, Y., Gabor, P. & Grinnell, R. (2007). Evaluation in social work: The art and science of practice . New York, New York: Oxford University Press. ↵

The systematic process by which we determine if social programs are meeting their goals, how well the program runs, whether the program had the desired effect, and whether the program has merit according to stakeholders (including in terms of the monetary costs and benefits)

individuals or groups who have an interest in the outcome of the study you conduct

the people or organizations who control access to the population you want to study

The people and organizations that have some interest in or will be effected by our program.

A graphic depiction (road map) that presents the shared relationships among the resources, activities, outputs, outcomes, and impact for your program

An analysis of how well your program ended up running, and sometimes how well it's going in real time.

The scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services.

The final condition that occurs at the end of an intervention or program.

Tthe long-term condition that occurs at the end of a defined time period after an intervention.

Graduate research methods in social work Copyright © 2020 by Matthew DeCarlo, Cory Cummings, Kate Agnelli is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• Emergency information
• Increase Font Size
• Decrease Font Size

Planning and Managing Program Evaluations

On this page

Introduction, when to evaluate and when to engage an independent evaluator, evaluation principles, appropriateness, stakeholder involvement, effective governance, methodological rigour, consideration of specific populations, ethical conduct, program logic, developing an evaluation plan, overview of the program, purpose of the evaluation, audience for the evaluation, evaluation questions, evaluation design and data sources, potential risks, resources and roles, procurement, preparing the request for proposal, engaging an evaluator, managing the development and implementation of the evaluation workplan, disseminating and using evaluation findings.

NSW Health is committed to the development of evidence-based policies and programs and the ongoing monitoring, review and evaluation of existing programs in line with NSW Government requirements. The NSW Treasury Policy and Guidelines: Evaluation (TPG22-22) 1 sets out mandatory requirements, recommendations and guidance for NSW General Government Sector agencies and other government entities to plan for and conduct the evaluation of policies, projects, regulations and programs. This guide aims to support NSW Health staff in planning and managing program* evaluations.

Evaluation can be defined as a rigorous, systematic and objective process to assess a program’s effectiveness, efficiency and appropriateness. 1 Evaluations are commonly undertaken to measure the outcomes of a program, and to reflect on its processes. Evaluation is considered to be distinct from ‘pure’ research. Both processes involve the rigorous gathering of evidence using robust and fit-for-purpose study designs and methods. However, research is typically guided by different sorts of questions than evaluations, with the broad aim to generate new knowledge, and research findings tend to be published in peer-reviewed journals. Evaluations are guided by ‘key evaluation questions’, with the aim to inform decision making around policies and programs. Involving key stakeholders in the design of evaluations and reporting findings back to stakeholders are also distinctive elements of evaluation.

Evaluation is also distinct from more operational assessments of programs, such as program monitoring. Monitoring is a continuous and systematic process of collecting and analysing information about the performance of a program. 1 Monitoring can help to inform ongoing improvement and identify trends, issues or gaps for further examination through evaluation. Although program monitoring and evaluation are unique activities, in practice it is best to take an integrated approach and work towards developing a program monitoring and evaluation framework in the initiative design phase.

The Treasury Evaluation Policy and Guidelines outline the requirements for suitable evaluation of NSW public programs to assess their effectiveness, efficiency, value and continued relevance, and to improve transparency. The online NSW Treasury evaluation workbooks and resources support the implementation of the Evaluation Guidelines and contain information to support monitoring and evaluation including templates for program logic models, data matrices, project management, and reporting. The NSW Health Guide to Measuring Value provides specific guidance about measuring improvements across the quadruple aim of value-based healthcare at NSW Health as part of monitoring and evaluation. 2

This guide to planning and managing program evaluations complements the NSW Treasury evaluation workbooks and resources. It promotes a proactive, planned and structured approach to planning and managing evaluations, including information on when and how to evaluate a program and how to make the most of the results. The guide draws on the principles and processes described in the Treasury Evaluation Policy and Guidelines, but it is framed specifically in relation to the health context, and it outlines the steps that should be taken when engaging an independent evaluator. The guide may be used to assist NSW Health staff in developing a complete evaluation plan, or in drafting an evaluation plan to which an evaluator can add value. The principles and proposed steps are also relevant for policy staff undertaking evaluations of their own programs. It should be noted that, in the field of evaluation, several terms are defined and used in different ways in different disciplines or contexts (for example: goal/aim and impact/outcome/benefits). This guide uses health-relevant language.

* In this guide the word ‘program’ is used interchangeably with ‘initiative’. The NSW Treasury Policy and Guidelines: Evaluation (TPG22-22) define an initiative as a program, policy, strategy, service, project, or any series of related events. Initiatives can vary in size and structure; from a small initiative at a single location, a series of related events delivered over a period, or whole-of-government reforms with many components delivered by different agencies or governments.

Whether or not a program should be formally evaluated will depend on factors such as the size of the program (including its scope and level of funding), its strategic significance, and the degree of risk. 1

Other important considerations include the program’s level of innovation and degree of complexity, and the extent to which any observed impacts will be able to be attributed to the program being evaluated, rather than to other external factors.

In some cases only certain components of a program will need to be evaluated, such as when a new implementation arm has been added to a program.

While some small-scale evaluations may be completed in-house, others will require engagement of an independent evaluator. An independent evaluator may be an individual or group external to the policy team managing the program or, for high priority/high risk programs, external to the program delivery agency. Engaging an independent evaluator is important where there is a need for special evaluation expertise and/or where independence needs to be demonstrated. 3 An independent evaluator is likely to be particularly important for programs that have involved a reasonable investment, and those being assessed for continuation, modification, or scaling up. †

Whether evaluations are completed in-house or by an independent evaluator, the active engagement of NSW Health staff who are overseeing the program remains important.

Figure 1 depicts Step 1 when planning an evaluation – a process for conducting a pre-evaluation assessment to determine whether a program should be evaluated and, if so, whether an independent evaluator should be used.

† Scaling up refers to deliberate efforts to increase the impact of successfully tested health interventions so as to benefit more people and to foster policy and program development on a lasting basis. For more information and a step-by-step process for scaling up interventions, refer to Increasing the Scale of Population Health Interventions: A Guide. 4

Figure 2 summarises Step 2, a process for planning a program evaluation where an executive sponsor with appropriate delegation has approved the engagement of an independent evaluator. The elements included in Figure 2 are explained in Sections 3 to 8 of this guide.

# Ideally a program logic model should be developed in the program planning phase. For more information about the development of program logic models and their use in planning program evaluations, refer to Developing and Using Program Logic: A Guide. 5

Best practice principles that underpin the conduct of effective evaluations should be incorporated where appropriate when planning and conducting an evaluation. 1 Considerations relevant to population health program evaluations include timeliness, appropriateness, stakeholder involvement, effective governance, methodological rigour, consideration of specific populations, and ethical conduct.

Evaluation planning should commence as early as possible during the program planning phase. 6 Incorporating evaluation planning into the broader process of program planning will help to ensure that the program has clear aims ‡ and objectives, a strong rationale, and can be properly evaluated. Planning an evaluation early also ensures that a robust evaluation can be built into the design of the program. This includes, for example, trialling and implementing data collection tools, modifying existing data collection instruments, providing appropriate training for staff responsible for collecting data, and collecting baseline data before program implementation, if relevant. In some cases, evaluation requirements may influence the way a program is rolled out across implementation sites. Although not ideal, an evaluation can still be developed after the program has commenced.

Evaluations should conclude before decisions about the program need to be made. To that end, consideration should be given to the realistic amount of time needed to conduct an evaluation to ensure findings will be available when needed to support decision making. 1 This is particularly relevant to outcome evaluations where the generation of measurable results may take some time.

‡ Program ‘aims’ may also be referred to as ‘goals’. In this guide, the term ‘aims’ will be used.

The scope of an evaluation should be realistic and appropriate with respect to the size, stage and characteristics of the program being evaluated, the available evaluation budget, and practical issues such as availability of data. 3 Scope refers to the boundaries around what an evaluation will and will not cover. 7 The scope may define, for example, the specific programs (or aspects of these) to be evaluated, the time period or implementation phase to be covered, the geographical coverage, and the target groups to be included.

The design of and approach to an evaluation should be fit for purpose. For example, it is not necessary to use a complex experimental design when a simple one will suffice, and methods for collecting data should be feasible within the time and resources available. 8 Focusing on the most relevant evaluation questions will help to ensure that evaluations are manageable, cost efficient and useful (see section on evaluation questions). 7

Stakeholders are people or organisations that have an investment in the conduct of the evaluation and its findings. Stakeholders can include the primary intended users of the evaluation, such as program decision makers or program and policy staff, as well as people affected by the program being evaluated, such as community members or organisations.

Evaluations should foster input and participation among stakeholders throughout the process to enable their contribution to planning and conducting the evaluation as well as interpreting and disseminating the findings. A review of NSW Health-funded population health intervention research projects demonstrated that involving end users of research from the inception of projects increased the likelihood of findings influencing policy. 9

See Table 5 in the Treasury Evaluation Policy and Guidelines for additional information on stakeholders and their potential roles in an evaluation. 1

An evaluation advisory group should be established to guide and inform the evaluation process. Depending on the scope of the evaluation, this group may include representatives from the Ministry of Health, non-government organisations, local health districts (LHDs) or industry bodies, along with consumers of the program, academics or individuals with evaluation skills and expertise. If a steering committee already exists for the overall program, this committee or a sub-group of its members may also take the role of the evaluation advisory group.

Where the program being evaluated affects the health or wellbeing of Aboriginal peoples or communities, the group should include Aboriginal representation (e.g. from the Aboriginal Health & Medical Research Council, an Aboriginal Community Controlled Health Service, or the community).

The evaluation advisory group should agree to terms of reference that set out its purpose and working arrangements, including members’ roles and responsibilities (see also section on governance). As the group may be provided with access to confidential information during the evaluation process, its members should also be requested to agree to a confidentiality undertaking, on appointment, to ensure that any information provided to them is kept confidential.

Evaluations should use appropriate methods and draw on relevant data that are valid and reliable. The methods for data collection and analysis should be appropriate to the purpose and scope of the evaluation (see section on evaluation design and data sources). A quantitative, qualitative, or mixed approach may be most suitable. For evaluations that aim to assess the outcomes of a program, approaches to attributing any changes to the program being evaluated (as opposed to other programs or activities and other environmental factors) are particularly important. In real world evaluations, as compared to ‘pure research’ studies, it can sometimes be difficult or not possible to implement studies of enough scientific rigour to make definitive claims regarding attribution – for example, where a program is already being implemented at scale and an appropriate comparison group cannot be identified. However, planning early will tend to increase the methodological options available. The NSW Treasury resource Outcome Evaluation Design Technical Note outlines approaches to investigating a program’s contribution to observed outcomes, sometimes referred to as ‘plausible contributions’.

The needs of specific populations, including Aboriginal peoples, should be considered in every stage of evaluation planning and implementation. Considerations for specific populations should include:

• the health context and health needs of specific populations who may be impacted by the evaluation
• engagement with specific populations throughout the design, development, implementation and dissemination of findings from the evaluation
• potential impacts of the evaluation on specific populations, including positive and negative impacts, and intended and unintended consequences.

Where a project affects Aboriginal peoples and communities, evaluation methods should be culturally appropriate and sensitive. Consider procuring evaluation services from Aboriginal providers to design and conduct the evaluation or contribute at key points (see Section 5.V in the Treasury Evaluation Policy and Guidelines). 1 If the evaluation is being conducted as part of a larger study or project, cultural engagement should be built into the larger project at its outset.

The evaluation must be conducted in an ethical manner. This includes consideration of relevant legislative requirements, particularly regarding the privacy of participants and the costs and benefits to individuals, the community or population involved.

The National Health and Medical Research Council (NHMRC) document Ethical Considerations in Quality Assurance and Evaluation Activities provides guidance on relevant ethical issues and assists in identifying triggers for the consideration of ethical review. 10 In addition, the NSW Health Guideline GL2007_020 Human Research Ethics Committees: Quality Improvement & Ethical Review: A Practice Guide for NSW provides a checklist to assist in identifying potential ethical risks. If the evaluation is determined to involve more than a low level of risk, full review by a human research ethics committee (HREC) is required. 11 A list of NSW Health HRECs is available online. NSW Health HRECs provide an expedited review process for certain research projects that are considered to involve low or negligible risk to participants. 12 Research or evaluation projects that have specific review requirements are outlined below.

The process of applying for, and obtaining, HREC approval can take some time and this should be factored into evaluation planning. Pre-submission conversations with an HREC officer can help in preparing a successful application and avoid unnecessary delays.

Where an evaluation is deemed to not require ethical review by an HREC, it is recommended that program staff prepare a statement affirming that an alternative approach to ethical review was considered to be appropriate, outlining the reasons for this decision.

Special ethical review requirements

• Population health research or evaluation projects utilising and/or linking routinely collected health (and other) data, including data collections owned or managed by the NSW Ministry of Health or the Cancer Institute NSW. Resources: How to apply to the NSW Population and Health Services Research Ethics Committee (PHSREC). HREC: NSW Population and Health Services Research Ethics Committee.
• Research or evaluations affecting the health and wellbeing of Aboriginal people and communities in NSW. Resources: Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders, 13 Keeping research on track II: A companion document to Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders, 14 AH&MRC Ethical Guidelines: Key Principles. 15 HREC: Aboriginal Health & Medical Research Council of NSW (AH&MRC) Ethics Committee.
• Research or evaluations involving persons in custody and/or staff of Justice Health NSW. Resources: Getting ethics approval from the Justice Health NSW Human Research Ethics Committee. HREC: Justice Health NSW Human Research Ethics Committee AH&MRC Ethics Committee.*

∆ See also PD2010_055 Research–Ethical & Scientific Review of Human Research in NSW Public Health Organisations.

* The Justice Health NSW HREC considers that all research involving people in custody in NSW will involve at least some Aboriginal peoples and will require review and approval by the AH&MRC Ethics Committee. AH&MRC Ethics Committee approval can be sought concurrently with Justice Health NSW HREC approval.

Developing a program logic model is an important early step in designing a program and planning a program evaluation.

A program logic model is a schematic representation that describes how a program is intended to work by linking activities with outputs and with short, medium and longer-term outcomes. Program logic aims to show the intended causal links for a program. Wherever possible, these causal links should be evidence based.

A program logic model can assist in planning an evaluation by helping to: 16,17

• determine what to evaluate
• identify key evaluation questions
• identify information needed to answer evaluation questions
• decide when to collect data
• provide a mechanism for ensuring acceptability among stakeholders.

A variety of methods are used to develop program logic models. One approach, known as ‘backcasting’, involves identifying the possible outcomes of the program, arranging them in a chain from short-term to long-term outcomes, and subsequently working backwards to identify the program outputs and activities required to achieve these outcomes. The outcomes defined through this process should correspond to the program aims and objectives respectively, as depicted below.

The process of developing a program logic model should be consultative and include consideration of available information about the program, the advice of program and evaluation stakeholders, as well as the insights of the team implementing the program and people affected by the program. The final model should be coherent, logical and clear so it can illustrate the program for both technical and nontechnical audiences. 16

An example of a simple program logic model is presented in Figure 3. For more information and a step-by-step process for constructing a program logic model, refer to Developing and Using Program Logic: A Guide. 5

Relationship between program components, program logic model, and evaluation plan 18

• Program aims (program component) correspond to long-term program outcomes (program logic model) measured by outcome evaluation (evaluation plan)
• Program objectives (program component) correspond to short- to medium-term outcomes (program logic model) measured by outcome evaluation (evaluation plan)
• Program strategies/activities (program component) correspond to inputs, activities, outputs (program logic model) measured by process evaluation (evaluation plan)

The evaluation plan is a document that sets out what is being evaluated, why the evaluation is being undertaken, how the evaluation should be conducted, and how the findings will be used.

An evaluation plan that is agreed in consultation with stakeholders can help ensure a clear, shared understanding of the purpose of an evaluation and its process. For evaluations where an independent evaluator is engaged, elements of the evaluation plan will form the basis for a request for proposal (RFP) document (see section on preparing an RFP) and a contract with the successful evaluator.

Note that all of the information required for a comprehensive evaluation plan may not be known when preparing the RFP, and the evaluator may help further develop or refine the plan. However, the clearer and more comprehensive the information supplied in the RFP, the more likely prospective evaluators will be able to provide a considered proposal. The evaluation plan should be developed with reference to the components of the program and the program logic model; these inform the evaluation plan by identifying aspects of the program that could be assessed using process and outcome measures, as outlined in Table 2.

The specific content and format of an evaluation plan will vary according to the program to be evaluated. It is suggested that, for population health programs, the following elements at least are included:

• overview of the program
• purpose of the evaluation
• audience for the evaluation
• evaluation questions
• evaluation design and data sources
• potential risks
• resources and roles

Proposed inclusions in each section of the evaluation plan are summarised as follows.

This section should include a brief overview of the broad aims and specific objectives of the program. The program objectives should be SMART: 19

• Specific: clear and precise, including the population group and setting of the program
• Measurable: can be assessed using existing or potential data collection methods
• Achievable: reasonable and likely to be achieved within the timeframe
• Relevant: likely to be achieved given the activities employed, and appropriate for realising the aims
• Time specific: having a timeframe for meeting the objective.

This section should also outline the program’s development history, its strategies and/or activities, key stakeholders, and the context in which it is being developed and implemented. The program logic model should be included.

The fundamental reason for conducting the evaluation should be clearly stated. In articulating the purpose of the evaluation, it is important to consider the decisions that will be made as a result of the findings (such as program adjustments to enhance efficiency, justification of investment to program funders, scaling up of a program) and when these decisions will be made. For example, the purpose of an evaluation may be to inform decisions about developing, improving, continuing, stopping, reducing or expanding a program.

A related consideration is the primary audience for the evaluation: the people or groups that will use the information produced by the evaluation. These may include decision makers, program implementation staff, organisations running similar programs in other jurisdictions or countries, and consumers. The primary users should be specified in this section of the evaluation plan.

Evaluation questions serve to focus an evaluation and provide direction for the collection and analysis of data. 3 Evaluation questions should be based on the most important aspects of the program to be examined. The program logic model can help in identifying these. For example, the program logic can help to convert general questions about the effectiveness of a program into specific questions that relate to particular outcomes in the causal pathway, and questions about the factors most likely to affect those outcomes. 16

The number of evaluation questions agreed upon should be manageable in relation to the time and resources available. It is important to think strategically when determining what information is needed most so that the evaluation questions can be prioritised and the most critical questions identified. 7

Different types of evaluation require different sorts of evaluation questions.

Types of evaluation and typical evaluation questions 18

Process evaluation.

• Investigates how the program is delivered: activities of the program, program quality, and who it is reaching
• Can identify failures of implementation, as distinct from program ineffectiveness

Typical questions

• How is the program being implemented?
• Is the program being implemented as planned?
• Is the program reaching the target group?
• Are participants satisfied with the program?

Outcome evaluation

• Measures the immediate effects of the program (does it meet its objectives?) and the longer-term effects of the program (does it meet its aims?)
• Can identify unintended effects
• Did the program produce the intended effects in the short, medium or long term?
• For whom, in what ways and in what circumstances?
• What unintended effects (posititive and negative) were produced?
• To what extent can changes be attributed to the program?
• What were the particular features of the program and context that made a difference?
• What was the influence of other factors?

Economic evaluation

• Considers efficiency by standardising outcomes, often in terms of dollar value
• Answers questions of value for money, cost-effectiveness or cost-benefit
• Was the intervention cost-effective (compared to alternatives)?
• What was the ratio of costs to benefits?

Depending on its purpose and scope, the evaluation may include process, outcome or economic measures≈ or a combination of these. For example, while an innovative program (such as the pilot of an intervention) may require an outcome evaluation to determine whether the program was effective, rollout of an existing successful program may only require a process evaluation to monitor its implementation. 6 Figure 4 illustrates where different types of evaluation are likely to fit in the planning and evaluation cycle. Note that an assessment of the outcomes of a program should be made only after it has been determined that the program is being implemented as planned and appropriate approval (per delegations) for outcome evaluation has been obtained. Consideration should be given to the likely time required for program redesign (where relevant) and the expected time lag until outcomes are realised.

For each evaluation question, one or more indicators should be identified that define how change or progress in relation to the question will be assessed (for example, ‘number of clients enrolled’, ‘client satisfaction with program’, ‘change in vegetable intake’, ‘changes in waist circumference’). The indicators should meet the SMART criteria (specific, measurable, achievable, relevant, time specific).

≈ For more information about when to commission an economic evaluation and an overview of economic evaluation techniques, refer to Engaging an Independent Evaluator for Economic Evaluations: A Guide. 20

The design of a program evaluation sets out the combination of research methods that will be used to provide evidence for key evaluation questions. The design informs the data needed for the evaluation, when and how the data will be collected, the data collection instruments to be used, and how the data will be analysed and interpreted. More detailed information about quantitative study designs used in outcome evaluations is provided in Study Design for Evaluating Population Health and Health Service Interventions: A Guide. 21 Data may be collected using quantitative, qualitative or mixed methods; the NSW Treasury resource Evidence in Evaluation technical note describes each of these.

Data that will provide the information required for each indicator in order to answer the evaluation questions should be identified and documented. Data sources may include both existing data (e.g. routinely collected administrative data, medical records) and data that will have to be generated for the evaluation (e.g. survey of staff, interviews with program participants). For new data, consideration should be given to data collection methods, when data should be collected, who will be responsible for data collection, and who will be the data custodian (i.e. who has administrative control over the data).

Details about data required for an evaluation are often presented alongside relevant evaluation questions and indicators in a table (or data matrix). An example is included below.

It may be useful to seek advice from data, research or evaluation specialists when considering possible evaluation designs and data sources. Alternatively, potential independent evaluators may be asked to propose a design or enhance an initial idea for a design as part of their response to a request for proposal.

Example of an evaluation data matrix

Evaluation question.

Did the program result in increased quit attempts among smokers?

• Number of quit attempts initiated in previous 3 months among LHD clients who were smokers
• Number of successful quit attempts in previous 3 months among LHD clients who were smokers

Data source

Client survey

Baseline, then 3, 6 and 12-months post-intervention

Responsibility

Potential risks to the evaluation and possible mitigation strategies should be identified early in the evaluation planning process.

Potential risks to the evaluation may include, for example, inability to recruit participants or low response rates; evaluation findings that are inconclusive; or difficulty in determining the extent to which the changes observed are attributable to the program. Potential independent evaluators may be asked to determine possible risks and strategies for managing them as part of their response to a request for proposal.

A matrix to analyse the likelihood and consequences of any risks, and strategies for their management, is presented below. The NSW Health policy directive PD2022_023 Enterprise-Wide Risk Management Policy includes further information and tools.

While the risk management matrix and policy directive relate primarily to program management and corporate governance, the principles are also relevant to program evaluation.

Risk management matrix

Risk source.

List risks here

• Almost certain

Consequence

• Catastrophic

Risk rating

Action to manage risk.

List action to manage risks here

The human, financial and other resources available for the evaluation should be documented. This includes both internal resources for planning, procurement and project management, and a budget for engaging an independent evaluator. Financial resourcing for an evaluation will need to be considered at an early stage to ensure funding is approved and allocated in the program budget. A rough estimate of cost for an evaluation is 1% to 5% of the program costs; 1 however, the actual cost will be informed by the type and breadth of evaluative work to be undertaken.

The roles of Ministry staff, stakeholders and the evaluator should also be clearly documented. The timeframe for the evaluation should be linked to the stated roles and resources; this should take into account any key milestones (e.g. decision points).

As noted previously, an evaluation advisory group should be established to guide the planning and conduct of the evaluation. The roles and responsibilities of this group should be clearly stated in its terms of reference and outlined in this section of the evaluation plan.

A plan for how the results of the evaluation will be reported and disseminated should be agreed at an early stage. The dissemination plan should consider the range of target audiences for the evaluation findings (e.g. program decision makers, community members), their specific information needs, and appropriate reporting formats for each audience (e.g. written or oral, printed or electronic).

Note that the public release of evaluation findings is recommended to foster accountability and transparency, contribute to the evidence base, and reduce duplication and overlap. 1

Timeliness of reporting should also be considered; for example, staged reporting during the course of an evaluation can help to ensure that information is available at crucial decision making points. 3

Preparation of a detailed evaluation report that describes the program and the evaluation design, activities and results in full is important to enable replication or wider implementation of the program. 22 In addition, more targeted reporting strategies should be considered as part of dissemination planning. These may include, for example, stakeholder newsletters, brief plain language reports, or presentations to decision makers or at conferences, workshops and other forums.

If appropriate, evaluation results may also be published in a peer-reviewed journal. If it is proposed to publish a journal paper, the evaluation advisory group should pre-plan the procedures for writing and authorship; review of the evaluation by an HREC should also be considered at an early stage, as some journals require ethics approval. Consideration should be given to publication in an open access journal to enhance the potential reach of the results.

While small-scale evaluations may be completed inhouse, evaluations of programs involving a reasonable investment, and those being reviewed for continuation or expansion, may require procurement of an independent evaluator.

NSW Health requirements for the procurement of goods and services, including engagement of consultants, are outlined in the policy directive PD2023_028 NSW Health Procurement (Goods and Services) Policy . The NSW Health Procurement Portal provides a step-by-step overview of the procurement process and includes links to a range of tools, templates and other resources to support procurement.

The approvals required for the procurement process should be determined, noting that the level of approval will depend on the estimated cost of the consultancy as per the Delegations Manual. 23 All of the necessary approvals (e.g. funding approval by an appropriately delegated officer, approval to issue a tender) should be obtained prior to commencing procurement.

The specifications of the project should be developed and documented in a request for proposal concurrently with a plan for assessing responses. This plan should include assessment criteria and weightings and should identify who will be part of the assessment panel.

The process for engaging an independent evaluator will require preparation of a request for proposal (RFP). The RFP document outlines the specifications of the evaluation project and should be developed with reference to the parts of the evaluation plan that have been agreed with program stakeholders. An RFP template is available from the Ministry of Health Procurement Portal.

The RFP should be clear and comprehensive. The more information that can be provided, the greater the likelihood that potential evaluators will understand what is required of them and prepare a considered and appropriate response. Table 6 in the Treasury Evaluation Policy and Guidelines has additional information with examples to assist in ensuring potential evaluators can effectively design and cost their proposal. 1 Consider the following when preparing an RFP document:

When describing the program to be evaluated:

• Include a comprehensive overview of the key features of the program, including:
• the aims and objectives of the program
• its development and implementation history, including any previous or concurrent evaluations, and current stage of development or implementation of the program
• components and/or activities of the program, its scale (e.g. LHD-specific, statewide), and who is delivering the program
• governance and key stakeholders
• the context in which the program is being developed and/or implemented
• Include the program logic model, if one exists
• Ensure any technical terms are defined
• Ensure key terms are used accurately and consistently (e.g. cost benefit versus cost effectiveness). 20

When describing the evaluation and specifying the work to be undertaken by the evaluator:

• Ensure that the purpose of the evaluation is expressed in a way that will not compromise the objectivity of the evaluator. The purpose should be couched in neutral terms (e.g. “to inform decisions about scaling up the program” rather than “to justify plans to scale up the program”)
• Specify any evaluation questions, indicators and data sources that have already been agreed. If appropriate, include a draft evaluation plan
• Clearly delineate which tasks are within scope for the evaluator and those that are out of scope
• Describe in detail the data that will be available for use by the evaluator, how the evaluator will be given access to the data, and any conditions on its use. Include as much information about these data sources as possible (e.g data collection methods, size of dataset, relevant variables, any limitations of the data, custodianship, confidentiality)
• Ensure that timeframes for deliverables are realistic and achievable. In determining timeframes, consider the size and complexity of tasks to be undertaken by the evaluator, any key decision points for which results will be required, and any mitigating factors that could impact on the completion of tasks (e.g. end of year)
• It is recommended that an indicative budget is specified. The budget should be estimated based on the tasks expected of the evaluator and the funds available
• Clearly outline the format in which evaluation findings should be reported by the evaluator. In particular, it is important to consider whether reports should include only results from the evaluation or also an interpretation and/or recommendations. Whether or not recommendations should be included will depend on the program, the purpose of the evaluation and the stakeholders involved.

When listing criteria for assessing applications:

It is suggested that the criteria listed in Potential criteria for assessing applications be considered.

Depending on whether the evaluator was asked to identify potential risks, and consider appropriate mitigation strategies, it may be desirable to include a relevant assessment criterion (e.g. “Demonstrated experience and expertise in risk identification and mitigation related to evaluations and appropriateness of the risk mitigation strategy for this evaluation project”).

Responses to the RFP should be assessed in accordance with the agreed plan. A report and recommendation should be prepared and approval for the recommendation obtained as per delegations.

Once an evaluator has been selected a contract will need to be signed. Advice on selecting, developing and maintaining contracts is available from the Legal and Regulatory Services intranet site or the NSW Health Procurement Portal.

For projects with a value of $150,000 (GST inclusive) or more, it is a requirement under the Government Information (Public Access) Act 2009 (GIPA Act) that contract information is disclosed on the NSW Government tenders website; see PD2018_021 Disclosure of Contract Information. The GIPA Disclosure Form is available from the NSW Health Procurement Portal.

Potential criteria for assessing applications

• Demonstrated experience on evaluation projects of comparable scale and complexity, and/or with specific techniques or approaches (e.g. "Significant relevant evaluation experience and capability to deliver the full scope of the project requirements including the experience of the designated staff in undertaking similar evaluations”, "Demonstrated experience with both quantitative and qualitative evaluation methods and in producing high-quality evaluation reports”)
• Demonstrated experience on projects in relevant sectors or settings (e.g. “Demonstrated experience in working in the general practice setting”, “Demonstrated understanding of family violence and the associated issues”)
• Quality, feasibility and appropriateness of the proposal for conducting the evaluation (e.g. “Quality and relevance of the proposal for achieving the required evaluation services and deliverables as identified in this RFP”, “Feasibility, appropriateness and scientific rigour of the proposed work plan and methodology for achieving the required Services and Deliverables”)
• Feasibility and value for money of proposed fee structure (e.g. “Proposed fee structure is feasible and represents value for money”).

Planning an evaluation requires project management skills including the development of a workplan with clear timeframes and deliverables. An independent evaluator will usually develop a draft workplan as part of their response to the RFP which can be refined with the evaluation advisory group after they are contracted.

Plan to establish an effective evaluation governance structure with clear terms of reference from the outset.

The active involvement of NSW Health staff throughout the evaluation is important for successful project management. Regular scheduled updates and meetings with the evaluator throughout the implementation of the evaluation will help communication and facilitate a shared understanding of the evaluation needs and the management of any problems that may arise. Consider using structured project management methods or systems to keep the evaluation on track.

A successful RFP process will identify an evaluator who has the skills and experiences to rigorously collect, analyse and report the data. The contract with the evaluator will include requirements for the provision of a draft report or reports for comment, as well as the writing of a final report incorporating feedback. The Ministry’s role in reviewing the draft report is not to veto the results but to comment on structure, accuracy and whether it has answered the evaluation questions.

The fundamental reason for conducting an evaluation is to inform health policy and program decisions for the benefit of the NSW public.

Factors that support the incorporation of results into program decision making include:

• the engagement of end-users of the evaluation findings through the program planning and evaluation cycle
• active dissemination strategies (not limited to publications in academic journals or presentations at academic conferences)
• the tailored communication of results and recommendations to decision makers
• an organisational culture supportive of the understanding and use of evidence. 9,24,25

Before dissemination, the final evaluation report will need to be approved for release by the appropriate Ministry delegate. Once approved for release, communicating the completed evaluation results is important to inform the development of the program as well as future population health programs. It is good practice to make results available to any stakeholders who have had input into the evaluation.

It is best to plan early for how the results of the evaluation will be reported and communicated (see section on reporting).

Dissemination of evaluation findings may take a number of approaches:

• evaluators provide a feedback session to stakeholders
• electronic newsletters tailored to stakeholders
• results reported to relevant Ministry committees and management structures
• placing the final report online
• conference papers
• peer review publication of results
• if suitable, communication to the media, with the involvement of the Ministry’s Public Affairs Unit.

Crucially, the results and/or recommendations from the evaluation report will need to be reviewed and responded to by the policy branch responsible for the program, and an implementation plan or policy brief developed.

• NSW Treasury. NSW Treasury Policy and Guidelines: Evaluation (TPG22-22) . Sydney: NSW Treasury; 2023.
• NSW Health. NSW Health Guide to Measuring Value. Sydney: NSW Ministry of Health; 2023.
• Owen JM. Program evaluation: forms and approaches. 3rd edition. New York: The Guilford Press; 2007.
• Centre for Epidemiology and Evidence. Increasing the Scale of Population Health Interventions: A Guide . Evidence and Evaluation Guidance Series, Population and Public Health Division. Sydney: NSW Ministry of Health; 2023.
• Centre for Epidemiology and Evidence. Developing and Using Program Logic: A Guide . Evidence and Evaluation Guidance Series, Population and Public Health Division. Sydney: NSW Ministry of Health; 2023.
• Bauman AE, Nutbeam D. Evaluation in a nutshell: a practical guide to the evaluation of health promotion programs. 2nd edition. North Ryde: McGraw Hill; 2014.
• United Nations Development Programme. Handbook on Planning, Monitoring and Evaluating for Development Results . New York: UNDP; 2009.
• Robson C. Small-Scale Evaluation: Principles and Practice. London: Sage Publications Inc.; 2000.
• Milat AJ, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Sys 2013; 11: 5.
• National Health and Medical Research Council. Ethical Considerations in Quality Assurance and Evaluation Activities . Canberra: NHMRC; 2014.
• National Health and Medical Research Council. National Statement on ethical conduct in human research 2007 (updated 2018) . Canberra: NHMRC; 2007.
• NSW Health. Guidance Regarding Expedited Ethical and Scientific Review of Low and Negligible Risk Research: New South Wales . Sydney: NSW Health; 2023.
• National Health and Medical Research Council. Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders . Canberra: Commonwealth of Australia; 2018.
• National Health and Medical Research Council. Keeping research on track II: a companion document to Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders . Canberra: Commonwealth of Australia; 2018.
• AH&MRC Ethics Committee. AH&MRC Ethical Guidelines: Key Principles V2.0. Sydney: AH&MRC; 2020.
• Funnell SC, Rogers PJ. Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Hoboken: Wiley; 2011.
• Holt L. Understanding program logic . Victorian Government Department of Human Services; 2009.
• Hawe P, Degeling D, Hall J. Evaluating Health Promotion: A Health Worker’s Guide. Sydney: Maclennan & Petty; 1990.
• Round R, Marshall B, Horton K. Planning for effective health promotion evaluation. Melbourne: Victorian Government Department of Human Services; 2005.
• Centre for Epidemiology and Evidence. Engaging an Independent Evaluator for Economic Evaluations: A Guide . Evidence and Evaluation Guidance Series, Population and Public Health Division. Sydney: NSW Ministry of Health; 2023.
• Centre for Epidemiology and Evidence. Study Design for Evaluating Population Health and Health Service Interventions: A Guide . Evidence and Evaluation Guidance Series, Population and Public Health Division. Sydney: NSW Ministry of Health; 2023.
• Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 2008; 337: a1655.
• NSW Department of Health. Delegations Manual– Combined–Administrative, Financial, Staff. Sydney: NSW Department of Health; 1997.
• Moore G, Todd A, Redman S. Strategies to increase the use of evidence from research in population health policy and programs: a rapid review. Sydney: NSW Health; 2009. Available from: www.health.nsw.gov. au/research/Documents/10-strategies-to-increase-research-use.pdf
• Moore G, Campbell D. Increasing the use of research in policymaking . Sydney: NSW Health; 2017.

Advancing scientific understanding of climate, improving society’s ability to plan and respond

Events/Webinars

FY25 ESSM NOFO Informational Webinar

FY25 NIDIS Coping with Drought Competition Post LOI Informational Webinar

Applicant Resources at a Glance

• Current Federal Funding Opportunity (FFO)
• Data Universal Numbering System (DUNS) Number
• System for Award Management (SAM)
• National Environmental Policy Act (NEPA) Guidance
• Application for Federal Assistance  SF-424 Form SF-424 Form Instructions
• Budget Information  SF-424A SF-424A Form Instructions
• Assurances  SF-424B
• Certifications Required by the Department of Commerce  CD-511
• Catalog of Federal Domestic Assistance (CFDA)
• Grants Online
• Code of Federal Regulations (CFR)
• Department of Commerce, Grants and Cooperative Agreements Manual
• Department of Commerce, Grants Management Division

Evaluation of Community Partnership for Self-reliance (CPS)/Community Research Partnerships

The Alaska Center for Climate Assessment and Policy (ACCAP) is working in partnership with staff, community members, and researchers to evaluate both the outcomes of the project but more importantly the process of co-production throughout the community-university partnerships. An all-hands workshop was held in May as part of this project and further data collection and analysis is underway.

Funding Opportunities News and Events

NOAA seeks proposals to advance climate research

• August 12, 2024

Modeling, Analysis, Predictions, and Projections (MAPP) Program: Early Career Award for Exceptional Research in Earth System Model Development and Application

Atmospheric Chemistry, Carbon Cycle, and Climate (AC4) Program

FY25 NIDIS Coping with Drought Competition Information Webinar

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) and National Research Council (US) Committee on the Review of NIOSH Research Programs. Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps. Washington (DC): National Academies Press (US); 2009.

Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps.

• Hardcopy Version at National Academies Press

2 The Program Evaluation Context 1

P rogram evaluation has been defined as “systematic inquiry that describes and explains the policies’ and program’s operations, effects, justifications, and social implications” (Mark et al., 2000, p. 3) or “… the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming” (Patton, 1997, p. 23). The evaluations of National Institute for Occupational Safety and Health (NIOSH) programs carried out under the guidance of the framework committee represent just one way of evaluating research programs. This chapter places the National Academies’ evaluations of NIOSH programs in context by providing a brief overview of the general principles involved in program evaluation and by describing where the process fits in the spectrum of current practices in evaluating research programs. At the conclusion of some of the overview sections throughout the chapter, the committee’s findings specific to the evaluation process used by the framework and evaluation committees are included in bold and italicized text.

• PROGRAM EVALUATION

Although formal program evaluations, especially of educational programs, preceded World War I, the profession as currently practiced in the United States has increased in prominence within the past 50 years. A major impetus to this growth was the need to assess the social programs instituted through the War on Poverty and Great Society policies of the 1960s (Shadish et al., 1991). Legislative requirements for the evaluation of many programs represented a turning point in the growth in the number of evaluations. Evaluation is now an established professional practice, reflected through organizations such as the American Evaluation Association and the European Evaluation Society (AEA, 2009; EES, 2009). Program evaluation is one element of results-oriented management, the approach to public management embodied in the past decade in the Government Performance and Results Act (OMB, 2009a) and the Office of Management and Budget’s (OMB’s) Program Assessment Rating Tool (OMB, 2009b).

Current efforts in program evaluation follow several schools of thought that differ in the evaluation processes used but are all focused on achieving a valid evaluation. The essence of evaluation is determining what is of value in a program. The work revolves around understanding program goals (if available), setting criteria for success, and gathering information to determine whether the criteria are being met as a result of program activities. Program evaluations focus on examining the characteristics of a portfolio of projects rather than assessing one project at a time and often use retrospective information about program outputs and outcomes. Program evaluation differs from a research project in being more tightly connected to practice; it is commissioned by a specific user or organization and designed to inform decision making. It also differs from performance measurement, which is an ongoing process that gathers indicators of what the program is accomplishing but may not assess why the indicators are changing.

Program evaluations can serve several functions. When the program is initially in development or is undergoing changes and is being evaluated with the goal of program improvement, the evaluation is termed a formative evaluation (Scriven, 1991). These evaluations are often initiated and used in-house. When the objective of the evaluation is to assess the program’s outcomes in order to determine whether the program is succeeding or has accomplished its goals, the evaluation is termed a summative evaluation (Scriven, 1967; Gredler, 1996). Users of summative evaluations are often decision makers outside of the program. Program evaluation often also helps communicate the program’s goals and accomplishments to external audiences. Evaluations provide information that contributes to decisions that shape program goals, strategic plans, and actions. In these cases, they serve instrumental functions. Often they also serve enlightenment functions, such as increasing general understanding of program operations, underlying assumptions, or social context (Weiss, 1977).

The practice of evaluating research programs has historically been somewhat separate from that of social program evaluation. Qualitative assessments of research programs in the United States date back to the 1950s (NAS, 1959). The evaluation of research programs took a more quantitative turn in the 1970s as evaluations started to draw on the new availability of large-scale databases to describe scientific activity. Research program evaluation is distinguished from social program evaluation in a number of ways, including the dominant use of peer-review panels and the use of specialized data, including publication and patent-based measures (see discussion later in this chapter).

The evaluations of NIOSH programs discussed in this report were un dertaken in the context of the externally mandated Program Assessment Rating Tool process, a summative evaluation process developed by OMB. However, NIOSH leadership established their primary goal as program improvement, making the evaluations primarily formative.

• LOGIC MODELS

The evaluations of NIOSH programs used logic models—both a general logic model for NIOSH research and specific logic models for each program evaluated. Prior to the work of the evaluation committees, NIOSH contracted with RAND Corporation to provide operational and analytical assistance with compiling the evidence packages for the reviews and developing the logic models; a detailed description of that effort can be found in a recent RAND report (Williams et al., 2009).

Logic models are widely used in program evaluation (W. K. Kellogg Foundation, 2000; World Bank, 2000) to represent visually what evaluators call “program theory.” This phrase refers to the understanding of how the program is supposed to work. How do the program resources become results, and through what channels do those results have their expected impacts? The logic model may be represented as a set of boxes and arrows or as a hierarchy of goals, intermediate outcomes, and final outcomes. The representation provides guidance for the evaluation by pointing to relevant kinds of information to be considered in the assessment and often to indicators in the various areas of the model.

McLaughlin and Jordan (1999) refer to logic models as a way of “telling your program’s performance story.” The common elements of logic models are inputs, activities, outputs, customers, and outcomes (short, medium, and long term), plus external influences (Wholey, 1983; Figure 2-1).

Building a logic model is a process that should involve a team of people with different roles in the program who interact with external stakeholders at many points. After collecting relevant information and clearly identifying the problem the program addresses, the team organizes its information into various elements and composes a diagram that “captures the logical flow and linkages that exist in any performance story” (McLaughlin and Jordan, 1999, p. 68).

Elements of the logic model. Reprinted from McLaughlin and Jordan, 1999, with permission from Elsevier.

Logic models are nearly always predominantly linear and causal because agencies use them to think through how programs will achieve their public goals. In research planning and evaluation, this linearity is ironic. The widespread consensus is that research does not create its effects in a linear fashion. Rather, it is embedded in a complex ecology of relationships that shape and spread knowledge through a variety of channels, not just research knowledge.

Additionally, it is challenging for logic models to capture some outputs such as the development of human capital. Over time, a program may have a significant impact on a field by helping to build a community of practitioners and researchers. For example, NIOSH’s impact on the existence and growth of the occupational safety and health research community is hard to capture in a logic model. In addition, ongoing dialogues with external stakeholders shape research activities and spread research knowledge in ways that are hard to track. Program evaluations that solely rely on the logic model almost inevitably miss information on some of the nonlinear effects of program activities.

The logic models used in the evaluation of NIOSH programs helped pro gram staff and evaluators organize information into steps in the flow of program logic. However, because some of the NIOSH programs spanned several NIOSH divisions and laboratories, the logic model sometimes made it hard for the evaluation committee to grasp the full picture of the research program. Furthermore, the logic models focused a great deal of attention on the most readily observable short- and medium-term outcomes, perhaps missing information on nonlinear and more diffuse contributions of the programs to the development of knowledge and hu man capital in occupational safety and health.

• ROLE OF STAKEHOLDERS

The practice of program evaluation has paid special attention to external stakeholders and the role they play in the evaluation process. Sometimes stakeholders are direct beneficiaries of the program; for example, for a day-care center program, the major stakeholders are the families whose children receive care. Sometimes the stakeholders are organizations with whom the program must work to achieve its goals. In the case of research on occupational safety and health, key stakeholders include workers, employers, and regulatory agencies.

Stakeholder participation in evaluating research programs has come more slowly than in social program evaluation. Early evaluation panels tended to consist entirely of scientists and engineers. But as research policy became more focused on making research relevant to the private sector, evaluation panels also began to include industry and labor representation, often scientists and engineers working in industry and labor organizations. Individuals and families exposed to environmental hazards often organize to increase research and remediation efforts, and stakeholders from these groups also participate in evaluation processes.

Just as social program evaluation pays particular attention to differences in knowledge and expertise between evaluators and stakeholders, in the evaluation of research programs the different contributions of scientific experts and external stakeholders both need to be respected. When the research being evaluated is intended to serve vulnerable populations, current standard practice in the evaluation of research programs, as described in the last paragraph, is not sufficient to give voice to these groups and additional attention needs to be paid to obtaining adequate input.

The National Academies evaluation committees included a variety of members with strong connections to NIOSH’s external stakeholder groups, such as manufacturers of safety equipment, labor organiza tions, and employers. The committees also reached out to a wide range of external stakeholder groups for input, including vulnerable worker populations.

• METHODS OF EVALUATION

Evaluations of research programs necessarily use a variety of approaches. Expert panel review is the “bread-and-butter” approach worldwide, but there is also a long track record of evaluation studies, in which external consultants gather and analyze primary data to inform the expert deliberations.

Within the range of evaluation approaches for research programs, the National Academies’ evaluations of NIOSH programs clearly fall among expert panel evaluations, rather than evaluation studies.

Expert Panel Review

Merit review, peer review, and expert panels are used widely for both ex ante and ex post evaluations of the productivity, quality, and impact of funding organizations, research programs, and scientific activity. Benefits and limitations of this approach have been reviewed extensively (Bozeman, 1993; Guston, 2003; Hackett and Chubin, 2003). Expert panel review is the oldest—and still most common—form of research and development evaluation. In fact, the expert panel is very much a historical development from the National Academies itself, which was established in the 19th century to provide scientific and technical policy advice to the federal government. The underlying evaluative theory of the expert panel is that scientists are uniquely able to evaluate the quality and importance of scientific research (Kostoff, 1997). The preeminence of scientists to evaluate the quality and importance of scientific research was further codified in the research agencies that developed under the philosophy of Vannevar Bush in the 1940s (Bush, 1945).

Expert judgment is particularly capable of evaluating the quality of discrete scientific research activities and the relevance of such discrete activities to particular bodies of knowledge. For example, toxicologists and biochemists—through their scientific training—are uniquely capable of assessing the contributions of particular theories, research methodologies, and evidence to answer specific scientific questions and problems. The major limitation of expert panel review is that traditional training and experience in the natural and physical sciences do not prepare scientists to address questions related to the management, effectiveness, and impact of the types of broad research portfolios that federal agencies typically manage.

Although expert panel reviews work to balance conflicting values, objectives, or viewpoints, they also may lead to tensions in the same areas they are expected to resolve. As noted above, the review process may be broadened to include other stakeholders beyond “experts” or “peers.” Expert panels usually operate with an evaluation protocol developed by an outside group, including evaluation procedures, questions to be answered, and evaluation criteria (e.g., the evaluation of the Sea Grant College Program, Box 2-1 ). The panels usually review a compilation of data on the program, including plans, input counts (budget, staffing), project descriptions, and lists of results. They then talk with individuals connected to the program, both inside and outside the given agency, and use their own experience and judgment in reaching conclusions.

Evaluation of the National Sea Grant College Program. The National Sea Grant College Program, funded by the National Oceanic and Atmospheric Administration, is a nationwide network of 30 university programs aimed at conducting research, education, and (more...)

Closely tied to review processes is the assignment of various types of ratings. For example, the Research Assessment Exercise of the United Kingdom uses 15 panels and 67 subpanels following a common protocol to assess university research programs and assign scores by discipline area (RAE, 2009). Rating scales are be ing used more frequently as evaluations have become more and more oriented to demonstrating performance to outside audiences or to allocating resources. Rating scales capture qualitative judgments on ordinal scales and allow for descriptions of performance at the various levels.

Characteristics that are sought in expert panel reviews include a panel with a balanced set of expertise and credibility among various stakeholder groups and independence and avoidance of conflict of interest among panel members to the extent possible. Selection of panel members can involve trade-offs between recruiting independent reviewers or recruiting reviewers with knowledge and understanding of the program and its field of science. For this reason, expert review panels are seldom completely free of bias and may have conflicts of interest; the preferred practice, of course, is for conflicts to be considered and disclosed. Independence is also reinforced when the panel is commissioned by, and reports to, a user located at least one level above the program in the management hierarchy. The panel adds value by including its perspectives and insights in its report. The panel makes the evidence base for its conclusions explicit in the report and usually makes a limited number of realistic recommendations, phrased broadly enough to allow management to adapt the recommendations to specific circumstances.

The National Academies committees follow a thorough bias and conflict- of-interest process that includes completion of disclosure forms and the bias and conflict-of-interest discussion held at the first meeting.

Other Methods of Evaluating Research Programs

Other types of evaluations generally involve hiring consultants to provide analyses of specific outputs of the program. Because the goal of a research program is new knowledge, publications represent a concrete and observable manifestation of new knowledge and are frequently used as a convenient measure of research program outputs. Publications in peer-reviewed journals provide an indication of quality control, and citations to published articles are used to assess the scientific impact of the work. Patents provide a similar set of measures for technology development. Thus, evaluations of research programs have extensive relevant datasets on which to base their assessments.

Statistical analyses of data on publications (e.g., books, journal articles, review articles, book chapters, notes, letters) range from fairly simple counts and comparisons of publications to highly sophisticated factor analyses and correlations of many types of terms, such as keywords, institutions, and addresses, that lead to the development of networks or maps of the ways in which the research outputs are connected. These bibliometric methods are used extensively to evaluate research activities and compare research output across institutions, disciplines, fields, funding programs, countries, and groups of researchers (Kostoff, 1995; Georghiou and Roessner, 2000; Hicks et al., 2004; Weingart, 2005). Bibliometric methods also can be used to assess the extent of collaboration. Visualization techniques now produce “maps of science” allowing organizations that support research to “see” where the work they have supported fits into research in a specific field or the extent to which it is being used in other research endeavors. An important strength of bibliometric analyses is that they are data-based analyses following a fixed set of rules or algorithms. The analyses are often used as a complement to peer-review techniques, surveys, or impact analyses of research activities. An important weakness, however, is that the measures are incomplete. They do not capture all the dimensions of performance or its context, factors that an evaluation usually needs to consider. In general, a composite set of measures is used to determine the effectiveness of the research activities, institutions, or national programs ( Box 2-2 ).

Review of the National Science Foundation’s Science and Technology Center Programs. Beginning in 1989, the National Science Foundation (NSF) established 25 Science and Technology Centers (STCs) across the United States. The goal was to promote (more...)

Other methods used in evaluating research programs include methodologies drawn from the social sciences, including case studies, interviews, and surveys. One special application of case studies in the evaluation of a research program, for example, is the TRACES approach, named for an early study of Technology in Retrospect and Critical Events in Science (IIT, 1968). This approach starts from a recent accomplishment or success, then tracks the complex set of earlier research results and technologies that made it possible. Programs with economic goals have also used case studies to illustrate the return on investment in advanced technology projects (Ruegg, 2006).

In summary, the evaluation of research programs is an established branch of program evaluation. The National Academies’ evaluation of NIOSH research programs used one of the most common approaches: expert panel review. As is common in evaluations of applied research programs, this process involved stakeholders as members of the evaluation committees and also sought external stakeholder input. The evaluation framework described in Chapter 3 organizes data into a common evaluation tool based on a logic model approach and provides for consideration of external factors. Similar to many research program evaluation efforts, the evaluation committees used this structured rating tool to provide some consistency in ratings across programs. The process did not, however, expand into an evaluation study by gathering new data or extensively analyzing external data sources. The evaluations of NIOSH programs fall well within the range of acceptable practice in evaluating research programs and are compiled in comprehensive reports that went through peer review under the National Academies’ report review process.

• AEA (American Evaluation Association). 2009. American Evaluation Association . http://www ​.eval.org (accessed March 23, 2009).
• Bozeman, B. 1993. Peer review and evaluation of R&D impacts . In Evaluating R&D impacts: Methods and practice . Edited by B. Bozeman, editor; and J. Melkers, editor. . Boston, MA: Kluwer Academic. Pp.79–98.
• Bush, V. 1945. Science: The endless frontier . Washington, DC: U.S. Government Printing Office.
• EES (European Evaluation Society). 2009. European Evaluation Society . http://www ​.europeanevaluation.org/ (accessed March 23, 2009).
• Fitzsimmons, S. J., O. Grad, and B. Lal. 1996. An evaluation of the NSF Science Technology Centers Pro gram . Vol. 1, Summary. Washington, DC: Abt Associates. http://www ​.nsf.gov/od ​/oia/programs/stc/reports/abt.pdf (accessed March 23, 2009).
• Georghiou, L., and D. Roessner. 2000. Evaluating technology programs: Tools and methods . Research Policy 29(4–5):657–678.
• Gredler, M. E. 1996. Program evaluation . Englewood Cliffs, NJ: Merrill.
• Guston, D. 2003. The expanding role of peer review processes in the United States . In Learning from science and technology policy evaluation: Experiences from the United States and Europe . Edited by P. Shapira, editor; and S. Kuhlmann, editor; . Cheltenham, UK, editor. , and Northampton, MA: Edward Elgar. Pp.81–97.
• Hackett, E., and D. Chubin. 2003. Peer review for the 21st century: Applications to education research . Paper presented at the National Research Council Workshop, Washington, DC, February 25, 2003. http://www7 ​.nationalacademies ​.org/core/HacketChubin ​_peer_review_paper.pdf (accessed November 16, 2008).
• Hicks, D., H. Tomizawa, Y. Saitoh, and S. Kobayashi. 2004. Bibliometric techniques in the evaluation of federally funded research in the United States . Research Evaluation 13(2):78–86.
• IIT (Illinois Institute of Technology). 1968. Technology in retrospect and critical events in science . Vol. 1. Chicago, IL: IIT Research Institute.
• Kostoff, R. N. 1995. Federal research impact assessment—axioms, approaches, applications . Scien tometrics 34(2):163–206.
• Kostoff, R. N. 1997. Peer review: The appropriate GPRA metric for research . Science 277:651–652.
• Mark, M., G. Henry, and G. Julnes. 2000. Evaluation: An integrated framework for understanding, guiding, and improving policies and programs . San Francisco: Jossey-Bass.
• McLauglin, J. A., and G. B. Jordan. 1999. Logic models: A tool for telling your program’s performance story . Evaluation and Program Planning 22:65–72.
• NAPA (National Academy of Public Administration). 1995. National Science Foundation’s Science and Technology Centers: Building an interdisciplinary research program . Washington, DC: NAPA.
• NAS (National Academy of Sciences). 1959. Panel reports of the NAS-NRC panels advisory to the National Bureau of Standards . Washington, DC: National Academy of Sciences.
• NRC (National Research Council). 1994. A review of the NOAA National Sea Grant College Program . Washington, DC: National Academy Press.
• NRC. 1996. An assessment of the National Science Foundation’s Science and Technology Centers Program . Washington, DC: National Academy Press.
• NRC. 2006. Evaluation of the Sea Grant Program review process . Washington, DC: The National Academies Press.
• OMB (Office of Management and Budget). 2009. a. Government Performance and Results Act . http://www ​.whitehouse ​.gov/omb/mgmt-gpra/gplaw2m.html (accessed March 20, 2009).
• OMB. 2009. b. Assessing program performance . http://www.whitehouse. gov/omb/part/ (accessed March 20, 2009).
• Patton, M. Q. 1997. Utilization-focused evaluation: The new century text . Thousand Oaks, CA: Sage.
• RAE (Research Assessment Exercise). 2009. Research Assessment Exercise . http://www ​.rae.ac.uk/ (accessed March 23, 2009).
• Ruegg, R. 2006. Bridging from project case study to portfolio analysis in a public R&D program—A framework for evaluation and introduction . NIST GCR 06-891. http://www ​.atp.nist.gov ​/eao/gcr06-891/gcr06-891report.pdf (accessed March 23, 2009).
• Scriven, M. 1967. The methodology of evaluation . In Curriculum evaluation: AERA monograph series on evaluation . Edited by R. E. Stake, editor. . Chicago: Rand McNally. Pp.39–85.
• Scriven, M. 1991. Beyond formative and summative evaluation . In Evaluation and education: At quarter century . Edited by M. W. McLaughlin and, editor; D. C. Phillips, editor. . Chicago: University of Chicago Press. Pp.19–64.
• Shadish, W. R., T. D. Cook, and L. C. Leviton. 1991. . Foundations of program evaluation: Theories of practice . Newbury Park, CA: Sage.
• Weingart, P. 2005. Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics 62(1):117–131.
• Weiss, C. H. 1977. Research for policy’s sake: The enlightenment function of social research . Policy Analysis : 3:531–545.
• Wholey, J. S. 1983. Evaluation and effective public management . Boston: Little, Brown.
• Williams, V. L., E. Eiseman, E. Landree, and D. M. Adamson. 2009. Demonstrating and communicating research impact: Preparing NIOSH programs for external review . Santa Monica, CA: RAND.
• W. K. Kellogg Foundation. 2000. Logic model development guide . http://www ​.wkkf.org/Pubs ​/Tools/valuation/Pub3669.pdf (accessed March 23, 2009).
• World Bank. 2000. Logframe handbook: A logical framework approach to project cycle management . http://www ​.wau.boku.ac ​.at/fileadmin/_/H81 ​/H811/Skripten/811332 ​/811332_G3_log-framehand book.pdf (accessed March 23, 2009).

This chapter draws on background papers commissioned by the committee from Sonia Gatchair, Georgia Institute of Technology, and Monica Gaughan, University of Georgia.

• Cite this Page Institute of Medicine (US) and National Research Council (US) Committee on the Review of NIOSH Research Programs. Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps. Washington (DC): National Academies Press (US); 2009. 2, The Program Evaluation Context.
• PDF version of this title (3.6M)

In this Page

Recent activity.

• The Program Evaluation Context - Evaluating Occupational Health and Safety Resea... The Program Evaluation Context - Evaluating Occupational Health and Safety Research Programs

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Open access
• Published: 04 September 2024

Targeting emotion dysregulation in depression: an intervention mapping protocol augmented by participatory action research

• Myungjoo Lee   ORCID: orcid.org/0000-0002-8301-7996 1 ,
• Han Choi   ORCID: orcid.org/0000-0003-0406-5605 2 &
• Young Tak Jo   ORCID: orcid.org/0000-0002-0561-2503 1  

BMC Psychiatry volume  24 , Article number:  595 ( 2024 ) Cite this article

Metrics details

Depression is a highly prevalent and often recurrent condition; however, treatment is not always accessible or effective in addressing abnormalities in emotional processing. Given the high prevalence of depression worldwide, identifying and mapping out effective and sustainable interventions is crucial. Emotion dysregulation in depression is not readily amenable to improvement due to the complex, time-dynamic nature of emotion; however, systematic planning frameworks for programs addressing behavioral changes can provide guidelines for the development of a rational intervention that tackles these difficulties. This study proposes an empirical and theoretical art-based emotion regulation (ER) intervention using an integrated approach that combines intervention mapping (IM) with participatory action research (PAR).

We used the IM protocol to identify strategies and develop an intervention for patients with major depressive disorder (MDD). As applied in this study, IM comprises six steps: (a) determining the need for new treatments and determinants of risk; (b) identifying changeable determinants and assigning specific intervention targets; (c) selecting strategies to improve ER across relevant theories and research disciplines; (d) creating a treatment program and refining it based on consultations with an advisory group; (e) developing the implementation plan and conducting a PAR study to pilot-test it; and (f) planning evaluation strategies and conducting a PAR study for feedback on the initial testing.

Following the steps of IM, we developed two frameworks for an art-based ER intervention: an individual and an integrative framework. The programs include four theory- and evidence-based ER strategies aimed mainly at decreasing depressive symptoms and improving ER in patients with MDD. We also developed a plan for evaluating the proposed intervention. Based on our preliminary PAR studies, the intervention was feasible and acceptable for adoption and implementation in primary care settings.

The application of IM incorporated with PAR has resulted in an intervention for improving ER in depression. While changing behavior is perceived as a challenging and elaborate task, this method can be useful in offering a clear structure for developing rational interventions. Further refinement is necessary through rigorous research.

Peer Review reports

Depression is a highly prevalent and often recurrent condition that severely impairs psychological functioning and quality of life. According to the Global Health Data Exchange, depression affects 3.8% of the world’s population and, as “a major contributor to the overall global burden of disease,” is associated with substantial societal and personal costs [ 1 , 2 ]. Due to its enormous impact on public health, the World Health Organization (WHO) predicts that depression will rank first among all causes of the burden of disease by 2030 [ 3 ]. As depression is frequently comorbid with other mental and physical disorders, it is particularly challenging to identify risk factors and develop effective interventions.

Depression is a disorder of emotion. Disordered affect is a hallmark of depressive episodes, characterized by complex but apparent abnormalities of emotional functioning [ 4 , 5 ]. Many factors may be associated with the disorder; however, its symptoms evidently indicate failures in emotional self-regulation [ 6 ]. Emotion regulation (ER) refers to an individual’s ability to modulate the intensity, frequency, and duration of emotional responses [ 7 , 8 ]. Decades of empirical research have shown that depression is associated with increases in unpleasant emotions and decreases in positive emotions [ 9 , 10 ]. It has been proposed that difficulties in ER in depression significantly contribute to dysfunctional emotions [ 10 , 11 ].

The complexity and time-dynamic nature of emotion make emotion dysregulation in depression particularly challenging to tackle. Most situations in daily life that evoke emotions are ambiguous. It remains unclear how patients can enhance their ER abilities in treatment [ 12 ]. Dysfunctional ER is a fundamental risk factor for the onset of depression and a range of psychiatric disorders [ 13 , 14 ]; however, the evidence base is diffuse and broad, as its mechanisms remain poorly specified [ 12 , 15 , 16 ]. Although some studies have developed psychological interventions to improve ER, research in this area remains limited [ 12 , 17 , 18 ]. Some have argued that teaching a wide range of ER strategies might not be effective in enhancing patients’ emotional functioning [ 12 , 17 ]. Of note, there is a lack of research on the use of art psychotherapy in this context.

An intervention mapping (IM) study systematically rooted in the evidence and theories of basic affective science is required to increase the likelihood of changing behaviors in ER. To target emotional dysregulation, a systematic, participatory, and integrated approach that benefits from efficient behavior change is crucial [ 19 ]. Accordingly, this study determines effective ways of enhancing patients’ ER capacities and developing an optimized art-based psychotherapy intervention for depression. For this purpose, we followed the standard IM protocol [ 20 ]. While developing a treatment may be time-consuming and burdensome, this study provides a straightforward, stepwise decision-making procedure. Along with its use of participatory action research (PAR), this study aims to benefit from the engagement of patients and mental health professionals in a collaborative manner. This type of collaboration is a practical and powerful tool for developing specialized interventions.

Intervention mapping protocol

This study mapped out the process of development based on IM, a program-planning framework. IM provides a step-by-step process for planning theory/evidence-based interventions from the needs to potential methods addressing those needs [ 20 , 21 ]. Since its development in the healthcare field in 1998, IM has been widely used and applications have emerged in other fields, including health promotion. It has been used to develop intervention programs to better target specific behaviors, including health, discrimination, and safety behaviors [ 22 ]. In particular, mental health researchers have largely applied the IM approach for either creating new interventions or adapting existing ones: strategies have been developed for the treatment and prevention of depression through IM, such as an internet-based intervention for postpartum depression [ 23 ], an online-coaching blended program for depression in middle-aged couples [ 24 ], a return-to-work intervention for depression [ 25 ], music therapy for depressive symptoms in young adults [ 26 ], and life-review therapy for depressive symptoms in nursing home residents [ 27 ]. The use of IM has proven to be a useful instrument for the development and optimization of treatments for depression that are tailored to different contexts and target populations.

Over the course of the development of the entire program, four perspectives characterizing IM are applied: (a) a participation approach that engages intended participants, allies, and implementers in program development, implementation, and evaluation; (b) a systems approach acknowledging that an intervention is an event occurring in a system that includes other factors influencing the intervention; (c) a multi-theory approach that stimulates the use of multiple theories; and (d) an ecological approach recognizing the relevance of social, physical, and political environmental conditions.

The IM protocol includes six core steps: (i) justifying the rationale for developing a new treatment; (ii) selecting targeted determinants and setting treatment goals; (iii) determining theoretical and empirical methods for behavior change; (iv) developing a treatment and program materials; (v) planning for adoption and implementation; and (vi) specifying the evaluation design [ 20 , 21 , 28 ]. The development process is cumulative: subsequent steps are based on completed tasks from the previous step. Figure  1 presents the six steps of IM. This article presents the details of our study methods and the results as the six steps of the IM process.

Overview of the intervention mapping (IM) process [ 20 ]

Steps 1–3 of IM: Literature review

To address Steps 1, 2, and 3, we conducted a literature review using PubMed, ProQuest, Scopus, PsycArticles, and Google Scholar. Search strategies were devised using subject headings such as “emotion regulation,” “depression,” “emotional psychopathology,” “emotion regulation therapy,” and “art psychotherapy” as appropriate for each database. Furthermore, the program planners identified and included additional free text words. Due to the heterogeneity of emotion-related processes, the search strategies for Steps 1–3 were broad [ 15 ]. Additionally, we conducted an inclusive literature review of relevant databases to identify articles related to art-based interventions for ER, limited to published articles in English. This literature study identified effective ER strategies for improving regulatory capacities in depression. We describe the theoretical details related to ER and ER strategies in the Results section.

Steps 4–6 of IM: Participatory action research combined

Steps 4–6 of IM occasionally incorporate further studies for pilot testing and refining the intervention under development. As such, our study added participatory components to the IM process. PAR is “a participatory and consensual approach towards investigating problems and developing plans to deal with them.” [ 29 ] PAR empowers research participants compared with other approaches, where study participants are often considered subjects who passively follow directions [ 30 ]. The involvement of patients, care providers, and health professionals in research design is increasingly recognized as an essential approach for improving the quality of primary care [ 31 ] and bridging the gap between research and health care [ 32 ]. Indeed, PAR has been applied in many fields and achieved successful results, particularly in the field of mental health [ 33 ].

In particular, patient involvement is a meaningful partnership with stakeholders, including patients, carers, and public stakeholders, who actively participate in improving healthcare practices [ 31 ]. Involvement can occur at different levels and commonly includes patient engagement and advisory boards [ 32 ]. We conducted participatory action studies to combine systematic studies with the development of practical treatments [ 33 ] and anticipated the benefits of experiential knowledge. Figure  2 elaborates on how we incorporated PAR in the IM framework. It also presents our strategies to address the IM protocol and the results from each step. As described in Fig.  2 , the PAR in this current study comprises three phases:(a) consultation with an advisory board; (b) initial testing of intervention; and (c) mixed methods feedback studies using focus group interviews and survey research.

Study procedure combined with PAR, strategies applied for each step, and results for each step

Noted. The figure specifies strategies to adopt in addressing the six steps of IM protocol and the actions for each step. It represents how IM can be applied and how it can augment its protocols through PAR. In the application of IM, this study relied on literature research and empirical studies: we conducted a literature study to address Steps 1–3 and combined the participatory action approach with IM methodology to address Steps 4–6

(a) PAR 1: Consultation with the advisory board

First, we established an advisory board that included a psychiatrist, an expert on methodology, a trained integrative medicine professional, and a professor in a graduate art psychotherapy program. The advisory board provided feedback at the individual level and comments during subsequent consultations. We engaged and managed the advisory board throughout Step 4, the intervention development process.

(b) PAR 2: Initial testing of intervention being developed 

In addition, we conducted a participatory action study to facilitate patient engagement and elicit their voices in a collaborative relationship with researchers. Based on voluntary participation, this study aimed to pretest art-based ER strategies and treatment designs. We conducted an art therapy program as part of routine inpatient therapeutic programs involving willing patients. The participants’ reports of their experiences during the sessions were obtained using structured questionnaires and unstructured interviews. For research purposes, we conducted a retrospective chart review for therapeutic sessions between February 2023 and February 2024. This review was approved by the Institutional Review Board of Kangdong Sacred Heart Hospital (IRB no. 2024–02-019) and exempted from requiring patients’ informed consent because it was part of a routine clinical practice.

(c) PAR 3: A mixed-method approach

In this study, we employed a mixed-methods approach to plan evaluation strategies by combining a quantitative online survey with focus group interviews. The primary aim of this study is to ensure that the intervention developed in Step 4 can be adopted and maintained over time. For this purpose, we are gathering feedback regarding the initial interventions from clinic staff, consisting of nurses and psychiatrists. This PAR study is currently ongoing and will last for four months. At the environmental level of the organization, the process will be managed to best leverage the intervention in primary care settings. This study was approved by the Institutional Review Board of Kangdong Sacred Heart Hospital (IRB no. 2023–12-002). PAR 2 and PAR 3 are currently being conducted; the results of those studies will be available after their completion.

This section focuses on the explanation of outputs obtained through the IM protocol. The details of the theoretical and empirical bases, designed frameworks, and strategies for the implementation and evaluation of the program are categorized into six steps:

Step 1. Needs and Logic for the Program

For the first step, we identified the target group and analyzed their determinants. This step included determining the rationale and need for a new art-based ER intervention for depression. The target population comprised patients diagnosed with major depressive disorder (MDD). Predefined behaviors targeted were core symptoms of major depression, namely, consistent depressed mood and anhedonia [ 6 ].

Theoretical evidence

Prior research has highlighted difficulties in ER contributing to the etiology and maintenance of numerous psychiatric symptoms, such as depression, chronic anxiety, post-traumatic stress disorder, eating disorders, and worry [ 15 , 34 , 35 , 36 , 37 , 38 , 39 , 40 ]. In particular, research on depression has emphasized that apparent failure to modulate emotions is a hallmark of this disorder [ 6 ] and has attempted to link it to emotional abnormalities in depression [ 10 , 11 ]. ER, which influences the onset, magnitude, and duration of emotional response [ 41 ], is a distinct and differentiated higher-order construct from emotion itself (i.e., fear, anxiety, and depression) at different levels of analysis (e.g., behavioral or neural) [ 42 , 43 ]. From this perspective, ER is an important determinant affecting lower-order factor variability, whereas emotion determines variance downwards in the lower-order indicators [ 42 ].

A literature review revealed that ER difficulties play a role in understanding psychological health in major depression. This suggests the importance of altering problematic patterns of emotional reactivity in depression and identifies emotion dysregulation as a determinant of the predefined target behaviors [ 17 , 44 , 45 , 46 , 47 ]. According to imaging studies utilizing functional magnetic resonance imaging (fMRI), functional abnormalities in specific neural systems support the processing of emotion and ER in patients with depressive disorders [ 6 ]. Moreover, decades of empirical evidence supports the notion that depressive symptoms, characterized by consistently elevated depressed mood and relatively low positive mood, are associated with difficulties in ER [ 9 , 10 , 16 ]. Our review allowed us to analyze and specify the determinants of depressive symptoms (Fig.  3 ). Without this analysis, it would be challenging for psychological treatments to address emotion dysregulation in MDD.

Summary of the determinants influencing symptoms of major depression

Needs assessment for a new intervention

Although emotion dysregulation is a critical target in psychological treatments, intervention research examining ER is limited [ 18 , 48 ]. Psychotherapeutic approaches, including cognitive-behavioral and acceptance-based behavioral treatments, have positive effects on overall ER, and studies suggest that these improvements may mediate further improvements for psychiatric outcomes [ 18 , 48 ]: examples include cognitive behavioral therapy approaches (CBT) [ 49 , 50 ], acceptance and commitment therapy (ACT) [ 51 ], dialectical behavioral therapy (DBT) [ 52 , 53 ], and acceptance-based behavioral therapy (ABBT) [ 54 ]. However, most research assessing treatment efficacy precludes making any decisions about clinical mechanisms essential for improving ER. This is because they examine the impact of non-ER-focused interventions or interventions to target ER as part of a comprehensive program [ 18 , 48 ]. Due to the multi-component nature of the interventions, the specific components contributing to changes in ER remains unclear and whether the changes underlie improvements in other distressing symptoms has not yet been clarified. Thus, efforts to identify and inform the development of interventions leading to adaptive ER based on these studies are limited.

At present, patients who have distress disorders, such as generalized anxiety disorder (GAD), MDD, and particularly GAD diagnosed along with comorbid depression, often fail to respond well or experience sufficient gains from treatments: however, the reason for their lack of response is unknown [ 17 , 55 ]. Between 50 and 80% of patients receiving interventions for emotional disorders achieve the status of “responder.” [ 17 ] Between 50 and 60% of GAD patients showed meaningful improvement in response to treatment with traditional CBT [ 55 ]. While ER-focused interventions, such as the Unified Protocol (UP) [ 56 ], Emotion Regulation Group Therapy (ERGT) [ 57 ], and enhanced CBT emphasizing ER [ 58 ] were found to be effective in improving ER, research investigating these remains limited [ 18 , 59 , 60 ]. No substantial changes were found in the essential dimensions of ER after the application of several ER-focused interventions, implying that these were not present in a sufficient dose to promote ER [ 53 , 61 , 62 ]. Further, recent research identifying treatment response predictors for ERGT showed relatively few significant predictors [ 63 ]. In particular, the findings from a study that examined a treatment designed to enhance inpatient CBT for depression suggest that the addition of ER skills to CBT may not sufficiently change ER, although improvements were noted in ER strategies and depressive symptoms [ 58 ]. Another problem arises from the manualized CBT protocols, which are distinct and complex to use [ 17 , 64 ]. These protocols make it difficult to access and use CBT.

The limitations of the current interventions suggest the need for developing an ER-specific treatment. Designing more effective and targeted interventions requires a specific understanding of affective science to provide a broad framework for ER treatments. For example, recently, it has been identified that emotions can be generated and regulated not only through a top-down process but also through a bottom-up process: [ 65 ] current models of emotion generation and its regulation are based on these two processes, which are opposed but interactive [ 66 ]. The top-down mechanism is based on a view that focuses on cognition, where either individuals’ goal states or cognitive evaluations are thought to influence the variations in their emotional responses [ 67 ]. These processes are mapped to prefrontal cortical areas. Meanwhile, bottom-up mechanisms refer to processes based on a stimuli-focused view: in this mode of processing, emotions are mostly elicited by perceptions [ 68 ]. In everyday life, emotion can be processed through interactions between the bottom-up and top-down mechanisms [ 69 ].

Most research to date, however, has focused on top-down ER strategies, and few studies have focused on bottom-up regulation procedures [ 65 ]. In particular, CBT-based treatments, which are mainstream psychotherapies, focus on instruction in an array of cognitive means of coping with emotions; CBT traditionally tends to deal more directly with cognitive rather than emotional processes. One top-down strategy is cognitive reappraisal, an active component of most CBT-based treatments [ 70 ]. However, studies suggest that relying primarily on this strategy may be less effective for certain disorders, including depression, than treatments employing a flexible approach [ 65 ]. Such an approach would be straightforward and essential for researchers as they synthesize different research results, such as findings concerning bottom-up ER and its clinical implications for the investigation of interventions.

One intervention approach to bottom-up experiential ER is art psychotherapy. This type of treatment, which targets emotion dysregulation, may hold promise for improving ER in cases of depression. Patients with depression can benefit from experiential ER that emphasizes bottom-up means of coping with their emotional experiences over the course of art-based ER intervention. This perspective is supported by behavioral and neurocognitive findings indicating difficulties in top-down regulatory processes in individuals with depression [ 71 , 72 , 73 , 74 ]. Research examining neural activities between individuals with and without depression indicated different patterns between them: when downregulating negative emotions, individuals with depression show bilateral prefrontal cortex (PFC) activation, whereas individuals without depression show left-lateralized activation [ 74 ]. When given an effortful reappraisal task, moreover, the relationship patterns of individuals with depression between activation in the left ventrolateral PFC and the amygdala are different from those of individuals without depression. These findings indicate that the pathophysiology of depression underlies struggles of downregulation [ 74 ].

Thus, it is vital to design a new intervention for depression that focuses not only on top-down ER but also on bottom-up ER. In particular, this study examines art-based ER in the form of a client-centered and experiential psychotherapeutic approach allowing patients to attempt top-down and bottom-up regulation. While pursuing active engagement in art-based ER practices, patients can process their emotional experiences in a way that produces greater fine-tuning and depth. Art-based treatment is open and non-interventional as well as less demanding cognitively, enabling it to reach a diverse population with depressive symptoms. More promisingly, art-based ER primarily deals with visible and tangible works leading to visual representations. Emotional memory is perceptual [ 75 ], implying that art-based practices can influence its retrieval and manipulative process: the artworks that patients make in treatments are visual representations that are identical or similar to their emotional experiences. Importantly, creation involves colors, images, and spaces acting as new stimuli, allowing patients to manipulate and generate new emotions through a bottom-up process. As processes of emotion generation interact with those of ER [ 67 ], an art-based experiential approach can facilitate adaptive ER, potentially benefiting individuals who have emotional dysfunction.

However, few studies have explored ER in depression within the field of art psychotherapy [ 76 ]. The therapeutic strategies applied in relevant studies [ 77 , 78 , 79 ] are not explicitly identified or targeted with respect to the mechanisms of ER. For instance, earlier literature tested the effects of art therapy on ER in psychiatric disorders; most of these approaches focused on improving psychopathological symptoms related to specific disorders and considered ER to be a secondary therapeutic outcome. Thus, we identified a need to develop an effective art-based intervention specifically targeting emotion dysregulation in major depression.

Step 2: Formulation of change objectives

The second step required the specification of intervention goals, which involved moving from understanding what influences depressive symptoms, especially in terms of emotional abnormalities in depression, to clarifying what needs to be changed. Based on the needs assessment, the overall expected outcome was “a decrease in depressive symptoms and an improvement in ER.” In this process, the analysis of the determinants in Step 1 resulted in selecting key determinants to target, which were provided by a comprehensive review of the empirical literature and research evidence. It is difficult to understand generative and regulatory emotion processes that are enacted internally without the instigation of extrinsic stimuli [ 80 ]. Thus, it can be challenging to identify the right determinants to target and design an effective treatment that addresses problems related to ER. Based on our review, we determined and chose four important and changeable determinants and further divided them into five key determinants (see Table  1 ).

To apply IM, the construction of matrices of change consisting of performance and change objectives forms the basis for program development [ 20 , 81 ]. Overall, the program objectives were subdivided into performance objectives expected to be accomplished by the target group in the proposed intervention. While drawing on the key determinants and performance objectives, more general objectives, namely, change objectives, were formulated. The result of Step 2 is this change matrix, which further forms the basic factors for designing the intervention for major depression.

Step 3: Theory- and evidence-based strategies selection

In IM, Step 3 entails selecting theoretically grounded and evidence-based methods and strategies. For this process, we first conducted a comprehensive review of theories and empirical studies for therapeutic strategies, including the following characteristics: (i) they need to be confirmed as an efficient ER strategy based on empirical research evidence; (ii) they need to be effective not only in decreasing depressive symptoms but also in improving ER capacities of patients; and (iii) they can be translated into art-based practices. In iterations of reviewing theories related to and research evidence with regard to emotion regulatory strategies, we identified appropriate, theoretically sound therapeutic strategies for at least one program target.

Once an ER method was selected, we translated this method into art-based emotion regulation (ABER) strategies for practical applications. Practical applications refer to the practical translation of the chosen behavior change methods [ 19 , 20 , 21 , 81 ]. The end product of Step 3 is an initial set of theory- and evidence-based strategies selected and translated to address emotion dysregulation in major depression. Table 2  lists the strategies with supporting evidence and applications: art-based distraction, art-based positive rumination, art-based self-distancing (SD), and art-based acceptance. Based on an integrative view of emotional processing, which posits interactions between top-down and bottom-up systems [ 67 , 69 , 82 , 83 ], these strategies aim to modulate emotions through the use of top-down and bottom-up mechanisms.

In particular, as art-based ER involves visual-spatial processing that could exert influence as new sets of stimuli, this approach could lead to a more experiential bottom-up ER. For instance, distraction and cognitive defusion are usually considered cognitive forms of ER; however, both are translated and applied to art-based strategies. Individuals’ performance in art-based ER would differ from that on a given cognitive task, as their immersion experiences in the artistic and creative process involve the generation of colors, images, and spatial features, which may elicit new bottom-up processing. This may be associated with the superior ER effects of art-based distraction, as shown in some studies that compared the ER effects of artistic activities with those of non-artistic activities, such as completing verbal puzzles [ 98 , 99 , 100 ].

In addition, art-based SD promotes intuitive and experiential ER. Individuals are trained to adopt a self-distanced perspective in some treatments while reflecting on their emotions, such as mindfulness-based stress reduction (MBSR) and ERT. They meditate to take a decentered stance. Art-based SD may help those who have difficulty creating an internal distance. As individuals create visual forms of their inner feelings and thoughts, a spatially generated distance from the artworks representing their experiences allows them to adopt and maintain a more self-distanced perspective. As such, art-based SD is more intuitive but requires less mental energy. Importantly, this art-based experiential distancing may reconstrue individuals’ appraisals by facilitating a bottom-up mechanism.

Step 4: Program development

Step 4 concerns creating an actual program plan, leading to the ABER intervention model proposed in the current study. The intervention's elementary components, organization, and structure were created based on the findings of the preparation steps (Steps 1–3). Once the list of therapeutic strategies and their practical applications was generated, we designed a structured intervention framework that would be feasible and realistic to deliver in primary care settings.

The intervention framework developed in Step 4 is based on the process model of ER [ 7 ], supported by considerable empirical research [ 101 , 102 , 103 ]. Based on the extended model, a series of steps involved in the process of regulation with different ER strategies are considered while designing the conceptual framework. Accordingly, the primary areas of the intervention involve emotion perception, attention, and cognition. We developed specific art-based ER strategies, focusing primarily on antecedent- rather than response-focused regulation. Further, this intervention is meant to complement the process model in a framework that is designed to apply one or more strategies in a single session: this would be ideal for improving ER in real life, as current research on ER has found that people generally try multiple strategies simultaneously [ 104 ], whereas the process model examines a within-situation context, within which a single ER strategy is utilized [ 12 ]. In addition, we find that this treatment will be effective in improving ER as it attempts both top-down and bottom-up ER: actively engaging in artworks through the use of the body, a patient can apply experiential self-focus [ 64 ]. In treatment with art-making, patients can be provided with sufficient time and space to find personal meaning in their experiences and process emotions, which enables them to achieve change.

Table 3 presents an overview of the proposed intervention frameworks. As shown, we designed two frameworks to guide the intervention: an individual framework for short-term intervention and an integrative framework for long-term intervention. Each style of the ABER model draws on a different implementation design to build the framework, and each model has slightly different aims. In Step 4, the advisory board reviewed the draft frameworks, including the determinants, performance and change objectives, and therapeutic strategies. The advisory board acted as a support group throughout the review process, helping tailor the program to the target population. In response to the board’s reviews, supplementary resources were added.

Individual framework

First, a plan for an individual framework was devised that accounted for the scope and phase of a short-term intervention. As shown in Fig.  4 , this framework focuses on producing initial or short-term behavioral changes pertaining to achieving short-term clinical efficacy. That is, the individual model does not aim only at emotional changes in patients, such as increases or decreases in specific emotions. The therapeutic aim is not set in an emotion-specific manner, but in terms of effectiveness, it relates to the use of regulatory strategies [ 105 ]. Accordingly, an expected outcome is to increase the quantity and frequency of adaptive ER strategies. Patients are trained in rudimentary ER skills, including one or several combination ABER strategies, as designed in the previous step. These practices aim to enhance attentional, followed by cognitive control. The expected duration of individual sessions is around 1–1.5 h.

Individual intervention model diagram. Noted. The panel shows the individual intervention model in an inpatient setting as an example: each patient (patient i ) has a different time of admission (t 0 ) and inpatient discharge (t d ). Thus, the number of participating patients can differ per session. During the hospital stay, patients are trained in rudimentary emotion regulation (ER) skills, including one or a combination of several art-based ER strategies (aber i ). The application of the therapeutic strategies is flexible: it depends on the patient’s cognitive functions, depressive symptoms, and severity of the symptoms. The time of inpatient discharge (t d ) affects each patient’s treatment duration

Integrative framework

While an individual framework comprises a single phase, an integrative framework includes stepwise sequential phases. In addition to skill development in the individual treatment, three phases of the integrative model are designed to foster adaptive motivational responses and cognitive-behavioral flexibility, which enables patients to achieve greater emotional clarity [ 106 ]. In the integrative treatment, all three phases are performed for 6–12 weeks.

The first phase of the integrative model begins with psychoeducation, in which the patient is taught the concept of ER and the importance of identifying his or her habitual reactions, such as in terms of rumination and dampening [ 91 ], that have characterized his or her life. This therapeutic process is important because ER is an automatic process requiring the consideration of motivation [ 107 ]. Psychoeducation regarding ER and monitoring patients’ responses to emotional experiences precede the skill development procedure. For instance, for patients’ self-monitoring, retrospective self-report questionnaires can capture data on ER skill use. While these methodologies are easy to use and cost-efficient [ 108 ], they are demanding tools for use in capturing natural fluctuating patterns in ER [ 109 ]. As an alternative, ecological momentary assessment can be used in treatment to capture situational context and adaptiveness of the skill use [ 108 ]. In addition to patients’ self-monitoring, a psychotherapist should monitor their emotional responses during and between therapy sessions: psychotherapists function as human raters. Because self-monitoring may not be feasible for all patients, assessing the typical patterns with which patients use maladaptive emotion regulatory strategies is important. Specifically, therapists need to assess a patient’s ER repertoire: the quantity of ER strategies, the frequency of strategy use, and how the patient’s strategy use changes.

The second phase entails adopting and implementing ER strategies with processes resembling those of the individual model. These processes entail the selection and repetition of adaptive strategies. They differ from the individual model in that the duration of Phase II can vary from one patient to another depending on the severity of depressive symptoms and the frequency of maladaptive strategies used. The ER practices delivered in Phase II are art-based tasks through which therapists and patients explore and try adaptive strategies. As shown in Fig.  5 , the intervention program includes four ABER strategies selected and translated in Step 3: art-based distraction, art-based SD, art-based positive rumination, and art-based acceptance. The patients work with therapists in 4–8 1.5-h sessions to engage in art-based practices.

Summary steps and components for the integrative intervention model

Finally, the integrative framework includes a third phase for evaluation. While the previous sessions in Phase II focus on skill development, the sessions in Phase III focus on assessing changes in patients. All individual progress in ER is tracked and monitored. In this task, therapists help patients assess changes in their emotion-regulatory skill use and their achievements in terms of self-perception, effectiveness, and adaptiveness. Patients are given opportunities to take a broad view of the changes in their artworks during all treatment phases. Furthermore, patients receive a few tasks as homework to briefly review their strategy use in daily life from the beginning of the treatment until the current moment. The review process helps them assess their progress and supports their strengthening. It takes 6–12 weeks to complete the integrative treatment course, depending on the clinical impression. For instance, the duration of Phase II is expected to take 4–8 weeks, according to the clinical impression. A therapist or clinician renders his or her impression regarding the degree of the patients’ severity of depressive symptoms, use of maladaptive ER strategies, willingness to participate in the intervention, and insight into their treatment.

Step 5: Adoption and implementation

Implementation is an essential aspect of program development. In Step 5 of IM, the focus is on planning the adoption and implementation of the proposed intervention. This process is required at the environmental level [ 21 ] and ensures successful adoption and sustainable use in collaborating organizations. Thus, pilot tests can be conducted to gain practical insights into implementation decisions and refine the intervention. Using a PAR framework, we pilot-tested the individual model to ensure that the intervention is appropriate and helpful for patients. This PAR pilot study was performed to inform future practices while connecting intervention research with actual action in a primary care setting.

The advisory group’s results, which indicated that the intervention needed to be sufficiently pliable to be used in a variety of primary care settings, informed and supported the step for pretesting. Implementation was prepared in a primary care setting, in which the program was pretested with a steering group of psychiatrists, nurses, and an art psychotherapist. Two clinicians were in charge of informing the intervention program and facilitating patient involvement. The therapist, who had received appropriate training and instruction, was responsible for delivering the intervention and supporting all practical aspects of patient engagement. With support from the therapist, the patients were in charge of applying one or a combination of two strategies in therapeutic sessions.

We performed this initial testing in a psychiatric ward in Seoul. Between February 2023 and February 2024, during the first two phases of the pilot testing, approximately 24 sessions were conducted, and 45 inpatients, including 16 patients with depressive disorders, voluntarily participated in the program. At the end of each session, the participants were asked to report their experiences through free narratives and complete a short questionnaire survey (quantitative and free-text comments) that provided additional information regarding their involvement. The mean time expenditure for the patients was 1.1 h (SD: 18.0; range: 0.5–2). Patients’ emotional experiences were reflected in their artworks, and Fig.  6 shows a short overview of their art products. The detailed findings from these pilot trials are outside the scope of the IM protocol and will be available in a future publication.

Examples of the art products of the participating patients with depression. Noted. Figure 6 briefly outlines patient engagement through their artworks made during the treatment sessions in the first pilot phase: a shows an artwork a patient made in a treatment session, which applied art-based acceptance; b shows an artwork showing a patient’s reflection on his experience, which applied art-based self-distancing and acceptance; c and d show artworks in which patients apply art-based positive rumination and distraction. Different art materials were provided in each session depending on the ER strategies used. The art-based practices of ER promoted relaxation and expression of the patient’s inner feelings and thoughts

Step 6. Evaluation plan

The sixth step of IM is the planning of evaluation strategies to assess the potential impacts of the proposed intervention [ 20 ]. For this purpose, we designed two phases based on a PAR framework: patient feedback and expert feedback. The rationale for this plan was that comprehensive evaluations could investigate the necessity of refinement and what is needed to produce a more feasible and effective intervention. In particular, we expected that the engagement of patients as well as health professionals in the evaluation process would integrate the organizational perspective into patient-oriented quality improvements. From these two phases, we developed questions and measures for evaluation, conducting preliminary PAR studies to determine the feasibility and efficacy of the complete program. Table 4 presents the evaluation strategies for gaining patient and expert feedback. Meanwhile, Table  5 presents an overview and timeline of PAR 2 and PAR 3.

First, we developed a set of patient-reported outcome measures to obtain patient feedback. Quantitative assessments of treatment satisfaction, perceived helpfulness of treatment, and perceived difficulty were conducted following the end of a therapeutic session. Patient evaluations must be carried out regularly during treatment to assess the efficacy of the integrative model. At the end of the program, unstructured or semi-structured interviews are recommended to explore patients’ experiences of the treatment process. In addition, we planned a two-phase mixed-methods study to obtain feedback from participating healthcare professionals using an online survey and focus group interviews. The assessments included process measures, such as perceived difficulty, program appropriateness, and recommendations for improvements to its implementation on a professional level. A web-based survey was disseminated among clinicians and nurses to assess the feasibility of the intervention. Together, this enabled us to increase the time efficiency and cost-effectiveness of the evaluation process.

Feasibility was assessed in five ways. First, the feasibility with which patients participated in the program was described. In our preliminary study, for instance, we calculated the percentage of patients approached for program participation relative to those who did not. Second, the feasibility of retaining patients in a treatment session was reported. To capture the feasibility of retention in treatment, we calculated the percentage of patients who failed to complete treatment compared with the percentage of those who completed it. Third, the feasibility of administering treatment was measured with a self-reported survey of patients’ perceived difficulty in participation and a survey of healthcare professionals’ perceived difficulty in implementation. To report the feasibility of administering treatment, we calculated the mean hours a patient spent in completing treatment. In addition to feasibility, acceptability was operationalized in three ways: a quantitative self-report survey of patient satisfaction, patient perceptions of helpfulness of treatment, and patient willingness to recommend program participation were used. In our preliminary study, we developed responses for the patient survey and calculated the means and standard deviations for each item.

We received patient feedback in the first two pilot phases (PAR 2), and the results showed that the intervention program was feasible and acceptable for implementation in the primary care setting (the mean scores were as follows: Treatment satisfaction = 4.82, Perceived helpfulness of treatment = 4.57, Perceived difficulty = 4.45). The patients provided further recommendations for improved intervention in free-text comments. In addition to this patient feedback, we began conducting PAR3 in February 2024. The feedback research is being conducted through an online questionnaire that includes multiple-choice questions and open-ended questions, with focus group interviews being conducted virtually through Zoom. The results for PAR 2 and PAR 3 will be reported in separate articles.

In this paper, we proposed conceptual frameworks for an intervention that targets emotion dysregulation in depression. IM was used as the conceptual protocol to develop the intervention. To the best of our knowledge, this is the first art-based ER intervention incorporating previous theories, research evidence, and review data in relation to affective science and intervention research, combining PAR components with IM. We developed the intervention following the rationale and stepwise process of IM, which identifies theory- and evidence-based strategies to address key barriers to ER. In addition, to evaluate the developed intervention, preliminary PAR studies were conducted, including the acceptability of the trials and the ABER intervention to patients; the rate of recruitment, attendance, and attrition; perceived difficulties in intervention implementation; and psychological outcomes. Consequently, the intervention is theoretically underpinned and supported by empirical evidence regarding ER and the results of our pilot studies.

The current study benefits from integrating the PAR approach into the IM framework in two ways. First, using PAR studies in the IM resulted in the cogeneration of knowledge among academic researchers, implementers, and the intended participants. PAR ensured experiential knowledge to deliver content that addressed difficulties in ER in collaborative partnerships. Another contribution was enhancing the feasibility and acceptability of the proposed intervention. In particular, preliminary PAR studies helped investigate whether modifications were needed before the intervention’s adoption. Even though IM is a time-consuming process, the use of PAR made it more cost-effective and time-efficient.

In addition to these strengths, it is crucial to acknowledge and affirm the study’s limitations. First, the current study offers only preliminary evidence for the given conceptual framework. Although the proposed intervention may precisely target emotional dysfunction in depression, such as in the restrictive use of adaptive ER skills with repetitive use of maladaptive strategies, the integrative and individual frameworks of ABER have not been evaluated through randomized clinical trials. As the current study pilot-tested the intervention in an inpatient setting that served an acute, transdiagnostic population, implementers could extend the use of these frameworks by performing a fine-grained analysis of treatment contexts (e.g., by adapting the model for depressed outpatients in primary care). As such, the intervention must be examined and refined on the basis of the results of empirical studies on multidisciplinary design. In addition, this article did not examine the therapists’ capability of delivering treatment, fidelity of implementation, and feasibility of measuring tools. Intervention researchers interested in these variables are encouraged to extend our models by testing the broad contextual variables that influence its process. Similarly, further research is required to investigate standardized forms of assessment in treatment (e.g., a measurable rating scale for patient monitoring) to increase the efficiency of the intervention.

Conclusions

This article proposes empirical and theoretical intervention frameworks that can improve ER in depression. This IM study is unique, as the development process incorporates PAR components. Moreover, the intervention consists of four art-based regulatory strategies that enrich the present literature on intervention research targeting dysfunctional ER in major depression. Our participatory action studies demonstrate that, in a primary care setting, the individual protocol is feasible and acceptable for implementation. This result represents a potential step forward toward filling a gap in current mental health treatments for patients with MDD. Despite the tiresome and time-consuming process of intervention development, the application of IM augmented by PAR is helpful in optimizing chances for an effective behavior change. Further testing is required to assess the impact of the therapeutic program proposed in this study.

Availability of data and materials

The author confirms that the data generated or analysed during this study are included in this published article: however, raw datasets are not publicly available due to local legal restrictions. Since the data being generated by PAR2 and PAR3 are outside the scope of the current intervention mapping study, they are available elsewhere.

Abbreviations

Art-based emotion regulation

Cognitive-behavioral therapy

• Emotion regulation

Emotion Regulation Group Therapy

Generalized anxiety disorder

• Intervention mapping

Mindfulness-based stress reduction

Major depressive disorder

• Participatory action research

The Self-Assessment Manikin

Self-distancing

World Health Organization

Collaborators GBDMD. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of Disease Study 2019. Lancet Psychiatry. 2022;9(2):137–50.

Article   Google Scholar  

World Health Organization. Depressive disorder(depression) . http://www.who.int/news-room/fact-sheets/detail/depression . Accessed 4th March 2024.

Malhi GS, Mann JJ. Depression Lancet. 2018;392(10161):2299–312.

Article   PubMed   Google Scholar  

Rottenberg J. Emotions in depression: What do we really know? Annu Rev Clin Psychol. 2017May;8(13):241–63.

Thompson RJ, Boden MT, Gotlib IH. Emotional variability and clarity in depression and social anxiety. Cogn Emot. 2017Jan 2;31(1):98–108.

American Psychiatric Association. Diagnostic and statistical manual of mental disorders (5th ed.). 2013. https://doi.org/10.1176/appi.books.9780890425596 .

Gross JJ. The emerging field of emotion regulation: An integrative review. Rev Gen Psychol. 1998Sep;2(3):271–99.

Gross JJ. Emotion regulation: Conceptual and empirical foundations. Handbook of emotion regulation. 2014;2:3–20.

Google Scholar  

Khazanov GK, Ruscio AM. Is low positive emotionality a specific risk factor for depression? A meta-analysis of longitudinal studies. Psychol Bull. 2016Sep;142(9):991.

Article   PubMed   PubMed Central   Google Scholar  

Vanderlind WM, Everaert J, Joormann J. Positive emotion in daily life: Emotion regulation and depression. Emotion. 2022Oct;22(7):1614.

Liu DY, Thompson RJ. Selection and implementation of emotion regulation strategies in major depressive disorder: An integrative review. Clin Psychol Rev. 2017Nov;1(57):183–94.

Southward MW, Sauer-Zavala S, Cheavens JS. Specifying the mechanisms and targets of emotion regulation: A translational framework from affective science to psychological treatment. Clin Psychol Sci Pract. 2021Jun;28(2):168.

Berenbaum H, Raghavan C, Le HN, Vernon LL, Gomez JJ. A taxonomy of emotional disturbances. Clin Psychol Sci Pract. 2003;10(2):206.

Jazaieri H, Urry HL, Gross JJ. Affective disturbance and psychopathology: An emotion regulation perspective. J Exp Psychopathol. 2013Dec;4(5):584–99.

Gross JJ, Jazaieri H. Emotion, emotion regulation, and psychopathology: An affective science perspective. Clinical psychological science. 2014Jul;2(4):387–401.

Joormann J, Stanton CH. Examining emotion regulation in depression: A review and future directions. Behav Res Ther. 2016Nov;1(86):35–49.

Barlow DH, Allen LB, Choate ML. Toward a unified treatment for emotional disorders. In The Neurotic Paradox, Volume 1. New York: Routledge; 2020. p. 141–166.

Gratz KL, Weiss NH, Tull MT. Examining emotion regulation as an outcome, mechanism, or target of psychological treatments. Curr Opin Psychol. 2015Jun;1(3):85–90.

Stutterheim SE, van der Kooij YL, Crutzen R, Ruiter RAC, Bos AER, Kok G. (2023). Intervention mapping as a guide to developing, implementing, and evaluating stigma reduction interventions. Stigma and Health. Advance online publication. https://doi.org/10.1037/sah0000445 .

Eldredge LK, Markham CM, Ruiter RA, Fernández ME, Kok G, Parcel GS. Planning health promotion programs: an intervention mapping approach. New York: Wiley; 2016.

Kok G, Bartholomew LK, Parcel GS, Gottlieb NH, Fernández ME. Finding theory-and evidence-based alternatives to fear appeals: Intervention Mapping. Int J Psychol. 2014Apr;49(2):98–107.

Kok G, Peters LW, Ruiter RA. Planning theory-and evidence-based behavior change interventions: a conceptual review of the intervention mapping protocol. Psicologia: Reflexão e Crítica. 2018;30.  https://www.scielo.br/j/prc/a/TkCBLGGQb7JRQRBbrpnjxgf/?lang=en&format=html .

Drozd F, Haga SM, Brendryen H, Slinning K. An internet-based intervention (Mamma Mia) for postpartum depression: mapping the development from theory to practice. JMIR research protocols. 2015;4(4): e4858.

Kim SS, Gil M, Kim D. Development of an online-coaching blended couple-oriented intervention for preventing depression in middle adulthood: An intervention mapping study. Front Public Health. 2022;10: 882576.

Wisenthal A, Krupa T. Using intervention mapping to deconstruct cognitive work hardening: a return-to-work intervention for people with depression. BMC Health Serv Res. 2014;14:1–11.

Aalbers S, Vink A, Freeman RE, Pattiselanno K, Spreen M, van Hooren S. Development of an improvisational music therapy intervention for young adults with depressive symptoms: An intervention mapping study. Arts Psychother. 2019;65: 101584.

van Venrooij I, Spijker J, Westerhof GJ, Leontjevas R, Gerritsen DL. Applying intervention mapping to improve the applicability of precious memories, an intervention for depressive symptoms in nursing home residents. Int J Environ Res Public Health. 2019;16(24):5163.

Roozen S, Stutterheim SE, Bos AE, Kok G, Curfs LM. Understanding the social stigma of fetal alcohol spectrum disorders: from theory to interventions. Found Sci. 2020May;29:1–9.

Bowling A. Research methods in health: investigating health and health services. UK: McGraw-hill education; 2014.

White MA, Verhoef MJ. Toward a patient-centered approach: incorporating principles of participatory action research into clinical studies. Integr Cancer Ther. 2005Mar;4(1):21–4.

Haesebaert J, Samson I, Lee-Gosselin H, Guay-Bélanger S, Proteau JF, Drouin G, Guimont C, Vigneault L, Poirier A, Sanon PN, Roch G. “They heard our voice!” patient engagement councils in community-based primary care practices: a participatory action research pilot study. Research involvement and engagement. 2020Dec;6:1–4.

Koskinas E, Gilfoyle M, Salsberg J. Exploring how patients, carers and members of the public are recruited to advisory boards, groups and panels as partners in public and patient involved health research: a scoping review protocol. BMJ Open. 2022Apr 1;12(4): e059048.

Dold CJ, Chapman RA. Hearing a voice: Results of a participatory action research study. J Child Fam Stud. 2012Jun;21:512–9.

Aldao A, Gee DG, De Los Reyes A, Seager I. Emotion regulation as a transdiagnostic factor in the development of internalizing and externalizing psychopathology: Current and future directions. Dev Psychopathol. 2016;28(4pt1):927–46.

Garnefski N, Kraaij V. Relationships between cognitive emotion regulation strategies and depressive symptoms: A comparative study of five specific samples. Personality Individ Differ. 2006Jun 1;40(8):1659–69.

Gross JJ, Muñoz RF. Emotion regulation and mental health. Clin Psychol Sci Pract. 1995;2(2):151.

Mennin DS, Holaway RM, Fresco DM, Moore MT, Heimberg RG. Delineating components of emotion and its dysregulation in anxiety and mood psychopathology. Behav Ther. 2007Sep 1;38(3):284–302.

Berking M, Wupperman P. Emotion regulation and mental health: recent findings, current challenges, and future directions. Curr Opin Psychiatry. 2012Mar 1;25(2):128–34.

Cloitre M, Khan C, Mackintosh MA, Garvert DW, Henn-Haase CM, Falvey EC, Saito J. Emotion regulation mediates the relationship between ACES and physical and mental health. Psychol Trauma Theory Res Pract Policy. 2019Jan;11(1):82.

Putnam KM, Silk KR. Emotion dysregulation and the development of borderline personality disorder. Dev Psychopathol. 2005Dec;17(4):899–925.

Gross JJ, editor. Handbook of emotion regulation. Hoboken: Guilford publications; 2013.

Cisler JM, Olatunji BO, Feldner MT, Forsyth JP. Emotion regulation and the anxiety disorders: An integrative review. J Psychopathol Behav Assess. 2010Mar;32:68–82.

Zinbarg RE, Barlow DH. Structure of anxiety and the anxiety disorders: a hierarchical model. J Abnorm Psychol. 1996May;105(2):181.

Article   CAS   PubMed   Google Scholar  

Campbell-Sills L, Barlow DH, Brown TA, Hofmann SG. Acceptability and suppression of negative emotion in anxiety and mood disorders. Emotion. 2006Nov;6(4):587.

Campbell-Sills L, Barlow DH, Brown TA, Hofmann SG. Effects of suppression and acceptance on emotional responses of individuals with anxiety and mood disorders. Behav Res Ther. 2006Sep 1;44(9):1251–63.

Kashdan TB, Steger MF. Expanding the topography of social anxiety: An experience-sampling assessment of positive emotions, positive events, and emotion suppression. Psychol Sci. 2006Feb;17(2):120–8.

Kashdan TB, Barrios V, Forsyth JP, Steger MF. Experiential avoidance as a generalized psychological vulnerability: Comparisons with coping and emotion regulation strategies. Behav Res Ther. 2006Sep 1;44(9):1301–20.

Moore R, Gillanders D, Stuart S. The impact of group emotion regulation interventions on emotion regulation ability: A systematic review. J Clin Med. 2022Apr 29;11(9):2519.

Forkmann T, Scherer A, Pawelzik M, Mainz V, Drueke B, Boecker M, Gauggel S. Does cognitive behavior therapy alter emotion regulation in inpatients with a depressive disorder? Psychol Res Behav Manag. 2014May;12:147–53.

Papa A, Boland M, Sewell MT. Emotion regulation and CBT. In: O'Donohue WT, Fisher JE, editors. Cognitive behavior therapy: Core principles for practice. Wiley. 2012. p. 273–323. https://doi.org/10.1002/9781118470886.ch11 .

Hayes SC. Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and cognitive therapies. Behav Ther. 2004Sep 1;35(4):639–65.

Linehan M. Cognitive-behavioral treatment of borderline personality disorder. Hoboken: Guilford press; 1993.

Neacsiu AD, Eberle JW, Kramer R, Wiesmann T, Linehan MM. Dialectical behavior therapy skills for transdiagnostic emotion dysregulation: A pilot randomized controlled trial. Behav Res Ther. 2014Aug;1(59):40–51.

Roemer L, Orsillo SM, Salters-Pedneault K. Efficacy of an acceptance-based behavior therapy for generalized anxiety disorder: evaluation in a randomized controlled trial. J Consult Clin Psychol. 2008Dec;76(6):1083.

Renna ME, Quintero JM, Fresco DM, Mennin DS. Emotion regulation therapy: a mechanism-targeted treatment for disorders of distress. Front Psychol. 2017Feb;6(8): 211118.

Bullis JR, Sauer-Zavala S, Bentley KH, Thompson-Hollands J, Carl JR, Barlow DH. The unified protocol for transdiagnostic treatment of emotional disorders: Preliminary exploration of effectiveness for group delivery. Behav Modif. 2015Mar;39(2):295–321.

Gratz KL, Gunderson JG. Preliminary data on an acceptance-based emotion regulation group intervention for deliberate self-harm among women with borderline personality disorder. Behav Ther. 2006Mar 1;37(1):25–35.

Berking M, Wupperman P, Reichardt A, Pejic T, Dippel A, Znoj H. Emotion-regulation skills as a treatment target in psychotherapy. Behav Res Ther. 2008Nov 1;46(11):1230–7.

Berking M, Whitley B. Emotion Regulation: Definition and Relevance for Mental Health. In: Affect Regulation Training. New York: Springer; 2014. https://doi.org/10.1007/978-1-4939-1022-9_2 .

Hall K, Simpson A, O’donnell R, Sloan E, Staiger PK, Morton J, Ryan D, Nunn B, Best D, Lubman DI. Emotional dysregulation as a target in the treatment of co‐existing substance use and borderline personality disorders: A pilot study. Clin Psychol. 2018;22(2):112–25.

Safer DL, Jo B. Outcome from a randomized controlled trial of group therapy for binge eating disorder: Comparing dialectical behavior therapy adapted for binge eating to an active comparison group therapy. Behav Ther. 2010;41(1):106–20.

Keuthen NJ, Rothbaum BO, Fama J, Altenburger E, Falkenstein MJ, Sprich SE, Kearns M, Meunier S, Jenike MA, Welch SS. DBT-enhanced cognitive-behavioral treatment for trichotillomania: A randomized controlled trial. J Behav Addict. 2012;1:106–14.

Gratz KL, Dixon-Gordon KL, Tull MT. Predictors of treatment response to an adjunctive emotion regulation group therapy for deliberate self-harm among women with borderline personality disorder. Personal Disord Theory Res Treat. 2014;5:97–107.

Barlow DH, Levitt JT, Bufka LF. The dissemination of empirically supported treatments: a view to the future. Behav Res Ther. 1999Jul;1(37):S147–62.

Wang Y, Vlemincx E, Vantieghem I, Dhar M, Dong D, Vandekerckhove M. Bottom-up and cognitive top-down emotion regulation: Experiential emotion regulation and cognitive reappraisal on stress relief and follow-up sleep physiology. Int J Environ Res Public Health. 2022Jun 22;19(13):7621.

Viviani R. Emotion regulation, attention to emotion, and the ventral attentional network. Front Hum Neurosci. 2013Nov;7(7):746.

PubMed   PubMed Central   Google Scholar  

McRae K, Misra S, Prasad AK, Pereira SC, Gross JJ. Bottom-up and top-down emotion generation: implications for emotion regulation. Social cognitive and affective neuroscience. 2012Mar 1;7(3):253–62.

Seligman ME. Phobias and preparedness. Behav Ther. 1971Jul 1;2(3):307–20.

Ochsner KN, Gross JJ. The neural architecture of emotion regulation. Handbook of emotion regulation. 2007;1(1):87–109.

Smits JA, Julian K, Rosenfield D, Powers MB. Threat reappraisal as a mediator of symptom change in cognitive-behavioral treatment of anxiety disorders: a systematic review. J Consult Clin Psychol. 2012Aug;80(4):624.

Bishop SJ. Trait anxiety and impoverished prefrontal control of attention. Nat Neurosci. 2009Jan;12(1):92–8.

Blair KS, Geraci M, Smith BW, Hollon N, DeVido J, Otero M, Blair JR, Pine DS. Reduced dorsal anterior cingulate cortical activity during emotional regulation and top-down attentional control in generalized social phobia, generalized anxiety disorder, and comorbid generalized social phobia/generalized anxiety disorder. Biol Psychiat. 2012Sep 15;72(6):476–82.

Lyubomirsky S, Kasri F, Zehm K. Dysphoric rumination impairs concentration on academic tasks. Cogn Ther Res. 2003Jun;27:309–30.

Johnstone T, Van Reekum CM, Urry HL, Kalin NH, Davidson RJ. Failure to regulate: counterproductive recruitment of top-down prefrontal-subcortical circuitry in major depression. J Neurosci. 2007Aug 15;27(33):8877–84.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Arntz A, De Groot C, Kindt M. Emotional memory is perceptual. J Behav Ther Exp Psychiatry. 2005Mar 1;36(1):19–34.

Gruber H, Oepen R. Emotion regulation strategies and effects in art-making: A narrative synthesis. Arts Psychother. 2018Jul;1(59):65–74.

Abbing A, Baars EW, De Sonneville L. The effectiveness of art therapy for anxiety in adult women: a randomized controlled trial. Front Psychol. 2019May;29(10): 436010.

Haeyen S, van Hooren S, Dehue F, Hutschemaekers G. Development of an art-therapy intervention for patients with personality disorders: an intervention mapping study. International Journal of Art Therapy. 2018Jul 3;23(3):125–35.

Nan JK, Ho RT. Effects of clay art therapy on adults outpatients with major depressive disorder: A randomized controlled trial. J Affect Disord. 2017Aug;1(217):237–45.

Mennin DS, Fresco DM. Advancing emotion regulation perspectives on psychopathology: The challenge of distress disorders. Psychol Inq. 2015Jan 2;26(1):80–92.

Peters GJ. A practical guide to effective behavior change: how to identify what to change in the first place. European Health Psychologist. 2014Oct 1;16(5):142–55.

Ochsner KN, Bunge SA, Gross JJ, Gabrieli JD. Rethinking feelings: an FMRI study of the cognitive regulation of emotion. J Cogn Neurosci. 2002Nov 15;14(8):1215–29.

Ochsner KN, Gross JJ. The cognitive control of emotion. Trends Cogn Sci. 2005May 1;9(5):242–9.

Thiruchselvam R, Blechert J, Sheppes G, Rydstrom A, Gross JJ. The temporal dynamics of emotion regulation: An EEG study of distraction and reappraisal. Biol Psychol. 2011Apr 1;87(1):84–92.

Forkosh J, Drake JE. Coloring versus drawing: Effects of cognitive demand on mood repair, flow, and enjoyment. Art Ther. 2017Apr 3;34(2):75–82.

Gerin W, Davidson KW, Christenfeld NJ, Goyal T, Schwartz JE. The role of angry rumination and distraction in blood pressure recovery from emotional arousal. Psychosom Med. 2006Jan 1;68(1):64–72.

Urry HL, Gross JJ. Emotion regulation in older age. Curr Dir Psychol Sci. 2010Dec;19(6):352–7.

Fresco DM, Moore MT, van Dulmen MH, Segal ZV, Ma SH, Teasdale JD, Williams JM. Initial psychometric properties of the experiences questionnaire: validation of a self-report measure of decentering. Behav Ther. 2007Sep 1;38(3):234–46.

Kross E, Ayduk O. Facilitating adaptive emotional analysis: Distinguishing distanced-analysis of depressive experiences from immersed-analysis and distraction. Pers Soc Psychol Bull. 2008Jul;34(7):924–38.

Ayduk Ö, Kross E. From a distance: implications of spontaneous self-distancing for adaptive self-reflection. J Pers Soc Psychol. 2010May;98(5):809.

Feldman GC, Joormann J, Johnson SL. Responses to positive affect: A self-report measure of rumination and dampening. Cogn Ther Res. 2008Aug;32:507–25.

Gentzler AL, Ramsey MA, Yuen Yi C, Palmer CA, Morey JN. Young adolescents’ emotional and regulatory responses to positive life events: Investigating temperament, attachment, and event characteristics. J Posit Psychol. 2014Mar 4;9(2):108–21.

Harding KA, Hudson MR, Mezulis A. Cognitive mechanisms linking low trait positive affect to depressive symptoms: A prospective diary study. Cogn Emot. 2014Nov 17;28(8):1502–11.

Mennin DS, Fresco DM. Emotion regulation therapy. Handbook of emotion regulation. 2014;2:469–90.

Levitt JT, Brown TA, Orsillo SM, Barlow DH. The effects of acceptance versus suppression of emotion on subjective and psychophysiological response to carbon dioxide challenge in patients with panic disorder. Behav Ther. 2004Sep 1;35(4):747–66.

Wolgast M, Lundh LG, Viborg G. Cognitive reappraisal and acceptance: An experimental comparison of two emotion regulation strategies. Behav Res Ther. 2011Dec 1;49(12):858–66.

Moon BL. The role of metaphor in art therapy: Theory, method, and experience. Charles C Thomas Publisher; 2007.

Kimport ER, Robbins SJ. Efficacy of creative clay work for reducing negative mood: A randomized controlled trial. Art Ther. 2012Jun 1;29(2):74–9.

Drake JE, Hodge A. Drawing versus writing: The role of preference in regulating short-term affect. Art Ther. 2015Jan 2;32(1):27–33.

De Petrillo L, Winner E. Does Art Improve Mood? A Test of a Key Assumption Underlying Art Therapy. Art Ther. 2005;22(4):205–12. https://doi.org/10.1080/07421656.2005.10129521 .

Gross JJ. Emotion regulation: Current status and future prospects. Psychol Inq. 2015Jan 2;26(1):1–26.

Schutte NS, Manes RR, Malouff JM. Antecedent-focused emotion regulation, response modulation and well-being. Curr Psychol. 2009Mar;28:21–31.

Sheppes G, Gross JJ. Emotion regulation effectiveness: What works when. Handb Psychol. 2012Sep;26(5):391–406.

Aldao A, Nolen-Hoeksema S. One versus many: Capturing the use of multiple emotion regulation strategies in response to an emotion-eliciting stimulus. Cogn Emot. 2013Jun 1;27(4):753–60.

Lee M, Choi H. Art-based emotion regulation in major depression: Framework for intervention. Arts Psychother. 2023Mar;24: 102018.

Gohm CL, Clore GL. Four latent traits of emotional experience and their involvement in well-being, coping, and attributional style. Cogn Emot. 2002Jul 1;16(4):495–518.

Tamir M, Vishkin A, Gutentag T. Emotion regulation is motivated. Emotion. 2020Feb;20(1):115.

Nauphal M, Curreri AJ, Cardona ND, Meyer ER, Southward MW, Sauer-Zavala S. Measuring emotion regulation skill use during treatment: A promising methodological approach. Assessment. 2023Apr;30(3):592–605.

Aldao A, Nolen-Hoeksema S, Schweizer S. Emotion-regulation strategies across psychopathology: A meta-analytic review. Clin Psychol Rev. 2010Mar 1;30(2):217–37.

Bradley MM, Lang PJ. Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry. 1994Mar 1;25(1):49–59.

Download references

Acknowledgements

The present researchers express their gratitude to the Kangdong Sacred Heart Hospital for its help and support in this research. Appreciation is also extended to all participating patients, clinicians, health care professionals, and the advisory board in all steps of the development. There are no individuals or funding organizations, other than the co-authors, who contributed directly or indirectly to this article.

Not applicable.

Author information

Authors and affiliations.

Department of Psychiatry, Kangdong Sacred Heart Hospital, 150, Seongan-Ro, Gangdong-Gu, Seoul, Republic of Korea

Myungjoo Lee & Young Tak Jo

Department of Bio-Medical Engineering, Ajou University, 206, World Cup-Ro, Yeongtong-Gu, Gyeonggi-do, Republic of Korea

You can also search for this author in PubMed   Google Scholar

Contributions

ML contributed to plan and design the study with support from the rest of the study team. YT registered the trial. ML collected, and analyzed participant data. ML drafted and edited the manuscript. All authors reviewed and/or approved the final manuscript for submission.

Corresponding author

Correspondence to Young Tak Jo .

Ethics declarations

Ethics approval and consent to participate.

The ethical approvals for the current research were obtained from the Institutional Review Board of Kangdong Sacred Heart Hospital, PAR2 (IRB no. 2023–12-002), and PAR3 (no. 2024–02-019). In PAR2, informed consent was exempted due to its retrospective nature. Nevertheless, all patients who participated in the therapeutic sessions were requested to sign a consent form for later use of their artwork for educational and research purposes. In PAR 3, informed consent was obtained from the healthcare professionals, including physicians and nurses, who were involved in the program.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Lee, M., Choi, H. & Jo, Y.T. Targeting emotion dysregulation in depression: an intervention mapping protocol augmented by participatory action research. BMC Psychiatry 24 , 595 (2024). https://doi.org/10.1186/s12888-024-06045-y

Download citation

Received : 05 April 2024

Accepted : 23 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1186/s12888-024-06045-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Psychotherapy

BMC Psychiatry

ISSN: 1471-244X

• General enquiries: [email protected]

An official website of the United States government

Here’s how you know

The .gov means it’s official.

Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure.

The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Funding Programs
• Public Wireless Supply Chain Innovation Fund

Innovation Fund Round 1 (2023) Research and Development, Testing and Evaluation

This section provides more information on the Innovation Fund’s grant program. Explore the full list of grant recipients and learn how their projects are driving wireless innovation. The Innovation Fund’s first NOFO focused on two areas: testing and evaluation (T&E) and research and development (R&D) into testing methods.

Research and Development

The Research & Development (R&D) focus area invests in the development of new/improved testing methods. These testing methods will assess the interoperability, performance, and/or security of networks.

Testing and Evaluation

The T&E focus area awards grants to proposals that streamline testing and evaluation across the U.S. Advanced access to affordable T&E lowers the barriers of entry for new and emerging entities, like small companies, start-ups, and SEDI businesses.

$140.4 million

Awarded to Date

Public Wireless Supply Chain Innovation Fund Grant Program Awards

Applications Received

Innovation Fund Program Snapshot

You can visit the Innovation Fund Program Snapshot page or download a one-page summary of the Public Wireless Supply Chain Innovation by clicking on the button below.

Program Documentation

The Program Documentation section has details on learning more about the Innovation Fund Grant Program and how to apply. Visit the following page to find out more.

See Program Awardees

Visit the Awardees page for the awardees funding allocations and related awardee information.

Related content

International briefing on the innovation fund.

• Open access
• Published: 02 September 2024

Evaluation of stress, bio-psycho-social response and coping strategies during the practical training in nursing students: a cross sectional study

• Müjgan Solak   ORCID: orcid.org/0000-0001-6201-3139 1 ,
• Sevcan Topçu   ORCID: orcid.org/0000-0002-6228-1720 2 ,
• Zuhal Emlek Sert   ORCID: orcid.org/0000-0002-2809-5617 2 ,
• Satı Doğan   ORCID: orcid.org/0000-0002-9935-3265 3 &
• Fatma Savan   ORCID: orcid.org/0000-0002-4846-9129 2  

BMC Nursing volume  23 , Article number:  610 ( 2024 ) Cite this article

Metrics details

The aim of the study was to identify stress level, bio-psycho-social response and coping behavior of nursing students during the practical training.

A cross-sectional study was carried out with the 1st, 2nd, 3rd, 4th-year nursing students ( n  = 1181) between September 2018-may 2019. Data was collected using by Socio-Demographic Questionnaire, The Student Nurse Stress Index, The Bio-Psycho-Social Response Scale and Coping Behavior Inventory.

The fourth-grade nursing students’ stress level was found to be statistically significantly higher than of other graders. Nursing students have shown emotional symptoms and social-behavioral symptoms the most. To cope with stress, nursing students used the strategies transference, staying optimistic, problem-solving and avoidance, respectively.

Conclusions

These findings highlight the need to routinely evaluate nursing students for stress, bio-psycho-social response, and coping strategies during practical training. Thus, counseling units can be constituted by the nursing schools, and nursing students who have higher stress levels and inadequate coping strategies benefit from these units.

Peer Review reports

According to Lazarus and Folkman’s transactional theory of stress and coping, stress is a two-way process. Stress is defined as exposure to stimuli (as harmful, threatening, or challenging) that exceed the individual’s coping capacity [ 1 ]. There is a complex transaction between individual subjective reactions to stressors and stressors produced by the environment complex transaction. Transactional theory consists of cognitive appraisal, and coping. After a primary appraisal of the threat or challenge is made, a secondary appraisal process of identifying and selecting available coping options is made. Coping processes produce an outcome, which is reappraised as favorable, unfavorable, or unresolved [ 1 , 2 ].

Stress is accepted as a disease of the 20th century that affects many professions [ 3 ]. Health professionals, especially nurses encounter higher levels of stress and stress factors when their level of exposure to stress and the number of stress-sources are evaluated [ 4 ]. For nurses, stress starts from the beginning of training period and they experience the negative effects of stress on health for many years [ 5 , 6 , 7 ].

Nursing students experience different levels of stress both during their theoretical and practical training [ 8 , 9 ]. Sources of theoretical stress are constantly subjected to examinations, assignments about courses, length of lecture time despite the lack of free times and preparation process before practical evaluations [ 10 , 11 , 12 ]. But sources of practical training stress comprise of the followings; starting to practice for the first-time, clinical evaluations, feeling inadequate in practice, scaring to give patients any harm, caring for patients, relationships with healthcare workers, friends and patients [ 13 , 14 ]. Although nursing students experience stress due to many reasons both in practical and theoretical settings, practical training periods are expressed as periods in which nursing students experience the highest levels of stress [ 15 , 16 ].

Stress can sometimes be a source of motivation, however, high stress can affect coping, self-confidence, concentration, motivation, academic performance [ 9 , 17 ]. In addition, high stress levels may cause students to experience health problems such as hypertension, heart diseases, nutritional disorders, stammering, nausea, vomiting, exhaustion and depression [ 5 , 6 ]. It is stated that nursing students experience higher levels of stress and relevant physical and psychosocial symptoms when compared with the students of other health-related disciplines [ 15 , 18 ].

This situation makes coping strategies crucial for stress management. Coping is defined as constantly changing cognitive and behavioral efforts to manage specific external and/or internal demands that are appraised as taxing or exceeding the resources of the person [ 1 ]. The impact of stress on health depends on the adequacy of coping strategies that play a vital role in managing the stress [ 6 ]. As a strategy to cope with stress, nursing students prefer problem solving the most [ 8 , 11 , 19 ] and avoiding the least [ 5 , 6 , 8 ].

It was found in previous studies that the stress levels of nursing students differed according to classes. It is reported that stress levels are higher in nursing students in the upper grades and the initial clinical practice affects their stress levels [ 20 , 21 , 22 ]. In order to reduce the stress and its negative effects in nursing students, first of all, to determine their stress levels, their responses to stress, coping strategies and the factors affecting their stress levels is very important.

Research questions

What are the stress levels, bio-psycho-social responses, and coping behavior of nursing students during the practical training?

Which variables affect the stress level of nursing students?

A cross-sectional design was used.

Procedure and samples

The study’s population consisted of 1st, 2nd, 3rd, and 4th-year students [ n  = 1181] of nursing school. A cross-sectional study was conducted between September 2018-May 2019.Since it was aimed to reach the entire population, no sample selection method was used. The inclusion criteria for the study were (1) voluntary acceptance of study participation (2) being during the period of practical training. The number of students was 300 for first grade, 309 for second grade, 285 for third grade and 287 for fourth grade. All of the students [ n  = 996] who meet inclusion criteria are included in the study. The response rate of the questionnaires is 84%. ( n  = 996/1181).

Data was collected during the practical training for each grade. The Faculty of Nursing has an integrated education system. The integrated education system is based on holistic learning. It enables the student to see the big picture instead of learning small parts and subject areas are associated according to a subject. The integrated education programme, which includes a structuring from health to disease, is organised to include basic knowledge, attitudes and skills related to the subjects related to care. In the first, second and third years of the integrated education programme, courses are conducted as modules, active education methods are used, and skills training is provided in laboratories and clinics. The fourth year is organised as an internship programme. Practical training starts to in the second term of the first year in the Faculty of Nursing. 1st-year students have practical training consist of 13 h per week for one month in Primary and Secondary Schools. 2nd and 3rd-year students have practical training in Hospitals and Primary Care. The practical training of 2st-year students in the third semester consists of 24 h per week for one month in dermatology, otolaryngology clinics, eye clinics, etc. In the fourth semester, their practical training includes 24 h per week for two months in İnternal Medicine and Surgery clinics. The practical training of 3rd-year students comprises 24 h per week for three months in pediatrics, obstetrics (fifth semester) clinics and psychiatry clinics, primary care (sixth semester). 4th-year students (internship) are in practical training (eight different nursing fields fundamentals of nursing, internal nursing, surgery nursing, pediatric nursing, obstetric and gynecological nursing, psychiatric nursing, public health nursing) during the seventh and eighth semesters. They have practical training 32 h per week each semester.

Data collection tools

Data was collected using by Socio-Demographic Questionnaire, The Student Nurse Stress Index (SNSI), The Bio-Psycho-Social Response Scale (BPSRS) and Coping Behavior Inventory (CBI) Socio-Demographic Questionnaire consists of seven questions such as age, gender, grade, employment status, smoking status, choosing nursing profession willingly and academic status.

The student nurse stress index (SNSI): SNSI that developed by Jones & Johnstone (1999), consisted of 22 items, and four subscales which include academic load, clinical concerns, personal problems, and interface worries [ 23 ]. SNSI is a five-point Likert-type scale ranging from 1 [not stressful] to 5 [extremely stressful]. The Turkish validity and reliability study was conducted by Sarıkoç, Demiralp, Oksuz, Pazar, [ 24 ]. Its Cronbach α coefficient was 0.86. Turkish version of the scale consists of four subscales as personal problems, clinical concerns, interface worries, and academic load. The higher scores obtained from SNSI indicate the high-stress level.

The bio-psycho-social response scale (BPSRS): The BPSRS, developed by Sheu, Lin, Hwang (2002), consist of 21 items and three subscales about symptoms relating to the students’ physical, psychological and social health [ 25 ]. BPSRS five-point Likert-type scale from 0 to 4. Its Cronbach’s alpha coefficient was 0.90. A higher score indicated the presence of more symptoms and poorer physio-psychosocial status [ 25 ]. The Turkish validity and reliability study was conducted by Karaca et al. [ 26 ]. The Cronbach’s alpha coefficient of the Turkish version was found to be 0.91 [ 26 ].

Coping behavior inventory (CBI): The original version of CBI that developed by Sheu, Lin, Hwang, (2002), consists of 19 items and four subscales as avoidance, problem solving, stay optimistic and transference [ 25 ]. The scale is a five-point Likert-type scale from 0 to 4. Its Cronbach’s alpha coefficient was 0.76. A higher score in one factor indicated more frequent use of this type of coping behavior [ 25 ]. The Turkish validity and reliability study was conducted by Karaca et al. (2015) and its Cronbach’s alpha coefficient was 0.69 [ 26 ].

Data analysis

The data were evaluated using the SPSS 21 (Statistical Package for the Social Sciences). Descriptive statistics was used as mean and standard deviation. One way anova test was used to compare scale scores (SNSI, BPSRS, CBI) according to graders. Multiple regression analysis was used to determine the variables (gender, employment status, smoking status, willingness of the choice of the nursing profession, academic achievement status) affecting stress level. For all effects, we used the standard significance level of α = 0.05.

Ethical considerations

This study was approved by Ege University Scientific Research and Publication Ethics Committee (Approval Number: 56/2018). The participants received information about the research objectives and procedures, and their written permission was obtained by means of informed consent form before data collection.

The mean age of nursing students is 21.32 ± 1.57 years. Of the students, 91.9% are females and 26.5% are freshmen, and 5% are working outside the school (Table  1 ).

When nursing students’ total and subscale SNSI mean scores were compared, a statistically significant difference was found between the mean scores of total SNSI and academic loads, interface worries and clinical concerns subscale (Table  2 ). The first grade nursing students’ mean score of academic load subscale was found to be statistically significantly higher than of second and third graders ( p  < 0.05). The third and fourth grade nursing students’ interface worries subscale scores were also statistically significantly higher than of the first and second graders. In the clinical concerns subscale, the second and fourth grade nursing students had significantly higher clinical anxiety than the other graders and the first-year nursing students had lower clinical concerns than other graders. When the total SNSI mean scores were compared, fourth grade nursing students’ stress level was found to be statistically significantly higher than of other graders, and the first grade nursing students’ stress level was statistically lower than of other graders.

It was established that nursing students have shown emotional symptoms and social-behavioral symptoms the most, whereas physical symptoms were shown the least (Table  3 ). When the total and subscale mean scores of BPSRS were compared according to nursing students’ grades, a statistically significant difference was detected in subscales of total BPSRS, emotional symptoms and social behavioral symptoms. In the emotional symptoms subscale, the first year nursing students had less emotional symptoms than other graders. In the social behavioral symptoms subscale, the mean scores of fourth grade nursing students were found to be significantly higher than of other graders. When total BPSRS mean scores were compared, it was observed that the fourth grade students had more bio-psycho-social behavioral symptoms than the first grade students.

It was found that to cope with stress, nursing students used the strategies transference, staying optimistic, problem-solving and avoidance, respectively (Table  4 ). When nursing students’ behaviors related to coping with stress were evaluated according to grades, no statistically significant difference was found between the subscale scores of avoidance, staying optimistic and transference, whereas only the problem-solving subscale was statistically significant. In the problem-solving subscale, the problem-solving skills have increased significantly as the class increased ( F  = 72.63; p  = 0.00).

The relationship between nursing students’ stress level and gender, willingness to choose nursing profession, smoking status, employment status and academic achievement status was evaluated using regression analysis (Table  5 ). The extent to which nursing students’ stress levels were predicted by variables such as gender ( β =-0.22, p  = 0.00), choosing nursing profession willingly ( β =-0.27, p  = 0.00), smoking status ( β  = 0.28, p  = 0.00), employment status ( β  = 0.14, p  = 0.00) and academic achievement status ( β =-0.34, p  = 0.00) was determined by applying linear multiple regression. As a result of this process was detected as R  = 0.84, R2  = 0.70, and 70% of the total variance on stress level was explained by these variables. The stress level was found significantly higher in female students, working students, smokers, those who did not want to choose the nursing profession and those with low academic achievement.

One of the most important stress factors for nursing students is practical training periods especially an initial period of practical training [ 21 ]. It is stated that nursing students experience more stress in clinical practice periods than other periods [ 16 , 21 ]. In the literature, studies investigating the effects of grade on the stress level of nursing students have shown mixed results. Eswi, Radi, Youssri reported that there was no relationship between grade and stress level [ 27 ]. In a study conducted by Shaban, Khater, Akhu-Zaheya, it was found that nursing students were more sensitive to stress due to reasons such as transition to university life, managing their own needs and gaining new social skills, especially during the first years of education. In this study, unlike other studies, the first-year nursing students’ stress level was found lower than of other graders [ 6 ]. Aedh, Elfaki & Mohamed, reported that nursing students who are in the second year of nursing education have experienced higher level of stress than other grades [ 28 ]. In this study, although the second grade was not the highest stress level group, the stress level showed a rapid increase compared to the first grade and the clinical concerns subscale scores were found higher than other grades. Third and fourth grade nursing students’ mean interface worries scores were found high the other grades. Several studies have similarly reported that, nursing students’ stress level was found higher in the last period of nursing education compared to other periods [ 15 , 22 ]. In a qualitative study conducted by Admi et al. (2018) it was found that conflict between professional beliefs and the reality of hospital practice were stressors for final year students [ 19 ]. In the study conducted by Bhat (2021) et al. it was reported that training on invasive procedures (safe catheter etc.) should be standardised in undergraduate education and this should be made part of the annual or biannual compulsory training for healthcare personnel [ 29 ]. Similarly, in this study, the stress level of fourth-grade nursing students was found higher than of other graders, and fourth-grade nursing students’ mean scores of clinical concerns and interface worries were higher than of other graders. The results of our study indicate that the first-grade nursing students had problems adapting to the intensive pace of nursing education and that they experienced stress; accordingly, second-grade nursing students who first-time took to practical training and fourth-grade nursing students who had the longest practical training period also experienced stress due to practical training.

In several studies found that nursing students experienced higher levels of stress, physical and psychological symptoms than the students in other health disciplines [ 6 , 30 ]. Chen & Hung reported that nursing students demonstrated physical symptoms toward stress mostly, and social-behavioral symptoms the least [ 8 ]. In the study carried out by Kassem & Abdou, when the bio-psycho-social responses experienced by nursing students were evaluated, it was found that emotional symptoms were the most common and social-behavioral symptoms were the least [ 11 ]. In another study conducted by Durmuş & Gerçek with nursing students, it that bio-psycho-social responses were found to be occurred mostly in fourth grade students [ 31 ]. In all classes, the most often emotional symptoms were observed in nursing students followed by social behavior symptoms and physical symptoms respectively [ 31 ]. The present study showed that nursing students demonstrated emotional symptoms and social-behavioral symptoms the most, whereas physical symptoms were demonstrated the least, and these results were consistent with results from most of previous similar studies. It was found that fourth-grade nursing students experienced more Bio-Psycho-Social Responses than freshmen and emotional symptoms were higher in second, third and fourth grade nursing students and social behavioral symptoms were higher in fourth-grade nursing students. This difference may be explained by the fact that because fourth-grade nursing students’ stress levels were higher than of other graders, they showed more Bio-Psycho-Social Responses.

Durmuş & Gerçek found that first, and the third-year nursing students have usually used strategies for coping with stress such as stay optimistic and avoidance, respectively [ 31 ]. Also, the same study showed that second and fourth-year nursing students have used problem-solving most [ 31 ]. Many studies found that nursing students have generally used problem solving as a coping strategy [ 5 , 8 , 11 , 19 , 32 ] and the avoidance at least [ 5 , 6 , 8 ]. Sheu, Lin, Hwang reported that using effective ways of coping with the problem will facilitate returning to stable status by allowing reduction of negative consequences of stress [ 25 ]. The present study showed that nursing students most often used transference and least avoidance strategies to cope with stress, and as the students’ grade levels increased, also the level of using problem-solving skills increased. This situation indicates that the problem-solving competencies involving in nursing education are being provided to the students. The fourth grade of nursing students who has highest practical-training hours possess problem-solving skills more than other grades because of the positive effects of the practical applications encountered in a large number of complicated situations on the problem-solving skills of the nursing students.

In the present study, when the interaction between nursing students’ stress level and gender, working status, smoking status, willingness to choose nursing profession and academic achievement status was evaluated, it was found that female students, employees, smokers, those that have chosen nursing profession unwillingly, and those with low academic achievement had significantly higher stress levels. It was reported in different studies that academic success [11,20,], gender [ 20 , 21 , 33 ] have affected students’ stress levels and also their working hours outside of nursing education have affected their stress level [ 11 ]. Although it is important for all students to reduce stressors and to provide support for the use of coping mechanisms; especially female students, employees, smokers, those that choose the nursing profession unwillingly, and those with poor academic achievement should be supported more.

Limitations

This study has some limitations. Unlike other nursing schools in our country, this research was carried out in a nursing school where an integrated education system was applied. The findings could be specific to this college of nursing. Therefore, the generalizability of results may be limited. Besides, the small number of male students is another limitation of the study. SNSI, BPSRS, and CBI are a self-reported questionnaire. This can lead to social desirability bias in respondents.

Reccommendations

It is recommended that long-term studies be conducted to understand the long-term effects of stress experienced during nursing education and to develop sustainable support mechanisms. Support mechanisms may decrease stress levels and their negative effects on nursing students and can promote nursing students’ well-being and academic success, especially during practical training. Exploring what is nursing students of stress levels and coping strategies during education, can inform post-graduation preventive strategies. Also, evaluating the current stress levels and coping strategies in different nursing education programs is crucial for identifying gaps and areas for improvement. Interventional and qualitative studies are crucial to providing concrete recommendations for educational institutions and policymakers to address stress among nursing students.

According to results of the present study, the stress levels of fourth-grade nursing students were higher than of other graders and causes of stress varied as regards grades. The higher level of stress in the senior nursing students that have the maximum responsibilities and stay times of practical training and the bio-psycho-social responses given by students associated depending on this stress indicate that those clinical practices are one of the main sources of stress for nursing students. Due to the nature of nursing education and nursing practices, students use their problem solving skills as a coping strategy. However, the presence of stress-related emotional and social-behavioral symptoms in nursing students indicates that they cannot cope with stress sufficiently. Internship, which is the preparation period for the transition to professional life for nursing, is the period in which nursing students experience the most stress. Students’ learning to cope with stress in this period will enable them to use these strategies in their professional lives. Nursing schools can consider this period as an opportunity period to reduce and cope with stress, which is one of the important risk factors for nurses.

To develop stress management and the stress-coping mechanism of nursing students, it was recommended that courses or counseling units should be available, nursing educators should support students in the clinical areas, receive regular feedback from the students about practical training, and cooperate with clinical nurses to increase nursing students’ clinical compliance. Also, in particular, female students, working students, smokers, those that have chosen nursing profession unwillingly, and those with low academic achievement should be encouraged to receive individualized or group support for stress management and in coping with stress.

Data availability

The data that support the findings of this study are not openly available due to reasons of sensitivity and are available from the corresponding author upon reasonable request.

Lazarus RS, Folkman S. Stress, appraisal, and coping. Springer publishing company; 1984.

Biggs A, Brough P, Drummond S. Lazarus and Folkman’s psychological stress and coping theory. The handbook of stress and health: A guide to research and practice , 2017;349–364.

Evans W, Kelly B. Pre-registration diploma student nurse stress and coping measures. Nurse Educ Today. 2004;24(6):473–82.

Article   PubMed   Google Scholar  

Sossah L, Asiedu Y. Stress management and resilience in junior and senior nursing students in Ghana. EJRRES. 2015;3(2):46–53.

Google Scholar  

Al-Gamal E, Alhosain A, Alsunaye K. Stress and coping strategies among Saudi nursing students during clinical education. Perspect Psychiatr Care. 2018;54(2):198–205.

Shaban IA, Khater WA, Akhu-Zaheya LM. Undergraduate nursing students’ stress sources and coping behaviours during their initial period of clinical training: a Jordanian perspective. Nurse Educ Pract. 2012;12(4):204–9.

Watson R, Deary I, Thompson D, Li G. A study of stress and burn out in nursing students in Hong Kong. Int J Nurs Stud. 2008;45(10):1534–42.

Chen YW, Hung CH. Predictors of Taiwanese baccalaureate nursing students’ bio–psycho-social responses during clinical practicum. Nurse Educ Today. 2014;34(1):73–7.

Article   CAS   PubMed   Google Scholar  

McCarthy B, Trace A, O’Donovan M, Brady-Nevin C, Murphy M, O’Shea M, et al. Nursing and midwifery students’ stress and coping during their undergraduate education programmes: an integrative review. Nurse Educ Today. 2018;61:197–209.

Gibbons C, Dempster M, Moutray M. Index of sources of stress in nursing students: a confirmatory factor analysis. J Adv Nurs. 2009;65(5):1095–102.

Kassem F, Abdou R. Stress, coping strategies and perceived faculty support among nursing students in alexandria. ASJN. 2015;17(1):113–30.

Yucha C, Kowalski S, Cross C. Student stress and academic performance: home hospital program. J Nurs Educ. 2009;48(11):631–7.

Al-Zayyat A, Al-Gamal E. A review of the literature regarding stress among nursing students during their clinical education. Int Nurs Rev. 2014;61(3):406–15.

Article   CAS   Google Scholar  

Labrague LJ, McEnroe–Petitte DM, De Los Santos JAA, Edet OB. Examining stress perceptions and coping strategies among Saudi nursing students: a systematic review. Nurse Educ Today. 2018;65:192–200.

Edwards D, Burnard P, Bennett K, Hebden U. A comparative, longitudinal study of stress in nursing students in five countries: Albania, Brunei, the Czech Republic, Malta and Wales. Nurse Educ Today. 2010;28(2):134–45.

Singh C, Sharma S, Sharma R. Level of stress and coping strategies used by nursing interns. Nurs Midwifery Res J. 2011;7(4):152–60.

Article   Google Scholar  

Wolf L, Stidham A, Ross R. Predictors of stress and coping strategies of US accelerated vs. generic baccalaureate nursing students: an embedded mixed methods study. Nurse Educ Today. 2015;35(1):201–05.

Pulido-Martos M, Augusto-Landa JM, Lopez-Zafra E. Sources of stress in nursing students: a systematic review of quantitative studies. Int Nurs Rev. 2012;59:15–25.

Al-Zayyat A, Al-Gamal E. Perceived stress and coping strategies among Jordanian nursing students during clinical practice in psychiatric/mental health course. Int J Ment Health Nurs. 2014;23:326–35.

Admi H, Moshe-Eilon Y, Sharon D, Mann M. Nursing students’ stress and satisfaction in clinical practice along different stages: a cross-sectional study. Nurse Educ Today. 2018;68:86–92.

Doğan S, Yıldırım D. The relationship between vocational motivation and sources of stress among interns at a faculty of nursing. STED. 2019;28(6):418–29.

Ismaile S. Perceived clinical stressors among Saudi nursing students. Open J Nur. 2017;7(04):463.

Jones MC, Johnston DW. The derivation of a brief student nurse stress index. Work Stress. 1999;13(2):162e81.

Sarikoc G, Demiralp MB, Oksuz E, Pazar B. Turkish version of the student nurse stress index: validity and reliability. Asian Nurs Res. 2017;11(2):128–33.

Sheu S, Lin H, Hwang S. Perceived stress and physio-psycho-social status of nursing students during their initial period of clinical practice: the effect of coping behaviors. Int J Nurs Stud. 2002;39(2):165–75.

Karaca A, Yıldırım N, Ankaralı H, Acikgoz F, Akkuş D. The Turkish adaptaion of perceived stress scale, bio-psycho-social response and coping behaviours of stress scales for nursing students. J Psychiatr Nurs. 2015;6:15–25.

Eswi AS, Radi S, Youssri H. Stress/stressors as perceived by baccalaureate Saudi nursing students. Middle East J Sci Res. 2013;14(2):193–202.

Aedh AI, Elfaki NK, Mohamed IA. Factors associated with stress among nursing students. IOSR-JNHS. 2015;4(6):33–8.

Bhatt NR, Davis NF, Thorman H, et al. Knowledge, skills, and confidence among healthcare staff in urinary catheterization. Can Urol Assoc J. 2021;15(9):E488–94.

Article   PubMed   PubMed Central   Google Scholar  

Sheu S, Lin HS, Hwang SL. The relationships among stress, physio–psychosocial status, coping behaviors and personality traits of nursing students during first time clinical practice. J Nurs Res. 2001;9(3):233–45.

CAS   Google Scholar  

Durmuş M, Gerçek A. A research on assessing the factors affecting perceived stress situations, bio-psychological social situations and coping behaviors of stress in university students’. JASSS. 2017;3:616–33.

Onieva-Zafra MD, Fernández-Muñoz JJ, Fernández-Martínez E, García-Sánchez FJ, Abreu-Sánchez A, Parra-Fernández ML. Anxiety, perceived stress and coping strategies in nursing students: a cross-sectional, correlational, descriptive study. BMC Med Educ. 2020;20:1–9.

Hamaideh SH, Abuhammad S, Khait AA, Al-Modallal H, Hamdan-Mansour AM, Masa’deh R, Alrjoub S. (2024). Levels and predictors of empathy, self-awareness, and perceived stress among nursing students: a cross sectional study. BMC Nurs 2024;23(1):131.

Download references

The authors declared research is not funded.

Author information

Authors and affiliations.

Surgical Nursing, Faculty of Nursing, Ege University, Izmir, 35030, Turkey

Müjgan Solak

Public Health Nursing, Faculty of Nursing, Ege University, Izmir, 35030, Turkey

Sevcan Topçu, Zuhal Emlek Sert & Fatma Savan

Psychiatric Nursing, Faculty of Nursing, Ege University, Izmir, 35030, Turkey

You can also search for this author in PubMed   Google Scholar

Contributions

M. S Conception and design, data acquisition, data analysis and interpretation, writing, give final approvals. S. T Conception and design, data acquisition, data analysis, writing, give final approvals. Z. E. S Data acquisition, data interpretation, give final approvals. S. D Data acquisition, data analysis, give final approvals. F. S Conception, writing, give final approvals.

Corresponding author

Correspondence to Sevcan Topçu .

Ethics declarations

Ethical approval.

All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of Ege University [Approval Number: 56/2018].

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Solak, M., Topçu, S., Sert, Z.E. et al. Evaluation of stress, bio-psycho-social response and coping strategies during the practical training in nursing students: a cross sectional study. BMC Nurs 23 , 610 (2024). https://doi.org/10.1186/s12912-024-02265-5

Download citation

Received : 30 June 2024

Accepted : 14 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s12912-024-02265-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Practical training
• Coping strategies
• Nursing students

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]