quality of research thesis

/images/cornell/logo35pt_cornell_white.svg" alt="quality of research thesis"> Cornell University --> Graduate School

Guide to writing your thesis/dissertation, definition of dissertation and thesis.

The dissertation or thesis is a scholarly treatise that substantiates a specific point of view as a result of original research that is conducted by students during their graduate study. At Cornell, the thesis is a requirement for the receipt of the M.A. and M.S. degrees and some professional master’s degrees. The dissertation is a requirement of the Ph.D. degree.

Formatting Requirement and Standards

The Graduate School sets the minimum format for your thesis or dissertation, while you, your special committee, and your advisor/chair decide upon the content and length. Grammar, punctuation, spelling, and other mechanical issues are your sole responsibility. Generally, the thesis and dissertation should conform to the standards of leading academic journals in your field. The Graduate School does not monitor the thesis or dissertation for mechanics, content, or style.

“Papers Option” Dissertation or Thesis

A “papers option” is available only to students in certain fields, which are listed on the Fields Permitting the Use of Papers Option page , or by approved petition. If you choose the papers option, your dissertation or thesis is organized as a series of relatively independent chapters or papers that you have submitted or will be submitting to journals in the field. You must be the only author or the first author of the papers to be used in the dissertation. The papers-option dissertation or thesis must meet all format and submission requirements, and a singular referencing convention must be used throughout.

ProQuest Electronic Submissions

The dissertation and thesis become permanent records of your original research, and in the case of doctoral research, the Graduate School requires publication of the dissertation and abstract in its original form. All Cornell master’s theses and doctoral dissertations require an electronic submission through ProQuest, which fills orders for paper or digital copies of the thesis and dissertation and makes a digital version available online via their subscription database, ProQuest Dissertations & Theses . For master’s theses, only the abstract is available. ProQuest provides worldwide distribution of your work from the master copy. You retain control over your dissertation and are free to grant publishing rights as you see fit. The formatting requirements contained in this guide meet all ProQuest specifications.

Copies of Dissertation and Thesis

Copies of Ph.D. dissertations and master’s theses are also uploaded in PDF format to the Cornell Library Repository, eCommons . A print copy of each master’s thesis and doctoral dissertation is submitted to Cornell University Library by ProQuest.

• Cookies & Privacy
• GETTING STARTED
• Introduction
• FUNDAMENTALS

Getting to the main article

Choosing your route

Setting research questions/ hypotheses

Assessment point

Building the theoretical case

Setting your research strategy

Data collection

Data analysis

Research quality

All research has limitations , which negatively impact upon the quality of the findings you arrive at from your data analysis . This is the case whether you are an undergraduate or master's level student doing a dissertation, a doctoral student, or a seasoned academic researcher. The main journal article you are interested in will also have a number of limitations, some of which will have inevitably become the justifications for your chosen route , and the approach you selected within that route.

You need to think about research quality at this stage in the dissertation process because many of the problems experienced during the dissertation process can be avoided. The trick is to (a) understand the types of research limitation you may face when doing a dissertation, (b) anticipate what these will be in your dissertation, and (c) avoid them becoming a reality (where possible). Quite simply, the better the research quality of your dissertation, (a) the fewer problems you will experience when carrying out your dissertation research, (b) the less time you will need to write up the Research Limitations section of your Discussion/Conclusions chapter (i.e., Chapter Five: Discussion/Conclusions ), and (c) the greater the likelihood of a high mark.

To improve the research quality of your dissertation, you need to follow four steps : (a) understand the five factors through which research quality is assessed - internal validity , external validity , construct validity , reliability and objectivity ; (b) assess the research quality of the main journal article; (c) consider the potential research quality of your research strategy; and (d) determine how you will overcome such weaknesses in your dissertation, considering the practical aspects of your dissertation, and the implications that these may have on the quality of your findings .

• STEP ONE: Understand the five factors through which research quality is assessed
• STEP TWO: Assess the research quality of the main journal article
• STEP THREE: Consider the potential research quality of your research strategy
• STEP FOUR: Determine how you will overcome such weaknesses in your dissertation

STEP ONE Understand the five factors through which research quality is assessed

In quantitative dissertations, research quality is assessed based on the internal validity , external validity , construct validity , reliability and objectivity of the research. Irrespective of the route that you are following, or the approach within that route, it is important that (a) your dissertation is as internally and externally validity as possible, (b) the measurement procedure you used (i.e., the research method and its measures) are construct valid and reliable, and (c) your research was carried out in an objective way. If you are already confident that you understand these five means through which the quality of quantitative research is assessed, jump to STEP TWO: Assess the research quality of the main journal article . If not, we would suggest that you learn about these terms in the Research Quality section of the Fundamentals part of Lærd Dissertation before reading on. After all, in STEP TWO below, you will need to assess the research quality of the main journal article, before being able to consider the potential weaknesses in research quality in your dissertation, and how you will overcome these weaknesses. To do this, you first need to understand these five main factors through which research quality is assessed.

STEP TWO Assess the research quality of the main journal article

Irrespective of the route that you are following, the person marking your work will expect that you have critically analysed the research strategy used in the main journal article. Even if limitations in the research strategy do not act as the main justification for your choice of route , or the approach within that route (i.e., as is the case in method and measurement-based extensions , or design-based extensions within Route C: Extension ), being able to critically analyse the research strategy used is typically a very important part of the marking scheme for dissertations. Just as you were expected to critically analyse the literature in STAGE FIVE: Building the theoretical case , you have to demonstrate an equally good knowledge of the weaknesses (and strengths) of the research strategy of the main journal article.

You can critically analyse the research strategy used by assessing the research quality of the research strategy used in the main journal article in terms of (a) the internal and external validity of the research strategy, and (b) the construct validity and reliability of the measurement procedure that was used (i.e., the research method and its measures). In most cases, since you did not witness the way that the research in the main journal article was carried out in practice, it will be difficult to assess the objectivity of the research.

Therefore, in order to assess the research quality of the main journal article, you should read up about internal validity , external validity , construct validity , reliability , and even the objectivity of research in the Research Quality section of the Fundamentals part of Lærd Dissertation. However, it is worth mentioning that:

To assess the internal and external validity of the research strategy , assess the threats to such internal and external validity in the main journal article. For example, in the article, Internal validity , we discuss 14 potential threats to internal validity, which include (a) history effects, (b) maturation, (c) testing effects, (d) instrumentation, (e) statistical regression, (f) selection biases, (g) experimental mortality, (h) causal time order, (i) diffusion (or imitation) of treatments, (j) compensation, (k) compensatory rivalry, (l) demoralization, (m) experimenter effects and (n) subject effects. Since any of these 14 threats could have affected the internal validity of the main journal article, you should briefly read up about each one, and then assess whether you think these threats were present in the main journal article. You should note that it will not always be possible to tell whether such a threat was a problem because whilst some are more evident (e.g., the authors of the main journal article should have specified how they selected individuals to be included in their sample, which could expose potential selection biases ), many are not so obvious (e.g., experimenter effects could have occurred as a result of the personal characteristics of the researchers in the main journal article, or some non-verbal cues that they gave off, which influenced the choices participants made when they were being studied, but this would be extremely difficult to spot, especially if the authors did not explicitly try to assess such bias, which is uncommon). Again, you can learn about internal validity and external validity in the Research Quality section of the Fundamentals part of Lærd Dissertation.

Construct validity and reliability are two different ways of assessing the measurement procedure used in the main journal article. Construct validity is important because we want to make sure that the measurement procedure (e.g., a survey, structured interview, structured observation, etc.) that was used to measure the constructs we are interested in (e.g., sexism, obesity, famine, outsourcing, etc.) are valid. By construct valid , we mean that there is (a) a clear link between the constructs you are interested in and the measures and interventions that are used to operationalize them (i.e., measure them), and (b) a clear distinction between different constructs. Construct validity is an overarching term used to refer to the process of assessing the validity of the measurement procedure that was used, and you will need to read up about other types of validity that you will need to consider (i.e., content validity , convergent and divergent validity , criterion validity ), especially if you are taking on a method or measurement-based extension , or design-based extension within Route C: Extension . You can learn more about construct validity in the article: Construct validity . Reliability is important because in order for the results from a study to be considered valid , the measurement procedure must first be reliable . There are a number of types of reliability that you may need to consider when assessing the main journal article, depending on whether the measurement procedure involved (a) successive measurements; (b) simultaneous measurements by more than one researcher; and/or (c) multi-measure procedures. You can learn more about these types of reliability in the article: Reliability in research . When reading up about construct validity and reliability in these articles, you will learn how to assess a piece of research (i.e., your main journal article) in terms of its construct validity and reliability.

When you understand the five factors through which research quality is assess (i.e., STEP ONE ), and have assessed the research quality of your main journal article (i.e., STEP TWO ), you will be well-equipped to consider the potential research quality of your research strategy, based on the route you adopted, and the approach within that route (i.e., STEP THREE next).

Dallas Baptist University

Logins Student Faculty/Staff

Library Campus News Visit DBU Confirm Enrollment

• University Writing Center
• Writing Workshops

Writing a Quality Research Paper

Organize the information.

• Computer Databases, Internet, Encyclopedias, Dictionaries, Journals, Books, etc. 
• After printing, highlight the relevant information.
• Paraphrase important information in the margins.
• Cite Bibliography from each resource
• After reviewing the data, develop basic contentions and sub-points. 
• Group research into the contention that each bit of data falls under. 
• Develop your own thoughts and analysis, expanding on what your research says.
• Construct a thesis statement: a sentence that is an umbrella statement including each contention.
• Make an outline and fit contentions and sub-points together in Roman Numeral fashion.
• Introduction with a developed thesis. 
• Cite quotations according to the chosen documentation style (MLA, APA, Turabian).
• Do not plagiarize by failing to credit the author with his or her thoughts. 
• Your own analysis and your perception of the relevance to the thesis after each bit of data. 
• Original conclusion with a recapitulation of major points.

Add Finishing Touches

• Follow the step-by-step guidelines provided in the documentation manual. 
• Keep in mind specific assignment constraints. 
• Finish early enough to review your final draft with the Writing Center or other peers. 
• Make the appropriate changes.
• Before Visiting
• Quick Reference Flyers
• Writing Resource Links
• Frequently Asked Questions

• Privacy Policy

Home » Thesis – Structure, Example and Writing Guide

Thesis – Structure, Example and Writing Guide

Table of contents.

Definition:

Thesis is a scholarly document that presents a student’s original research and findings on a particular topic or question. It is usually written as a requirement for a graduate degree program and is intended to demonstrate the student’s mastery of the subject matter and their ability to conduct independent research.

History of Thesis

The concept of a thesis can be traced back to ancient Greece, where it was used as a way for students to demonstrate their knowledge of a particular subject. However, the modern form of the thesis as a scholarly document used to earn a degree is a relatively recent development.

The origin of the modern thesis can be traced back to medieval universities in Europe. During this time, students were required to present a “disputation” in which they would defend a particular thesis in front of their peers and faculty members. These disputations served as a way to demonstrate the student’s mastery of the subject matter and were often the final requirement for earning a degree.

In the 17th century, the concept of the thesis was formalized further with the creation of the modern research university. Students were now required to complete a research project and present their findings in a written document, which would serve as the basis for their degree.

The modern thesis as we know it today has evolved over time, with different disciplines and institutions adopting their own standards and formats. However, the basic elements of a thesis – original research, a clear research question, a thorough review of the literature, and a well-argued conclusion – remain the same.

Structure of Thesis

The structure of a thesis may vary slightly depending on the specific requirements of the institution, department, or field of study, but generally, it follows a specific format.

Here’s a breakdown of the structure of a thesis:

This is the first page of the thesis that includes the title of the thesis, the name of the author, the name of the institution, the department, the date, and any other relevant information required by the institution.

This is a brief summary of the thesis that provides an overview of the research question, methodology, findings, and conclusions.

This page provides a list of all the chapters and sections in the thesis and their page numbers.

Introduction

This chapter provides an overview of the research question, the context of the research, and the purpose of the study. The introduction should also outline the methodology and the scope of the research.

Literature Review

This chapter provides a critical analysis of the relevant literature on the research topic. It should demonstrate the gap in the existing knowledge and justify the need for the research.

Methodology

This chapter provides a detailed description of the research methods used to gather and analyze data. It should explain the research design, the sampling method, data collection techniques, and data analysis procedures.

This chapter presents the findings of the research. It should include tables, graphs, and charts to illustrate the results.

This chapter interprets the results and relates them to the research question. It should explain the significance of the findings and their implications for the research topic.

This chapter summarizes the key findings and the main conclusions of the research. It should also provide recommendations for future research.

This section provides a list of all the sources cited in the thesis. The citation style may vary depending on the requirements of the institution or the field of study.

This section includes any additional material that supports the research, such as raw data, survey questionnaires, or other relevant documents.

How to write Thesis

Here are some steps to help you write a thesis:

• Choose a Topic: The first step in writing a thesis is to choose a topic that interests you and is relevant to your field of study. You should also consider the scope of the topic and the availability of resources for research.
• Develop a Research Question: Once you have chosen a topic, you need to develop a research question that you will answer in your thesis. The research question should be specific, clear, and feasible.
• Conduct a Literature Review: Before you start your research, you need to conduct a literature review to identify the existing knowledge and gaps in the field. This will help you refine your research question and develop a research methodology.
• Develop a Research Methodology: Once you have refined your research question, you need to develop a research methodology that includes the research design, data collection methods, and data analysis procedures.
• Collect and Analyze Data: After developing your research methodology, you need to collect and analyze data. This may involve conducting surveys, interviews, experiments, or analyzing existing data.
• Write the Thesis: Once you have analyzed the data, you need to write the thesis. The thesis should follow a specific structure that includes an introduction, literature review, methodology, results, discussion, conclusion, and references.
• Edit and Proofread: After completing the thesis, you need to edit and proofread it carefully. You should also have someone else review it to ensure that it is clear, concise, and free of errors.
• Submit the Thesis: Finally, you need to submit the thesis to your academic advisor or committee for review and evaluation.

Example of Thesis

Example of Thesis template for Students:

Title of Thesis

Table of Contents:

Chapter 1: Introduction

Chapter 2: Literature Review

Chapter 3: Research Methodology

Chapter 4: Results

Chapter 5: Discussion

Chapter 6: Conclusion

References:

Appendices:

Note: That’s just a basic template, but it should give you an idea of the structure and content that a typical thesis might include. Be sure to consult with your department or supervisor for any specific formatting requirements they may have. Good luck with your thesis!

Application of Thesis

Thesis is an important academic document that serves several purposes. Here are some of the applications of thesis:

• Academic Requirement: A thesis is a requirement for many academic programs, especially at the graduate level. It is an essential component of the evaluation process and demonstrates the student’s ability to conduct original research and contribute to the knowledge in their field.
• Career Advancement: A thesis can also help in career advancement. Employers often value candidates who have completed a thesis as it demonstrates their research skills, critical thinking abilities, and their dedication to their field of study.
• Publication : A thesis can serve as a basis for future publications in academic journals, books, or conference proceedings. It provides the researcher with an opportunity to present their research to a wider audience and contribute to the body of knowledge in their field.
• Personal Development: Writing a thesis is a challenging task that requires time, dedication, and perseverance. It provides the student with an opportunity to develop critical thinking, research, and writing skills that are essential for their personal and professional development.
• Impact on Society: The findings of a thesis can have an impact on society by addressing important issues, providing insights into complex problems, and contributing to the development of policies and practices.

Purpose of Thesis

The purpose of a thesis is to present original research findings in a clear and organized manner. It is a formal document that demonstrates a student’s ability to conduct independent research and contribute to the knowledge in their field of study. The primary purposes of a thesis are:

• To Contribute to Knowledge: The main purpose of a thesis is to contribute to the knowledge in a particular field of study. By conducting original research and presenting their findings, the student adds new insights and perspectives to the existing body of knowledge.
• To Demonstrate Research Skills: A thesis is an opportunity for the student to demonstrate their research skills. This includes the ability to formulate a research question, design a research methodology, collect and analyze data, and draw conclusions based on their findings.
• To Develop Critical Thinking: Writing a thesis requires critical thinking and analysis. The student must evaluate existing literature and identify gaps in the field, as well as develop and defend their own ideas.
• To Provide Evidence of Competence : A thesis provides evidence of the student’s competence in their field of study. It demonstrates their ability to apply theoretical concepts to real-world problems, and their ability to communicate their ideas effectively.
• To Facilitate Career Advancement : Completing a thesis can help the student advance their career by demonstrating their research skills and dedication to their field of study. It can also provide a basis for future publications, presentations, or research projects.

When to Write Thesis

The timing for writing a thesis depends on the specific requirements of the academic program or institution. In most cases, the opportunity to write a thesis is typically offered at the graduate level, but there may be exceptions.

Generally, students should plan to write their thesis during the final year of their graduate program. This allows sufficient time for conducting research, analyzing data, and writing the thesis. It is important to start planning the thesis early and to identify a research topic and research advisor as soon as possible.

In some cases, students may be able to write a thesis as part of an undergraduate program or as an independent research project outside of an academic program. In such cases, it is important to consult with faculty advisors or mentors to ensure that the research is appropriately designed and executed.

It is important to note that the process of writing a thesis can be time-consuming and requires a significant amount of effort and dedication. It is important to plan accordingly and to allocate sufficient time for conducting research, analyzing data, and writing the thesis.

Characteristics of Thesis

The characteristics of a thesis vary depending on the specific academic program or institution. However, some general characteristics of a thesis include:

• Originality : A thesis should present original research findings or insights. It should demonstrate the student’s ability to conduct independent research and contribute to the knowledge in their field of study.
• Clarity : A thesis should be clear and concise. It should present the research question, methodology, findings, and conclusions in a logical and organized manner. It should also be well-written, with proper grammar, spelling, and punctuation.
• Research-Based: A thesis should be based on rigorous research, which involves collecting and analyzing data from various sources. The research should be well-designed, with appropriate research methods and techniques.
• Evidence-Based : A thesis should be based on evidence, which means that all claims made in the thesis should be supported by data or literature. The evidence should be properly cited using appropriate citation styles.
• Critical Thinking: A thesis should demonstrate the student’s ability to critically analyze and evaluate information. It should present the student’s own ideas and arguments, and engage with existing literature in the field.
• Academic Style : A thesis should adhere to the conventions of academic writing. It should be well-structured, with clear headings and subheadings, and should use appropriate academic language.

Advantages of Thesis

There are several advantages to writing a thesis, including:

• Development of Research Skills: Writing a thesis requires extensive research and analytical skills. It helps to develop the student’s research skills, including the ability to formulate research questions, design and execute research methodologies, collect and analyze data, and draw conclusions based on their findings.
• Contribution to Knowledge: Writing a thesis provides an opportunity for the student to contribute to the knowledge in their field of study. By conducting original research, they can add new insights and perspectives to the existing body of knowledge.
• Preparation for Future Research: Completing a thesis prepares the student for future research projects. It provides them with the necessary skills to design and execute research methodologies, analyze data, and draw conclusions based on their findings.
• Career Advancement: Writing a thesis can help to advance the student’s career. It demonstrates their research skills and dedication to their field of study, and provides a basis for future publications, presentations, or research projects.
• Personal Growth: Completing a thesis can be a challenging and rewarding experience. It requires dedication, hard work, and perseverance. It can help the student to develop self-confidence, independence, and a sense of accomplishment.

Limitations of Thesis

There are also some limitations to writing a thesis, including:

• Time and Resources: Writing a thesis requires a significant amount of time and resources. It can be a time-consuming and expensive process, as it may involve conducting original research, analyzing data, and producing a lengthy document.
• Narrow Focus: A thesis is typically focused on a specific research question or topic, which may limit the student’s exposure to other areas within their field of study.
• Limited Audience: A thesis is usually only read by a small number of people, such as the student’s thesis advisor and committee members. This limits the potential impact of the research findings.
• Lack of Real-World Application : Some thesis topics may be highly theoretical or academic in nature, which may limit their practical application in the real world.
• Pressure and Stress : Writing a thesis can be a stressful and pressure-filled experience, as it may involve meeting strict deadlines, conducting original research, and producing a high-quality document.
• Potential for Isolation: Writing a thesis can be a solitary experience, as the student may spend a significant amount of time working independently on their research and writing.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Tables in Research Paper – Types, Creating Guide...

Data Analysis – Process, Methods and Types

Dissertation vs Thesis – Key Differences

Problem Statement – Writing Guide, Examples and...

Research Topics – Ideas and Examples

Thesis Outline – Example, Template and Writing...

• Search Menu
• Sign in through your institution
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

1. introduction, 4. synthesis, 4.1 principles of tdr quality, 5. conclusions, supplementary data, acknowledgements, defining and assessing research quality in a transdisciplinary context.

• Article contents
• Figures & tables
• Supplementary Data

Brian M. Belcher, Katherine E. Rasmussen, Matthew R. Kemshaw, Deborah A. Zornes, Defining and assessing research quality in a transdisciplinary context, Research Evaluation , Volume 25, Issue 1, January 2016, Pages 1–17, https://doi.org/10.1093/reseval/rvv025

• Permissions Icon Permissions

Research increasingly seeks both to generate knowledge and to contribute to real-world solutions, with strong emphasis on context and social engagement. As boundaries between disciplines are crossed, and as research engages more with stakeholders in complex systems, traditional academic definitions and criteria of research quality are no longer sufficient—there is a need for a parallel evolution of principles and criteria to define and evaluate research quality in a transdisciplinary research (TDR) context. We conducted a systematic review to help answer the question: What are appropriate principles and criteria for defining and assessing TDR quality? Articles were selected and reviewed seeking: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, proposed principles of research quality, proposed criteria for research quality assessment, proposed indicators and measures of research quality, and proposed processes for evaluating TDR. We used the information from the review and our own experience in two research organizations that employ TDR approaches to develop a prototype TDR quality assessment framework, organized as an evaluation rubric. We provide an overview of the relevant literature and summarize the main aspects of TDR quality identified there. Four main principles emerge: relevance, including social significance and applicability; credibility, including criteria of integration and reflexivity, added to traditional criteria of scientific rigor; legitimacy, including criteria of inclusion and fair representation of stakeholder interests, and; effectiveness, with criteria that assess actual or potential contributions to problem solving and social change.

Contemporary research in the social and environmental realms places strong emphasis on achieving ‘impact’. Research programs and projects aim to generate new knowledge but also to promote and facilitate the use of that knowledge to enable change, solve problems, and support innovation ( Clark and Dickson 2003 ). Reductionist and purely disciplinary approaches are being augmented or replaced with holistic approaches that recognize the complex nature of problems and that actively engage within complex systems to contribute to change ‘on the ground’ ( Gibbons et al. 1994 ; Nowotny, Scott and Gibbons 2001 , Nowotny, Scott and Gibbons 2003 ; Klein 2006 ; Hemlin and Rasmussen 2006 ; Chataway, Smith and Wield 2007 ; Erno-Kjolhede and Hansson 2011 ). Emerging fields such as sustainability science have developed out of a need to address complex and urgent real-world problems ( Komiyama and Takeuchi 2006 ). These approaches are inherently applied and transdisciplinary, with explicit goals to contribute to real-world solutions and strong emphasis on context and social engagement ( Kates 2000 ).

While there is an ongoing conceptual and theoretical debate about the nature of the relationship between science and society (e.g. Hessels 2008 ), we take a more practical starting point based on the authors’ experience in two research organizations. The first author has been involved with the Center for International Forestry Research (CIFOR) for almost 20 years. CIFOR, as part of the Consultative Group on International Agricultural Research (CGIAR), began a major transformation in 2010 that shifted the emphasis from a primary focus on delivering high-quality science to a focus on ‘…producing, assembling and delivering, in collaboration with research and development partners, research outputs that are international public goods which will contribute to the solution of significant development problems that have been identified and prioritized with the collaboration of developing countries.’ ( CGIAR 2011 ). It was always intended that CGIAR research would be relevant to priority development and conservation issues, with emphasis on high-quality scientific outputs. The new approach puts much stronger emphasis on welfare and environmental results; research centers, programs, and individual scientists now assume shared responsibility for achieving development outcomes. This requires new ways of working, with more and different kinds of partnerships and more deliberate and strategic engagement in social systems.

Royal Roads University (RRU), the home institute of all four authors, is a relatively new (created in 1995) public university in Canada. It is deliberately interdisciplinary by design, with just two faculties (Faculty of Social and Applied Science; Faculty of Management) and strong emphasis on problem-oriented research. Faculty and student research is typically ‘applied’ in the Organization for Economic Co-operation and Development (2012) sense of ‘original investigation undertaken in order to acquire new knowledge … directed primarily towards a specific practical aim or objective’.

An increasing amount of the research done within both of these organizations can be classified as transdisciplinary research (TDR). TDR crosses disciplinary and institutional boundaries, is context specific, and problem oriented ( Klein 2006 ; Carew and Wickson 2010 ). It combines and blends methodologies from different theoretical paradigms, includes a diversity of both academic and lay actors, and is conducted with a range of research goals, organizational forms, and outputs ( Klein 2006 ; Boix-Mansilla 2006a ; Erno-Kjolhede and Hansson 2011 ). The problem-oriented nature of TDR and the importance placed on societal relevance and engagement are broadly accepted as defining characteristics of TDR ( Carew and Wickson 2010 ).

The experience developing and using TDR approaches at CIFOR and RRU highlights the need for a parallel evolution of principles and criteria for evaluating research quality in a TDR context. Scientists appreciate and often welcome the need and the opportunity to expand the reach of their research, to contribute more effectively to change processes. At the same time, they feel the pressure of added expectations and are looking for guidance.

In any activity, we need principles, guidelines, criteria, or benchmarks that can be used to design the activity, assess its potential, and evaluate its progress and accomplishments. Effective research quality criteria are necessary to guide the funding, management, ongoing development, and advancement of research methods, projects, and programs. The lack of quality criteria to guide and assess research design and performance is seen as hindering the development of transdisciplinary approaches ( Bergmann et al. 2005 ; Feller 2006 ; Chataway, Smith and Wield 2007 ; Ozga 2008 ; Carew and Wickson 2010 ; Jahn and Keil 2015 ). Appropriate quality evaluation is essential to ensure that research receives support and funding, and to guide and train researchers and managers to realize high-quality research ( Boix-Mansilla 2006a ; Klein 2008 ; Aagaard-Hansen and Svedin 2009 ; Carew and Wickson 2010 ).

Traditional disciplinary research is built on well-established methodological and epistemological principles and practices. Within disciplinary research, quality has been defined narrowly, with the primary criteria being scientific excellence and scientific relevance ( Feller 2006 ; Chataway, Smith and Wield 2007 ; Erno-Kjolhede and Hansson 2011 ). Disciplines have well-established (often implicit) criteria and processes for the evaluation of quality in research design ( Erno-Kjolhede and Hansson 2011 ). TDR that is highly context specific, problem oriented, and includes nonacademic societal actors in the research process is challenging to evaluate ( Wickson, Carew and Russell 2006 ; Aagaard-Hansen and Svedin 2009 ; Andrén 2010 ; Carew and Wickson 2010 ; Huutoniemi 2010 ). There is no one definition or understanding of what constitutes quality, nor a set guide for how to do TDR ( Lincoln 1995 ; Morrow 2005 ; Oberg 2008 ; Andrén 2010 ; Huutoniemi 2010 ). When epistemologies and methods from more than one discipline are used, disciplinary criteria may be insufficient and criteria from more than one discipline may be contradictory; cultural conflicts can arise as a range of actors use different terminology for the same concepts or the same terminology for different concepts ( Chataway, Smith and Wield 2007 ; Oberg 2008 ).

Current research evaluation approaches as applied to individual researchers, programs, and research units are still based primarily on measures of academic outputs (publications and the prestige of the publishing journal), citations, and peer assessment ( Boix-Mansilla 2006a ; Feller 2006 ; Erno-Kjolhede and Hansson 2011 ). While these indicators of research quality remain relevant, additional criteria are needed to address the innovative approaches and the diversity of actors, outputs, outcomes, and long-term social impacts of TDR. It can be difficult to find appropriate outlets for TDR publications simply because the research does not meet the expectations of traditional discipline-oriented journals. Moreover, a wider range of inputs and of outputs means that TDR may result in fewer academic outputs. This has negative implications for transdisciplinary researchers, whose performance appraisals and long-term career progression are largely governed by traditional publication and citation-based metrics of evaluation. Research managers, peer reviewers, academic committees, and granting agencies all struggle with how to evaluate and how to compare TDR projects ( ex ante or ex post ) in the absence of appropriate criteria to address epistemological and methodological variability. The extent of engagement of stakeholders 1 in the research process will vary by project, from information sharing through to active collaboration ( Brandt et al. 2013) , but at any level, the involvement of stakeholders adds complexity to the conceptualization of quality. We need to know what ‘good research’ is in a transdisciplinary context.

As Tijssen ( 2003 : 93) put it: ‘Clearly, in view of its strategic and policy relevance, developing and producing generally acceptable measures of “research excellence” is one of the chief evaluation challenges of the years to come’. Clear criteria are needed for research quality evaluation to foster excellence while supporting innovation: ‘A principal barrier to a broader uptake of TD research is a lack of clarity on what good quality TD research looks like’ ( Carew and Wickson 2010 : 1154). In the absence of alternatives, many evaluators, including funding bodies, rely on conventional, discipline-specific measures of quality which do not address important aspects of TDR.

There is an emerging literature that reviews, synthesizes, or empirically evaluates knowledge and best practice in research evaluation in a TDR context and that proposes criteria and evaluation approaches ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Klein 2008 ; Carew and Wickson 2010 ; ERIC 2010; de Jong et al. 2011 ; Spaapen and Van Drooge 2011 ). Much of it comes from a few fields, including health care, education, and evaluation; little comes from the natural resource management and sustainability science realms, despite these areas needing guidance. National-scale reviews have begun to recognize the need for broader research evaluation criteria but have had difficulty dealing with it and have made little progress in addressing it ( Donovan 2008 ; KNAW 2009 ; REF 2011 ; ARC 2012 ; TEC 2012 ). A summary of the national reviews that we reviewed in the development of this research is provided in Supplementary Appendix 1 . While there are some published evaluation schemes for TDR and interdisciplinary research (IDR), there is ‘substantial variation in the balance different authors achieve between comprehensiveness and over-prescription’ ( Wickson and Carew 2014 : 256) and still a need to develop standardized quality criteria that are ‘uniquely flexible to provide valid, reliable means to evaluate and compare projects, while not stifling the evolution and responsiveness of the approach’ ( Wickson and Carew 2014 : 256).

There is a need and an opportunity to synthesize current ideas about how to define and assess quality in TDR. To address this, we conducted a systematic review of the literature that discusses the definitions of research quality as well as the suggested principles and criteria for assessing TDR quality. The aim is to identify appropriate principles and criteria for defining and measuring research quality in a transdisciplinary context and to organize those principles and criteria as an evaluation framework.

The review question was: What are appropriate principles, criteria, and indicators for defining and assessing research quality in TDR?

This article presents the method used for the systematic review and our synthesis, followed by key findings. Theoretical concepts about why new principles and criteria are needed for TDR, along with associated discussions about evaluation process are presented. A framework, derived from our synthesis of the literature, of principles and criteria for TDR quality evaluation is presented along with guidance on its application. Finally, recommendations for next steps in this research and needs for future research are discussed.

2.1 Systematic review

Systematic review is a rigorous, transparent, and replicable methodology that has become widely used to inform evidence-based policy, management, and decision making ( Pullin and Stewart 2006 ; CEE 2010). Systematic reviews follow a detailed protocol with explicit inclusion and exclusion criteria to ensure a repeatable and comprehensive review of the target literature. Review protocols are shared and often published as peer reviewed articles before undertaking the review to invite critique and suggestions. Systematic reviews are most commonly used to synthesize knowledge on an empirical question by collating data and analyses from a series of comparable studies, though methods used in systematic reviews are continually evolving and are increasingly being developed to explore a wider diversity of questions ( Chandler 2014 ). The current study question is theoretical and methodological, not empirical. Nevertheless, with a diverse and diffuse literature on the quality of TDR, a systematic review approach provides a method for a thorough and rigorous review. The protocol is published and available at http://www.cifor.org/online-library/browse/view-publication/publication/4382.html . A schematic diagram of the systematic review process is presented in Fig. 1 .

Search process.

2.2 Search terms

Search terms were designed to identify publications that discuss the evaluation or assessment of quality or excellence 2 of research 3 that is done in a TDR context. Search terms are listed online in Supplementary Appendices 2 and 3 . The search strategy favored sensitivity over specificity to ensure that we captured the relevant information.

2.3 Databases searched

ISI Web of Knowledge (WoK) and Scopus were searched between 26 June 2013 and 6 August 2013. The combined searches yielded 15,613 unique citations. Additional searches to update the first searchers were carried out in June 2014 and March 2015, for a total of 19,402 titles scanned. Google Scholar (GS) was searched separately by two reviewers during each search period. The first reviewer’s search was done on 2 September 2013 (Search 1) and 3 September 2013 (Search 2), yielding 739 and 745 titles, respectively. The second reviewer’s search was done on 19 November 2013 (Search 1) and 25 November 2013 (Search 2), yielding 769 and 774 titles, respectively. A third search done on 17 March 2015 by one reviewer yielded 98 new titles. Reviewers found high redundancy between the WoK/Scopus searches and the GS searches.

2.4 Targeted journal searches

Highly relevant journals, including Research Evaluation, Evaluation and Program Planning, Scientometrics, Research Policy, Futures, American Journal of Evaluation, Evaluation Review, and Evaluation, were comprehensively searched using broader, more inclusive search strings that would have been unmanageable for the main database search.

2.5 Supplementary searches

References in included articles were reviewed to identify additional relevant literature. td-net’s ‘Tour d’Horizon of Literature’, lists important inter- and transdisciplinary publications collected through an invitation to experts in the field to submit publications ( td-net 2014 ). Six additional articles were identified via supplementary search.

2.6 Limitations of coverage

The review was limited to English-language published articles and material available through internet searches. There was no systematic way to search the gray (unpublished) literature, but relevant material identified through supplementary searches was included.

2.7 Inclusion of articles

This study sought articles that review, critique, discuss, and/or propose principles, criteria, indicators, and/or measures for the evaluation of quality relevant to TDR. As noted, this yielded a large number of titles. We then selected only those articles with an explicit focus on the meaning of IDR and/or TDR quality and how to achieve, measure or evaluate it. Inclusion and exclusion criteria were developed through an iterative process of trial article screening and discussion within the research team. Through this process, inter-reviewer agreement was tested and strengthened. Inclusion criteria are listed in Tables 1 and 2 .

Inclusion criteria for title and abstract screening

Inclusion criteria for abstract and full article screening

Article screening was done in parallel by two reviewers in three rounds: (1) title, (2) abstract, and (3) full article. In cases of uncertainty, papers were included to the next round. Final decisions on inclusion of contested papers were made by consensus among the four team members.

2.8 Critical appraisal

In typical systematic reviews, individual articles are appraised to ensure that they are adequate for answering the research question and to assess the methods of each study for susceptibility to bias that could influence the outcome of the review (Petticrew and Roberts 2006). Most papers included in this review are theoretical and methodological papers, not empirical studies. Most do not have explicit methods that can be appraised with existing quality assessment frameworks. Our critical appraisal considered four criteria adapted from Spencer et al. (2003): (1) relevance to the review question, (2) clarity and logic of how information in the paper was generated, (3) significance of the contribution (are new ideas offered?), and (4) generalizability (is the context specified; do the ideas apply in other contexts?). Disagreements were discussed to reach consensus.

2.9 Data extraction and management

The review sought information on: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, principles of research quality, criteria for research quality assessment, indicators and measures of research quality, and processes for evaluating TDR. Four reviewers independently extracted data from selected articles using the parameters listed in Supplementary Appendix 4 .

2.10 Data synthesis and TDR framework design

Our aim was to synthesize ideas, definitions, and recommendations for TDR quality criteria into a comprehensive and generalizable framework for the evaluation of quality in TDR. Key ideas were extracted from each article and summarized in an Excel database. We classified these ideas into themes and ultimately into overarching principles and associated criteria of TDR quality organized as a rubric ( Wickson and Carew 2014 ). Definitions of each principle and criterion were developed and rubric statements formulated based on the literature and our experience. These criteria (adjusted appropriately to be applied ex ante or ex post ) are intended to be used to assess a TDR project. The reviewer should consider whether the project fully satisfies, partially satisfies, or fails to satisfy each criterion. More information on application is provided in Section 4.3 below.

We tested the framework on a set of completed RRU graduate theses that used transdisciplinary approaches, with an explicit problem orientation and intent to contribute to social or environmental change. Three rounds of testing were done, with revisions after each round to refine and improve the framework.

3.1 Overview of the selected articles

Thirty-eight papers satisfied the inclusion criteria. A wide range of terms are used in the selected papers, including: cross-disciplinary; interdisciplinary; transdisciplinary; methodological pluralism; mode 2; triple helix; and supradisciplinary. Eight included papers specifically focused on sustainability science or TDR in natural resource management, or identified sustainability research as a growing TDR field that needs new forms of evaluation ( Cash et al. 2002 ; Bergmann et al. 2005 ; Chataway, Smith and Wield 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Andrén 2010 ; Carew and Wickson 2010 ; Lang et al. 2012 ; Gaziulusoy and Boyle 2013 ). Carew and Wickson (2010) build on the experience in the TDR realm to propose criteria and indicators of quality for ‘responsible research and innovation’.

The selected articles are written from three main perspectives. One set is primarily interested in advancing TDR approaches. These papers recognize the need for new quality measures to encourage and promote high-quality research and to overcome perceived biases against TDR approaches in research funding and publishing. A second set of papers is written from an evaluation perspective, with a focus on improving evaluation of TDR. The third set is written from the perspective of qualitative research characterized by methodological pluralism, with many characteristics and issues relevant to TDR approaches.

The majority of the articles focus at the project scale, some at the organization level, and some do not specify. Some articles explicitly focus on ex ante evaluation (e.g. proposal evaluation), others on ex post evaluation, and many are not explicit about the project stage they are concerned with. The methods used in the reviewed articles include authors’ reflection and opinion, literature review, expert consultation, document analysis, and case study. Summaries of report characteristics are available online ( Supplementary Appendices 5–8 ). Eight articles provide comprehensive evaluation frameworks and quality criteria specifically for TDR and research-in-context. The rest of the articles discuss aspects of quality related to TDR and recommend quality definitions, criteria, and/or evaluation processes.

3.2 The need for quality criteria and evaluation methods for TDR

Many of the selected articles highlight the lack of widely agreed principles and criteria of TDR quality. They note that, in the absence of TDR quality frameworks, disciplinary criteria are used ( Morrow 2005 ; Boix-Mansilla 2006a , b ; Feller 2006 ; Klein 2006 , 2008 ; Wickson, Carew and Russell 2006 ; Scott 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Oberg 2008 ; Erno-Kjolhede and Hansson 2011 ), and evaluations are often carried out by reviewers who lack cross-disciplinary experience and do not have a shared understanding of quality ( Aagaard-Hansen and Svedin 2009 ). Quality is discussed by many as a relative concept, developed within disciplines, and therefore defined and understood differently in each field ( Morrow 2005 ; Klein 2006 ; Oberg 2008 ; Mitchell and Willets 2009 ; Huutoniemi 2010 ; Hellstrom 2011 ). Jahn and Keil (2015) point out the difficulty of creating a common set of quality criteria for TDR in the absence of a standard agreed-upon definition of TDR. Many of the selected papers argue the need to move beyond narrowly defined ideas of ‘scientific excellence’ to incorporate a broader assessment of quality which includes societal relevance ( Hemlin and Rasmussen 2006 ; Chataway, Smith and Wield 2007 ; Ozga 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ). This shift includes greater focus on research organization, research process, and continuous learning, rather than primarily on research outputs ( Hemlin and Rasmussen 2006 ; de Jong et al. 2011 ; Wickson and Carew 2014 ; Jahn and Keil 2015 ). This responds to and reflects societal expectations that research should be accountable and have demonstrated utility ( Cloete 1997 ; Defila and Di Giulio 1999 ; Wickson, Carew and Russell 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Stige 2009 ).

A central aim of TDR is to achieve socially relevant outcomes, and TDR quality criteria should demonstrate accountability to society ( Cloete 1997 ; Hemlin and Rasmussen 2006 ; Chataway, Smith and Wield 2007 ; Ozga 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ; de Jong et al. 2011 ). Integration and mutual learning are a core element of TDR; it is not enough to transcend boundaries and incorporate societal knowledge but, as Carew and Wickson ( 2010 : 1147) summarize: ‘…the TD researcher needs to put effort into integrating these potentially disparate knowledges with a view to creating useable knowledge. That is, knowledge that can be applied in a given problem context and has some prospect of producing desired change in that context’. The inclusion of societal actors in the research process, the unique and often dispersed organization of research teams, and the deliberate integration of different traditions of knowledge production all fall outside of conventional assessment criteria ( Feller 2006 ).

Not only do the range of criteria need to be updated, expanded, agreed upon, and assumptions made explicit ( Boix-Mansilla 2006a ; Klein 2006 ; Scott 2007 ) but, given the specific problem orientation of TDR, reviewers beyond disciplinary academic peers need to be included in the assessment of quality ( Cloete 1997 ; Scott 2007 ; Spappen et al. 2007 ; Klein 2008 ). Several authors discuss the lack of reviewers with strong cross-disciplinary experience ( Aagaard-Hansen and Svedin 2009 ) and the lack of common criteria, philosophical foundations, and language for use by peer reviewers ( Klein 2008 ; Aagaard-Hansen and Svedin 2009 ). Peer review of TDR could be improved with explicit TDR quality criteria, and appropriate processes in place to ensure clear dialog between reviewers.

Finally, there is the need for increased emphasis on evaluation as part of the research process ( Bergmann et al. 2005 ; Hemlin and Rasmussen 2006 ; Meyrick 2006 ; Chataway, Smith and Wield 2007 ; Stige, Malterud and Midtgarden 2009 ; Hellstrom 2011 ; Lang et al. 2012 ; Wickson and Carew 2014 ). This is particularly true in large, complex, problem-oriented research projects. Ongoing monitoring of the research organization and process contributes to learning and adaptive management while research is underway and so helps improve quality. As stated by Wickson and Carew ( 2014 : 262): ‘We believe that in any process of interpreting, rearranging and/or applying these criteria, open negotiation on their meaning and application would only positively foster transformative learning, which is a valued outcome of good TD processes’.

3.3 TDR quality criteria and assessment approaches

Many of the papers provide quality criteria and/or describe constituent parts of quality. Aagaard-Hansen and Svedin (2009) define three key aspects of quality: societal relevance, impact, and integration. Meyrick (2006) states that quality research is transparent and systematic. Boaz and Ashby (2003) describe quality in four dimensions: methodological quality, quality of reporting, appropriateness of methods, and relevance to policy and practice. Although each article deconstructs quality in different ways and with different foci and perspectives, there is significant overlap and recurring themes in the papers reviewed. There is a broadly shared perspective that TDR quality is a multidimensional concept shaped by the specific context within which research is done ( Spaapen, Dijstelbloem and Wamelink 2007 ; Klein 2008 ), making a universal definition of TDR quality difficult or impossible ( Huutoniemi 2010 ).

Huutoniemi (2010) identifies three main approaches to conceptualizing quality in IDR and TDR: (1) using existing disciplinary standards adapted as necessary for IDR; (2) building on the quality standards of disciplines while fundamentally incorporating ways to deal with epistemological integration, problem focus, context, stakeholders, and process; and (3) radical departure from any disciplinary orientation in favor of external, emergent, context-dependent quality criteria that are defined and enacted collaboratively by a community of users.

The first approach is prominent in current research funding and evaluation protocols. Conservative approaches of this kind are criticized for privileging disciplinary research and for failing to provide guidance and quality control for transdisciplinary projects. The third approach would ‘undermine the prevailing status of disciplinary standards in the pursuit of a non-disciplinary, integrated knowledge system’ ( Huutoniemi 2010 : 313). No predetermined quality criteria are offered, only contextually embedded criteria that need to be developed within a specific research project. To some extent, this is the approach taken by Spaapen, Dijstelbloem and Wamelink (2007) and de Jong et al. (2011) . Such a sui generis approach cannot be used to compare across projects. Most of the reviewed papers take the second approach, and recommend TDR quality criteria that build on a disciplinary base.

Eight articles present comprehensive frameworks for quality evaluation, each with a unique approach, perspective, and goal. Two of these build comprehensive lists of criteria with associated questions to be chosen based on the needs of the particular research project ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ). Wickson and Carew (2014) develop a reflective heuristic tool with questions to guide researchers through ongoing self-evaluation. They also list criteria for external evaluation and to compare between projects. Spaapen, Dijstelbloem and Wamelink (2007) design an approach to evaluate a research project against its own goals and is not meant to compare between projects. Wickson and Carew (2014) developed a comprehensive rubric for the evaluation of Research and Innovation that builds of their extensive previous work in TDR. Finally, Lang et al. (2012) , Mitchell and Willets (2009) , and Jahn and Keil (2015) develop criteria checklists that can be applied across transdisciplinary projects.

Bergmann et al. (2005) and Carew and Wickson (2010) organize their frameworks into managerial elements of the research project, concerning problem context, participation, management, and outcomes. Lang et al. (2012) and Defila and Di Giulio (1999) focus on the chronological stages in the research process and identify criteria at each stage. Mitchell and Willets (2009) , , with a focus on doctoral s tudies, adapt standard dissertation evaluation criteria to accommodate broader, pluralistic, and more complex studies. Spaapen, Dijstelbloem and Wamelink (2007) focus on evaluating ‘research-in-context’. Wickson and Carew (2014) created a rubric based on criteria that span the research process, stages, and all actors included. Jahn and Keil (2015) organized their quality criteria into three categories of quality including: quality of the research problems, quality of the research process, and quality of the research results.

The remaining papers highlight key themes that must be considered in TDR evaluation. Dominant themes include: engagement with problem context, collaboration and inclusion of stakeholders, heightened need for explicit communication and reflection, integration of epistemologies, recognition of diverse outputs, the focus on having an impact, and reflexivity and adaptation throughout the process. The focus on societal problems in context and the increased engagement of stakeholders in the research process introduces higher levels of complexity that cannot be accommodated by disciplinary standards ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Klein 2008 ).

Finally, authors discuss process ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ; Boix-Mansilla 2006b ; Spaapen, Dijstelbloem and Wamelink 2007 ) and utilitarian values ( Hemlin 2006 ; Ernø-Kjølhede and Hansson 2011 ; Bornmann 2013 ) as essential aspects of quality in TDR. Common themes include: (1) the importance of formative and process-oriented evaluation ( Bergmann et al. 2005 ; Hemlin 2006 ; Stige 2009 ); (2) emphasis on the evaluation process itself (not just criteria or outcomes) and reflexive dialog for learning ( Bergmann et al. 2005 ; Boix-Mansilla 2006b ; Klein 2008 ; Oberg 2008 ; Stige, Malterud and Midtgarden 2009 ; Aagaard-Hansen and Svedin 2009 ; Carew and Wickson 2010 ; Huutoniemi 2010 ); (3) the need for peers who are experienced and knowledgeable about TDR for fair peer review ( Boix-Mansilla 2006a , b ; Klein 2006 ; Hemlin 2006 ; Scott 2007 ; Aagaard-Hansen and Svedin 2009 ); (4) the inclusion of stakeholders in the evaluation process ( Bergmann et al. 2005 ; Scott 2007 ; Andréen 2010 ); and (5) the importance of evaluations that are built in-context ( Defila and Di Giulio 1999 ; Feller 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; de Jong et al. 2011 ).

While each reviewed approach offers helpful insights, none adequately fulfills the need for a broad and adaptable framework for assessing TDR quality. Wickson and Carew ( 2014 : 257) highlight the need for quality criteria that achieve balance between ‘comprehensiveness and over-prescription’: ‘any emerging quality criteria need to be concrete enough to provide real guidance but flexible enough to adapt to the specificities of varying contexts’. Based on our experience, such a framework should be:

Comprehensive: It should accommodate the main aspects of TDR, as identified in the review.

Time/phase adaptable: It should be applicable across the project cycle.

Scalable: It should be useful for projects of different scales.

Versatile: It should be useful to researchers and collaborators as a guide to research design and management, and to internal and external reviews and assessors.

Comparable: It should allow comparison of quality between and across projects/programs.

Reflexive: It should encourage and facilitate self-reflection and adaptation based on ongoing learning.

In this section, we synthesize the key principles and criteria of quality in TDR that were identified in the reviewed literature. Principles are the essential elements of high-quality TDR. Criteria are the conditions that need to be met in order to achieve a principle. We conclude by providing a framework for the evaluation of quality in TDR ( Table 3 ) and guidance for its application.

Transdisciplinary research quality assessment framework

a Research problems are the particular topic, area of concern, question to be addressed, challenge, opportunity, or focus of the research activity. Research problems are related to the societal problem but take on a specific focus, or framing, within a societal problem.

b Problem context refers to the social and environmental setting(s) that gives rise to the research problem, including aspects of: location; culture; scale in time and space; social, political, economic, and ecological/environmental conditions; resources and societal capacity available; uncertainty, complexity, and novelty associated with the societal problem; and the extent of agency that is held by stakeholders ( Carew and Wickson 2010 ).

c Words such as ‘appropriate’, ‘suitable’, and ‘adequate’ are used deliberately to allow for quality criteria to be flexible and specific enough to the needs of individual research projects ( Oberg 2008 ).

d Research process refers to the series of decisions made and actions taken throughout the entire duration of the research project and encompassing all aspects of the research project.

e Reflexivity refers to an iterative process of formative, critical reflection on the important interactions and relationships between a research project’s process, context, and product(s).

f In an ex ante evaluation, ‘evidence of’ would be replaced with ‘potential for’.

There is a strong trend in the reviewed articles to recognize the need for appropriate measures of scientific quality (usually adapted from disciplinary antecedants), but also to consider broader sets of criteria regarding the societal significance and applicability of research, and the need for engagement and representation of stakeholder values and knowledge. Cash et al. (2002) nicely conceptualize three key aspects of effective sustainability research as: salience (or relevance), credibility, and legitimacy. These are presented as necessary attributes for research to successfully produce transferable, useful information that can cross boundaries between disciplines, across scales, and between science and society. Many of the papers also refer to the principle that high-quality TDR should be effective in terms of contributing to the solution of problems. These four principles are discussed in the following sections.

4.1.1 Relevance

Relevance is the importance, significance, and usefulness of the research project's objectives, process, and findings to the problem context and to society. This includes the appropriateness of the timing of the research, the questions being asked, the outputs, and the scale of the research in relation to the societal problem being addressed. Good-quality TDR addresses important social/environmental problems and produces knowledge that is useful for decision making and problem solving ( Cash et al. 2002 ; Klein 2006 ). As Erno-Kjolhede and Hansson ( 2011 : 140) explain, quality ‘is first and foremost about creating results that are applicable and relevant for the users of the research’. Researchers must demonstrate an in-depth knowledge of and ongoing engagement with the problem context in which their research takes place ( Wickson, Carew and Russell 2006 ; Stige, Malterud and Midtgarden 2009 ; Mitchell and Willets 2009 ). From the early steps of problem formulation and research design through to the appropriate and effective communication of research findings, the applicability and relevance of the research to the societal problem must be explicitly stated and incorporated.

4.1.2 Credibility

Credibility refers to whether or not the research findings are robust and the knowledge produced is scientifically trustworthy. This includes clear demonstration that the data are adequate, with well-presented methods and logical interpretations of findings. High-quality research is authoritative, transparent, defensible, believable, and rigorous. This is the traditional purview of science, and traditional disciplinary criteria can be applied in TDR evaluation to an extent. Additional and modified criteria are needed to address the integration of epistemologies and methodologies and the development of novel methods through collaboration, the broad preparation and competencies required to carry out the research, and the need for reflection and adaptation when operating in complex systems. Having researchers actively engaged in the problem context and including extra-scientific actors as part of the research process helps to achieve relevance and legitimacy of the research; it also adds complexity and heightened requirements of transparency, reflection, and reflexivity to ensure objective, credible research is carried out.

Active reflexivity is a criterion of credibility of TDR that may seem to contradict more rigid disciplinary methodological traditions ( Carew and Wickson 2010 ). Practitioners of TDR recognize that credible work in these problem-oriented fields requires active reflexivity, epitomized by ongoing learning, flexibility, and adaptation to ensure the research approach and objectives remain relevant and fit-to-purpose ( Lincoln 1995 ; Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Mitchell and Willets 2009 ; Andreén 2010 ; Carew and Wickson 2010 ; Wickson and Carew 2014 ). Changes made during the research process must be justified and reported transparently and explicitly to maintain credibility.

The need for critical reflection on potential bias and limitations becomes more important to maintain credibility of research-in-context ( Lincoln 1995 ; Bergmann et al. 2005 ; Mitchell and Willets 2009 ; Stige, Malterud and Midtgarden 2009 ). Transdisciplinary researchers must ensure they maintain a high level of objectivity and transparency while actively engaging in the problem context. This point demonstrates the fine balance between different aspects of quality, in this case relevance and credibility, and the need to be aware of tensions and to seek complementarities ( Cash et al. 2002 ).

4.1.3 Legitimacy

Legitimacy refers to whether the research process is perceived as fair and ethical by end-users. In other words, is it acceptable and trustworthy in the eyes of those who will use it? This requires the appropriate inclusion and consideration of diverse values, interests, and the ethical and fair representation of all involved. Legitimacy may be achieved in part through the genuine inclusion of stakeholders in the research process. Whereas credibility refers to technical aspects of sound research, legitimacy deals with sociopolitical aspects of the knowledge production process and products of research. Do stakeholders trust the researchers and the research process, including funding sources and other sources of potential bias? Do they feel represented? Legitimate TDR ‘considers appropriate values, concerns, and perspectives of different actors’ ( Cash et al. 2002 : 2) and incorporates these perspectives into the research process through collaboration and mutual learning ( Bergmann et al. 2005 ; Chataway, Smith and Wield 2007 ; Andrén 2010 ; Huutoneimi 2010 ). A fair and ethical process is important to uphold standards of quality in all research. However, there are additional considerations that are unique to TDR.

Because TDR happens in-context and often in collaboration with societal actors, the disclosure of researcher perspective and a transparent statement of all partnerships, financing, and collaboration is vital to ensure an unbiased research process ( Lincoln 1995 ; Defila and Di Giulio 1999 ; Boaz and Ashby 2003 ; Barker and Pistrang 2005 ; Bergmann et al. 2005 ). The disclosure of perspective has both internal and external aspects, on one hand ensuring the researchers themselves explicitly reflect on and account for their own position, potential sources of bias, and limitations throughout the process, and on the other hand making the process transparent to those external to the research group who can then judge the legitimacy based on their perspective of fairness ( Cash et al. 2002 ).

TDR includes the engagement of societal actors along a continuum of participation from consultation to co-creation of knowledge ( Brandt et al. 2013 ). Regardless of the depth of participation, all processes that engage societal actors must ensure that inclusion/engagement is genuine, roles are explicit, and processes for effective and fair collaboration are present ( Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Hellstrom 2012 ). Important considerations include: the accurate representation of those involved; explicit and agreed-upon roles and contributions of actors; and adequate planning and procedures to ensure all values, perspectives, and contexts are adequately and appropriately incorporated. Mitchell and Willets (2009) consider cultural competence as a key criterion that can support researchers in navigating diverse epistemological perspectives. This is similar to what Morrow terms ‘social validity’, a criterion that asks researchers to be responsive to and critically aware of the diversity of perspectives and cultures influenced by their research. Several authors highlight that in order to develop this critical awareness of the diversity of cultural paradigms that operate within a problem situation, researchers should practice responsive, critical, and/or communal reflection ( Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Mitchell and Willets 2009 ; Carew and Wickson 2010 ). Reflection and adaptation are important quality criteria that cut across multiple principles and facilitate learning throughout the process, which is a key foundation to TD inquiry.

4.1.4 Effectiveness

We define effective research as research that contributes to positive change in the social, economic, and/or environmental problem context. Transdisciplinary inquiry is rooted in the objective of solving real-word problems ( Klein 2008 ; Carew and Wickson 2010 ) and must have the potential to ( ex ante ) or actually ( ex post ) make a difference if it is to be considered of high quality ( Erno-Kjolhede and Hansson 2011 ). Potential research effectiveness can be indicated and assessed at the proposal stage and during the research process through: a clear and stated intention to address and contribute to a societal problem, the establishment of the research process and objectives in relation to the problem context, and the continuous reflection on the usefulness of the research findings and products to the problem ( Bergmann et al. 2005 ; Lahtinen et al. 2005 ; de Jong et al. 2011 ).

Assessing research effectiveness ex post remains a major challenge, especially in complex transdisciplinary approaches. Conventional and widely used measures of ‘scientific impact’ count outputs such as journal articles and other publications and citations of those outputs (e.g. H index; i10 index). While these are useful indicators of scholarly influence, they are insufficient and inappropriate measures of research effectiveness where research aims to contribute to social learning and change. We need to also (or alternatively) focus on other kinds of research and scholarship outputs and outcomes and the social, economic, and environmental impacts that may result.

For many authors, contributing to learning and building of societal capacity are central goals of TDR ( Defila and Di Giulio 1999 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Carew and Wickson 2010 ; Erno-Kjolhede and Hansson 2011 ; Hellstrom 2011 ), and so are considered part of TDR effectiveness. Learning can be characterized as changes in knowledge, attitudes, or skills and can be assessed directly, or through observed behavioral changes and network and relationship development. Some evaluation methodologies (e.g. Outcome Mapping ( Earl, Carden and Smutylo 2001 )) specifically measure these kinds of changes. Other evaluation methodologies consider the role of research within complex systems and assess effectiveness in terms of contributions to changes in policy and practice and resulting social, economic, and environmental benefits ( ODI 2004 , 2012 ; White and Phillips 2012 ; Mayne et al. 2013 ).

4.2 TDR quality criteria

TDR quality criteria and their definitions (explicit or implicit) were extracted from each article and summarized in an Excel database. These criteria were classified into themes corresponding to the four principles identified above, sorted and refined to develop sets of criteria that are comprehensive, mutually exclusive, and representative of the ideas presented in the reviewed articles. Within each principle, the criteria are organized roughly in the sequence of a typical project cycle (e.g. with research design following problem identification and preceding implementation). Definitions of each criterion were developed to reflect the concepts found in the literature, tested and refined iteratively to improve clarity. Rubric statements were formulated based on the literature and our own experience.

The complete set of principles, criteria, and definitions is presented as the TDR Quality Assessment Framework ( Table 3 ).

4.3 Guidance on the application of the framework

4.3.1 timing.

Most criteria can be applied at each stage of the research process, ex ante , mid term, and ex post , using appropriate interpretations at each stage. Ex ante (i.e. proposal) assessment should focus on a project’s explicitly stated intentions and approaches to address the criteria. Mid-term indicators will focus on the research process and whether or not it is being implemented in a way that will satisfy the criteria. Ex post assessment should consider whether the research has been done appropriately for the purpose and that the desired results have been achieved.

4.3.2 New meanings for familiar terms

Many of the terms used in the framework are extensions of disciplinary criteria and share the same or similar names and perhaps similar but nuanced meaning. The principles and criteria used here extend beyond disciplinary antecedents and include new concepts and understandings that encapsulate the unique characteristics and needs of TDR and allow for evaluation and definition of quality in TDR. This is especially true in the criteria related to credibility. These criteria are analogous to traditional disciplinary criteria, but with much stronger emphasis on grounding in both the scientific and the social/environmental contexts. We urge readers to pay close attention to the definitions provided in Table 3 as well as the detailed descriptions of the principles in Section 4.1.

4.3.3 Using the framework

The TDR quality framework ( Table 3 ) is designed to be used to assess TDR research according to a project’s purpose; i.e. the criteria must be interpreted with respect to the context and goals of an individual research activity. The framework ( Table 3 ) lists the main criteria synthesized from the literature and our experience, organized within the principles of relevance, credibility, legitimacy, and effectiveness. The table presents the criteria within each principle, ordered to approximate a typical process of identifying a research problem and designing and implementing research. We recognize that the actual process in any given project will be iterative and will not necessarily follow this sequence, but this provides a logical flow. A concise definition is provided in the second column to explain each criterion. We then provide a rubric statement in the third column, phrased to be applied when the research has been completed. In most cases, the same statement can be used at the proposal stage with a simple tense change or other minor grammatical revision, except for the criteria relating to effectiveness. As discussed above, assessing effectiveness in terms of outcomes and/or impact requires evaluation research. At the proposal stage, it is only possible to assess potential effectiveness.

Many rubrics offer a set of statements for each criterion that represent progressively higher levels of achievement; the evaluator is asked to select the best match. In practice, this often results in vague and relative statements of merit that are difficult to apply. We have opted to present a single rubric statement in absolute terms for each criterion. The assessor can then rank how well a project satisfies each criterion using a simple three-point Likert scale. If a project fully satisfies a criterion—that is, if there is evidence that the criterion has been addressed in a way that is coherent, explicit, sufficient, and convincing—it should be ranked as a 2 for that criterion. A score of 2 means that the evaluator is persuaded that the project addressed that criterion in an intentional, appropriate, explicit, and thorough way. A score of 1 would be given when there is some evidence that the criterion was considered, but it is lacking completion, intention, and/or is not addressed satisfactorily. For example, a score of 1 would be given when a criterion is explicitly discussed but poorly addressed, or when there is some indication that the criterion has been considered and partially addressed but it has not been treated explicitly, thoroughly, or adequately. A score of 0 indicates that there is no evidence that the criterion was addressed or that it was addressed in a way that was misguided or inappropriate.

It is critical that the evaluation be done in context, keeping in mind the purpose, objectives, and resources of the project, as well as other contextual information, such as the intended purpose of grant funding or relevant partnerships. Each project will be unique in its complexities; what is sufficient or adequate in one criterion for one research project may be insufficient or inappropriate for another. Words such as ‘appropriate’, ‘suitable’, and ‘adequate’ are used deliberately to encourage application of criteria to suit the needs of individual research projects ( Oberg 2008 ). Evaluators must consider the objectives of the research project and the problem context within which it is carried out as the benchmark for evaluation. For example, we tested the framework with RRU masters theses. These are typically small projects with limited scope, carried out by a single researcher. Expectations for ‘effective communication’ or ‘competencies’ or ‘effective collaboration’ are much different in these kinds of projects than in a multi-year, multi-partner CIFOR project. All criteria should be evaluated through the lens of the stated research objectives, research goals, and context.

The systematic review identified relevant articles from a diverse literature that have a strong central focus. Collectively, they highlight the complexity of contemporary social and environmental problems and emphasize that addressing such issues requires combinations of new knowledge and innovation, action, and engagement. Traditional disciplinary research has often failed to provide solutions because it cannot adequately cope with complexity. New forms of research are proliferating, crossing disciplinary and academic boundaries, integrating methodologies, and engaging a broader range of research participants, as a way to make research more relevant and effective. Theoretically, such approaches appear to offer great potential to contribute to transformative change. However, because these approaches are new and because they are multidimensional, complex, and often unique, it has been difficult to know what works, how, and why. In the absence of the kinds of methodological and quality standards that guide disciplinary research, there are no generally agreed criteria for evaluating such research.

Criteria are needed to guide and to help ensure that TDR is of high quality, to inform the teaching and learning of new researchers, and to encourage and support the further development of transdisciplinary approaches. The lack of a standard and broadly applicable framework for the evaluation of quality in TDR is perceived to cause an implicit or explicit devaluation of high-quality TDR or may prevent quality TDR from being done. There is a demonstrated need for an operationalized understanding of quality that addresses the characteristics, contributions, and challenges of TDR. The reviewed articles approach the topic from different perspectives and fields of study, using different terminology for similar concepts, or the same terminology for different concepts, and with unique ways of organizing and categorizing the dimensions and quality criteria. We have synthesized and organized these concepts as key TDR principles and criteria in a TDR Quality Framework, presented as an evaluation rubric. We have tested the framework on a set of masters’ theses and found it to be broadly applicable, usable, and useful for analyzing individual projects and for comparing projects within the set. We anticipate that further testing with a wider range of projects will help further refine and improve the definitions and rubric statements. We found that the three-point Likert scale (0–2) offered sufficient variability for our purposes, and rating is less subjective than with relative rubric statements. It may be possible to increase the rating precision with more points on the scale to increase the sensitivity for comparison purposes, for example in a review of proposals for a particular grant application.

Many of the articles we reviewed emphasize the importance of the evaluation process itself. The formative, developmental role of evaluation in TDR is seen as essential to the goals of mutual learning as well as to ensure that research remains responsive and adaptive to the problem context. In order to adequately evaluate quality in TDR, the process, including who carries out the evaluations, when, and in what manner, must be revised to be suitable to the unique characteristics and objectives of TDR. We offer this review and synthesis, along with a proposed TDR quality evaluation framework, as a contribution to an important conversation. We hope that it will be useful to researchers and research managers to help guide research design, implementation and reporting, and to the community of research organizations, funders, and society at large. As underscored in the literature review, there is a need for an adapted research evaluation process that will help advance problem-oriented research in complex systems, ultimately to improve research effectiveness.

This work was supported by funding from the Canada Research Chairs program. Funding support from the Canadian Social Sciences and Humanities Research Council (SSHRC) and technical support from the Evidence Based Forestry Initiative of the Centre for International Forestry Research (CIFOR), funded by UK DfID are also gratefully acknowledged.

Supplementary data is available here

The authors thank Barbara Livoreil and Stephen Dovers for valuable comments and suggestions on the protocol and Gillian Petrokofsky for her review of the protocol and a draft version of the manuscript. Two anonymous reviewers and the editor provided insightful critique and suggestions in two rounds that have helped to substantially improve the article.

Conflict of interest statement . None declared.

1. ‘Stakeholders’ refers to individuals and groups of societal actors who have an interest in the issue or problem that the research seeks to address.

2. The terms ‘quality’ and ‘excellence’ are often used in the literature with similar meaning. Technically, ‘excellence’ is a relative concept, referring to the superiority of a thing compared to other things of its kind. Quality is an attribute or a set of attributes of a thing. We are interested in what these attributes are or should be in high-quality research. Therefore, the term ‘quality’ is used in this discussion.

3. The terms ‘science’ and ‘research’ are not always clearly distinguished in the literature. We take the position that ‘science’ is a more restrictive term that is properly applied to systematic investigations using the scientific method. ‘Research’ is a broader term for systematic investigations using a range of methods, including but not restricted to the scientific method. We use the term ‘research’ in this broad sense.

Aagaard-Hansen J. Svedin U. ( 2009 ) ‘Quality Issues in Cross-disciplinary Research: Towards a Two-pronged Approach to Evaluation’ , Social Epistemology , 23 / 2 : 165 – 76 . DOI: 10.1080/02691720902992323

Google Scholar

Andrén S. ( 2010 ) ‘A Transdisciplinary, Participatory and Action-Oriented Research Approach: Sounds Nice but What do you Mean?’ [unpublished working paper] Human Ecology Division: Lund University, 1–21. < https://lup.lub.lu.se/search/publication/1744256 >

Australian Research Council (ARC) ( 2012 ) ERA 2012 Evaluation Handbook: Excellence in Research for Australia . Australia : ARC . < http://www.arc.gov.au/pdf/era12/ERA%202012%20Evaluation%20Handbook_final%20for%20web_protected.pdf >

Google Preview

Balsiger P. W. ( 2004 ) ‘Supradisciplinary Research Practices: History, Objectives and Rationale’ , Futures , 36 / 4 : 407 – 21 .

Bantilan M. C. et al.  . ( 2004 ) ‘Dealing with Diversity in Scientific Outputs: Implications for International Research Evaluation’ , Research Evaluation , 13 / 2 : 87 – 93 .

Barker C. Pistrang N. ( 2005 ) ‘Quality Criteria under Methodological Pluralism: Implications for Conducting and Evaluating Research’ , American Journal of Community Psychology , 35 / 3-4 : 201 – 12 .

Bergmann M. et al.  . ( 2005 ) Quality Criteria of Transdisciplinary Research: A Guide for the Formative Evaluation of Research Projects . Central report of Evalunet – Evaluation Network for Transdisciplinary Research. Frankfurt am Main, Germany: Institute for Social-Ecological Research. < http://www.isoe.de/ftp/evalunet_guide.pdf >

Boaz A. Ashby D. ( 2003 ) Fit for Purpose? Assessing Research Quality for Evidence Based Policy and Practice .

Boix-Mansilla V. ( 2006a ) ‘Symptoms of Quality: Assessing Expert Interdisciplinary Work at the Frontier: An Empirical Exploration’ , Research Evaluation , 15 / 1 : 17 – 29 .

Boix-Mansilla V. . ( 2006b ) ‘Conference Report: Quality Assessment in Interdisciplinary Research and Education’ , Research Evaluation , 15 / 1 : 69 – 74 .

Bornmann L. ( 2013 ) ‘What is Societal Impact of Research and How can it be Assessed? A Literature Survey’ , Journal of the American Society for Information Science and Technology , 64 / 2 : 217 – 33 .

Brandt P. et al.  . ( 2013 ) ‘A Review of Transdisciplinary Research in Sustainability Science’ , Ecological Economics , 92 : 1 – 15 .

Cash D. Clark W.C. Alcock F. Dickson N. M. Eckley N. Jäger J . ( 2002 ) Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making (November 2002). KSG Working Papers Series RWP02-046. Available at SSRN: http://ssrn.com/abstract=372280 .

Carew A. L. Wickson F. ( 2010 ) ‘The TD Wheel: A Heuristic to Shape, Support and Evaluate Transdisciplinary Research’ , Futures , 42 / 10 : 1146 – 55 .

Collaboration for Environmental Evidence (CEE) . ( 2013 ) Guidelines for Systematic Review and Evidence Synthesis in Environmental Management . Version 4.2. Environmental Evidence < www.environmentalevidence.org/Documents/Guidelines/Guidelines4.2.pdf >

Chandler J. ( 2014 ) Methods Research and Review Development Framework: Policy, Structure, and Process . < http://methods.cochrane.org/projects-developments/research >

Chataway J. Smith J. Wield D. ( 2007 ) ‘Shaping Scientific Excellence in Agricultural Research’ , International Journal of Biotechnology 9 / 2 : 172 – 87 .

Clark W. C. Dickson N. ( 2003 ) ‘Sustainability Science: The Emerging Research Program’ , PNAS 100 / 14 : 8059 – 61 .

Consultative Group on International Agricultural Research (CGIAR) ( 2011 ) A Strategy and Results Framework for the CGIAR . < http://library.cgiar.org/bitstream/handle/10947/2608/Strategy_and_Results_Framework.pdf?sequence=4 >

Cloete N. ( 1997 ) ‘Quality: Conceptions, Contestations and Comments’, African Regional Consultation Preparatory to the World Conference on Higher Education , Dakar, Senegal, 1-4 April 1997 .

Defila R. DiGiulio A. ( 1999 ) ‘Evaluating Transdisciplinary Research,’ Panorama: Swiss National Science Foundation Newsletter , 1 : 4 – 27 . < www.ikaoe.unibe.ch/forschung/ip/Specialissue.Pano.1.99.pdf >

Donovan C. ( 2008 ) ‘The Australian Research Quality Framework: A Live Experiment in Capturing the Social, Economic, Environmental, and Cultural Returns of Publicly Funded Research. Reforming the Evaluation of Research’ , New Directions for Evaluation , 118 : 47 – 60 .

Earl S. Carden F. Smutylo T. ( 2001 ) Outcome Mapping. Building Learning and Reflection into Development Programs . Ottawa, ON : International Development Research Center .

Ernø-Kjølhede E. Hansson F. ( 2011 ) ‘Measuring Research Performance during a Changing Relationship between Science and Society’ , Research Evaluation , 20 / 2 : 130 – 42 .

Feller I. ( 2006 ) ‘Assessing Quality: Multiple Actors, Multiple Settings, Multiple Criteria: Issues in Assessing Interdisciplinary Research’ , Research Evaluation 15 / 1 : 5 – 15 .

Gaziulusoy A. İ. Boyle C. ( 2013 ) ‘Proposing a Heuristic Reflective Tool for Reviewing Literature in Transdisciplinary Research for Sustainability’ , Journal of Cleaner Production , 48 : 139 – 47 .

Gibbons M. et al.  . ( 1994 ) The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies . London : Sage Publications .

Hellstrom T. ( 2011 ) ‘Homing in on Excellence: Dimensions of Appraisal in Center of Excellence Program Evaluations’ , Evaluation , 17 / 2 : 117 – 31 .

Hellstrom T. . ( 2012 ) ‘Epistemic Capacity in Research Environments: A Framework for Process Evaluation’ , Prometheus , 30 / 4 : 395 – 409 .

Hemlin S. Rasmussen S. B . ( 2006 ) ‘The Shift in Academic Quality Control’ , Science, Technology & Human Values , 31 / 2 : 173 – 98 .

Hessels L. K. Van Lente H. ( 2008 ) ‘Re-thinking New Knowledge Production: A Literature Review and a Research Agenda’ , Research Policy , 37 / 4 , 740 – 60 .

Huutoniemi K. ( 2010 ) ‘Evaluating Interdisciplinary Research’ , in Frodeman R. Klein J. T. Mitcham C. (eds) The Oxford Handbook of Interdisciplinarity , pp. 309 – 20 . Oxford : Oxford University Press .

de Jong S. P. L. et al.  . ( 2011 ) ‘Evaluation of Research in Context: An Approach and Two Cases’ , Research Evaluation , 20 / 1 : 61 – 72 .

Jahn T. Keil F. ( 2015 ) ‘An Actor-Specific Guideline for Quality Assurance in Transdisciplinary Research’ , Futures , 65 : 195 – 208 .

Kates R. ( 2000 ) ‘Sustainability Science’ , World Academies Conference Transition to Sustainability in the 21st Century 5/18/00 , Tokyo, Japan .

Klein J. T . ( 2006 ) ‘Afterword: The Emergent Literature on Interdisciplinary and Transdisciplinary Research Evaluation’ , Research Evaluation , 15 / 1 : 75 – 80 .

Klein J. T . ( 2008 ) ‘Evaluation of Interdisciplinary and Transdisciplinary Research: A Literature Review’ , American Journal of Preventive Medicine , 35 / 2 Supplment S116–23. DOI: 10.1016/j.amepre.2008.05.010

Royal Netherlands Academy of Arts and Sciences, Association of Universities in the Netherlands, Netherlands Organization for Scientific Research (KNAW) . ( 2009 ) Standard Evaluation Protocol 2009-2015: Protocol for Research Assessment in the Netherlands . Netherlands : KNAW . < www.knaw.nl/sep >

Komiyama H. Takeuchi K. ( 2006 ) ‘Sustainability Science: Building a New Discipline’ , Sustainability Science , 1 : 1 – 6 .

Lahtinen E. et al.  . ( 2005 ) ‘The Development of Quality Criteria For Research: A Finnish approach’ , Health Promotion International , 20 / 3 : 306 – 15 .

Lang D. J. et al.  . ( 2012 ) ‘Transdisciplinary Research in Sustainability Science: Practice , Principles , and Challenges’, Sustainability Science , 7 / S1 : 25 – 43 .

Lincoln Y. S . ( 1995 ) ‘Emerging Criteria for Quality in Qualitative and Interpretive Research’ , Qualitative Inquiry , 1 / 3 : 275 – 89 .

Mayne J. Stern E. ( 2013 ) Impact Evaluation of Natural Resource Management Research Programs: A Broader View . Australian Centre for International Agricultural Research, Canberra .

Meyrick J . ( 2006 ) ‘What is Good Qualitative Research? A First Step Towards a Comprehensive Approach to Judging Rigour/Quality’ , Journal of Health Psychology , 11 / 5 : 799 – 808 .

Mitchell C. A. Willetts J. R. ( 2009 ) ‘Quality Criteria for Inter and Trans - Disciplinary Doctoral Research Outcomes’ , in Prepared for ALTC Fellowship: Zen and the Art of Transdisciplinary Postgraduate Studies ., Sydney : Institute for Sustainable Futures, University of Technology .

Morrow S. L . ( 2005 ) ‘Quality and Trustworthiness in Qualitative Research in Counseling Psychology’ , Journal of Counseling Psychology , 52 / 2 : 250 – 60 .

Nowotny H. Scott P. Gibbons M. ( 2001 ) Re-Thinking Science . Cambridge : Polity .

Nowotny H. Scott P. Gibbons M. . ( 2003 ) ‘‘Mode 2’ Revisited: The New Production of Knowledge’ , Minerva , 41 : 179 – 94 .

Öberg G . ( 2008 ) ‘Facilitating Interdisciplinary Work: Using Quality Assessment to Create Common Ground’ , Higher Education , 57 / 4 : 405 – 15 .

Ozga J . ( 2007 ) ‘Co - production of Quality in the Applied Education Research Scheme’ , Research Papers in Education , 22 / 2 : 169 – 81 .

Ozga J . ( 2008 ) ‘Governing Knowledge: research steering and research quality’ , European Educational Research Journal , 7 / 3 : 261 – 272 .

OECD ( 2012 ) Frascati Manual 6th ed. < http://www.oecd.org/innovation/inno/frascatimanualproposedstandardpracticeforsurveysonresearchandexperimentaldevelopment6thedition >

Overseas Development Institute (ODI) ( 2004 ) ‘Bridging Research and Policy in International Development: An Analytical and Practical Framework’, ODI Briefing Paper. < http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/198.pdf >

Overseas Development Institute (ODI) . ( 2012 ) RAPID Outcome Assessment Guide . < http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/7815.pdf >

Pullin A. S. Stewart G. B. ( 2006 ) ‘Guidelines for Systematic Review in Conservation and Environmental Management’ , Conservation Biology , 20 / 6 : 1647 – 56 .

Research Excellence Framework (REF) . ( 2011 ) Research Excellence Framework 2014: Assessment Framework and Guidance on Submissions. Reference REF 02.2011. UK: REF. < http://www.ref.ac.uk/pubs/2011-02/ >

Scott A . ( 2007 ) ‘Peer Review and the Relevance of Science’ , Futures , 39 / 7 : 827 – 45 .

Spaapen J. Dijstelbloem H. Wamelink F. ( 2007 ) Evaluating Research in Context: A Method for Comprehensive Assessment . Netherlands: Consultative Committee of Sector Councils for Research and Development. < http://www.qs.univie.ac.at/fileadmin/user_upload/qualitaetssicherung/PDF/Weitere_Aktivit%C3%A4ten/Eric.pdf >

Spaapen J. Van Drooge L. ( 2011 ) ‘Introducing “Productive Interactions” in Social Impact Assessment’ , Research Evaluation , 20 : 211 – 18 .

Stige B. Malterud K. Midtgarden T. ( 2009 ) ‘Toward an Agenda for Evaluation of Qualitative Research’ , Qualitative Health Research , 19 / 10 : 1504 – 16 .

td-net ( 2014 ) td-net. < www.transdisciplinarity.ch/e/Bibliography/new.php >

Tertiary Education Commission (TEC) . ( 2012 ) Performance-based Research Fund: Quality Evaluation Guidelines 2012. New Zealand: TEC. < http://www.tec.govt.nz/Documents/Publications/PBRF-Quality-Evaluation-Guidelines-2012.pdf >

Tijssen R. J. W. ( 2003 ) ‘Quality Assurance: Scoreboards of Research Excellence’ , Research Evaluation , 12 : 91 – 103 .

White H. Phillips D. ( 2012 ) ‘Addressing Attribution of Cause and Effect in Small n Impact Evaluations: Towards an Integrated Framework’. Working Paper 15. New Delhi: International Initiative for Impact Evaluation .

Wickson F. Carew A. ( 2014 ) ‘Quality Criteria and Indicators for Responsible Research and Innovation: Learning from Transdisciplinarity’ , Journal of Responsible Innovation , 1 / 3 : 254 – 73 .

Wickson F. Carew A. Russell A. W. ( 2006 ) ‘Transdisciplinary Research: Characteristics, Quandaries and Quality,’ Futures , 38 / 9 : 1046 – 59

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• How to Write a Thesis Statement | 4 Steps & Examples

How to Write a Thesis Statement | 4 Steps & Examples

Published on January 11, 2019 by Shona McCombes . Revised on August 15, 2023 by Eoghan Ryan.

A thesis statement is a sentence that sums up the central point of your paper or essay . It usually comes near the end of your introduction .

Your thesis will look a bit different depending on the type of essay you’re writing. But the thesis statement should always clearly state the main idea you want to get across. Everything else in your essay should relate back to this idea.

You can write your thesis statement by following four simple steps:

• Start with a question
• Write your initial answer
• Develop your answer
• Refine your thesis statement

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is a thesis statement, placement of the thesis statement, step 1: start with a question, step 2: write your initial answer, step 3: develop your answer, step 4: refine your thesis statement, types of thesis statements, other interesting articles, frequently asked questions about thesis statements.

A thesis statement summarizes the central points of your essay. It is a signpost telling the reader what the essay will argue and why.

The best thesis statements are:

• Concise: A good thesis statement is short and sweet—don’t use more words than necessary. State your point clearly and directly in one or two sentences.
• Contentious: Your thesis shouldn’t be a simple statement of fact that everyone already knows. A good thesis statement is a claim that requires further evidence or analysis to back it up.
• Coherent: Everything mentioned in your thesis statement must be supported and explained in the rest of your paper.

Prevent plagiarism. Run a free check.

The thesis statement generally appears at the end of your essay introduction or research paper introduction .

The spread of the internet has had a world-changing effect, not least on the world of education. The use of the internet in academic contexts and among young people more generally is hotly debated. For many who did not grow up with this technology, its effects seem alarming and potentially harmful. This concern, while understandable, is misguided. The negatives of internet use are outweighed by its many benefits for education: the internet facilitates easier access to information, exposure to different perspectives, and a flexible learning environment for both students and teachers.

You should come up with an initial thesis, sometimes called a working thesis , early in the writing process . As soon as you’ve decided on your essay topic , you need to work out what you want to say about it—a clear thesis will give your essay direction and structure.

You might already have a question in your assignment, but if not, try to come up with your own. What would you like to find out or decide about your topic?

For example, you might ask:

After some initial research, you can formulate a tentative answer to this question. At this stage it can be simple, and it should guide the research process and writing process .

Now you need to consider why this is your answer and how you will convince your reader to agree with you. As you read more about your topic and begin writing, your answer should get more detailed.

In your essay about the internet and education, the thesis states your position and sketches out the key arguments you’ll use to support it.

The negatives of internet use are outweighed by its many benefits for education because it facilitates easier access to information.

In your essay about braille, the thesis statement summarizes the key historical development that you’ll explain.

The invention of braille in the 19th century transformed the lives of blind people, allowing them to participate more actively in public life.

A strong thesis statement should tell the reader:

• Why you hold this position
• What they’ll learn from your essay
• The key points of your argument or narrative

The final thesis statement doesn’t just state your position, but summarizes your overall argument or the entire topic you’re going to explain. To strengthen a weak thesis statement, it can help to consider the broader context of your topic.

These examples are more specific and show that you’ll explore your topic in depth.

Your thesis statement should match the goals of your essay, which vary depending on the type of essay you’re writing:

• In an argumentative essay , your thesis statement should take a strong position. Your aim in the essay is to convince your reader of this thesis based on evidence and logical reasoning.
• In an expository essay , you’ll aim to explain the facts of a topic or process. Your thesis statement doesn’t have to include a strong opinion in this case, but it should clearly state the central point you want to make, and mention the key elements you’ll explain.

If you want to know more about AI tools , college essays , or fallacies make sure to check out some of our other articles with explanations and examples or go directly to our tools!

• Ad hominem fallacy
• Post hoc fallacy
• Appeal to authority fallacy
• False cause fallacy
• Sunk cost fallacy

College essays

• Choosing Essay Topic
• Write a College Essay
• Write a Diversity Essay
• College Essay Format & Structure
• Comparing and Contrasting in an Essay

 (AI) Tools

• Grammar Checker
• Paraphrasing Tool
• Text Summarizer
• AI Detector
• Plagiarism Checker
• Citation Generator

A thesis statement is a sentence that sums up the central point of your paper or essay . Everything else you write should relate to this key idea.

The thesis statement is essential in any academic essay or research paper for two main reasons:

• It gives your writing direction and focus.
• It gives the reader a concise summary of your main point.

Without a clear thesis statement, an essay can end up rambling and unfocused, leaving your reader unsure of exactly what you want to say.

Follow these four steps to come up with a thesis statement :

• Ask a question about your topic .
• Write your initial answer.
• Develop your answer by including reasons.
• Refine your answer, adding more detail and nuance.

The thesis statement should be placed at the end of your essay introduction .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, August 15). How to Write a Thesis Statement | 4 Steps & Examples. Scribbr. Retrieved September 9, 2024, from https://www.scribbr.com/academic-essay/thesis-statement/

Is this article helpful?

Shona McCombes

Other students also liked, how to write an essay introduction | 4 steps & examples, how to write topic sentences | 4 steps, examples & purpose, academic paragraph structure | step-by-step guide & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Adv Pract Oncol
• v.12(4); 2021 May

Quality Improvement Projects and Clinical Research Studies

Every day, I witness firsthand the amazing things that advanced practitioners and nurse scientists accomplish. Through the conduct of quality improvement (QI) projects and clinical research studies, advanced practitioners and nurse scientists have the opportunity to contribute exponentially not only to their organizations, but also towards personal and professional growth.

Recently, the associate editors and staff at JADPRO convened to discuss the types of articles our readership may be interested in. Since we at JADPRO believe that QI projects and clinical research studies are highly valuable methods to improve clinical processes or seek answers to questions, you will see that we have highlighted various QI and research projects within the Research and Scholarship column of this and future issues. There have also been articles published in JADPRO about QI and research ( Gillespie, 2018 ; Kurtin & Taher, 2020 ). As a refresher, let’s explore the differences between a QI project and clinical research.

Quality Improvement

As leaders in health care, advanced practitioners often conduct QI projects to improve their internal processes or streamline clinical workflow. These QI projects use a multidisciplinary team comprising a team leader as well as nurses, PAs, pharmacists, physicians, social workers, and program administrators to address important questions that impact patients. Since QI projects use strategic processes and methods to analyze existing data and all patients participate, institutional review board (IRB) approval is usually not needed. Common frameworks, such as Lean, Six Sigma, and the Model for Improvement can be used. An attractive aspect of QI projects is that these are generally quicker to conduct and report on than clinical research, and often with quantifiable benefits to a large group within a system ( Table 1 ).

Clinical Research

Conducting clinical research through an IRB-approved study is another area in which advanced practitioners and nurse scientists gain new knowledge and contribute to scientific evidence-based practice. Research is intended for specific groups of patients who are protected from harm through the IRB and ethical principles. Research can potentially benefit a larger group, but benefits to participants are often unknown during the study period.

Clinical research poses many challenges at various stages of what can be a lengthy process. First, the researcher conducts a review of the literature to identify gaps in existing knowledge. Then, the researcher must be diligent in their self-reflection (is this phenomenon worth studying?) and in developing the sampling and statistical methods to ensure validity and reliability of the research ( Higgins & Straub, 2006 ). A team of additional researchers and support staff is integral to completing the research and disseminating findings. A well-designed clinical trial is worth the time and effort it takes to answer important clinical questions.

So, as an advanced practitioner, would a QI project be better to conduct than a clinical research study? That depends. A QI project uses a specific process, measures, and existing data to improve outcomes in a specific group. A research study uses an IRB-approved study protocol, strategic methods, and generates new data to hopefully benefit a larger group.

In This Issue

Both QI projects and clinical research can provide evidence to base one’s interventions on and enhance the lives of patients in one way or another. I hope you will agree that this issue is filled with valuable information on a wide range of topics. In the following pages, you will learn about findings of a QI project to integrate palliative care into ambulatory oncology. In a phenomenological study, Carrasco explores patient communication preferences around cancer symptom reporting during cancer treatment.

We have two excellent review articles for you as well. Rogers and colleagues review the management of hematologic adverse events of immune checkpoint inhibitors, and Lemke reviews the evidence for use of ginseng in the management of cancer-related fatigue. In Grand Rounds, Flagg and Pierce share an interesting case of essential thrombocythemia in a 15-year-old, with valuable considerations in the pediatric population. May and colleagues review practical considerations for integrating biosimilars into clinical practice, and Moore and Thompson review BTK inhibitors in B-cell malignancies.

• Higgins P. A., & Straub A. J. (2006). Understanding the error of our ways: Mapping the concepts of validity and reliability . Nursing Outlook , 54 ( 1 ), 23–29. 10.1016/j.outlook.2004.12.004 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gillespie T. W. (2018). Do the right study: Quality improvement projects and human subject research—both valuable, simply different . Journal of the Advanced Practitioner in Oncology , 9 ( 5 ), 471–473. 10.6004/jadpro.2018.9.5.1 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kurtin S. E., & Taher R. (2020). Clinical trial design and drug approval in oncology: A primer for the advanced practitioner in oncology . Journal of the Advanced Practitioner in Oncology , 11 ( 7 ), 736–751. 10.6004/jadpro.2020.11.7.7 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

Research Question Examples 🧑🏻‍🏫

25+ Practical Examples & Ideas To Help You Get Started 

By: Derek Jansen (MBA) | October 2023

A well-crafted research question (or set of questions) sets the stage for a robust study and meaningful insights.  But, if you’re new to research, it’s not always clear what exactly constitutes a good research question. In this post, we’ll provide you with clear examples of quality research questions across various disciplines, so that you can approach your research project with confidence!

Research Question Examples

• Psychology research questions
• Business research questions
• Education research questions
• Healthcare research questions
• Computer science research questions

Examples: Psychology

Let’s start by looking at some examples of research questions that you might encounter within the discipline of psychology.

How does sleep quality affect academic performance in university students?

This question is specific to a population (university students) and looks at a direct relationship between sleep and academic performance, both of which are quantifiable and measurable variables.

What factors contribute to the onset of anxiety disorders in adolescents?

The question narrows down the age group and focuses on identifying multiple contributing factors. There are various ways in which it could be approached from a methodological standpoint, including both qualitatively and quantitatively.

Do mindfulness techniques improve emotional well-being?

This is a focused research question aiming to evaluate the effectiveness of a specific intervention.

How does early childhood trauma impact adult relationships?

This research question targets a clear cause-and-effect relationship over a long timescale, making it focused but comprehensive.

Is there a correlation between screen time and depression in teenagers?

This research question focuses on an in-demand current issue and a specific demographic, allowing for a focused investigation. The key variables are clearly stated within the question and can be measured and analysed (i.e., high feasibility).

Examples: Business/Management

Next, let’s look at some examples of well-articulated research questions within the business and management realm.

How do leadership styles impact employee retention?

This is an example of a strong research question because it directly looks at the effect of one variable (leadership styles) on another (employee retention), allowing from a strongly aligned methodological approach.

What role does corporate social responsibility play in consumer choice?

Current and precise, this research question can reveal how social concerns are influencing buying behaviour by way of a qualitative exploration.

Does remote work increase or decrease productivity in tech companies?

Focused on a particular industry and a hot topic, this research question could yield timely, actionable insights that would have high practical value in the real world.

How do economic downturns affect small businesses in the homebuilding industry?

Vital for policy-making, this highly specific research question aims to uncover the challenges faced by small businesses within a certain industry.

Which employee benefits have the greatest impact on job satisfaction?

By being straightforward and specific, answering this research question could provide tangible insights to employers.

Examples: Education

Next, let’s look at some potential research questions within the education, training and development domain.

How does class size affect students’ academic performance in primary schools?

This example research question targets two clearly defined variables, which can be measured and analysed relatively easily.

Do online courses result in better retention of material than traditional courses?

Timely, specific and focused, answering this research question can help inform educational policy and personal choices about learning formats.

What impact do US public school lunches have on student health?

Targeting a specific, well-defined context, the research could lead to direct changes in public health policies.

To what degree does parental involvement improve academic outcomes in secondary education in the Midwest?

This research question focuses on a specific context (secondary education in the Midwest) and has clearly defined constructs.

What are the negative effects of standardised tests on student learning within Oklahoma primary schools?

This research question has a clear focus (negative outcomes) and is narrowed into a very specific context.

Need a helping hand?

Examples: Healthcare

Shifting to a different field, let’s look at some examples of research questions within the healthcare space.

What are the most effective treatments for chronic back pain amongst UK senior males?

Specific and solution-oriented, this research question focuses on clear variables and a well-defined context (senior males within the UK).

How do different healthcare policies affect patient satisfaction in public hospitals in South Africa?

This question is has clearly defined variables and is narrowly focused in terms of context.

Which factors contribute to obesity rates in urban areas within California?

This question is focused yet broad, aiming to reveal several contributing factors for targeted interventions.

Does telemedicine provide the same perceived quality of care as in-person visits for diabetes patients?

Ideal for a qualitative study, this research question explores a single construct (perceived quality of care) within a well-defined sample (diabetes patients).

Which lifestyle factors have the greatest affect on the risk of heart disease?

This research question aims to uncover modifiable factors, offering preventive health recommendations.

Examples: Computer Science

Last but certainly not least, let’s look at a few examples of research questions within the computer science world.

What are the perceived risks of cloud-based storage systems?

Highly relevant in our digital age, this research question would align well with a qualitative interview approach to better understand what users feel the key risks of cloud storage are.

Which factors affect the energy efficiency of data centres in Ohio?

With a clear focus, this research question lays a firm foundation for a quantitative study.

How do TikTok algorithms impact user behaviour amongst new graduates?

While this research question is more open-ended, it could form the basis for a qualitative investigation.

What are the perceived risk and benefits of open-source software software within the web design industry?

Practical and straightforward, the results could guide both developers and end-users in their choices.

Remember, these are just examples…

In this post, we’ve tried to provide a wide range of research question examples to help you get a feel for what research questions look like in practice. That said, it’s important to remember that these are just examples and don’t necessarily equate to good research topics . If you’re still trying to find a topic, check out our topic megalist for inspiration.

Psst... there’s more!

This post was based on one of our popular Research Bootcamps . If you're working on a research project, you'll definitely want to check this out ...

Research ideas on Political Science

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Assessing and predicting the quality of research master’s theses: an application of scientometrics

• Published: 03 June 2020
• Volume 124 , pages 953–972, ( 2020 )

Cite this article

• Zheng Xie   ORCID: orcid.org/0000-0003-0391-8725 1 ,
• Yanwu Li 2 &
• Zhemin Li 1  

809 Accesses

4 Citations

Explore all metrics

The educational quality of research master’s degree can be in part reflected by the examiner score of the thesis. This study focuses on finding positive predictors of this score with the aim of developing assessment and prediction methods for the educational quality of postgraduates. This study is based on regression analysis of the characteristics extracted from publications and references involving 1038 research master’s theses written at three universities in China. The analysis indicates that for a thesis, the number and the integrated impact factor of its references in Science Citation Index Expanded (SCIE) journals are significantly positive predictors of having publications in such journals. Additionally, the number and the integrated impact factor of a thesis’ representative publications, defined as the publications authored by the master’s student as a first author or second author with tutors in lead position, in SCIE journals, are significantly positive predictors of its examiner score. Based on these predictors, a range of indicators is provided to assess thesis quality, to measure the contributions of disciplines to postgraduate education, to predict postgraduates’ research outcomes, and to provide benchmarks regarding the quality and quantity of their reading work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Quantitative study on australian academic science, refrain from adopting the combination of citation and journal metrics to grade publications, as used in the italian national research assessment exercise (vqr 2011–2014).

The role of citation networks to explain academic promotions: an empirical analysis of the Italian national scientific qualification

Impact factor (IF) of a journal at a given year is the average number of citations received at that year for its publications at two preceding years (Garfield 1994 , 2006 ). See https://clarivate.com/webofsciencegroup/essays/impact-factor/ .

Science Citation Index Expanded indexes over 9200 major journals across 178 scientific disciplines. In this study, these journals are called SCIE journals for short. See https://clarivate.com/webofsciencegroup/solutions/webofscience-scie/ .

See http://www.moj.gov.cn/Department/content/2004-09/03/592_201359.html

See http://old.moe.gov.cn/publicfiles/business/htmlfiles/moe/s6183/201112/128828.html .

See http://cdgdc.edu.cn/xwyyjsjyxx/sy/glmd/264462.shtml .

See http://cdgdc.edu.cn/xwyyjsjyxx/zlpj/ .

Abt, H. A. (2000). Do important papers produce high citation counts? Scientometrics , 48 (1), 65–70.

Google Scholar  

Aittola, H. (2008). Doctoral education and doctoral theses-changing assessment practices. In J. Välimaa & O. H. Ylijoki (Eds.), Cultural Perspectives on Higher Education (pp. 161–177). Dordrecht: Springer.

Anderson, C., Day, K., & McLaughlin, P. (2006). Mastering the dissertation: lecturers’ representations of the purposes and processes of master’s level dissertation supervision. Studies in Higher Education , 31 (2), 149–168.

Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics , 1 (5), 228–230.

Bourke, S. (2007). Ph.D. thesis quality: the views of examiners. South African Journal of Higher Education , 21 (8), 1042–1053.

Bourke, S., & Holbrook, A. P. (2013). Examining PhD and research masters theses. Assessment & Evaluation in Higher Education , 38 (4), 407–416.

Bouyssou, D., & Marchant, T. (2011). Bibliometric rankings of journals based on impact factors: An axiomatic approach. Journal of Informetrics , 5 (1), 75–86.

Bouyssou, D., & Marchant, T. (2011). Ranking scientists and departments in a consistent manner. Journal of the American Society for Information Science and Technology , 62 (9), 1761–1769.

Braun, T., & Glänzel, W. (1990). United Germany: The new scientific superpower? Scientometrics , 19 , 513–521.

De Bruin, R. E., Kint, A., Luwel, M., & Moed, H. F. (1993). A study of research evaluation and planning: The university of Ghent. Research Evaluation , 3 (1), 25–41.

Böhning, D. (1992). Multinomial logistic regression algorithm. Annals of the Institute of Statistical Mathematics , 44 (1), 197–200.

MathSciNet   MATH   Google Scholar  

Eng, J. (2003). Sample size estimation: How many individuals should be studied? Radiology , 227 (2), 309–313.

Fernández-Cano, A., & Bueno, A. (1999). Synthesizing scientometric patterns in Spanish educational research. Scientometrics , 46 (2), 349–367.

Freedman, D. A. (2009). Statistical models: Theory and practice . Cambridge: Cambridge University Press.

MATH   Google Scholar  

Garfield, E. (1970). Citation indexing for studying science. Nature , 227 (5259), 669–671.

Garfield, E. (1994). The impact factor. Current Contents , 25 (20), 3–7.

Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA , 295 (1), 90–93.

Hagen, N. (2010). Deconstructing doctoral dissertations: How many papers does it take to make a PhD? Scientometrics , 85 (2), 567–579.

Hansford, B. C., & Maxwell, T. W. (1993). A masters degree program: Structural components and examiners’ comments. Higher Education Research and Development , 12 (2), 171–187.

Hemlin, S. (1993). Scientific quality in the eyes of the scientist: a questionnaire study. Scientometrics , 27 (1), 3–18.

Holbrook, A., Bourke, S., Fairbairn, H., & Lovat, T. (2014). The focus and substance of formative comment provided by PhD examiners. Studies in Higher Education , 39 (6), 983–1000.

Holbrook, A., Bourke, S., Lovat, T., & Dally, K. (2004). Investigating PhD thesis examination reports. International Journal of Educational Research , 41 , 98–120.

Holbrook, A., Bourke, S., Lovat, T., & Fairbairn, H. (2008). Consistency and inconsistency in PhD thesis examination. Australian Journal of Education , 52 (1), 36–48.

Kamler, B. (2008). Rethinking doctoral publication practices: Writing from and beyond the thesis. Studies in Higher Education , 33 (3), 283–294.

Kyvik, S., & Thune, T. (2015). Assessing the quality of PhD dissertations: a survey of external committee members. Assessment & Evaluation in Higher Education , 40 (5), 768–782.

Lariviére, V. (2012). On the shoulders of students? The contribution of PhD students to the advancement of knowledge. Scientometrics , 90 (2), 463–481.

Leydesdorff, L., & Bornmann, L. (2011). Integrated impact indicators compared with impact factors: An alternative research design with policy implications. Journal of the American Society for Information Science and Technology , 62 (11), 2133–2146.

Lisee, C., Lariviere, V., & Archambault, E. (2008). Conference proceedings as a source of scientific information: A bibliometric analysis. Journal of the American Society for Information Science and Technology , 59 (11), 1776–1784.

MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science , 40 (5), 342–349.

MacRoberts, M. H., & MacRoberts, B. R. (2018). The mismeasure of science: Citation analysis. Journal of the American Society for Information Science , 69 (3), 474–482.

Mason, S., Merga, M. K., & Morris, J. E. (2019). Choosing the thesis by publication approach: Motivations and influencers for doctoral candidates. The Australian Educational Researcher ,. https://doi.org/10.1007/s13384-019-00367-7 .

Article   Google Scholar  

Mason, S., Merga, M. K., & Morris, J. E. (2020). Typical scope of time commitment and research outputs of thesis by publication in Australia. Higher Education Research & Development , 39 (2), 244–258.

Moed, H. F., De Bruin, R. E., & Van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications. Scientometrics , 33 (3), 381–422.

Mullins, G., & Kiley, M. (2002). It’s a PhD, not a Nobel Prize: How experienced examiners assess research theses. Studies in Higher Education , 27 (4), 369–386.

Nelder, J. A., & Wedderburn, R. W. (1972). Generalized linear models. Journal of the Royal Statistical Society: Series A (General) , 135 (3), 370–384.

Pilcher, N. (2011). The UK postgraduate masters dissertation: An elusive chameleon? Teaching in Higher Education , 16 (1), 29–40.

Prieto, E., Holbrook, A., & Bourke, S. (2016). An analysis of PhD examiners’ reports in engineering. European Journal of Engineering Education , 41 (2), 192–203.

Stracke, E., & Kumar, V. (2010). Feedback and self-regulated learning: insights from supervisors’ and PhD examiners’ reports. Reflective Practice , 11 (1), 19–32.

Tinkler, P., & Jackson, C. (2000). Examining the doctorate: institutional policy and the PhD examination process in Britain. Studies in Higher Education , 25 , 167–180.

Tinkler, P., & Jackson, C. (2004). The doctoral examination process: A handbook for students, examiners and supervisors . Maidenhead: Open University Press.

Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. (2011). Towards a new crown indicator: An empirical analysis. Scientometrics , 87 , 467–481.

Winter, R., Griffiths, M., & Green, K. (2000). The academic qualities of practice: What are the criteria for a practice-based PhD? Studies in Higher Education , 25 (1), 25–37.

Xie, Z. (2020). Predicting the number of coauthors for researchers: A learning model. Journal of Informetrics , 14 (2), 101036.

Xie, Z., & Xie, Z. (2019). Modelling the dropout patterns of MOOC learners. Tsinghua Science and Technology , 25 (3), 313–324.

Zong, Q. J., Shen, H. Z., Yuan, Q. J., Hu, X. W., Hou, Z. P., & Deng, S. G. (2013). Doctoral dissertations of Library and Information Science in China: A co-word analysis. Scientometrics , 94 (2), 781–799.

Download references

Acknowledgements

The authors are grateful to Professor Shannon Mason in the Nagasaki University and anonymous reviewers for their helpful comments and feedback. LYW is supported by National Education Science Foundation of China (Grant No. DIA180383). XZ is supported by National Natural Science Foundation of China (Grant No. 61773020).

Author information

Authors and affiliations.

College of Liberal Arts and Sciences, National University of Defense Technology, Changsha, Hunan, China

Zheng Xie & Zhemin Li

Graduate School, National University of Defense Technology, Changsha, Hunan, China

You can also search for this author in PubMed   Google Scholar

Contributions

LYW motivated this study and provided empirical data. LZM preprocessed the data. XZ designed the methods to analyze the data, and wrote the manuscript. All authors discussed the research and approved the final version of the manuscript.

Corresponding author

Correspondence to Zheng Xie .

Ethics declarations

Conflicts of interest.

The authors declare that they have no conflicts of interest.

Appendix A: Minimum sample size

Assume the size of group from which a sample is taken to be infinite. Let the confidence level be \(1-\alpha\) . Denote the corresponding z -score of \(\alpha\) by \(z_{ {\alpha }/{2}}\) , the expected proportion by p , the population standard deviation by \(\sigma\) , and the margin of error by E . If the expected proportion and population standard deviation are not known, the sample proportion and sample standard deviation can be used (Eng 2003 ).

The formula of the minimum sample size required for estimating the population proportion is

Let \(\alpha =5\%\) , \(p=\)  sample population proportion, and \(E=0.15\) . For regression analysis on having representative publications, \(n=42,33,29, 38\) for Biological , Engineering , Information , and Physical sciences respectively.

The corresponding formula for estimating the population mean is

Let \(\alpha =5\%\) , \(\sigma =\)  sample standard deviation, and \(E= 1.5\%\) . For the regression analysis on examiner score, \(n=52, 53, 51, 43\) for Biological , Engineering , Information , and Physical sciences respectively.

Appendix B: More results of regression

Figure  10 shows linear regression results between the examiner score and the indexes derived from the references of theses. The number and the integrated impact factor of SCIE references are significantly positive predictors of the examiner score in information sciences , and the number is significantly positive in engineering . There are no significant relationship in the other cases.

The relationship between the examiner score and the indexes derived from references. The panels show the mean examiner score of theses with the same index value (red squares), the predicted score (solid dot lines), and confidence intervals (dashed lines). The p value is that of \(\chi ^2\) -test. (Color figure online)

Figure  11 shows that for each disciplinary group, the number of representative publications of a thesis follows a Gamma distribution. Therefore, Gamma regression can be utilized to analyse the relationship between the number of representative publications and the indexes derived from references. Gamma regression is a generalized linear model that assumes that the response variable follows a Gamma distribution. The negative reciprocal of the expected value of the Gamma distribution is fitted by a linear combination of predictors (Nelder and Wedderburn 1972 ).

Figure  12 shows Gamma regression results. Except for biological sciences , the number of SCIE references is a significantly positive predictor of the number of representative publications. And there is no significant relationship between the number of non-SCIE references and the number of representative publications. These results may be statistically meaningless due to the small sample size of theses having a given number of representative publications.

The distribution of the number of representative publications. The panels show the empirical distributions (red circles) and Gamma distributions (blue squares). The KS test cannot reject the hypothesis that the number of representative publications follows a Gamma distribution, p value \(>5\%\) . (Color figure online)

The relationship between the number of representative publications and that of SCIE/non-SCIE references. The panels show the average number of representative publications of theses with the same index value (red squares), the predicted value (solid dot lines), and confidence intervals (dashed lines). The p value is that of \(\chi ^2\) -test. (Color figure online)

Rights and permissions

Reprints and permissions

About this article

Xie, Z., Li, Y. & Li, Z. Assessing and predicting the quality of research master’s theses: an application of scientometrics. Scientometrics 124 , 953–972 (2020). https://doi.org/10.1007/s11192-020-03489-3

Download citation

Received : 15 June 2019

Published : 03 June 2020

Issue Date : August 2020

DOI : https://doi.org/10.1007/s11192-020-03489-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data science applications in education
• Higher education
• Assessment methodologies
• Find a journal
• Publish with us
• Track your research

• Search News and Events
• News overview
• Media relations

How light and sleep quality affect burnout

Sophia Frick defended her PhD thesis at the Department of Industrial Engineering and Innovation Sciences on September 6th.

Burnout is becoming increasingly common with many people reporting that they feel exhausted and detached from their work. Mild symptoms related to burnout are probably familiar to us all: most of us have moments, days, or even weeks during which we feel more exhausted and less engaged in our work than usual. Past research has shown that sleep helps us to recover from work, and that light regulates our sleep patterns, and affects our energy and mood throughout the day. Similarly, getting good quality sleep and the right amount of light may be important for reducing or preventing mild burnout symptoms, whereas poor sleep and insufficient light could worsen such symptoms. The focus of Sophia Frick’s PhD research was to investigate if and how sleep and light contribute to, and could potentially be used to mitigate, the experience and development of mild burnout-related symptoms.

Do burnout symptoms fluctuate throughout the day and week?

Although clinical burnout has traditionally been viewed as a phenomenon that does not change much over time, the PhD research of Sophia Frick explored whether mild burnout-related experiences fluctuate from moment to moment and day to day.

First, along with her collaborators at the Department of of Industrial Engineering and Innovation Sciences, Frick repeatedly asked 65 participants to report how burnt-out they felt while working during their regular routine.

Findings from this part of her research showed that burnout-related experiences indeed fluctuate from moment to moment in working participants.

Energy levels peaked around midday and decreased towards the evening, suggesting that our most demanding work tasks should perhaps be tackled before noon.

What comes first: bad sleep or burnout?

When we sleep poorly, we often wake up in a bad mood and with low levels of energy. However, when we feel stressed from work and exhausted, this can also affect how well we sleep at night.

To understand which comes first (poor sleep or burnout), Frink then conducted a one-week field study involving 72 participants in which she employed surveys and wrist-worn sensors to monitor sleep. She found that poor sleep appears to precede burnout-related experiences, not the other way around.

Additionally, Frick found that a poor night’s sleep affects everyone differently: those experiencing high levels of burnout the previous months were more affected by sleeping poorly. This suggests that obtaining good quality sleep may be particularly important for those who are already (susceptible to) feeling burnt-out.

In addition, she was also interested in investigating relationships between sleep and burnout across longer periods of time. For this study, 150 people took part and the whole study lasted 12 months. Participants were repeatedly assessed over the course of the study.

Results from this longitudinal study showed that when participants felt more detached from their work they had more disturbed sleep three months later. Yet, when participants had more disturbed sleep, they experienced more exhaustion three months later.

Poor sleep may thus be one of the mechanisms whereby burnout worsens over time, suggesting that sleep health may help increase resilience against burnout.

Light and burnout

In addition to helping us to see, light is one of the most important factors determining many of our daily rhythms, including when we feel alert and when get sleepy.

In this part of her PhD research, Frick was interested in knowing whether light also influences burnout-related symptoms.

When people were exposed to more intense/brighter light (than they are used to normally during the day), she found that they felt less exhausted at the end of the hour. Basically, being in brighter light than usual for one hour made people feel less exhausted than they did before being in the brighter light.

In addition, Frick also uncovered that while light may be a helpful tool for reducing exhaustion at work for everyone. Light was particularly important for those who already felt burnt-out.

Overall, Frick’s findings suggest that improving sleep quality and ensuring adequate light exposure, could be effective strategies for preventing and managing burnout, particularly for those who are already vulnerable.

Importance and motivation

For Frick, working on this topic was easy to relate to. “We all experience work stress every once in a while, and we have all felt the consequences of not sleeping well. I really appreciated that this research is relevant to so many people and has the potential to contribute to enhancing employee wellbeing and sustainable employment – topics I find very important.”

Added to that, the interdisciplinary nature of the project combined several topics that Frick is interested in. “I have a background in cognitive neuroscience, so I enjoyed linking psychology to more biological and physiological processes via sleep and light. I also appreciated the societal relevance and applied value of the project.”

Be adaptable

During any PhD project, there are key lessons to be learned with regards to content and the research process. For Frick, she learned that the key is to be adaptable.

“Many things do not go as planned (e.g., COVID-19). It is important not to be deterred by that but to find ways to work with and around it,” says Frick. “This may mean that the end result looks a bit different than originally planned, but can also lead to new and unexpected insights.”

Title of PhD-thesis: Fluctuations in Burnout: Uncovering the Role of Light and Sleep . Supervisors: Karin Smolders (TU/e), Leander van der Meij (TU/e), Evangelia Demerouti (TU/e), and Yvonne de Kort (TU/e).

Media contact