Shapiro Library

Higher Education Administration (HEA) Guide

A review of quantitative and qualitative analysis.

Need a refresher on Quantitative and Qualitative Analysis? Click below to get a review of both research methodologies.

Program Evaluation and Planning

Close up on hand writing out numbered plans on paper

Image by Kelly Sikkema, retrieved via Unsplash

From data analysis to program management methods and more, evaluating and planning for the success of each program is a crucial aspect of Higher Education Administration. Below you will find some useful articles and reports to help bring context to this important element of higher education leadership. 

Useful Articles

Below you will find a sample of reports, case studies and articles that outline the process of program evaluation, planning and analysis. Click through and read on for more information. 

  • The Feasibility of Program-Level Accountability in Higher Education: Guidance for Policymakers. Research Report Policymakers have expressed increased interest in program-level higher education accountability measures as a supplement to, or in place of, institution-level metrics. But it is unclear what these measures should look like. In this report, the authors assess the ways program-level data could be developed to facilitate federal accountability.
  • Improving Institutional Evaluation Methods: Comparing Three Evaluations Using PSM, Exact and Coarsened Exact Matching Policymakers and institutional leaders in higher education too often make decisions based on descriptive data analyses or even anecdote when better analysis options could produce more nuanced and more valuable results. Employing the setting of higher education program evaluation at a midwestern regional public university, for this study we compared analysis approaches using basic descriptive analyses, regression, standard propensity score matching (PSM), and a mixture of PSM with continuous variables, coarsened exact matching, and exact matching on categorical variables. We used three examples of program evaluations: a freshman seminar, an upper division general education program intended to improve cultural awareness and respect for diverse groups, and multiple living learning communities. We describe how these evaluations were conducted, compare the different results for each type of method employed, and discuss the strengths and weaknesses of each in the context of program evaluation.
  • Data-Informed Policy Innovations in Tennessee: Effective Use of State Data Systems This link opens in a new Analysis of student-level data to inform policy and promote student success is a core function of executive higher education agencies. Postsecondary data systems have expanded their collection of data elements for use by policymakers, institutional staff and the general public. State coordinating and governing boards use these data systems for strategic planning, to allocate funding, establish performance metrics, evaluate academic programs, and inform students and their families. This report discusses efforts at the Tennessee Higher Education Commission to support policy innovation with data and information resources.

Other Resources

case study higher education evaluation

  • << Previous: Contemporary Issues In Higher Education
  • Next: Legal and Regulatory Research >>

Learning analytics in higher education: an analysis of case studies

Asian Association of Open Universities Journal

ISSN : 2414-6994

Article publication date: 2 May 2017

The purpose of this paper is to present a systematic review of the mounting research work on learning analytics.

Design/methodology/approach

This study collects and summarizes information on the use of learning analytics. It identifies how learning analytics has been used in the higher education sector, and the expected benefits for higher education institutions. Empirical research and case studies on learning analytics were collected, and the details of the studies were categorized, including their objectives, approaches, and major outcomes.

The results show the benefits of learning analytics, which help institutions to utilize available data effectively in decision making. Learning analytics can facilitate evaluation of the effectiveness of pedagogies and instructional designs for improvement, and help to monitor closely students’ learning and persistence, predict students’ performance, detect undesirable learning behaviours and emotional states, and identify students at risk, for taking prompt follow-up action and providing proper assistance to students. It can also provide students with insightful data about their learning characteristics and patterns, which can make their learning experiences more personal and engaging, and promote their reflection and improvement.

Originality/value

Despite being increasingly adopted in higher education, the existing literature on learning analytics has focussed mainly on conventional face-to-face institutions, and has yet to adequately address the context of open and distance education. The findings of this study enable educational organizations and academics, especially those in open and distance institutions, to keep abreast of this emerging field and have a foundation for further exploration of this area.

  • Higher education
  • Learning analytics
  • Open and distance education

Wong, B.T.M. (2017), "Learning analytics in higher education: an analysis of case studies", Asian Association of Open Universities Journal , Vol. 12 No. 1, pp. 21-40. https://doi.org/10.1108/AAOUJ-01-2017-0009

Emerald Publishing Limited

Copyright © 2017, Billy Tak Ming Wong

Published in the Asian Association of Open Universities Journal . Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

Introduction

Learning analytics (LA) refers to the process of collecting, evaluating, analysing, and reporting organizational data for decision making ( Campbell and Oblinger, 2007 ). It involves the use of big data analysis for understanding and improving the performance of educational institutions in educational delivery. Open and distance learning (ODL) institutions present an ideal context for the use of LA as, with their large student numbers and the increasing use of the internet and mobile technologies, they already have a very substantial amount of data available for analysis with analytics.

Despite LA being increasingly applied in a wide range of educational organizations, the literature in this area has usually focussed on conventional face-to-face institutions. In the ODL setting, there is yet to be a systematic review summarizing existing work on the potential benefits of LA to open and distance institutions ( Firat and Yuzer, 2016 ; Prinsloo and Slade, 2014 ), and relevant research findings potentially applicable to these institutions ( Rienties et al. , 2016 ).

This paper gives a systematic review of the mounting research work on LA that has been published in recent years to provide an overview of this emerging field and serves as a foundation for further exploration. It addresses the potential problems of ODL institutions that could be solved by using LA, and the benefits that could be obtained according to the existing case studies. It also presents a meta-analysis of relevant empirical studies which shows the effect of intervention for at-risk students based on the use of LA.

Related studies

LA involves the use of a broad range of data and techniques for analysis – covering, for example, statistical tests, explanatory and predictive models, and data visualization ( Arroway et al. , 2016 ). Various stakeholders, such as administrators, teaching staff, and students, can then act on the data-driven analysis. Without a standardized methodology, LA has been implemented using diverse approaches for various objectives. Gašević et al. (2016) summarized three major themes in LA implementation, namely, the development of predicators and indicators for various factors (e.g. academic performance, student engagement, and self-regulated learning skills); the use of visualizations to explore and interpret data and to prompt remedial actions; and the derivation of interventions to shape the learning environment. The diversity in LA implementation poses a challenge for education institutions which plan to be involved in it, leading to a commonly voiced question – “How do we start the process for the adoption of institutional learning analytics?” ( Gašević et al. , 2016 , p. 4).

As an emerging field of study, an increasing number of case studies relevant to the implementation of LA in higher education have been published. However, only a small number of reviews summarize these individual case studies. Among them, Dyckhoff (2011) reviewed the research questions and methods of these studies. The findings showed that existing studies have focussed on six types of research questions: qualitative evaluation; quantitative measures of use and attendance; differentiation between groups of students; differentiation between learning offerings; data consolidation; and effectiveness. The research methods used include online surveys, log files, observations, group interviews, students’ class attendance, eye tracking, and the analysis of examination grades. Based on the results, suggestions were given on LA indicators for improving teaching.

Papamitsiou and Economides (2014) focussed on the impacts of LA and educational data mining on adaptive learning. They reviewed the experimental case studies between 2008 and 2013, and identified four distinct categories, namely, pedagogy-oriented issues, contextualization of learning, networked learning, and the handling of educational resources.

Also, Nunn et al. (2016) discussed LA’s methods, benefits, and challenges. It was found that the methods used included visual data analysis, social network analysis, semantic analysis, and educational data mining. The benefits of LA were seen to revolve around targeted course offerings; curriculum development; student learning outcomes; behaviours and processes; personalized learning; improvements in instructor performance; post-educational employment opportunities; and enhancement of educational research. The challenges included the tracking, collection, evaluation and analysis of data, as well as a lack of connection to learning science, the need for learning environment optimization, and issues concerning ethics and privacy.

Focussing on computer science courses, Ihantola et al. (2015) surveyed LA case studies in terms of their goals, approaches, contexts, subjects, tasks, data and collection, and methods of analysis. The goals were related to students, programming, and the learning environment. The approaches included case studies, constructive research, experimental studies, and survey research. They also found that most of the research work was undertaken in a course context, with the number of subjects ranging from 10 to 265,000, with 64 per cent of the studies having 500 or fewer subjects. In most of the studies, students were required to complete multiple programming tasks. Over 60 per cent of the studies used automated data collection that logged students’ actions, and a variety of data analysis methods such as descriptive and inferential statistics.

The existing reviews of LA case studies provide a basic descriptive summary. However, as a new area in education, there remain many uncertainties for ODL institutions about involving themselves in it. To make an informed decision on whether or not to implement LA, a key question is: “What are the expected benefits for the institution?” This paper addresses this issue by surveying the outcomes of LA implementation for institutions.

Methodology

the study reported one or more empirical cases of the use of LA in a higher education institution;

the institution in question was accredited by the government or government-related bodies;

the institution had 1,000 or more students; and

the source information contained the aims of using LA, a description of the analytics, its implementation and the outcomes.

An initial search returned 1,492 results. After screening, a total of 43 cases which fulfilled the criteria for inclusion were selected for further analysis. They were analysed in terms of their objectives, approaches, and major outcomes.

A meta-analysis was also conducted to synthesize the empirical findings reported in the case studies. Studies which included relevant quantitative data analysis were chosen, resulting in six studies on student support and analysis of learning behaviours, with the effect of LA intervention validated and reported.

Benefits for institutions, staff, and students

A summary of the objectives and approaches of the use of LA in the institutions chosen is presented in Table AI . The benefits of LA for the institutions, staff and students revolve around the following aspects.

Improving student retention

Table I presents the use of LA which improved student retention. By closely monitoring students’ learning and persistence, undesirable learning behaviours and emotional states can be detected, and students who are at risk can be identified early. Factors leading to student dropout or retention can be identified and prediction models developed. Staff can take prompt follow-up action and provide proper assistance to students who need extra support, such as counselling, suggesting learning resources, and formulating individual learning plans. Students’ level of achievement, as well as their retention, can be enhanced.

Supporting informed decision making

Table II shows the use of LA which supported informed decision making. Institutions are provided with information and analyses generated from a massive amount of data for informed decision making. For example, planning can be carried out on course development and resources allocation on the basis of information about the popularity of courses, and types and frequency of materials reviewed by students.

Increasing cost-effectiveness

Table III presents cases of LA use which increased cost-effectiveness. LA can be integrated with other platforms such as the learning management system. Instructors can then access various kinds of information online for providing feedback and support to students. Analyses and feedback on students’ study progress can be delivered to staff, students, or parents in an automatic and cost-effective manner.

Understanding students’ learning behaviours

Table IV presents the use of LA for understanding students’ learning behaviours. By analysing diverse sources of data (e.g. learning management systems and social networks), institutions and academic staff can understand the relationships among students’ utilization of resources, learning behaviours and characteristics, and learning outcomes, which helps them to evaluate the effectiveness of pedagogies and instructional designs for improvement. For instance, the use of LA helps to capture the students’ behaviours in watching course videos by highlighting the patterns of their preferences and behaviours as well as showing the parts of videos which were watched most and least frequently. Curriculum and learning materials can thus be better designed to address students’ preferences and needs.

Providing personalized assistance for students

Table V illustrates the use of LA for providing students with insightful data about their learning characteristics and patterns, which can make their learning experiences more personal and engaging, and facilitate their reflections and improvements while a course is still in progress. Early alerts can be automatically generated and sent to students if their academic performance is below a certain standard. Students can also be encouraged to engage more in the personalized learning activities which are conducive to success in their studies.

Timely feedback and intervention

Table VI presents the use of LA for timely feedback and intervention. Instructors can obtain up-to-date and holistic information about students’ study progress, so that timely feedback can be given and individualized interventions made. Students develop a sense of belonging to the learner community through personalized feedback given to them. For example, the use of social network analytics allows instructors to understand the development of the learner community and identify students who are performing poorly or are isolated from the main discussion, and then provide intervention during discussion in real time. This is especially important for ODL institutions, where students may be using different study modes and social media is a major communication channel.

Meta-analysis of the effect of interventions on student success

An important function of LA is to predict at-risk students and deliver early alerts and interventions to them, in order to improve their academic attainment, and their retention and graduation rate. This section provides a meta-analysis of the various prediction models utilized in LA systems, and the effect of the intervention solutions on enhancing students’ success.

Among the case studies examined, only six which provided quantitative analysis results were selected and the results are synthesized in this section. The effect sizes for each analysis were calculated where the data required for the calculation were available, and a descriptive comparison of the effect sizes across the studies was made. Table VII presents a summary of the predictive models and intervention solutions employed in the six case studies; and Table VIII summarizes the results of quantitative analyses for the intervention solutions and the effect sizes for each study.

To summarize, a common approach utilized in the cases of intervention for student success was to collect and analyse data from students’ learning activities and employ a specific computational model to predict and prioritize those students who were at-risk of dropping out or getting poor academic results. Based on the findings of the predictive modelling, subsequent measures can be taken for intervention. A common practice was to get academic staff to contact the at-risk students and provide personalized learning support to them. Such an approach to prediction and intervention was found to effectively enhance students’ success, as measured by various indicators such as GPA, study progress, the retention rate, and the graduation rate.

According to the meta-analysis of the quantitative results, all the institutions found improvement in the students’ success in the intervention group compared to the control group, although the effect size varied across different types of indicators for success and different institutions. For instance, the intervention groups in the case of Marist College showed a 6 per cent improvement in the students’ final grades compared to the non-intervention control groups ( Sclater et al. , 2016 ), while the effect size was in the range of small to medium based on Cohen’s (1988) convention. For the retention rate examined in Mattingly et al. (2012) for the Course Signal System of Purdue University, the intervention groups showed a nearly 50 per cent performance improvement compared to the control groups. In spite of the small sample size, the meta-analysis showed an encouraging result for the benefits of LA in aiding institutions to make effective informed decisions to improve students’ learning performance and success.

Discussion and conclusion

This study shows that positive outcomes have been widely reported in relevant case studies. The results suggest great potential for ODL institutions to utilize LA for analysing existing data, which is expected to benefit their operations in areas such as quality assurance and student support. This study also reviewed various predictive models for student success which were developed and validated to identify and prioritize students who may be in need of support. The quantitative analyses confirmed that the learning performance of these students improved after they had been approached for LA-based interventions. The findings of this study thus provide various stakeholders – institutions, staff, and students – with the benefits they may gain from LA.

In particular, the results related to student learning suggest that, to change students’ behaviours, it may suffice to simply make them aware of their learning engagement through LA tools in relation to other students or indicate that they are at risk ( Jayaprakash et al. , 2014 ; Sclater and Mullan, 2017 ). Complex data visualizations or dashboards may not be necessary. What is more important, as recommended in Gašević et al. (2016) , is to help students to interpret correctly the information from visualizations or dashboards.

The meta-analysis revealed that only a few case studies related to LA implementation provided quantitative analyses data – a limitation which may be caused by the relatively new development of LA. Therefore, empirical investigations and validation of many new models and new theories in this area remain to be carried out. While an increase in the quantity of empirical and quantitative research can be expected in future, it is also important to develop and test innovative solutions supported by LA. Present LA-based interventions, as reviewed in this paper, were mostly based on the interaction and discussion between students and instructors. Although such interventions were shown to be effective in general, their effectiveness may vary among different groups of students in different contexts.

A challenge in measuring the effectiveness of LA implementation lies in the difficulty of identifying the extent to which any change after the LA implementation is attributed to the LA itself. As discussed in Sclater and Mullan (2017) , it may not be feasible to isolate the influence of LA when it is part of a wider initiative to develop data-informed approaches in an institution. The case studies published and reviewed in this paper would thus be biased to the institutions which only deployed LA without other measures in their data-informed approaches.

In the ODL context, work on LA remains at an initial stage. Features of ODL, such as open admission which allows a broad range of students to study the same course with very limited face-to-face interaction, are yet to be studied in relation to LA implementation. It is therefore suggested that future research can involve more fine-grained validation studies to identify the effect of the various factors involved the implementation of LA. In particular, investigation on those factors related to ODL institutions, staff and students, as well as the plausible constraints on their use of LA, would shed light on how they can benefit more from involvement in LA.

Use of LA which improved student retention

Institution Major outcomes Source
Bowie State University More student activities and communication were initiated through the system (2012)
Edith Cowan University The student retention rate for those who got support was higher than the university’s average rate (2013)
Harvard University The results demonstrate the potential for natural language processing to contribute to predicting student success in MOOCs and other forms of open online learning (2016)
New York Institute of Technology An at-risk model of high predictive power was developed (2016)
Northern Arizona University Student-instructor interaction was increased and personal interventions were given; and students showed better academic performance, retention and graduation rates
Paul Smith’s College Students devoting more efforts in their studies resulted in a higher chance of success, and better persistence and graduation rates
Rio Salado Community College A 40% decrease in drop-out rate was obtained for students who received welcome e-mails compared with those who did not (2012)
The Open University (UK) A vast majority of students showed continuous engagement
Student retention was at an average to good level
Students demonstrated higher satisfaction
(2016)
University of New England The student attrition dropped from 18 to 12%
Students demonstrated an increase in their sense of belonging to the learner community and learning motivation
(2016)

Use of LA which supported informed decision making

Institution Major outcomes Source
Grand Rapids College Better decisions can be made about course delivery to help to ensure student success through a LA tool which is easy for end user analysis
The Open University (UK) Elements tacitly implicated in pedagogical decisions during course design were unpicked
University of Adelaide Educators were provided with guidelines to design collaborative learning activities (2015)
University of Edinburgh Through identification of socially engaged students, the instructional team can identify suitable teaching assistants (2016)
University of North Bengal Counsellors and faculty members were provided with useful inputs to advise learners on the best possible completion options
University of Salamanca Visual analytics was shown to help to lead to better understanding of what is happening in a student. Informed decisions can be made that help students to succeed (2015)
The Technical University of Madrid Information was provided by the LA system which helped to prevent problems, carry out corrective measures and make informed decisions to improve students’ learning (2015)

Use of LA which increased cost-effectiveness

Institution Major outcomes Source
Bridgewater College Notifications were automatically generated and sent to students and their parents to recognize students’ good performance (2016)
Drexel University Faculty, programme developers, and programme administrators were able to analyse the connections between a specific programme outcome and data related to that outcome
Georgia Institute of Technology and Carnegie Mellon University High reliability was achieved for analysing students’ online discussion data (2016)
Harvard University A machine learning prediction model was shown to be effective for predicting students who would complete an online course (2016)
Lancaster University Tutors could efficiently access various kinds of data for providing students with timely support (2016)
New York Institute of Technology A dashboard simple and easy to use by staff was developed (2016)
Open University of Catalonia Information could be updated and maintained automatically (2015)
Portland State University Operation efficiency was increased, e.g. faster generation of reports
The system could easily be modified to fit the needs of other institutions
Purdue University Students who had engaged with the LA system sought more help and resources than other students
Rio Salado College The likelihood of successful course completion was accurately assessed (2012)
The Hong Kong Institute of Education There was greater interaction between teachers and students
University of Adelaide Lecturers were allowed to assess and monitor students’ collaboration in an online environment, without having to traverse a large discussion forum (2015)
University of Michigan The system demonstrated high scalability and extensibility (2012)
University of Salamanca The system allowed the provision of learning support to students in an automatic manner (2014)
University of the South Pacific The utilization of open source resources could be modified and adapted by anyone to meet specific user needs (2016)
University of Sydney LA features such as instant feedback and auto-grading are especially useful for instructors teaching subjects in computer science education (2016)

Use of LA which helped in understanding students’ learning behaviours

Institution Major outcomes Source
Ball State University Data analyses showed the consistent predictive power of the LA system on students’ academic performance, persistence, retention and graduation
Georgia Institute of Technology and Carnegie Mellon University Students who displayed more higher-order thinking behaviours learnt more through deeper engagement with course materials displayed by their discussion behaviours
These students in turn also learnt more than students who were constantly off topic in the forums
Social-oriented topics triggered richer discussion compared with biopsychology oriented topics, and higher-order thinking behaviours tended to appear together within threads in the forums
(2016)
McGill University It provides an unprecedented opportunity to use data from real learners in authentic learning situations to better understand learning processes
The study demonstrated how to detect learner misconceptions
Prediction precision and weighted relative accuracy were significantly increased
(2016)
Oxford Brookes University Problems were identified with ethnic minority students in particular courses (2016)
The Hong Kong Institute of Education Potential indicators were found for predicting student performance, such as the contribution of in-depth contents in online discussion
The Open University (UK) Common pedagogical patterns were identified from learning designs, showing the relationship between learning activities and students’ learning outcomes
The Technical University of Madrid Relationship between student interaction and individual performance was identified (2015)
The University of Melbourne Relationships among students’ motivation, participation and performance in MOOCs were found (2016)
The University of Melbourne Learners’ learning progress could be visualized showing their development from novice to expert
University of Adelaide Lecturers could track the evolution of team roles across each study group and identify various sentiments within each group (2015)
University of Edinburgh Patterns of students’ engagement in MOOC learning activities were found, showing differences in their learning behaviours between enrolments in the same courses (2016)
University of North Bengal Factors leading to students’ dropout were identified, such as pregnancy and the remoteness of residence locations
University of Rijeka Student activities on the learning management system (e.g. assignment uploads and course views) were shown as predictors of academic success (2015)
University of Santiago de Compostela Teachers could understand more clearly how students behave during a course that facilitated the evaluation process (2014)

Use of LA for providing personalized assistance to students

Institution Major outcomes Source
Albany Technical College Based on analysis of students’ study results, demographics and social data, at-risk students were identified for providing individual counselling
Bridgewater College Tutors were provided with detailed information to discuss with students on their progress against targets and suggested actions (2016)
Open Universities Australia Students obtained from the system recommended content and activities and a personalized learning environment (2013)
The Technical University of Madrid The LA system provided information for preventing problems, carrying out corrective measures and improving students’ learning (2015)
University of Michigan Customized recommendations were provided, including suggestions on study habits, assignment practice, feedback on progress and encouragement (2012)

Use of LA for timely feedback and intervention

Institution Major outcomes Source
Edith Cowan University Students likely to need support were automatically identified and support staff could efficiently reach them for interventions (2016)
Marist College Interventions resulted in a 6% improvement in final grades for the treatment group compared to the control group (2014)
Northern Arizona University Instructors’ feedback was available to individual students and to university personnel, facilitating a comprehensive support network for all students
Purdue University Interventions were provided to at-risk students, and a higher student retention rate was achieved
San Diego State University Interventions through e-mails were shown to be the best treatment within constraints, while having an impact on student achievement (2015)
University of Adelaide The LA system allowed instructors to be aware when particular students are behaving differently from the others for making appropriate and timely interventions (2015)
University of Edinburgh Instant feedback was shown to be a useful LA feature for students in courses on computer programming (2016)
University of Michigan Students were provided with feedback (e.g. grade prediction) for self-reflection (2012)
University of Wollongong Students who are isolated from the main discussion could be identified, and interventions could be provided during discussion in real time (2013)

Summary of predictive model and intervention solution for selected case studies

Institution Learning analytics system (s) Predictive model Intervention solution
Georgia Institute of Technology and Carnegie Mellon University ( , 2016) Interactive-Constructive-Active-Passive (ICAP) framework It was predicted that engaging in higher-order thinking behaviours results in better learning outcomes than paying general or focussed attention to course materials Students’ online discussion behaviours were categorized into three types:
 Higher-order – the student has contributed at least one constructive or interactive post during a course
 Paying-attention – the student has contributed at least one active post during the course but has not displayed any constructive or interactive posts
 No contribution to any on-topic discussion during the course
Together with the students’ other persistent characteristics, treatment and control groups were formed to investigate differences in their learning outcomes
Hong Kong Institute of Education ( ) KeyGraph algorithm and Polaris (a software tool) A test-mining analytical tool was used to predict students’ academic performance. The tool visualizes the hidden patterns and linkages among students’ learning activities. The findings of the study showed that this approach can provide insights into predicting students’ performance, and students with a higher grade tended to contribute more in-depth contents in an online learning environment Students’ posts in an online learning forum were extracted and analysed – how the students presented concepts, specifically whether they can make linkage among various concepts. Such a pattern was correlated with the grades they obtained. The findings can be used to guide interventions on students’ learning process, and inform ways to give feedback to improve teaching and learning
Marist College ( , 2014) Open Academic Analytics Initiative A machine learning algorithm and logistic regression were used to predict whether students are at risk based on their demographic details, aptitude data, and various aspects of their usage of the virtual learning environment obtained from the LA system An online academic support environment was developed containing study skills materials and community support for specialists and student mentors. At-risk students identified by the predictive model were directed to the support environment
Nottingham Trent University ( , 2016) NTU Student Dashboard Students’ engagement was assessed using indicators, such as door swipes into academic buildings, visits to the virtual learning environment, the submission of assignments, and the frequency of borrowing library resources. Each student received one of five engagement ratings: high, good, partial, low and not fully enroled Tutors are prompted to contact students to give assistance when the students’ engagement drops off. Students can view their own engagement scores on the dashboard so that they will be self-motivated
Paul Smith’s College ( ) Rapid Insight’s Veera, Starfish EARLY ALERT, and CONNECT Rapid Insight’s Veera combines different file types and uses automatic analyses and predictive modelling to identify at-risk students prior to their enrolment. Starfish EARLY ALERT automates data collection and uses analytics to increase the identification of at-risk students The Starfish EARLY ALERT and CONNECT automatically prioritize students who are identified as at-risk and facilitate intervention and outreach
Purdue University ( ) Course Signal System The Course Signal System predicted students’ performance relying on a series of variables, including students’ demographic characteristics, academic performance, past academic history, and students’ efforts devoted to study Instructors provided real-time personalized feedback to each student based on the outcomes generated from LA, in which the student is informed about how he/she is doing

Summary of quantitative analysis results for selected case studies

Institution Independent variable Dependent variable Statistical method Description of result Effect size type Effect size [95% CI] Interpretation of effect size
Georgia Institute of Technology and Carnegie Mellon University Higher-order thinking behaviours Test score Regression The average posttest score of the treatment group (with higher-order thinking behaviour) was significantly higher than that of the control group (without higher-order thinking behaviour) Hedge’s 0.237 [0.018, 0.492] Small-to-medium effect size
Hong Kong Institute of Education “Contribution” and “innovation” from students’ postings in discussion forum Final grade test of independence Students who obtained better grades usually contributed more in-depth contents in their posts which linked to other concepts compared to those with lower grades who tended to provide isolated facts with little or no connection or transition from one concept to another Odds ratio (OR) 0.634 [0.504, 0.798] The students who contributed more in-depth contents were 63.4% more likely to get a higher grade than those contributing isolated facts
Marist College Intervention Final grade One-way ANOVA Groups receiving intervention obtained significantly higher final grade than groups receiving no intervention Hedge’s 0.373 [0.176, 0.571] Small-to-medium effect size
Nottingham Trent University Level of engagement rating Progression status Descriptive categorical data analysis A much larger proportion of students with satisfactory to high engagement ratings obtained progression status than those with low engagement ratings
Paul Smith’s College Intervention Grade, suspension or probation rate, graduation rate Descriptive categorical data analysis Student groups receiving intervention were less likely to get a grade D or below, to end a semester with probation or suspension, and more likely to get good standing by GPA and to graduate on time
Purdue University Intervention Retention rate test of independence Student groups receiving intervention had a higher retention rate than those receiving no intervention Odds ratio (OR) 0.455 [0.427, 0.485] The intervention group was 45.5% less likely to dropout than the non-intervention group
The results presented in the case studies of these two institutions did not involve any statistical tests and complete information for the data – that is, sample size for each category was not provided. Therefore, no effect size could be calculated from the available data; the effect size was computed by combining the data for the second-year retention rate for three cohorts (2007, 2008, 2009) from the original tables in (2012)

Institution Approaches Objectives Source
1. Albany Technical College Monitoring, intervention Identify at-risk students and provide them with counselling
2. Ball State University Monitoring, intervention Identify at-risk students and provide them with counselling
Increase effectiveness by reducing the time required to diagnose problems and targeting specific issues
Help the institution to make informed decisions about student success programmes and retention services
Allow students to become aware of the gaps between their behaviours and expected outcomes, to understand elements of their academic success, and to utilize on-campus resources to solve their problems
3. Bowie State University Monitoring, intervention Support student retention
Track students’ progress towards graduation to facilitate decision making
Provide early alerts for staff to intervene to prevent dropout
(2012)
4. Bridgewater College Monitoring, intervention Track students’ attainment level
Support students to do better than the national average
(2016)
5. California State University Monitoring Analyse how students use the learning management system (2012)
6. Drexel University Updating data and curriculum Measure the effectiveness of specific course components through maintaining data records aligned with the curriculum, courses and syllabi, course learning objectives and assessment strategies
Manage student learning outcomes and performance criteria
7. Edith Cowan University Monitoring, intervention Identify students who need support
Establish a system to contact a large number of students and manage interventions
Improve student retention
Improve graduation rates
(2016)
8. Georgia Institute of Technology and Carnegie Mellon University Monitoring, analysis Better scaffolded online discussion to improve learning in a MOOC context
Explore effects of higher-order thinking behaviours in learning
Identify kinds of discussion behaviours associated with learning
Investigate types of learning materials which trigger richer discussion
(2016)
9. Harvard University Monitoring, prediction Analyse the extent to which students’ responses about motivation and utility value can predict persistence and completion of study (2016)
10. Lancaster University Monitoring, intervention, feedback Allow tutors to access the transcripts of their students
Allow early intervention
Ensure student work is graded and feedback given to students in a timely manner
(2016)
11. Loughborough University Feedback Provide academics with a better and more holistic picture of student engagement
Provide staff with actionable insights into student learning experience
Provide students with their own educational data in a meaningful way
(2016)
12. Manchester Metropolitan University Monitoring, curriculum design Improve student experience as reflected in the National Student Survey
Provide data for improving the undergraduate curriculum
(2016)
13. Marist College Prediction, intervention Predict academic success
Provide interventions
(2014)
14. McGill University Monitoring, analysis Identify misconceptions of medical students as reflected in their interactions in the online learning environment (2016)
15. New York Institute of Technology Prediction, intervention Create an at-risk model to identify students in need of support
Improve student retention in their first year of study
Provide information that could support counsellor in their work
(2016)
16. Northern Arizona University Feedback Facilitate online interaction between students and instructors
Allow students to receive direct feedback on issues such as academic concerns and grades
17. Nottingham Trent University Intervention Enhance retention and improve attainment
Increase students’ sense of belonging within the course community, particularly with tutors
(2016)
18. Open Universities Australia Intervention Identify at-risk students
Suggest alternative modules to students which are more appropriate for their needs
(2013)
19. Open University of Catalonia Information collection and management Identify automatically pieces of knowledge taught in each subject
Gather students’ information
Keep information updated
(2015)
20. Oxford Brookes University Monitoring Improve student experience
Support progress evaluation of modules and programmes, and the identification of priorities at an institutional level
(2016)
21. Paul Smith’s College Monitoring, intervention Identify at-risk students and prioritize outreach for them
Provide more efficient and effective interventions for student success
22. Portland State University Information management Make information more accessible and easier to use
23. Purdue University Monitoring, intervention Give students early and frequent performance notifications
Help faculty members to steer students towards additional campus resources as needed
24. Rio Salado College Prediction Identify factors having a significant statistical correlations with final course outcomes
25. San Diego State University Intervention Identify methods and interventions that would alleviate students’ failure
Discover approaches that could be applied with minimal support and are scalable to a large number of courses
(2015)
26. The Hong Kong Institute of Education Monitoring, feedback Provide insights into predicting students’ performance
Develop measures to assess students’ online learning
Boost teachers’ and students’ interaction
Allow students to realize their knowledge discovery
Facilitate teachers to assess students’ performance
27. The Open University (UK) Monitoring, intervention, personalization Identify learners at risk and needing support
Improve learning design
Deliver personalized intervention for students
Achieve cost-effectiveness
(2016)
Identifying patterns Identify common patterns in course design
Find out pedagogical implications for various patterns and learning designs
28. The Technical University of Madrid Monitoring, evaluation Support teachers’ monitoring and evaluation of individual students’ progress within a team (2015)
29. The University of Adelaide Monitoring, feedback Analyse students’ online discussion data, such as team mood, role distribution and emotional climate
Develop students’ soft skills necessary for collaborative work
(2015)
30. The University of East London Monitoring, feedback Monitor student attendance and learning activities
Collect student data, such as demographic information, library activities, coursework, and download of free books
Send automated e-mails to students showing their attendance, and warnings to students without satisfactory attendance
(2016)
31. The University of Melbourne Monitoring, analysis Investigate how motivation and participation influence students’ performance in a MOOC Barba (2016)
Analyse how MOOC participants use online forums to support learning
Investigate how students interpret feedback delivered via learning analytics dashboard and the relevant influence on their learning strategies and motivation
32. Universidad a Distancia de Madrid Monitoring, analysis Find predictors of teamwork and commitment as cross-curricular competences (2015)
33. University of Edinburgh Analysis, prediction Examine MOOC data about students who enroled in the same course at least twice
Identify changes in their behaviours between the two enrolments to the same course
(2016)
34. University of Maryland, Baltimore County Monitoring, feedback, reflection Reduce student barriers
Create a community of learners
Improve students’ self-awareness by providing feedback
Provide early alerts to students if their GPA falls below a level
(2012)
35. University of Michigan Monitoring, personalization, reflection Identify at-risk students
Provide personalized feedback to students
(2012)
36. University of New England Monitoring, intervention Foster a sense of community among students studying part-time, at a distance as well as on-campus
Identify students who are struggling in order to provide timely support
Develop a dynamic, systematic and automated process to capture the learning well-being status of students
Encourage peer-to-peer student networking
Disseminate information and connect support staff with the students
(2016)
37. University of North Bengal Prediction Examine the predictive relationship between learners’ pre-entry demographic information and their dropout behaviours
38. University of Rijeka Data mining, analysis Find out factors leading to student success in study
Identify problems timely and increase the course pass rate
(2015)
39. University of Salamanca Information extraction, analysis Extract information useful for teaching/administrative staff, such as interaction of students with peers, teachers, the system, and course contents
Provide teachers with tools to facilitate managerial tasks
(2015)
Support practical learning in a 3D virtual environment, analyse the problems that arisen, and report relevant data to students and teachers (2014)
40. University of Santiago de Compostela Analysis, evaluation Generate automatically reports of learners’ activities that take place in a virtual learning environment
Improve the efficiency of the evaluation process
(2014)
41. University of Sydney Analysis, observation Identify the relationship among student performance, choices of programming languages for study, and times at which a student starts and stops working on an assignment (2016)
42. University of the South Pacific Monitoring Track individual learners’ online and offline interactions with open learning resources (2016)
43. University of Wollongong Analysis, intervention, reflection Visualize patterns of student interactions on discussion forums
Allow instructors to identify at-risk students and potentially high and low performing students for planning interventions, and the extent to which a learner community is developing in a class
(2013)

Allen , W.R. , Fernandes , K. and Whitmer , J. ( 2012 ), “ Analytics in progress: technology use, student characteristics, and student achievement ”, EDUCAUSE Review , 12 August, available at: http://er.educause.edu/articles/2012/8/analytics-in-progress-technology-use-student-characteristics-and-student-achievement (accessed 28 December 2016 ).

Arnold , K.E. and Pistilli , M.D. ( 2012 ), “ Course signals at Purdue: using learning analytics to increase student success ”, The 2nd International Conference on Learning Analytics and Knowledge , Vancouver , pp. 267 - 270 .

Arroway , P. , Morgan , G. , O’Keefe , M. and Yanosky , R. ( 2016 ), “ Learning analytics in higher education ”, EDUCAUSE, available at: https://library.educause.edu/~/media/files/library/2016/2/ers1504la.pdf (accessed 28 February 2017 ).

Atif , A. , Richards , D. , Bilgin , A. and Marrone , M. ( 2013 ), “ A panorama of learning analytics featuring the technologies for the learning and teaching domain ”, The 30th Ascilite Conference , Sydney , pp. 68 - 72 .

Barba , P.D. , Kennedy , G. and Ainley , M. ( 2016 ), “ The role of students’ motivation and participation in predicting performance in a MOOC ”, Journal of Computer Assisted Learning , Vol. 32 No. 3 , pp. 218 - 231 .

Blanton , S.E. ( 2012 ), “ Datamaster: success and failure on a journal of business intelligence ”, EDUCAUSE case studies, available at: http://er.educause.edu/articles/2012/7/datamaster-success-and-failure-on-a-journey-to-business-intelligence (accessed 28 December 2016 ).

Campbell , J.P. and Oblinger , D.G. ( 2007 ), “ Academic analytics ”, EDUCAUSE, available at: https://net.educause.edu/ir/library/pdf/PUB6101.pdf (accessed 28 December 2016 ).

Chacon , F. , Spicer , D. and Valbuena , A. ( 2012 ), “ Analytics in support of student retention and success ”, available at: https://net.educause.edu/ir/library/pdf/ERB1203.pdf (accessed 28 December 2016 ).

Cohen , J. ( 1988 ), Statistical Power Analysis for the Behavioral Sciences , 2nd ed. , Lawrence Erlbaum , Hillsdale, NJ .

Conde , M.A. , Garcia-Penalvo , F.J. , Gomez-Aguilar , D. and Theron , R. ( 2015 ), “ Exploring software engineering subjects by using visual learning analytics techniques ”, IEEE Revista Iberoamericana De Tecnologias Del Aprendizaje , Vol. 10 No. 4 , pp. 242 - 252 .

Corrin , L. and Barba , P. ( 2015 ), “ How do students interpret feedback delivered via dashboards? ”, Proceedings of the 5th International Conference on Learning Analytics and Knowledge, Poughkeepsie, New York, NY, 16-20 March , pp. 430 - 431 .

Cruz-Benito , J. , Theron , R. , Garcia-Penalvo , F.J. , Maderuelo , C. , Perez-Blanco , J.S. , Zazo , H. and Martin-Suarez , A. ( 2014 ), “ Monitoring and feedback of learning processes in virtual worlds through analytics architectures: a real case ”, The 9th Iberian Conference on Information Systems and Technologies , Barcelona , pp. 1126 - 1131 .

Dodge , B. , Whitmer , J. and Frazee , J.P. ( 2015 ), “ Improving undergraduate student achievement in large blended courses through data-driven interventions ”, The 5th International Conference on Learning Analytics and Knowledge , New York, NY , pp. 412 - 413 .

Dyckhoff , A.L. ( 2011 ), “ Implications for learning analytics tools: a meta-analysis of applied research questions ”, International Journal of Computer Information Systems and Industrial Management Applications , Vol. 3 , pp. 594 - 601 .

Fidalgo-Blanco , Á. , Sein-Echaluce , M.L. , García-Peñalvo , F.J. and Conde , M.Á. ( 2015 ), “ Using learning analytics to improve teamwork assessment ”, Computers in Human Behavior , Vol. 47 , June , pp. 149 - 156 .

Firat , M. and Yuzer , T.V. ( 2016 ), “ Learning analytics: assessment of mass data in distance education ”, International Journal on New Trends in Education and their Implications , Vol. 7 No. 2 , available at: www.ijonte.org/FileUpload/ks63207/File/01.mehmet_firat_.pdf (accessed 28 December 2016 ).

Fritz , J. and Kunnen , E. ( 2010 ), “ Using analytics to intervene with underperforming college students ”, available at: www.educom.edu/eli/events/eli-annual-meeting/2010/using-analytics-intervene-underperforming-college-students-innovative-practice (accessed 28 December 2016 ).

Gašević , D. , Dawson , S. and Pardo , A. ( 2016 ), “ How do we start? State and directions of learning analytics adoption ”, International Council for Open and Distance Education, available at: https://icde.memberclicks.net/assets/RESOURCES/dragan_la_report%20cc%20licence.pdf (accessed 28 February 2017 ).

Gewerc , A. , Montero , L. and Lama , M. ( 2014 ), “ Collaboration and social networking in higher education ”, Media Education Research Journal , Vol. 21 No. 42 , pp. 55 - 63 .

Gramoli , V. , Charleston , M. , Jeffries , B. , Koprinska , I. , McGrane , M. , Radu , A. , Viglas , A. and Yacef , K. ( 2016 ), “ Mining autograding data in computer science education ”, The Eighteenth Australasian Computing Education Conference, Canberra , 1-5 February , available at: http://dl.acm.org/citation.cfm?id=2843070 (accessed 28 December 2016 ).

Grush , M. ( 2011 ), “ Monitoring the PACE of student learning: analytics at Rio Salado College ”, Campus Technology , 14 December, available at: https://campustechnology.com/articles/2011/12/14/monitoring-the-pace-of-student-learning-analytics-at-rio-salado-college.aspx

Guitart , I. , More , J. , Duran , J. , Conesa , J. , Baneres , D. and Ganan , D. ( 2015 ), “ A semi-automatic system to detect relevant learning content for each subject ”, The 7th International Conference on Intelligent Networking and Collaborative Systems , Taipei , pp. 301 - 307 .

Harvey , F. ( 2013 ), “ Technology tools for developing, delivering, updating and assessing sustainable high-quality academic programs ”, Society for Information Technology & Teacher Education International Conference , New Orleans, LA , pp. 2131 - 2133 .

Iglesias-Pradas , S. , Ruiz-de-Azcárate , C. and Agudo-Peregrina , Á.F. ( 2015 ), “ Assessing the suitability of student interactions from moodle data logs as predictors of cross-curricular competencies ”, Computers in Human Behavior , Vol. 47 , June , pp. 81 - 89 .

Ihantola , P. , Vihavainen , A. , Ahadi , A. , Butler , M. , Börstler , J. , Edwards , S.H. , Isohanni , E. , Korhonen , A. , Petersen , A. , Rivers , K. , Rubio , M.Á. , Sheard , J. , Skupas , B. , Spacco , J. , Szabo , C. and Toll , D. ( 2015 ), “ Educational data mining and learning analytics in programming: literature review and case studies ”, The 20th Annual Conference on Innovation and Technology in Computer Science Education – Working Group Reports, ACM , New York, NY , pp. 41 - 63 , available at: http://dl.acm.org/citation.cfm?doid=2858796.2858798 (accessed 28 December 2016 ).

Jayaprakash , S.M. , Moody , E.W. , Lauria , E.J.M. , Regan , R. and Baron , J.D. ( 2014 ), “ Early alert of academically at-risk students: an open source analytics initiative ”, Journal of Learning Analytics , Vol. 1 No. 1 , pp. 6 - 47 .

Jones , D. and Woosley , S. ( 2011 ), “ The foundation of MAP-Works: research and theoretical underpinnings of MAP-Works ”, Educational Benchmarking (EBI), available at: www2.indstate.edu/studentsuccess/pdf/The Foundation of MAP-Works.pdf (accessed 28 December 2016 ).

Karkhanis , P.S. and Dumbre , S.S. ( 2015 ), “ A study of application of data mining and analytics in education domain ”, International Journal of Computer Applications , Vol. 120 No. 22 , pp. 23 - 29 .

Kovanović , V. , Joksimović , S. , Gašević , D. , Owers , J. , Scott , A. and Woodgate , A. ( 2016 ), “ Profiling MOOC course returners: how does student behavior change between two course enrollments? ”, The Third ACM Conference on Learning @ Scale , Edinburgh , pp. 269 - 272 .

McAleese , V. and Taylor , L. ( 2012 ), “ Beyond retention: using targeted analytics to improve student success ”, EDUCAUSE Review , 17 July, available at: http://er.educause.edu/articles/2012/7/beyond-retention-using-targeted-analytics-to-improve-student-success (accessed 28 December 2016 ).

Mat , U.B. , Buniyamin , N. , Arsad , P.M. and Kassim , R. ( 2013 ), “ An overview of using academic analytics to predict and improve students’ achievement: a proposed proactive intelligent intervention ”, IEEE 5th Conference on Engineering Education (ICEED) , pp. 126 - 130 .

Mattingly , K.D. , Rice , M.C. and Berge , Z.L. ( 2012 ), “ Learning analytics as a tool for closing the assessment loop in higher education ”, Knowledge Management & E-learning: An International Journal , Vol. 4 No. 3 , pp. 236 - 247 .

Milligan , S. ( 2015 ), “ Crowd-sourced learning in MOOCs ”, The 5th International Conference on Learning Analytics and Knowledge , New York, NY , pp. 151 - 155 .

Nunn , S. , Avella , J.T. , Kanai , T. and Kebritchi , M. ( 2016 ), “ Learning analytics methods, benefits, and challenges in higher education: a systematic literature review ”, Online Learning , Vol. 20 No. 2 , available at: https://olj.onlinelearningconsortium.org/index.php/olj/article/view/790 (accessed 28 December 2016 ).

Papamitsiou , Z. and Economides , A.A. ( 2014 ), “ Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence ”, Educational Technology & Society , Vol. 17 No. 4 , pp. 49 - 64 .

Poitras , E.G. , Naismith , L.M. , Doleck , T. and Lajoie , S.P. ( 2016 ), “ Using learning analytics to identify medical student misconceptions in an online virtual patient environment ”, Online Learning , Vol. 20 No. 2 , pp. 239 - 250 .

Prasad , D. , Totaram , R. and Usagawa , T. ( 2016 ), “ Development of open textbooks learning analytics system ”, International Review of Research in Open and Distributed Learning , Vol. 17 No. 5 , pp. 215 - 234 .

Prinsloo , P. and Slade , S. ( 2014 ), “ Educational triage in open distance learning: walking a moral tightrope ”, International Review of Research in Open and Distance Learning , Vol. 15 No. 4 , pp. 306 - 331 .

Rienties , B. , Boroowa , A. , Cross , S. , Kubiak , C. , Mayles , K. and Murphy , S. ( 2016 ), “ Analytics4action evaluation framework: a review of evidence-based learning analytics interventions at the Open University UK ”, Journal of Interactive Media in Education , Vol. 1 No. 2 , pp. 1 - 13 .

Rienties , B. , Boroowa , A. , Cross , S. , Farrington-Flint , L. , Herodotou , C. , Prescott , L. , Mayles , K. , Olney , T. , Toetenel , L. and Woodthorpe , J. ( 2016 ), “ Reviewing three case-studies of learning analytics interventions at the Open University UK ”, The 6h International Conference on Learning Analytics & Knowledge, ACM , New York, NY , pp. 534 - 535 .

Robinson , C. , Yeomans , M. , Reich , J. , Hulleman , C. and Gehlbach , H. ( 2016 ), “ Forecasting student achievement in MOOCs with natural language processing ”, The 6th International Conference on Learning Analytics & Knowledge , Edinburgh , pp. 383 - 387 .

Sclater , N. and Mullan , J. ( 2017 ), “ Jisc briefing: learning analytics and student success – assessing the evidence ”, available at: http://repository.jisc.ac.uk/6560/1/learning-analytics_and_student_success.pdf (accessed 28 February 2017 ).

Sclater , N. , Peasgood , A. and Mullan , J. ( 2016 ), “ Learning analytics in higher education: a review of UK and international practice ”, available at: www.jisc.ac.uk/reports/learning-analytics-in-higher-education (accessed 28 December 2016 ).

Sisovic , S. , Matetic , M. and Bakaric , M.B. ( 2015 ), “ Mining student data to assess the impact of moodle activities and prior knowledge on programming course success ”, The 16th International Conference on Computer Systems and Technologies , Dublin , pp. 366 - 373 .

Smith , V.C. , Lange , D.R.H. and Huston , D.R. ( 2012 ), “ Predictive modelling to forecast student outcomes and drive effective interventions in online community college courses ”, Journal of Asynchronous Learning Networks , Vol. 16 No. 3 , pp. 51 - 61 .

Star , M. and Collette , L. ( 2010 ), “ GPS: shaping student success one conversation at a time ”, EDUCAUSE, available at: http://er.educause.edu/articles/2010/12/gps-shaping-student-success-one-conversation-at-a-time (accessed 28 December 2016 ).

Tarmazdi , H. , Vivian , R. , Szabo , C. , Falkner , K. and Falkner , N. ( 2015 ), “ Using learning analytics to visualise computer science teamwork ”, The 2015 ACM Conference on Innovation and Technology in Computer Science Education , Vilnius , pp. 165 - 170 .

Toetenel , L. and Rienties , B. ( 2016 ), “ Analysing 157 learning designs using learning analytic approaches as a means to evaluate the impact of pedagogical decision making ”, British Journal of Educational Technology , Vol. 47 No. 5 , pp. 981 - 992 .

Wang , X. , Wen , M. and Rosé , C.P. ( 2016 ), “ Towards triggering higher-order thinking behaviors in MOOCs ”, The 6th International Conference on Learning Analytics & Knowledge , Edinburgh , pp. 398 - 407 .

Wong , G.K.W. and Li , S.Y.K. ( 2016 ), “ Academic performance prediction using chance discovery from online discussion forums ”, IEEE 40th Annual Computer Software and Applications Conference , GA , pp. 706 - 711 .

Yasmine , D. ( 2013 ), “ Application of the classification tree model in predicting learner dropout behaviour in open and distance learning ”, Distance Education , Vol. 34 No. 2 , pp. 218 - 231 .

Further reading

Pirani , J.A. and Albrecht , B. ( 2005 ), University of Phoenix: Driving decisions Through Academic Analytics , Educause Center for Applied Research , available at: https://net.educause.edu/ir/library/pdf/ers0508/cs/ecs0509.pdf (accessed 28 December 2016 ).

Acknowledgements

The work described in this paper was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (UGC/IDS16/15).

Corresponding author

Related articles, all feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

Metaverse Integration in COIL to Improve Intercultural Competence in Higher Education: A Case Study

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, index terms.

Social and professional topics

User characteristics

Cultural characteristics

Recommendations

The use of metaverse for intercultural collaboration in higher education.

The development of intercultural competencies enables individuals to have appropriate knowledge and understanding of diverse cultures and to interact and relate effectively in different multicultural settings. This document aims to present the status of ...

Flipped Classroom Method in Higher Education: A Case of Kazakhstan

This study is set to explore flipped classroom method in Kazakhstan higher education. Teachers from eleven universities participated in the survey. Observation, questioning, counting, and comparison were used as the main research methods. The ...

Exploring the Use of Metaverse for Collaborative Learning in Higher Education: A Scoping Review

Student engagement in higher education is strongly associated with positive learning outcomes; however, not all learning methods are equally engaging or stimulating. Recent technological developments hold the potential to make learning more ...

Information

Published in.

cover image ACM Other conferences

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates, author tags.

  • Collaborative online international learning
  • Educational innovation
  • Higher education
  • Immersive technology
  • Intercultural education
  • Research-article
  • Refereed limited

Contributors

Other metrics, bibliometrics, article metrics.

  • 0 Total Citations
  • 0 Total Downloads
  • Downloads (Last 12 months) 0
  • Downloads (Last 6 weeks) 0

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

HTML Format

View this article in HTML Format.

Share this Publication link

Copying failed.

Share on social media

Affiliations, export citations.

  • Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
  • Download citation
  • Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Information

  • Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

  • Active Journals
  • Find a Journal
  • Proceedings Series
  • For Authors
  • For Reviewers
  • For Editors
  • For Librarians
  • For Publishers
  • For Societies
  • For Conference Organizers
  • Open Access Policy
  • Institutional Open Access Program
  • Special Issues Guidelines
  • Editorial Process
  • Research and Publication Ethics
  • Article Processing Charges
  • Testimonials
  • Preprints.org
  • SciProfiles
  • Encyclopedia

education-logo

Article Menu

case study higher education evaluation

  • Subscribe SciFeed
  • Google Scholar
  • on Google Scholar
  • Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Systematic review: revisiting challenge-based learning teaching practices in higher education.

case study higher education evaluation

1. Introduction

1.1. entrepreneurial mindset and innovative capabilities, 1.2. teaching practices, 2. materials and methods, 2.1. protocol, 2.2. eligibility criteria, information sources, and search strategy, 2.3. selection process, 2.4. analysis, 2.5. study risk of bias assessment, 3.1. teaching practice insights from each reviewed paper, 3.2. four dimensions of teaching practices in challenge-based learning, 3.2.1. pedagogical approaches in cbl, 3.2.2. technological integration in cbl, 3.2.3. industry and professional engagement in cbl, 3.2.4. support and development in cbl, 3.3. core teaching practices within the context of cbl, 4. discussion, limitations, 5. conclusions, supplementary materials, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

  • Van den Beemt, A.; Vázquez-Villegas, P.; Puente, S.G.; O’Riordan, F.; Gormley, C.; Chiang, F.K.; Leng, C.T.; Caratozzolo, P.; Zavala, G.; Membrillo-Hernández, J. Taking the Challenge: An Exploratory Study of the Challenge-Based Learning Context in Higher Education Institutions across Three Different Continents. Educ. Sci. 2023 , 13 , 234. [ Google Scholar ] [ CrossRef ]
  • Leijon, M.; Gudmundsson, P.; Staaf, P.; Christersson, C. Challenge-Based Learning in Higher Education–A Systematic Literature Review. Innov. Educ. Teach. Int. 2022 , 59 , 609–618. [ Google Scholar ] [ CrossRef ]
  • Gyldendahl, C.; Georgsen, M.; Dau, S. A Systematic Review of Concepts Related to Reflective Practice-Based Learning with a Focus on Theoretical Positions. In Proceedings for the European Conference on Reflective Practice-Based Learning ; Georgsen, M., Dau, S., Helverskov, L., Eds.; Aalborg University Press: Aalborg, Denmark, 2023; pp. 31–51. [ Google Scholar ]
  • Gallagher, S.; Savage, T. Challenge-Based Learning in Higher Education: An Exploratory Literature Review. Teach. High. Educ. 2020 , 28 , 1135–1157. [ Google Scholar ] [ CrossRef ]
  • Pepin, B.; Kock, Z.J. Students’ Use of Resources in a Challenge-Based Learning Context Involving Mathematics. Int. J. Res. Undergrad. Math. Educ. 2021 , 7 , 306–327. [ Google Scholar ] [ CrossRef ]
  • López-Caudana, E.; Ruiz, S.; Calixto, A.; Nájera, B.; Castro, D.; Romero, D.; Luna, J.; Vargas, V.; Legorreta, I.; Lara-Prieto, V.; et al. A Personalized Assistance System for the Location and Efficient Evacuation in Case of Emergency: TECuidamos, A Challenge-Based Learning Derived Project Designed to Save Lives. Sustainability 2022 , 14 , 4931. [ Google Scholar ] [ CrossRef ]
  • Gudoniene, D.; Paulauskaite-Taraseviciene, A.; Daunoriene, A.; Sukacke, V. A Case Study on Emerging Learning Pathways in SDG-Focused Engineering Studies through Applying CBL. Sustainability 2021 , 13 , 8495. [ Google Scholar ] [ CrossRef ]
  • Nizami, M.Z.I.; Xue, V.W.; Wong, A.W.Y.; Yu, O.Y.; Yeung, C.S.; Chu, C.H. Challenge-Based Learning in Dental Education. Dent. J. 2023 , 11 , 14. [ Google Scholar ] [ CrossRef ]
  • Darsø, L. Is There a Formula for Innovation? In Børsens Ledelseshåndbøger ; Børsen Forlag: Copenhagen, Denmark, 2003. [ Google Scholar ]
  • Drucker, P. Innovation and Entrepreneurship ; HarperCollins: New York, NY, USA, 1985. [ Google Scholar ]
  • Scharmer, O. Theory U: Leading from the Future as It Emerges. In The Social Technology of Presencing ; Berrett-Koehler Publishers: San Francisco, CA, USA, 2009. [ Google Scholar ]
  • Sørensen, E.; Torfing, J. Enhancing Collaborative Innovation in the Public Sector. Adm. Soc. 2011 , 43 , 842–868. [ Google Scholar ] [ CrossRef ]
  • Kirketerp, A. Fortagsomhedsdidaktik. Ph.D. Thesis, Aarhus University, Aarhus, Denmark, 2010. Available online: http://cei.au.dk/fileadmin/cei/Billeder/Publikationer/PHD_Foretagsomhedsdidaktik_310510.pdf (accessed on 12 June 2024).
  • Bandura, A. Perceived Self-Efficacy in Cognitive Development and Functioning. Educ. Psychol. 1993 , 28 , 117–148. [ Google Scholar ] [ CrossRef ]
  • Fardella, C.; Carvajal, F. Los Estudios Sociales de la Práctica y la Práctica Como Unidad de Estudio. Psicoperspectivas 2018 , 17 , 91–102. [ Google Scholar ] [ CrossRef ]
  • Galdames-Calderón, M. Prácticas Directivas de Liderazgo Distribuido: Creación de Oportunidades de Desarrollo Profesional Docente para la Mejora Escolar. Ph.D. Thesis, Universitat Autònoma de Barcelona, Barcelona, Spain, 2021. Available online: http://hdl.handle.net/10803/671984 (accessed on 12 June 2024).
  • Galdames-Calderón, M. Distributed Leadership: School Principals’ Practices to Promote Teachers’ Professional Development School Improvement. Educ. Sci. 2023 , 13 , 715. [ Google Scholar ] [ CrossRef ]
  • Stavnskær, A. The Innovation Circle: A Guide to Innovative Learning Processes ; Huset Venture: Aalborg, Denmark, 2019. [ Google Scholar ]
  • Lozano Rodríguez, A.; Alvarado García, M.; Llaven Aguilar, M. Desarrollo de Competencias en el Contexto del Semestre i: Un Estudio de Caso. Educ. Knowl. Soc. 2019 , 20 , 14. [ Google Scholar ] [ CrossRef ]
  • Doulougeri, K.; Vermunt, J.; Bombaerts, G.; Bots, M. Analyzing Student-Teacher Interactions in Challenge-Based Learning. In Towards a New Future in Engineering Education, New Scenarios that European Alliances of Tech Universities Open Up ; Universitat Politècnica de Catalunya: Barcelona, Spain, 2022; pp. 252–262. [ Google Scholar ] [ CrossRef ]
  • Campos, E.; Núñez, S.; Enríquez, J.; Castaño, R.; Escamilla, J.; Hosseini, S. Educational Model Transition: Student Evaluation of Teaching Amid the COVID-19 Pandemic. Front. Educ. 2022 , 7 , 991654. [ Google Scholar ] [ CrossRef ]
  • Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.; Akl, E.; Brennan, S.; et al. The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ]
  • Newman, M.; Gough, D. Systematic Reviews in Educational Research Methodology, Perspectives, and Application. In Systematic Reviews in Educational Research Methodology, Perspectives, and Application ; Zawacki-Richter, O., Kerres, M., Bedenlier, S., Bond, M., Buntins, K., Eds.; Springer: Berlin/Heidelberg, Germany, 2020; pp. 3–22. [ Google Scholar ]
  • Galdames-Calderón, M. Protocol for Systematic Review: Revisiting Challenge-Based Learning—Teaching Practices in Higher Education. INSPLAY Protocol 2024 , 202450140. [ Google Scholar ] [ CrossRef ]
  • Barrington, M.; D’Souza, R.; Mascha, E.; Narouze, S.; Kelley, G. Systematic Reviews and Meta-Analyses in Regional Anesthesia and Pain Medicine (Part I): Guidelines for Preparing the Review Protocol. Anesth. Analg. 2023 , 138 , 379–394. [ Google Scholar ] [ CrossRef ]
  • Pranckute, R. Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World. Publications 2021 , 9 , 12. [ Google Scholar ] [ CrossRef ]
  • Zhu, J.; Liu, W. A Tale of Two Databases: The Use of Web of Science and Scopus in Academic Papers. Scientometrics 2020 , 123 , 321–335. [ Google Scholar ] [ CrossRef ]
  • Haddaway, N.R.; Page, M.J.; Pritchard, C.C.; McGuinness, L.A. PRISMA2020: An R Package and Shiny App for Producing PRISMA 2020-Compliant Flow Diagrams, with Interactivity for Optimized Digital Transparency and Open Synthesis. Campbell Syst. Rev. 2022 , 18 , e1230. [ Google Scholar ] [ CrossRef ]
  • Braun, V.; Clarke, V. Conceptual and Design Thinking for Thematic Analysis. Qual. Psychol. 2022 , 9 , 3. [ Google Scholar ] [ CrossRef ]
  • Sundler, A.J.; Lindberg, E.; Nilsson, C.; Palmér, L. Qualitative Thematic Analysis Based on Descriptive Phenomenology. Nurs. Open 2019 , 6 , 733–739. [ Google Scholar ] [ CrossRef ] [ PubMed ]
  • Abril-López, D.; Carrillo, D.L.; González-Moreno, P.M.; Delgado-Algarra, E.J. How to Use Challenge-Based Learning for the Acquisition of Learning to Learn Competence in Early Childhood Preservice Teachers: A Virtual Archaeological Museum Tour in Spain. Front. Educ. 2021 , 6 , 714684. [ Google Scholar ] [ CrossRef ]
  • Agüero, M.; López, L.; Pérez, J. Challenge-Based Learning as a Professional Learning Model: Universidad Europea and Comunica +A Program Case Study. Rev. Comun. 2019 , 149 , 1–24. [ Google Scholar ] [ CrossRef ]
  • De Aldecoa, C.Y.; Gómez-Trigueros, I.M. Challenges with Complex Situations in the Teaching and Learning of Social Sciences in Initial Teacher Education. Soc. Sci. 2022 , 11 , 295. [ Google Scholar ] [ CrossRef ]
  • De Stefani, P.; Han, L. An Inter-University CBL Course and Its Reception by the Student Body: Reflections and Lessons Learned (in Times of COVID-19). Front. Educ. 2022 , 7 , 853699. [ Google Scholar ] [ CrossRef ]
  • Dieck-Assad, G.; Ávila-Ortega, A.; Peña, G. Comparing Competency Assessment in Electronics Engineering Education with and without Industry Training Partner by Challenge-Based Learning Oriented to Sustainable Development Goals. Sustainability 2021 , 13 , 10721. [ Google Scholar ] [ CrossRef ]
  • Franco, E.; González-Peño, A.; Trucharte, P.; Martínez-Majolero, V. Challenge-Based Learning Approach to Teach Sports: Exploring Perceptions of Teaching Styles and Motivational Experiences among Student Teachers. J. Hosp. Leis. Sport Tour. Educ. 2023 , 32 , 100432. [ Google Scholar ] [ CrossRef ]
  • Gaskins, W.B.; Johnson, J.; Maltbie, C.; Kukreti, A.R. Changing the Learning Environment in the College of Engineering and Applied Science Using Challenge-Based Learning. Int. J. Eng. Pedagogy 2015 , 5 , 33–41. [ Google Scholar ] [ CrossRef ]
  • Khambari, M.N.M. Instilling Innovativeness, Building Character, and Enforcing Camaraderie through Interest-Driven Challenge-Based Learning Approach. Res. Pract. Technol. Enhanc. Learn. 2019 , 14 , 19. [ Google Scholar ] [ CrossRef ]
  • Kohn Rådberg, K.; Lundqvist, U.; Malmqvist, J.; Svensson, O.H. From CDIO to Challenge-Based Learning Experiences—Expanding Student Learning as Well as Societal Impact? Eur. J. Eng. Educ. 2020 , 45 , 22–37. [ Google Scholar ] [ CrossRef ]
  • Membrillo-Hernández, J.; Ramírez-Cadena, M.D.J.; Caballero-Valdés, C.; Ganem-Corvera, R.; Bustamante-Bello, R.; Ordoñez-Díaz, J.A.B.; Elizalde, H. Challenge-Based Learning: The Case of Sustainable Development Engineering at the Tecnologico de Monterrey, Mexico City Campus. Int. J. Eng. Pedagogy 2018 , 8 , 137–144. [ Google Scholar ] [ CrossRef ]
  • Mesutoglu, C.; Bayram-Jacobs, D.; Vennix, J.; Limburg, A.; Pepin, B. Exploring Multidisciplinary Teamwork of Applied Physics and Engineering Students in a Challenge-Based Learning Course. Res. Sci. Technol. Educ. 2022 , 42 , 639–657. [ Google Scholar ] [ CrossRef ]
  • Meyer, H. Teachers’ Thoughts on Student Decision Making during Engineering Design Lessons. Educ. Sci. 2018 , 8 , 9. [ Google Scholar ] [ CrossRef ]
  • Nguyen, H.; Gijlers, H.; Pisoni, G. Identifying Struggling Teams in Online Challenge-Based Learning. High. Educ. Skills Work.-Based Learn. 2023 , 13 , 233–248. [ Google Scholar ] [ CrossRef ]
  • Piccardo, C.; Goto, Y.; Koca, D.; Aalto, P.; Hughes, M. Challenge-Based, Interdisciplinary Learning for Sustainability in Doctoral Education. Int. J. Sustain. High. Educ. 2022 , 23 , 1482–1503. [ Google Scholar ] [ CrossRef ]
  • Tang, A.C.Y.; Chow, M.C.M. Learning Experience of Baccalaureate Nursing Students with Challenge-Based Learning in Hong Kong: A Descriptive Qualitative Study. Int. J. Environ. Res. Public Health 2021 , 18 , 6293. [ Google Scholar ] [ CrossRef ]
  • Yoong, S.; Turon, H.; Grady, A.; Hodder, R.; Wolfenden, L. The Benefits of Data Sharing and Ensuring Open Sources of Systematic Review Data. J. Public Health 2022 , 44 , e582–e587. [ Google Scholar ] [ CrossRef ]
  • Galdames-Calderón, M.; Stavnskær Pedersen, A.; Rodriguez Gomez, D. Replication Data for: “Systematic Review: Revisiting Challenge-Based Learning Teaching Practices in Higher Education”. CORA. Repositori de Dades de Recerca 2024 , V1 . [ Google Scholar ] [ CrossRef ]
  • Sablatzky, T. The Delphi Method. Hypothesis: Res. J. Health Inf. Prof. 2022 , 34 , 1–6. [ Google Scholar ] [ CrossRef ]
  • Lozano, R.; Barreiro-Gen, M.; Lozano, F.; Sammalisto, K. Teaching Sustainability in European Higher Education Institutions: Assessing the Connections Between Competences and Pedagogical Approaches. Sustainability 2019 , 11 , 1602. [ Google Scholar ] [ CrossRef ]
  • Oyetade, K.; Zuva, T.; Harmse, A. Technology Adoption in Education: A Systematic Literature Review. Adv. Sci. Technol. Eng. Syst. J. 2020 , 5 , 108–112. [ Google Scholar ] [ CrossRef ]
  • Wagino, W.; Maksum, H.; Purwanto, W.; Krismadinata, K.; Suhendar, S.; Koto, R. Exploring the Full Potential of Collaborative Learning and E-Learning Environments in Universities: A Systematic Review. TEM J. 2023 , 12 , 1772–1785. [ Google Scholar ] [ CrossRef ]
  • Alonso-Muñoz, S.; Torrejón-Ramos, M.; Medina-Salgado, M.S.; González Sánchez, R. Trends in Mentoring at Higher Education: A Bibliometric Analysis. In Proceedings Book: International Conference on Innovation, Documentation and Education ; INNODOCT/22 ; Editorial Universitat Politècnica de València: Valencia, Spain, 2022. [ Google Scholar ] [ CrossRef ]
  • Roberts, C.; Khanna, P.; Bleasel, J.; Lane, S.; Burgess, A.; Charles, K.; Howard, R.; O’Mara, D.; Haq, I.; Rutzou, T. Student Perspectives on Programmatic Assessment in a Large Medical Programme: A Critical Realist Analysis. Med. Educ. 2022 , 56 , 901–914. [ Google Scholar ] [ CrossRef ] [ PubMed ]
  • Børte, K.; Nesje, K.; Lillejord, S. Barriers to Student Active Learning in Higher Education. Teach. High. Educ. 2023 , 28 , 597–615. [ Google Scholar ] [ CrossRef ]
  • Oonk, C.; Gulikers, J.T.; den Brok, P.J.; Wesselink, R.; Beers, P.J.; Mulder, M. Teachers as Brokers: Adding a University-Society Perspective to Higher Education Teacher Competence Profiles. High. Educ. 2020 , 80 , 701–718. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

ReferenceTitleCountryContextStudy Design
1Abril-López et al. [ ]How to Use Challenge-Based Learning for the Acquisition of Learning to Learn Competence in Early Childhood Preservice Teachers: A Virtual Archaeological Museum Tour in SpainSpainEducation level: Higher Education
Field of degree: Teaching and learning of social sciences and teaching and learning of natural sciences with early childhood preservice teachers
Format: Presential
Quantitative, quasi-experimental design
2Agüero et al. [ ]Challenge based learning as a professional learning model. Universidad Europea and Comunica +A program case studySpainEducation level: Higher Education
Field of degree: Advertising communication degree
Format: Presential
Qualitative, questionnaire data
3De Aldecoa and Gómez-Trigueros [ ]Challenges with Complex Situations in the Teaching and Learning of Social Sciences in Initial Teacher EducationAndorraEducation level: Higher Education
Field of degree: Bachelor’s degree in teaching and learning
Format: Presential
Qualitative.
4De Stefani and Han [ ]An Inter-University CBL Course and Its Reception by the Student Body: Reflections and Lessons Learned (in Times of COVID-19)Austria, France, Germany, Italy, Lithuania, Norway, and SpainEducation level: Higher Education
Field of degree: Many disciplinary fields, including social sciences and natural sciences
Format: Online
Qualitative
5Dieck-Assad et al. [ ]Comparing competency assessment in electronics engineering education with and without industry training partner by Challenge-Based Learning oriented to sustainable development goalsMexicoEducation level: Higher Education
Field of degree: Mechatronics engineering, digital systems and robotics engineering, biomedical engineering, and other engineering such as innovation engineering
Format: Presential
Quantitative
6Franco et al. [ ]Challenge-Based Learning approach to teach sports: Exploring perceptions of teaching styles and motivational experiences among students teachersSpainEducation level: Higher Education
Field of degree: Physical activity and sport sciences
Format: Presential
Quasi-experimental study with
experimental and control groups
7Gaskins et al. [ ].Changing the Learning Environment in the College of Engineering and Applied Science Using Challenge Based LearningUSAEducation level: Higher Education
Field of degree: Department of biomedical, chemical, and environmental engineering
Format: Presential
Experimental design
8Gudoniene et al. [ ].A Case Study on Emerging Learning Pathways in SDG-Focused Engineering Studies through Applying CBLLithuaniaEducation level: Higher Education
Field of degree: Engineering education
Format: Presential
Qualitative, case study
9Khambari [ ]Instilling innovativeness, building character, and enforcing camaraderie through interest-driven Challenge-Based Learning approachMalaysiaEducation level: Higher Education
Field of degree: Educational technology course
Format: Presential
Qualitative
10Kohn Radberg et al. [ ]From CDIO to Challenge-Based Learning experiences-expanding student learning as well as societal impact?SwedenEducation level: Higher Education
Field of degree: Engineering degree
Format: Presential
Qualitative, case study
11López-Caudana et al. [ ].A Personalized Assistance System for the Location and Efficient Evacuation in Case of Emergency: TECuidamos, a Challenge-Based Learning Derived Project Designed to Save LivesMexicoEducation level: Higher Education
Field of degree: Telecommunications and electronic systems engineering
Format: Presential
Experimental design
12Membrillo-Hernández et al. [ ]Challenge-Based Learning: The Case of Sustainable Development Engineering at the Tecnologico de Monterrey, Mexico City Campus.MexicoEducation level: Higher Education
Field of degree: Sustainable development engineering
Format: Presential (i-week and i-semester)
Experimental design
13Mesutoglu et al. [ ]Exploring multidisciplinary teamwork of applied physics and engineering students in a Challenge-Based Learning courseNetherlandsEducation level: Higher Education
Field of degree: Applied physics and engineering
Format: Presential
Qualitative, case study
14Meyer [ ]Teachers’ Thoughts on Student Decision Making During Engineering Design LessonsUSAEducation level: Higher Education
Field of degree: Engineering design
Format: Presential
Mixed methods
15Nguyen et al. [ ]Identifying struggling teams in online Challenge-Based LearningNetherlandsEducation level: Higher Education
Field of degree: Financial technology course
Format: Online
Qualitative, questionnaire data
16Nizami et al. [ ]Challenge-Based Learning in Dental Education.ChinaEducation level: Higher Education
Field of degree: Dental education
Conceptual design
17Pepin and Kock [ ]Students’ Use of Resources in a Challenge-Based Learning Context Involving MathematicsNetherlandsEducation level: Higher Education
Field of degree: Mechanical Engineering, data science, industrial engineering, psychology, and technology
Format: Online
Qualitative, case study
18Piccardo et al. [ ]Challenge-Based, interdisciplinary learning for sustainability in doctoral education.Finland and SwedenEducation level: Higher Education
Field of degree: Life sciences, physical sciences and engineering, and social sciences and humanities
Format: Presential
Qualitative, questionnaire data
19Tang and Chow [ ]Learning Experience of Baccalaureate Nursing Students with Challenge-Based Learning in Hong Kong: A Descriptive Qualitative StudyChinaEducation level: Higher Education
Field of degree: Nursing program
Format: Presential
Qualitative
20Van den Beemt et al. [ ]Taking the Challenge: An Exploratory Study of the Challenge-Based Learning Context in Higher Education Institutions across Three Different ContinentsMexico, Netherlands,
Ireland, and China
Education level: Higher Education
Field of degree: Engineering education
Format: Presential
Comparative case study
AuthorsPerspectives on Teaching Practices in CBL
Abril-López et al. [ ]; Dieck-Assad et al. [ ];
Gaskins et al. [ ];
Van den Beemt et al. [ ]
Emphasize the teacher’s role as a facilitator and guide, integrating support with resources to enhance students’ autonomous learning, critical thinking, problem-solving, and readiness for future challenges. Mention the need for teachers to adapt teaching strategies and develop “learning to learn” competencies.
Agüero et al. [ ];
De Stefani and Han, [ ];
Tang and Chow [ ]
Highlight the transition from knowledge source to facilitator, fostering a collaborative, participatory experience and preparing students for professional demands through the integration of theory and practice.
De Aldecoa and Gómez-Trigueros [ ];
Mesutoglu et al. [ ]
Discuss the multifaceted role of teachers in promoting interdisciplinary work and guiding students through social challenges using ICTs, enhancing digital competencies, and involving students in decision-making and innovative solution development.
Franco et al. [ ];
Gudoniene et al. [ ];
Meyer [ ];
Nguyen et al. [ ]
Describe the adaptive roles of teachers in enhancing engagement, supporting autonomy, and balancing structured support with student-led learning. Stress the importance of training for teachers and professional development.
Khambari [ ];
López-Caudana et al. [ ]; Membrillo Hernández et al. [ ]; Nizami et al. [ ];
Piccardo et al. [ ]
Focus on the critical importance of tutors as resources themselves, organizing project implementation, connecting students with external stakeholders, and guiding multidisciplinary collaboration.
Pepin and Kock [ ]; Kohn Radberg et al. [ ]Detail the shift of teachers to coach-like roles, fostering learning through feedback, taking a process-oriented perspective, and guiding students with different disciplinary backgrounds through challenges.
Teaching PracticesDescription and Conceptualization
Shifting from
instructor to
facilitator
Teachers’ roles evolve to focus on learning facilitation and support rather than direct instruction and shifting from a traditional teaching role to that of a coach or facilitator [ , , , , , ].
Facilitating the
learning process
Teachers guide students through CBL, fostering autonomy in learning [ ], nurturing entrepreneurial skills [ , ], and enhancing critical thinking abilities, thereby shaping proactive and dedicated community members [ , ].
Creating
collaborative learning environments
Teachers enhance collaborative learning [ ] by establishing positive classrooms that promote teamwork and guide problem-solving [ ] while also supporting student autonomy through valuing their feelings and choices and creating an open environment for expression [ ] and decision-making [ ].
Promoting critical thinking and
innovation
Teachers promote critical thinking and innovation [ ] through holistic methodologies, enhancing the practical application of theoretical knowledge beyond the confines of the classroom [ , ] and involving students in taking action and developing innovative solutions [ , ] for sustainable development [ , , ].
Guiding research questions and
problem-solving
Educators guide students through a multifaceted process in CBL [ ], where they assist in navigating complex questions and solving problems by immersing students in a mix of conceptual, procedural, and attitudinal learning [ ]. This approach includes an iterative cycle [ , ] of three phases of CBL framework: “engage”, “investigate”, and “act” [ ] and the related processes, such as analysis, diagnosis, observation, research, strategy development, decision-making, design, evaluating feasibility and environmental impact, implementation, and assessment. Consequently, it cultivates essential skills in research, analysis, and information management among students [ ].
Encouraging active learningThe teacher’s role encompasses empowering students to become self-directed learners [ , ] co-responsible for the creation of knowledge [ ] who take ownership of their education [ , ], preparing them to master the skill of learning to learn [ ] and fulfilling meaningful and lifelong learning [ ] through active learning [ ] or learning by doing [ ].
Designing challengesBy connecting students with real-world problems observed in their communities [ ], teachers create engaging [ ] and motivating challenges with global importance [ , ] based on students’ interests [ ], integrating adaptable difficulty levels to cater to diverse abilities [ ] and ensuring personalized and inclusive learning experiences [ ].
Teaching PracticesDescription and Conceptualization
Using digital
technology
In response to the shift from face-to-face to online delivery of CBL [ ] prompted by COVID-19 [ ] or the use of blended formats [ ], teachers have been pivotal in incorporating technology [ ] and ICTs to cultivate students’ digital competencies [ , , ], establishing ICT integration as an essential element of modern teaching practices in CBL [ ].
Teaching PracticesDescription and Conceptualization
Collaborating with industry
professionals
Teachers work with industry to define real-world challenges, integrating professional standards or stakeholders and resources into the learning experience [ , , , , , , ].
Facilitating the
integration of
professional practices
Teachers guide students in crafting projects that comply with both professional and ecological standards, thus supporting the Sustainable Development Goals (SDGs) [ , , , ] and enhancing student employability [ , ].
Guiding students in managing project
resources
Teachers have emerged as the pivotal resource [ ], securing and utilizing both external materials and their specialized knowledge to support practical learning effectively [ , ]. By collaborating with industrial partners and leveraging their expertise in technical domains, tutors form an integral part of the instructional team that significantly enhances the practical learning experience [ ].
Teaching PracticesDescription and Conceptualization
Encouraging
self-regulated learning
Teachers encourage students to regulate their own learning processes, fostering autonomy and self-regulated learning [ , ] and enhancing motivation [ , ] and persistence [ ].
Engaging in continued professional
development
Teachers undergo professional development to become facilitators and coaches in CBL environments [ , , , ], and additional training is useful to ensure a comprehensive understanding of CBL processes and their successful implementation [ , ].
Facilitating
interdisciplinary
communication
Teachers facilitate communication among students from different disciplines [ ], encouraging multidisciplinary collaboration [ , , , , , ] and inter-/transdisciplinary learning [ ].
Fostering resilience and providing supportTeachers aid students in overcoming challenges with supportive feedback [ ] and resilience-building [ ] while striving to develop their competence, fostering a sense of capability and accomplishment [ ].
Preparing learners for future challengesTeachers equip students for the demands of the real world, nurturing skills such as leadership [ , ], creativity [ , ], ethical problem-solving [ ], teamwork [ ], interpersonal skills [ ], and entrepreneurial skills [ , , ] to acquire 21st-century skills [ , , ].
Supporting student
decision-making
Teachers engage students in decision-making processes [ ], though they may require further training to support internal cognitive processes [ ].
Providing feedbackTeachers bear the responsibility of assessing student performance, offering structured guidance, and confirming that learning objectives are achieved [ ], striking a balance between the industrial partner’s needs and the competencies that students must acquire [ ]. Evaluations should prioritize learning, considering the shift away from simply meeting exam criteria [ ].
The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

Galdames-Calderón, M.; Stavnskær Pedersen, A.; Rodriguez-Gomez, D. Systematic Review: Revisiting Challenge-Based Learning Teaching Practices in Higher Education. Educ. Sci. 2024 , 14 , 1008. https://doi.org/10.3390/educsci14091008

Galdames-Calderón M, Stavnskær Pedersen A, Rodriguez-Gomez D. Systematic Review: Revisiting Challenge-Based Learning Teaching Practices in Higher Education. Education Sciences . 2024; 14(9):1008. https://doi.org/10.3390/educsci14091008

Galdames-Calderón, Marisol, Anni Stavnskær Pedersen, and David Rodriguez-Gomez. 2024. "Systematic Review: Revisiting Challenge-Based Learning Teaching Practices in Higher Education" Education Sciences 14, no. 9: 1008. https://doi.org/10.3390/educsci14091008

Article Metrics

Supplementary material.

ZIP-Document (ZIP, 665 KiB)

Further Information

Mdpi initiatives, follow mdpi.

MDPI

Subscribe to receive issue release notifications and newsletters from MDPI journals

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

  • We're Hiring!
  • Help Center

paper cover thumbnail

Issues in Higher Education Program Evaluation

Profile image of Melvin Hall

Related Papers

New Directions for Institutional Research

Elizabeth Wroblewski

case study higher education evaluation

American Journal of Evaluation

Yvonna Lincoln

Canadian Journal of Higher Education

Colin P Neufeldt

Post-secondary institutions carry out cyclical program reviews (CPRs) to assess educational effectiveness. CPRs often use both qualitative and quantitative data analyses with the aim of improving teaching and learning. Though most of the CPR review studies identify various factors for this purpose, they fail to identify measures/indicators that are relevant and practical for the institutional decision-making process. Our main objectives for this article are two-fold: first, we identify and list variables that are measurable and sort them into clusters/groups that are relevant to all programs, and second, we critically assess the relevance of these indicators to program review in a small-sized, post-secondary institution.

Ana Maria Graffigna

Evaluative practices at Argentine Universities have been realized in two ways: the first related to the accreditation of university programs and the second related to the evaluation of the universities as educational institutions. Bearing this in mind, this study presents an analysis of the concepts of evaluation, explicit and implicit in the processes, revealing the underlying logics. Also the procedures carried out by the institutions for implementing self-assessment processes and their relationships to the external evaluation are categorized focusing on the subjects and objects of evaluation in each case. A categorical discourse analysis is carried out, which is crossed with the documental study. The present work provides input for the strategic situational planning of the University, as a starting point to strenghten the evaluation process in the institutional context, thus enabling the articulation of practices, economy of effort and quality assurance of the assessment at the institutional level.

BMC Medical Education

Monica Lypson

Alan Lindsay

Munir Majdalawieh

The aim of this book is to propose a holistic conceptual framework for “program review” to assess the effectiveness and viability of an academic program and to improve the academic program and the education of the students. The proposed framework consists of three main components: program review process, program review principles and the program review measurement matrix. The framework and the methods used to create it will provide an opportunity to colleges and universities to undertake a robust and targeted approach to proactively and continuously review their academic programs to ensure that programs are functioning at the highest possible levels of academic quality and are consistent with the mission of the college or the university. The framework will transform the “program review” process into a continuing assessment activity rather than a periodic event.

Association For Institutional Research

Gita Pitter

JIm Carifio

Yüksek Öğretim

Prof. Dr. Nurdan Kalayci

Changing economic, cultural, political, and social conditions worldwide have a big impact on higher education. Under the influence of changing conditions, the functional scope of higher education institutions has expanded and new functions have been added. In addition to these changes, the demand for higher education institutions is increasing day by day in terms of education, research, and service to society. The expanding functions of higher education and its deepening impact on society call for quality activities of higher education institutions. Therefore, institutional quality evaluation processes are carried out in higher education institutions. This study aims to analyze and compare institutional quality evaluation processes applied in Turkish, European, and American higher education systems. The findings obtained are important as they will contribute to the Higher Education Quality Council of Turkey, quality commissions in higher education institutions, and other researchers who will conduct scientific studies on this subject. It is a descriptive and qualitative study whose sample consists of institutional quality evaluation agencies from Turkey, England, Norway, Finland, and the United States of America. The data in the study were collected and analyzed by applying the document analysis method. The findings indicate that institutional external evaluation or audit models are used in Turkey, England, Norway, and Finland while an institutional accreditation system is used in the USA. Although the quality evaluation processes applied are generally similar in terms of basic objectives, assessed dimensions, assessment approach, people involved in the implementation of the assessment, and assessment type, there are differences in aspects such as the management, coordination, and recognition practices of the countries’ higher education systems. Taking into account national circumstances, the institutional quality evaluation processes in the Turkish higher education system should be organized and implemented in a systematic way to ensure quality higher educational practice. Keywords: Higher education, higher education evaluation, institutional quality evaluation.

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED TOPICS

  •   We're Hiring!
  •   Help Center
  • Find new research papers in:
  • Health Sciences
  • Earth Sciences
  • Cognitive Science
  • Mathematics
  • Computer Science
  • Academia ©2024
  • DOI: 10.69593/ajahe.v4i03.84
  • Corpus ID: 271079373

Assessing English Language Proficiency: A Case Study of English Version Students of Higher Secondary Level of Bangladesh

  • Alia Rawshan Banu
  • Published in ACADEMIC JOURNAL ON ARTS &amp… 8 July 2024
  • Education, Linguistics
  • ACADEMIC JOURNAL ON ARTS &amp; HUMANITIES EDUCATION

Related Papers

Showing 1 through 3 of 0 Related Papers

IMAGES

  1. (PDF) Evaluation of Higher Education Pedagogy for Continuous

    case study higher education evaluation

  2. Evaluation process of the higher education study programmes The

    case study higher education evaluation

  3. Case Study Evaluation Form Template

    case study higher education evaluation

  4. PPT

    case study higher education evaluation

  5. 49 Free Case Study Templates ( + Case Study Format Examples + )

    case study higher education evaluation

  6. (PDF) The Assessment of Higher Education Quality from the Perspective

    case study higher education evaluation

VIDEO

  1. Les Roches Case Study (Higher Education)

  2. Digital Leadership-A New Perspective of Higher Education Evaluation

  3. Center for World University Rankings (CWUR) Report 2024

  4. Video Case Study: Liverpool John Moores University

  5. Evaulate (HCC Libraries Academic Research Video 2)

  6. 4.1-Evaluation:-Continuous and comprehensive Evaluation (CCE) B.Ed 3rd semester #jammuuniversitybed

COMMENTS

  1. Competency-based assessment tools for engineering higher education: a

    The findings of this study highlight the potential of competency-based assessment tools to address the challenges faced by higher education, particularly in engineering disciplines. The two assessment instruments—a competence rubric and a checklist binary evaluation index—developed and tested in this research offer a significant departure ...

  2. A Case Study for the 10-Step Approach to Program Evaluation

    See Rubric for higher education program evaluation case study. Table 2Rubric for Higher Education Program Evaluation Case Study. There was one evaluator on this case that performed all analysis. Based on the interviews and survey responses, it was clear that the participants enjoyed the program and would have wanted more content that connected ...

  3. Assessment & Evaluation in Higher Education

    Assessment & Evaluation in Higher Education welcomes research-based, reflective or theoretical studies which help to illuminate the practice of assessment and evaluation in higher education. The journal is aimed at all higher education practitioners, irrespective of discipline. It sets out to provide readily accessible, up-to-date information ...

  4. Higher Education Administration (HEA) Guide

    Below you will find a sample of reports, case studies and articles that outline the process of program evaluation, planning and analysis. Click through and read on for more information. ... Employing the setting of higher education program evaluation at a midwestern regional public university, for this study we compared analysis approaches ...

  5. Full article: Time to Revisit Existing Student's Performance Evaluation

    1.1. Study Objectives. This article presents a case study using an experiment-based approach to observe whether ChatGPT can provide full-baked solutions to the assortment of assessment tools used for developing students' skills in critical thinking, problem solving, communication, knowledge application, research, teamwork, ethics, and so on—critical for their professional and career ...

  6. Making Sense of Assessment Feedback in Higher Education

    Abstract. This article presents a thematic analysis of the research evidence on assessment feedback in higher education (HE) from 2000 to 2012. The focus of the review is on the feedback that students receive within their coursework from multiple sources. The aims of this study are to (a) examine the nature of assessment feedback in HE through ...

  7. Student Learning Outcomes Assessment in Higher Education and in

    Application from higher education to the academic library. Assessment culture is more broadly discussed in the higher education literature than in library studies (Hufford, 2016). The most salient positive practice of note centers on faculty inclusion. Diverse perspectives and expertise are vital components to a healthy assessment culture.

  8. Peer evaluation: A case study

    Peer evaluation is the process whereby students critique the performances of other students. A peer evaluation format emphasizes skills, encourages involvement, focuses on learning, establishes a reference, promotes excellence, provides increased feedback, fosters attendance, and teaches responsibility. The process of peer evaluation is explained, the criteria are specified, the training for ...

  9. A Protocol to Assess Contextual Factors During Program Impact

    In our case study of a university gender equity intervention, EMA generated useful evidence of competing causes to augment program evaluation. Top-down administrative culture, poor experiences with hiring and promotion, and workload were identified as impeding forces that might have reduced program benefits.

  10. Higher Education Student Engagement Scale

    The Higher Education Student Engagement Scale (HESES; Zhoc et al., 2019) was adapted from the 61-item First Year Engagement Scales (FYSES; Krause & Coates, 2008), to assess the components of student engagement in higher education. The framework of the HESES was guided by a five-factor model of student engagement proposed by Finn and Zimmer (2012): (1) academic engagement, (2) cognitive ...

  11. Evaluations of Educational Practice, Programs, Projects ...

    For example, many large projects in higher education funded by the USA's National Science Foundation and Institute for Education Sciences require an external examiner, as do most European Commission education projects. ... While it might not be possible to generalize findings from such a program evaluation case study, it is often possible to ...

  12. The Value of Assessing Higher Education Student Learning Outcomes

    Two articles focus on the challenges associated with creating nation- or systemwide assessment systems. Martin and colleague present a case study that reflects on development of the field in Australia. It discusses insights from a review of institutional websites and a survey of leaders regarding learning outcomes identified by institutions.

  13. PDF Assessment in Higher Education and Student Learning

    Smith's (2013) formula determined that the number of respondents needed for a reliable representation of the population. The formula for an unknown population was 'Necessary sample size = (Z-score)2 x StdDev x (1-StdDev) / (margin of error)2'. The Z-score was 1.96, which corresponds to a confidence level of 95%.

  14. Learning analytics in higher education: an analysis of case studies

    Introduction. Learning analytics (LA) refers to the process of collecting, evaluating, analysing, and reporting organizational data for decision making (Campbell and Oblinger, 2007). It involves the use of big data analysis for understanding and improving the performance of educational institutions in educational delivery.

  15. PDF Student perspectives: evaluating a higher education administration program

    The Ph.D. program in Higher Education Administration is primarily for those applicants whose interests lie in gaining a tenure-track position in a Higher Education department at a college or university. Furthermore, the Ph.D. program prepares students to conduct research both inside and outside of college/university settings.

  16. The case method evaluated in terms of higher education research: A

    This study indicates that the case method is more effective than a lecture-based course, when evaluated in terms of approaches to learning, an internationally accepted measure of course quality in higher education. In particular, the case method promotes a deep approach (learning with intent and strategies to understand) and discourages a ...

  17. Gamified Evaluation in STEAM for Higher Education: A Case Study

    The process of converting non-game educational content and processes into game-like educational content and processes is called gamification. This article describes a gamified evaluation software for university students in Science, Technology, Engineering, the Arts and Mathematics (STEAM) courses, based on competence profiles of students and problems. The traditional learning management ...

  18. The case method evaluated in terms of higher education research: A

    Taking the results of this study with the concept of constructive alignment (Biggs, 1996; Hrivnak, 2019), we can apply the findings of higher education research to the design of case method courses and of management education courses that use other pedagogies. Again, the R-SPQ-2F could be used to evaluate changes made as a result of this approach.

  19. Data Envelopment Analysis and Higher Education: A Systematic Review of

    The interest in Data Envelopment Analysis (DEA) has grown since its first put forward in 1978. In response to the overwhelming interest, systematic literature reviews, as well as bibliometric studies, have been performed in describing the state-of-the-art and offering quantitative outlines with regard to the high-impact papers on global applications of DEA and the higher education system (DEA-HE).

  20. Metaverse Integration in COIL to Improve Intercultural Competence in

    Design and validation of a questionnaire for the evaluation of educational experiences in the metaverse in Spanish students (METAEDU). Heliyon. 8, 11 (Nov. 2022). Crossref. Google Scholar ... Metaverse Integration in COIL to Improve Intercultural Competence in Higher Education: A Case Study. Social and professional topics. User characteristics.

  21. Education Sciences

    In recent years, Higher Education institutions have reviewed learning and teaching methodologies to align competencies with evolving socioeconomic scenarios. Challenge-Based Learning (CBL) has emerged as a key method for developing competencies and self-regulating capacities in university students. This study aimed to identify the teaching practices associated with CBL in Higher Education.

  22. Formative peer assessment in a CSCL environment: a case study

    The course concerned a higher education case‐based virtual seminar, in which students were asked to conduct research and write a report in small multidisciplinary teams. The assessment assignments contained the discussion of assessment criteria, the assessment of a group report of a fellow group and writing an assessment report.

  23. Case Studies in Higher Education

    Gale Case Studies was created by university faculty and developed specifically for the classroom. This new higher education tool gives undergraduate students the chance to sharpen their critical-thinking skills by using historical content to evaluate and discuss contemporary social issues within the educational context of a case study.

  24. Issues in Higher Education Program Evaluation

    A comparative case study.of seven universities initially focused on: (1) whethen and how program evaluation systeas differ from'other systems approaches, such as management'infarmati3n and progtam budgeting; (2) desirable degree of overlap between evaluative recommendations and administrative'action; and (31 limitations of program evaluation.

  25. PDF Quality assurance in higher education. A case study ...

    The Finnish Higher Education Evaluation Council (FINHEEC) is an independent expert body assisting universities, polytechnics and the Minis­ try of Education in matters relating to evaluation. FINHEEC is appointed by the Ministry of Education for a four year pe­ riod. The accreditation of higher education in Finland is one element of the

  26. Institutional repository of scholarship, research, and publications at

    Institutional repository of scholarship, research, and publications at ...

  27. Assessment Effectiveness on the Job Training in Higher Education (Case

    Conclusion This study examined the evaluation framework of training elements in relation to the effectiveness of training and development in the Higher Education sector. The findings revealed that on the use of a four level evaluation model for employee and Teachers training program; at level 1, most of the respondents were satisfied with the ...

  28. Assessing English Language Proficiency: A Case Study of English Version

    Assessment is an integral part of language learning. Language proficiency can be judged by a well-organised evaluation procedure. A systematic assessment process paves the path of successful language learning. The main focus of this study is to explore the current English Language Assessment practice in Higher Secondary institutions located in Dhaka city.

  29. A Case Study on The Evaluation of Maturity Class in Potato Breeding

    In potato breeding, maturity class (MC) is a crucial selection criterion because this is a critical aspect of commercial potato production. Currently, the classification of potato genotypes into MCs is done visually, which is time- and labor-consuming. The objective of this research was to use vegetation indices (VIs) derived from unmanned aerial vehicle (UAV) imagery to remotely assign MCs to ...