1.  Fundamentals of Evaluation in Medical Education

Presenter:  Brian Mavis, PhD - Michigan State University College of Human Medicine

Fundamentals of Evaluation in Medical Education

In the context of medical education, the term evaluation encompasses a broad territory of inquiry.  Evaluation questions and strategies can range from a focus on learner's experiences and abilities to larger organizational concerns characterized by questions about the curriculum, students, faculty, institutional processes or organizational mission. Regardless of the focus of a specific evaluation effort, the purpose of an evaluation is quality improvement.   Central to evaluation is the systematic collection of information for decision-making, usually as a process for documenting outcomes.  Since the early 1990s, there have been a variety of new strategies implemented to collect information for decision-making, moving beyond data that are readily available to efforts to identify and collect information that is more meaningful.  

In this one-hour audio seminar, Dr. Mavis reviews the basic principles of evaluation, distinguishes common evaluation methods and approaches, outlines questions to ask when planning an evaluation, as well as present some current issues related to evaluation in medical education. His goal for this session is provide medical school faculty without formal training in evaluation with the basic concepts and vocabulary necessary to be knowledgeable consumers of evaluation.  Subsequent sessions in this audio series will build on the concepts presented in this session.

2.  Evaluating Student Learning in the Didactic Setting

Presenter:  Byron Crawford, MD - Tulane University Health Sciences Center

Evaluating Student Learning in the Didactic Setting

This course will cover multiple methods of assessing student learning in the didactic setting.  Once it is determined what is to be learned, exam development is very important.  Different methods of evaluating exam results will be presented along with evaluation of national board type exams to assess student learning.  Subjective mechanisms as peer review, retrospective student perceptions, reflections of other course directors requiring your course as a prerequisite and tips on generating interest in providing feedback from these individuals will be covered with systems to evaluate student learning after teaching methodology and/or personal responsible for teaching has changed.

3.  Evaluating Student Learning in the Clinical Setting

Presenter:  Debra DeRosa, PhD - Northwestern University Feinburg School of Medicine

Evaluating Student Learning in the Clinical Setting

The ability to assess medical students or residents in various clinical settings requires a sound performance rating system.  The purpose of this session is to equip the participants with the knowledge and skills needed to methodically review their current performance evaluation system used to assess clinical performance so they can enhance their system's usefulness and quality.  In this one-hour seminar, Dr. DaRosa will start by briefly summarizing lessons learned from the literature in and outside of medical education about common errors in performance rating systems.  The review will describe those cognitive, social, and environmental factors that influence ratings of clinical performance.  Participants will learn how to use a diagnostic checklist designed to identify problems in their performance rating systems, and guide and prioritize improvement efforts.  An overview will be provided on how to calibrate faculty expectations and ratings by employing  frame of reference training.  The results of this type of training promote more consistent ratings among faculty.  Acquiring faculty consensus on the meaning of a one versus a three versus a five on a five-point rating scale is critical for fair, reliable, and judicious ratings.  Finally, participants will learn about a strategy for helping faculty members write narratives about their learners' clinical performance that help distinguish between outstanding, average, and marginal learners. 

In this seminar, Dr. DaRosa will address the pros and cons of performance ratings, the most common form of evaluation used to assess the clinical performance of medical students and residents.  The aim of this session is to heighten awareness of sources of error, and make recommendations on how participants might enhance the clinical performance systems used at their institutions.

4.  Options for Evaluating Student Learning in PBL Programs

Presenter:  Phyllis Blumberg, PhD - University of the Sciences in Philadelphia

Options for Evaluating Student Learning in PBL Programs

PBL offers varied opportunities for evaluating student learning beyond traditional tests including evaluating professional behaviors, information literacy, evidence-based practice, the ability and desire for lifelong learning, interpersonal and communication skills. According to the LCME, appropriate learning outcomes for medical students include the application of knowledge to complex problem solving and critical thinking. These outcomes are difficult to measure directly.  Student's demonstration of their ability to perform prerequisites of these outcomes is a measurable approach. Deep learning is a prerequisite for these abilities and can be measured through validated questionnaires readily available.

Self-directed learning and information literacy are prerequisites for lifelong learning.  Information literacy means the determination of information needs, acquisition of information effectively and efficiently, evaluation of information and its sources critically, and incorporation of selected information into knowledge base.  Information literacy, self-directed learning, and evidence-based practice, all skills deemed essential for competent physicians, can be evaluated by looking at the written summaries students develop from their reading on their learning issues.  The ability to communicate succinctly can also be evaluated from these summaries.

In this seminar, Dr. Blumberg will apply a framework for determining specific measures to evaluate learning in PBL. This evaluation framework is based on 1) a rationale for selection of the outcome based upon literature, accreditation pressures, and institutional values; 2) how the outcome should be measured; and 3) what data can be collected practically and efficiently. The presentation will give specific examples of evaluation tools that can be used to measure different learning outcomes.  Often the same tool can be used to evaluate different learning outcomes.

5.  Ensuring that your Student Evaluations are Effective

Presenter:  Rebecca Henry, PhD - Michigan State University College of Human Medicine

Ensuring that your Student Evaluations are Effective

In this one-hour audio seminar, Dr. Henry will address what makes for a healthy evaluation system and then review what to look for in your own institution's evaluation systems that may indicate problems.  Finally, she will discuss recent trends in educational evaluation (e.g. ACGME evaluation tool-box) that may help us all make decisions on what to evaluate and what strategies may work best.

6.  Computer Based Assessment of Medical Knowledge and Skills

Presenter:  Sebastian Uijdehaage, PhD - UCLA David Geffen School of Medicine

Computer Based Assessment of Medical Knowledge and Skills

For centuries, medical educators have used traditional means for assessing medical knowledge and skills: paper-and-pencil tests, microscope-based exams, and clinical skills exams with (simulated) patients.  These trusted methods, however, have serious drawbacks.  For instance, in a typical microscope-based exam students are given little time to examine a specimen and are not allowed to review their answers.  Not uncommonly, specimens change or become damaged during the examination process. 

Recent advances in web-based and robotic technology have remedied some of the disadvantages of traditional assessment methods.  These new formats of assessment, however, are expensive and introduce a new set of challenges.  For instance, security concerns are raised because students need to be tested in shifts due to limited seating capacity in computer laboratories.

In this seminar, Dr. Uijtdehaage will review the advantages and disadvantages of web-based assessment.  In addition, he will discuss how security issues can be handled.  Furthermore, the question of home-grown versus off-the-shelf applications will be addressed.  Finally, he will give examples how patient computer simulations and robotic technology can be employed for assessment of clinical skills.