The newly formed Medical Student Curriculum Subcommittee of the EBM Resource Center kicked off its activities for the year 2000 by hosting the Resource Center's first educational program of the year. Dr. Dan Mayer of Albany Medical College was the featured speaker. His January 4^th presentation discussed a program, now seven years old, instituted at AMC, in which medical students were introduced to EBM as an integral part of their school curriculum.

This course of study, designed to encourage life-long learning, and to acquaint students with EBM and with the structure and function of the health care system, is a four-year program, which begins for students in their first year of medical school. In this introductory year, students are familiarized with biostatistics and with journal clubs. Year two expands the critical appraisal curriculum and specific teaching tasks are done in each clinical clerkship.

As the program developed, the students' desire for more lecture time was honored; not, however, at the expense of small group work. Due to the commitment of the AMC faculty, the hours of the course were increased regularly in an effort to facilitate student needs.

Student response to the course, and faculty involvement in EBM, which encouraged a number of AMC faculty members to attend the EBM Resource Center's "Teaching EBM" programs, has fostered the integration of EBM into other departments, and made it an integral part of the Albany Medical College educational process.

The Clinical Medical Library Program: Memorial Sloan-Kettering Cancer Center

by Gail Hendler, MLS

Memorial Sloan-Kettering Cancer Center

In an effort to support Memorial Sloan-Kettering Cancer Center staff with the current, best evidence from the literature, the Medical Library Director and Information Services Librarian developed a Clinical Medical Library (CML) program in the Spring, 1999. This program was designed to actively position the clinical medical librarian on the patient care team, to furnish clinicians with specialized training on evidence based and medical databases, and to provide busy health care professionals with rapid access to the medical literature for patient care and clinical research needs. The genesis of this program and its current status and future goals will be briefly discussed here.

Establishing a CML service for the Cancer Center was the primary goal of the Medical Library. Preparation for the program began with a literature review, followed by a consultation with veteran colleagues to address specific questions and target service levels. A presentation to the Disease Management Team Leaders in May delivered by Katherine Stemmer Frumento, Medical Library Director and Gail Hendler, Information Services Librarian, outlined the program's goals and objectives. As a result, the Medical Library was asked to initiate a CML program with the Gynecologic Disease Management Team. Currently, three departments, Gynecologic Oncology Disease Management Team, the Department of Psychiatry, and the Gastric and Mixed Tumor Service, participate in the CML program. The clinical librarian tailors each program and its services to the unique needs of the participating department.

In June, Gail Hendler initiated CML service with the Gynecologic Oncology Disease Management Team, at the request of Richard R. Barakat, MD, Associate Chief, Gynecology Service, Department of Surgery. The librarian attends a weekly Treatment Planning Conference where patient care questions, posed by clinicians, are later researched with a turnaround time of less than two hours. Citations, and when appropriate, quality-filtered articles that answer the clinical question are faxed to the clinician. However, since many more information requests arise during the course of real time patient care, clinicians either e-mail or call the clinical librarian from the clinic to request research that is delivered within one hour. Verbal feedback to the service has been overwhelmingly positive with an increase in one-on-one tutorials for database searching are provided for all of the Library's subscription and NLM databases. Future plans include participation in Journal Club and presentations on effective searching tips for end-users.

Evidence Based Oncology: Suggested Readings

Compiled by Gail Hendler, MLS

Memorial Sloan-Kettering Cancer Center

Browman GP. Evidence-based paradigms and opinions in clinical management and cancer research. Semin Oncol 26(3 Suppl 8):9-13, 1999 Jun.

Djulbegovic B. Loughran TP Jr. Hornung CA. Kloecker G. Efthimiadis EN. Hadley TJ. Englert J. Hoskins M. Goldsmith GH. The quality of medical evidence in hematology-oncology. Am J M 106(2):198-205, 1999 Feb.

Jensen N. How evidence-based medicine incorporates patient preferences. WMJ. 98(2):49-52, 1999 Mar-Apr.

Straus SE. Sackett DL. Applying evidence to the individual patient. Ann Oncol 10(1):29-32, 1999 Jan.

Djulbegovic B. Hadley T. Evaluating the quality of clinical guidelines. Linking decisions to medical evidence. Oncology (Huntingt) 12(11A):310-4, 1998 Nov.

Foy R. So J. Rous E. Scarffe JH. Perspectives of commissioners and cancer specialists in prioritising new cancer drugs: impact of the evidence threshold. BMJ. 318(7181):456-9, 1999 Feb 13.

McKee L. Putting cancer research into practice. Nurs Times 94(35):64-5, 1998 Sep 2-8.

Rutledge DN. Greene P. Mooney K. Nail LM. Ropka M. Use of research-based practices by oncology staff nurses. Oncol Nurs Forum 23(8):1235-44, 1996 Sep. (22 ref)

Fitch MI. Thompson L. Fostering the growth of research-based oncology nursing practice. Oncol Nurs Forum. 23(4):631-7, 1996 May.

McGuire DB. Walczak JR. Krumm SL. Development of a nursing research utilization program in a clinical oncology setting: organization, implementation, and evaluation. Oncol Nurs Forum 21(4):704-10, 1994 May.

McGuire DB. Walczak JR. Krumm SL. Haisfield ME. Beezley A. Reedy AM. Shivnan JC. Hanson JL. Gregory RE. Ashley B. Research utilization in oncology nursing: application of the Stetler model in a comprehensive cancer center. Oncol Nurs Forum 21(4):703-24, 1994 May.

Clarke M. Godwin J. Systematic reviews using individual patient data: a map for the minefields? Ann Oncol 9(8):827-33, 1998 Aug.

Williams CJ Evidence-based cancer care. Clin Oncol (R Coll Radiol) 10(3):144-9, 1998.

Williams CJ. The pitfalls of narrative reviews in clinical medicine. Ann Oncol 9(6):601-5, 1998 Jun.

Fijal BA. Witte JS. Evaluating the performance of a genetic test using information from previous studies in place of a gold standard. Med Care. 36(8 Suppl):AS46-56, 1998 Aug.

Moher D. Schulz KF. Randomized controlled trials in cancer: improving the quality of their reports will also facilitate better conduct. Ann Oncol 9(5):483-7, 1998 May.

Grimes DA. Atkins D. The U.S. Preventive Services Task Force: putting evidence-based medicine to work. Clin Obstet Gynecol 41(2):332-42, 1998 Jun.

Bentzen SM. Towards evidence based radiation oncology: improving the design, analysis, and reporting of clinical outcome studies in radiotherapy. Radiother Oncol 46(1):5-18, 1998 Jan.

"Teaching EBM" Course Held in 1999

The course, "Teaching Evidence-based Medicine' was held by the EBM Resource Center at The New York Academy of Medicine on October 5-8 1999.

This year, the attendees included 64 physicians in internal and pediatric medicine, 16 physicians in emergency medicine and 9 librarians.

The course, again held over a three and one half day period, featured large and small group sessions that afforded participants a total immersion in EBM for the duration of the course.

Plans for a future course are still pending. If you are interested, please save the dates of October 2 - 5, 2000 and watch your mail for future announcements.

How Well Did We Teach Emergency Medicine Physicians at the 1999 "Teaching EBM" Workshop?

By Barney Eskin, MD, PhD, FACEP

For the Evidence-based Emergency Medicine Working Group; and the Evaluation Subcommittee of the Evidence-based Medicine Committee

Sixteen Emergency Medicine (EM) physicians representing 12 different hospitals participated in the workshop. Although most were from New York City and Long Island, there were three from NJ, and one each from Pittsburgh and Montreal.

We attempted to assess the impact of the workshop on the participants in a number of ways. How can you measure this? A hierarchy of levels has been proposed by Kirkpatrick (referenced in BMJ 1999;318:1265-7), suggested by the following questions which could be posed to the participants: 1. How did you like the workshop? 2. How much did you learn? 3. How much did the workshop influence your clinical practice? 4. How much did the workshop benefit your patients? I will present preliminary results of attempts to assess the first 3 levels. Obtaining answers to the level 4 question is difficult and thus far we have not developed methodology for doing this.

The evaluations of the EM experience took advantage of and supplemented an innovative model developed and previously tested by Rosanne Leipzig and used an already exisiting pre-and-post 'needs assessment' survey developed as part of that model. Pre- and post-workshop surveys were returned by all 16 EM physicians. Since some questions were common to both surveys, I was able to look at the change in the answers between them.

LEVEL 1 QUESTION: "How did you like the workshop?"

The workshop overall, lecturers, tutors, cotutors and librarian cotutors were rated highly between "very good" and "excellent." All participants uniformly answered yes to the question, "Did the workshop provide ample opportunity for learner participation and interaction with faculty?" Narrative comments by the participants were also collected. Nine wrote favorably about the tutorial/small group format, some referring to it as a model for teaching residents. That the format was informal and interactive was cited by several participants.

LEVEL 2 QUESTION: "How much did you learn?"

This was assessed subjectively by looking at the pre and post workshop surveys and objectively by a quiz which I will describe later. On the surveys, the participants were asked about their familiarity with key concepts and key procedures. Key concepts included 11 terms such as likelihood ratio and absolute risk reduction and, for each, participants picked one of four levels of familiarity. On average, before the workshop, the average level was "understand the concepts" (second level). After the workshop, on average, they picked the third level, "can calculate or do most of the time." Key procedures included 11 items such as searching Medline and determining validity of an article. Participants picked one of five ability levels. On average, the level before the workshop was "needing some assistance" (third level); after the workshop it was "needing minimal assistance" (fourth level).

Six questions were only on the post-workshop questionnaires. The participants were asked to assess how well they could perform tasks such as leading a small group critical appraisal exercise and formulating a specific clinical question. Their answers on average were between "moderately" and "to a high degree" (third and fourth levels on a four level scale).

The participants were also asked about their attitude toward EBM. In general more felt that they would be able to keep up with the literature and choose EBM resources rather than specialist consultation after the workshop than before. Both before and after the workshop almost all participants felt strongly that they should base their clinical decisions on valid evidence when such evidence exists. Again, both before and after the workshop, few felt that publication of research in a respected medical journal ensures that the conclusions are valid.

The second way that the level 2 question, "How much did you learn?" was addressed was through use of a quiz consisting of 16 questions that attempted to test familiarity with EBM principles. The quiz was given to each participant twice, once before and once after the workshop. Of the 16 participants, all did the pre-workshop quiz and 14 returned the post-workshop quiz; differences in the scores before and after the workshop were analyzed for these 14. On average, the scores rose from 10.1 correct before the workshop to 11.4 after. When each participant's performance was examined individually, post-workshop scores were better than pre-workshop scores for 10 of the 14 participants, unchanged for 1, and worse for 3.

LEVEL 3 QUESTION: "How much did the workshop influence your clinical practice?"

A one day conference on evaluating teaching EBM was held last year at NYAM. One of the speakers, Geoff Norman from McMaster University, proposed that one approach to answering the level 3 question might be to survey participants unannounced some time after the workshop and ask how much they had been using EBM resources such as Medline, Ovid, MDConsult, the Cochrane library and Users' Guides. Phil Hubel conducted such a survey within the last month. He received replies from 6 of the 16 programs he tried to contact. The 6 programs, on average, each use about 5 of these resources regularly.

Obviously these results are a work in progress. I look forward to any feedback that you may have about what I have reported here, with regard to either the methods used or the results.

EBM Resource Center Executive Committee Forms New Subcommittee Groups

The EBM Resource Center Executive Committee has established five new subcommittees, in an effort to address some of the issues of concern to practitioners and teachers of EBM. These subcommittees and their chairs are:

If you would like to volunteer to serve on any of these committees, please write or fax, to the attention of the committee chair:

The New York Academy of Medicine

EBM Resource Center

1216 Fifth Avenue

New York, NY 10029

Fax: 212-423-0266

ebm@nyam.org

Please be sure to include your name, institutional affiliation and email address.

Natural Language Processing from OVID

Ovid is very excited to announce that it is in the final development stage of a Natural Language Processing (NLP) interface for MEDLINE, and several of its other databases. NLP is the ability for a user to enter a phrase as if making a spoken or written request to another person. The search engine will then "understand" that phrase and search appropriately. There are no syntax rules or conventions for the user to learn.

NLP is a major step beyond mapping. Mapping tries to match the term you enter to an appropriate term from a controlled vocabulary list, and then uses that term to search. NLP searches, however, use your original term, combined with synonyms found in its lexicon. It then evaluates the information retrieved for relevancy. Bringing the articles deemed most relevant to the top of the list.

Ovid will begin a beta-testing phase very soon. The pupose of this phase it to put the product in front of the experts (the end users) and test it in real world situations. Unfortunatly all of the details are not worked for the Beta test phase but below are some general ideas on the types of recruiting we will be doing.

• •We will be looking for Clinical Librarians to provide us with the real life queries that were presented to them. They will be asked to keep these questions as close to what the clinician asked as possible. Ovid Staff will run these queries against the NLP algorithm.
• •We will be looking for a group of health care providers (Physicians, Nurses, NursePractitioners) and Librarians to use the product in a setting where we can observe to see what type of questions are asked, how easy the interface is to use, and how satisfied the user is with hi/her results,
• •We may be looking to partner with specific institutions to introduce this product to their staff and collect data on use and user satisfaction.

If you are interested in being updated on beta testing as details develop, please contact Timothy Roberts at Ovid Technologies (timothy_roberts@ovid.com 212-563-5006 ext. 229 or 800-950-2035 ext. 229)

BOOK REVIEW:

Evidence-Based Clinical Practice: Concepts and Approaches

Geyman.JP. Deyo RA. Ramsey SD, ed. 177 pages. Boston: Butterworth-Heinemann; 2000. $35.00. ISBN 0750670975. Order Phone 781-904-2500.

This well written and accessible book offers an explanation of evidence - based medicine and strategies for applying its skills. Chapters address critical appraisal , computer aids in practice and how to access information at the point of care. Others tackle broader issues of cost-effectiveness and economic evidence, decision analysis and using outcomes to improve quality of care. The text is rooted in the realities of practice, focuses on patient oriented outcomes and the need for timely and easily recoverable answers to clinical questions. The emphasis is on use of secondary sources in abstract form to answer questions. There are a few arithmetic errors in demonstrating calculations of numbers needed to treat and post-test probability which may confuse new learners. This book differs enough in its emphasis from Sackett, Greenhalgh and Friedland's texts to warrant adding it to a list of useful aids for understanding and applying evidence-based medicine skills.

NOTE: The second issue of Clinical Evidence published by the BMJ and available through the ACP-ASIM is now available. This print series, updated every six months, is striving to provide evidence-based answers to a series of clinical questions in primary care in a format that, when learned, offers enormous amounts of information concisely summarized. The material is derived from the standard sources, critically appraised by experts in EBM and then presented in the context of the clinical question which generated the search. An electronic version is in the making but only a print version exists at this time. Cost is about $35.

Reviewer: Eleanor Z. Wallace, MD, MACP, SUNY HSC, Brooklyn, New York