Print

Frequently Asked Questions

 

What is CertLink?

CertLink is an online technology platform for the development and delivery of longitudinal assessment to support continuous learning and professional development. CertLink was created through a partnership of the American Board of Medical Specialties (ABMS) with FIGmd, a leading provider of clinical data registry platforms. The goal of ABPath CertLink is to transform assessment of our Diplomates’ knowledge and judgment into a program that is practical, meaningful, and relevant to their practice.


Through CertLink, participants answer questions, many with images, using a web browser. Once answered, they receive immediate feedback (an essential element to learning and retention). They have a personalized dashboard which summarizes their ongoing performance and identifies their knowledge gaps. ABPath CertLink leverages advances in technology to assess medical knowledge and judgement using a convenient, continuous, and dynamic program that incorporates learning into the assessment process.

 

What is longitudinal assessment? Is there research to support this form of assessment?

Longitudinal assessment is a testing approach supported by contemporary adult learning theory that fosters learning through a continuous assessment of knowledge. The method involves administering questions over a longer period of time in order to identify knowledge gaps. Longitudinal assessment can aid in comprehension, retention, and retrieval of knowledge. Longitudinal assessment models represent an evolution from traditional, point-in-time exams.

A variety of studies and literature support this approach in the adult learning and education arena. Two such studies that focused on medical professionals are:

The 2013 study, Brain Science Provides New Approach to Patient Safety Training, which found that in more than 16 randomized trials, physicians improved long-term knowledge retention by answering questions over spaced intervals of time; and

Comparative Effects of Test-Enhanced Learning and Self-Explanation on Long-Term Retention, which studied medical students, and found testing in combination with explaining the information is the most effective way to drive long-term retention of information.

What is the ABPath CertLink pilot?

The American Board of Pathology (ABP) is exploring the potential of CertLink longitudinal assessment as an alternative to the MOC Part III Assessment of Knowledge, Judgment, and Skills examination. The length of the pilot will be 3-5 years, during which time we will:


    • Implement technology infrastructure
    • Gather feedback about the design of the program and participant experience
    • Develop a large item bank of questions to make the program relevant to our subspecialists
    • Perform research to evaluate the pilot’s success and validity
    • Determine next steps for use as an alternative to the MOC exam


Participants will answer 25 multiple choice questions every quarter (3 months). Once a question is answered, Diplomates will be asked how relevant that question is to the general practice of pathology, how relevant the question is to their practice, and how confident they were of their answer. Diplomates then receive immediate feedback about answering the question right or wrong, a short narrative about the topic (critique), and references. If they answered the question incorrectly, they have an opportunity to answer the question a second time. They will have a dashboard showing performance over time as compared to other participants.

How will these longitudinal assessment questions be delivered to diplomates?

The ABP MOC Assessment pilot will be delivered via the ABMS Certlink™
platform, which will be accessible via web browser or through a mobile app. There will be 25 questions delivered at the beginning of each quarter, that may be answered anytime during that quarter, either all at once or any
number at a time based on the diplomate's availability. Diplomates will have a personalized dashboard summarizing performance and identifying areas of strength and weakness.

Why is the American Board of Pathology launching this pilot?

The ABP’s goal is to improve our Diplomates’ MOC Part III experience, while retaining a reliable and valid assessment that allows for credible summative decisions about medical knowledge, judgment, and skills for continuing certification. The ABP heard your concerns about the current MOC examination and relevance of MOC to your practice. Assessments offered through the CertLink platform offer exposure to relevant information to support Diplomates’ lifelong learning. It also helps fulfill the ABP mission to protect the public by ensuring continuing certification represents quality long after initial certification. Conducting the pilot will provide necessary information to help make changes to the MOC program.


An advisory and development group made up of ABP staff and experts at ABMS, has met on a regular basis to design and implement ABPath CertLink. Everything from question structure, quality and delivery, references, computer displays, test security, and relevance for subspecialists with multiple certifications have been addressed. Valuable feedback from ABP Diplomates collected through surveys and focus groups has also been integrated into the pilot development process.

Who can participate in the ABPath CertLink pilot?

An initial soft launch of the pilot, scheduled for October 2, 2017, will be conducted to test the functionality of the CertLink assessment-delivery platform. The soft launch will rely on a group of
Diplomate volunteers selected to help ensure our entire Diplomate population is represented. After the soft launch, the ABP will conduct a hard launch of the pilot that will be open to all Diplomates. With such diverse pathology practices, participants must understand that
the initial size of item pools may not permit presenting questions that are always relevant to their practice. As time goes by, the ABP will continually add items to our item banks thus increasing the content relevance to individual Diplomates.. If you are interested in contributing questions, please go to www.pathcertlink.com. After the anticipated three-to five-year pilot, during which the practicality, reliability, and validity of CertLink will be assessed, it’s anticipated that ABPath CertLink™ will be an alternative to the current MOC Part III Exam.

What is the cost to participate in ABPath CertLink pilot?

There is no cost for the first year of participation. After the first year, there is an annual fee of $100 to cover the cost of technology, item selection and delivery, item development and editing, item banking, staff, item writing committees, and overhead. For participants in the pilot, the $50 annual MOC reporting fee (paid every two years) is waived. Current MOC fees are $100 every two years + $700 for the MOC examination = $1200 over ten years. We anticipate that participants in ABPath CertLink will pay $1000 over ten years for MOC, if expenses remain stable. See the question Are incentives being offered for participation in the pilot.

What other ABMS Member Boards are launching a pilot?

Several of ABMS Member Boards are exploring the idea of, or actively conducting, pilots to evaluate longitudinal assessment. The ABMS Member Boards participating in CertLink pilots include:

    • Colon and Rectal Surgery
    • Otolaryngology
    • Dermatology
    • Pathology
    • Medical Genetics and Genomics
    • Physical Medicine and Rehabilitation
    • Nuclear Medicine

Are these assessments available for subspecialties?

The CertLink pilot will include general APCP questions along with some subspecialty items. In 2024, once many subspecialty questions have been developed, CertLink should fully accommodate subspecialty assessment.

When will the pilot start?

Development of the ABPath CertLink MOC Assessment Pilot is well underway, and the soft launch with volunteer Diplomates is scheduled for October 2, 2017. The soft launch will last four months, each month will simulate a quarter of the year, during which time the process and platform will be evaluated. Beginning April 2, 2018, the ABP will open the pilot to all Diplomates that wish to participate. The pilot will last for three to five years. During and after the pilot, participants will receive surveys and have the option to volunteer to participate in focus groups to gather feedback and revise
the program. If the pilot is determined to be a success, the longitudinal assessment model is expected to be fully operational by 2024.

What does participation in the pilot entail?

Pilot volunteers will be expected to answer a minimum number of items each year (25 per quarter), commit to the duration of the pilot, and provide feedback to the ABP. They will also be expected to
participate in surveys and have the option to volunteer to participate in focus groups (virtual and/or inperson) during the pilot. Pilot participation will allow Diplomates to have a voice and active role in
creating the next generation of MOC assessment.

Are there other ways to get involved beyond participating in CertLink?

The ABP is currently requesting that items be submitted through our crowdsourcing site. The tutorial found here: https://db.tt/KdcMmNyU should be viewed prior to submitting items. There is also a link to an ABMS-approved video on developing well-written test questions, which can be accessed here: http://www.abms.org/news-events/abms-osmosis-video-developing-well-written-test-questions/. After viewing the tutorial and video, go to www.pathcertlink.com, and create an account by clicking on Register and completing the form and Confidentiality Agreement.

How should I prepare to take the assessment through CertLink?

No preparation is necessary, because part of the process is to maintain knowledge and gain new knowledge. Answering the questions without preparation will help uncover areas of weakness. The simple act of participating in the assessment — answering test questions right or wrong, and then seeing them repeated in slightly changed format at a later time — has been shown to be one of the best methods for retaining information. Participants will be allowed to use resources (books, internet, etc.) while answering questions, but cannot answer questions along with another person.

How will the American Board of Pathology determine the pilot’s success?

There will be several factors that determine whether the pilot is successful including, but not limited to:
    • Diplomates report that ABPath CertLink is relevant to their practice
    • CertLink™ becomes a “go-to” resource Diplomates use to maintain or grow their knowledge
    • ABPath CertLink reliably measures the knowledge and judgment of Diplomates, assuring the public that certified pathologists are engaged in lifelong learning and staying current in the field of pathology.

How does the ABPath CertLink allow users to customize the questions to fit their daily practice?

To increase relevancy, the ABP has identified over 30 pathology practice areas from which Diplomates can elect to receive questions. Participants will be required to answer 20% of questions in their area of primary certification (AP and/or CP) and then may elect to receive the remaining 80% of questions from up to 8 elective practice areas in 10% increments. If a Diplomate has subspecialty certification, an additional 20% of questions will be for that subspecialty. If a Diplomate has more than one subspecialty certification, then an additional 10% of questions will be for each subspecialty area. The remaining percentage of questions can be selected from the numerous practice areas in 10% increments. All aspects of program design will be reviewed during the pilot and are subject to change. See examples below.

 CertLink Examples

 

A Diplomate’s performance on CertLink during the soft or hard launch of the pilot will not have any bearing on the Diplomate’s certification status.

How is the CertLink assessment scored?

Test questions will be presented to Diplomates as parent questions and clone questions that together form test question families. Each parent question presented to a Diplomate during a given quarter measures a key objective or topic. Diplomates are asked to answer the parent question correctly on their first attempt. If a Diplomate answers incorrectly on the first attempt, the Diplomate will be permitted an immediate second attempt to answer the parent test question after having the opportunity to read the question critique. Only their first attempt to answer, however, will be scored as being a correct or incorrect response. If a Diplomate fails to answer a parent test question correctly, the Diplomate will be presented one or more clone test question(s) of the same family during a subsequent quarter. A correct response to a parent question will be scored as one full point while an incorrect response will be scored as zero. Scores on clone test questions that are part of the same family will be averaged. Presenting test question critiques, permitting immediate second attempts to answer parent test questions, and presenting clones when Diplomates incorrectly answer parent questions is intended to provide repeated exposure and assessment of important knowledge and skills for practice.

Diplomates’ responses to test questions will be initially used to calculate both the number and percent of questions answered correctly on a cumulative basis. Norm-referenced interpretation of Diplomates’ scores will be facilitated by reporting how well Diplomates perform in relation to appropriate peer groups. In the future, criterion-referenced interpretation of Diplomates’ performances will be made by reporting whether Diplomates achieve performance standards as set by subject matter experts and scale scores may be reported when feasible.

Results of Diplomates’ performances will be displayed via a Diplomate dashboard that is a component of the online CertLink platform. The dashboard will display results of Diplomates’ performances on the assessment overall, as well as results for elective practice areas to help target areas for improvement and further assessment.

How will assessment security be addressed?

Longitudinal assessment models represent an evolution from traditional, point-in-time secure exams. While the nature and format of longitudinal assessment does not conform to traditional examination security measures (due to the frequency of engagement required by the Diplomate), the longitudinal format and design helps mitigate concerns of unauthorized assistance in answering questions. With the help of ABMS, the ABP is evaluating several additional security measures. Ultimately, Diplomates are professionals and it is essential that they hold each other to the standards of professionalism listed in the CertLink End User License Agreement (EULA).