The Path from Beta Exam to Live Exam
Posted 01 Jul 2016 by Liberty Munson (Microsoft)
You may have heard that it can take 8 weeks from taking a beta exam to getting your result... Sometimes, it takes longer, and on rare occasions, it can actually be faster. So, being a psychologist and, as commonly assumed, mind reader, I know that you're wondering "why does it take so long?"
Here are some of the factors that affect the amount of time an exam stays in beta:
How quickly we fill beta seats. We need at least 250 qualified candidates (meaning that they meet the audience profile experience description) to take the exam while in beta. More is always better because I know that some unqualified people will also attempt the beta, and these responses can affect what happens at the next phase of exam development (see #2 below). Sometimes, we are able to deliver 250+ exams quickly; often, it takes longer. In the past, the voucher that allows you to take the beta exam for free was valid until the exam went live, but we need to speed up this process, so we have limited its use to a 3 week period. The faster we meet the goal of 250 exam results, the faster we can move to the next stage of exam development. If we don't have 250 results by the end of the 3 week period, we have to extend the beta, and, obviously, that means the time it takes before we can move to the next stage of exam development is also pushed out.
Why do we need 250+ results? Because we want stable estimates of the psychometric performance of each item, form, and the overall exam. Evaluating these psychometrics (statistics) is the next phase of exam development called "item selection." We calculate a wide variety of statistics for each item and form to evaluate how effective they are in assessing candidate skills and abilities. These include p-value (percent of people answering correctly), point biserial correlations (essentially tells me if the question differentiates high and low performers), item measures, reliabilities, median time spent, alpha, means, median, mode scores, test characteristic curves, etc. All of which give me insight into the validity and quality of each item and the overall item pool. In addition, I read every beta comment to better understand the candidate feedback and potential issues with items that may not have been identified through the statistics. This analysis takes about a week.
Once I have done this review, we meet with subject matter experts to review items that I'm unsure about. I'm not a technical expert, so I rely on their expertise to help me determine if a questionable item can be kept, fixed, or must be killed. During this meeting, we also gather information from the SMEs to establish the cut or passing score.
After this meeting, I compile all of this information, finalize the item pool and cut score, and then assemble the live forms (well, I work with a vendor who does the last bit based on the guidelines that I provide about content distribution across the forms). [A form is the instance of the exam that you see when you sit it—that is, the combination of items that are delivered during your administration.] This process takes a little more than a week.
Once we have the specifications for the live version of the exam, we "build" them, hand them off to our exam delivery provider, who then sets the exam up in their system and tests it to ensure that everything is functioning (including scoring) as expected. Someone on my team also reviews this "pre-published" version. This takes about 2 weeks.
The exam is now live, and the exam delivery provider can then rescore all beta candidates. This takes about a week.
So, if you add this all up . . . best case is 8 weeks. Now, you know the magic that's behind the curtain!
What new questions does this raise for you?
