The Process: Exam Creation
Live Form Creation
The next stage of development involves assembling items into exams for global deployment.
Each test has multiple forms. If candidates fails one form and retake the exam, they receive the different form of the exam on retake attempt.
The Pearson VUE test engine randomly orders the questions of each form when someone takes the exam to ensure two candidates taking the same exam are not tested on the same questions in the same order.
Initial Exam Publishing
Once the LPI psychometric staff has determined the composition of forms, the exam must be converted from text-based items into the actual exam file format to deployed globally through LPI’s network of testing centers.
The exam enter a period of initial testing to determine if the questions are in fact measuring skills and competencies. In IT certification, this period is known as the beta testing period.
During the beta period, candidates can register for tests and complete them at local events. They receive credit, but candidates do not receive scores back immediately after the exam. Beta exams often involve extra questions with an extended-time format as well as additional survey and demographic questions. Several simultaneous processes determine the cut score, so that exams may be evaluated and scored.
Obtaining Enough Exams
Before the passing score can be set, LPI had to accumulate an adequate number of exams taken by people who are similar to the target job description. As our support has grown, our target data numbers continue to grow, helping to generate the most accurate results.
As part of the beta exam process, demographic data is taken into account by psychometric staff when reviewing the validity of questions.
Reviewing the Questions
As tests results roll in, psychometric staff start to examine the data. Asking questions such as: Are there questions that everyone gets correct? Are there questions that everyone fails? to determine if there anything wrong with the questions.
Exam comments collected during the process are reviewed and questions and concerns are addressed.
Modified Angoff Study
While psychometric staff review incoming data, a separate pool of subject-matter experts simultaneously participated in a modified Angoff study. Their goal, to provide the psychometric staff with additional data to validate questions and assist in setting the passing score.
During this process, the experts:
- Receive copies of the exam questions on each form.
- Look at each question independently and in consultation with each other and make judgments about how likely a minimally qualified person meeting the job requirements described in a specification sheet would be able to answer the question correctly. In other words, the experts consider the question from the perspective of someone who is at the bottom of the competence scale for job performance.
- Rate each question with their estimate of what percentage of people will answer correctly, keeping in mind that on multiple choice questions, some people will get it right by virtue of guessing.
Ideally, the results from the Angoff study should parallel the actual results from the exams in the beta period. Beyond validating item performance, the results of the Angoff study are also used in helping to establish the passing score for exams.
Distributing Score Results
After all of the data collection, the analysis and the Angoff study, the psychometric staff set a passing score, and distribute scores to exam takers who participated in the beta.
Once the beta has been completed, the passing score has been set, and any bad items have been removed or fixed, the exam is ready to be re-published. This work involves significant review and can take a month or more to complete. Once this final review process is complete, we coordinate with Pearson VUE and our partner network to publish the finalized exams for all test takers worldwide.