The Development of Examination Content - in Brief 

    Published July 29, 2013

    Overview

    The content of the Uniform CPA Examination is developed in an extensive and integrated process. At each step in the process expertise in various disciplines is applied to ensure that the test materials are accurate and appropriate for use on the CPA Exam.

    The process incorporates expertise in a number of key areas.

    The first key area of expertise is in accounting. Individuals who draft, review, and finalize test materials are experienced CPAs.

    A second area of expertise is in the science of testing, called psychometrics. At each stage in the test development process, psychometricians are involved in the design, development, and implementation of test materials. These include test specifications, test questions, and data analysis.

    A third area of expertise is in test development. Experts in the design and development of test questions are involved in the process.

    The Role of Volunteer Committee Members

    As described below, throughout the process committee members review exam materials to help ensure that the materials meet high standards of quality. The committee members are CPAs, volunteers who serve a significant role throughout the development of examination content. These volunteers spend a great deal of time and effort to help ensure that test materials are of high quality. The test could not be developed without their valuable input.

    Steps in the Test Development Process

    There are a number of steps in the test development process. These include defining the material to be tested, developing test questions, trying out test questions and analyzing the results, constructing and reviewing test forms, and reviewing candidate comments.

    Definition of What to Test

    The first step is to define the knowledge and skills eligible for testing. This step involves determining what material should be covered in the exam.

    The process starts with expert input from CPAs who work to draft a set of statements defining the knowledge and skills that are required of entry-level CPAs. This is a key step since the purpose of the CPA Exam is to determine if a candidate has the knowledge and skills required for a CPA license. Multiple groups of CPAs are involved in drafting and reviewing the statements.

    After experts work to draft an outline of knowledge and skills, the outline is sent out to practicing CPAs in a survey. Each survey respondent rates each statement on a number of criteria, such as relevance, importance, and frequency. The purpose of the survey is to gather independent judgments from a wide sample of practitioners to help further develop the initial draft set of knowledge and skill statements.

    After the survey data is gathered it is analyzed for review. Committees of CPAs review the survey results to determine which statements should be included, which should be deleted, and which should be combined. As a further step, the committees recommend how the statements should be organized for testing, as well as the relative proportion that each area should represent on the Exam.

    After the committees of CPAs have completed their work, the revised outlines go to the Board of Examiners for review. The Board of Examiners is made up CPAs and other experts in testing.

    And yet there are still more steps in the process. The outline of content and skills to be tested is sent out to an even wider audience of stakeholders in what is known as an “exposure draft”. Stakeholders are encouraged to review the draft and provide comments. The comments are collected and analyzed, then presented to the Board of Examiners for a final review.

    It is only after this last review that the outline of content and skills eligible for testing is approved for use. This definition of exam content is undertaken periodically. Candidates are notified well in advance when the definition of test content is going to change.

    Development of Test Questions

    Once the outlines of knowledge and skills eligible for testing are finalized, the next step in the process is to develop questions to measure that material.

    The training of item writers starts the process. CPAs are recruited to draft test questions and they are provided with training in how to write test questions based on the approved outline of knowledge and skills. During training, item writers review and discuss sample questions. They also draft questions for review and discussion with the group. Item writers are provided with detailed materials including question writing guidelines and sample questions.

    After training, test questions are drafted for initial review. Each question is reviewed for content (match to specifications, accuracy, relevance). Questions are also reviewed for editorial quality and to make sure they meet test development guidelines for question construction. Questions are revised throughout this review process.

    When the initial reviews are completed, the questions are presented to review groups made up of CPAs. The CPAs review and discuss each draft question. In reviewing questions, reviewers consider accuracy, relevance, match to specifications, and other criteria. The review group may accept a question, revise it, or reject it.

    Pretesting

    Test questions are next revised, based on the group review, for pretesting. Pretesting is an important step because it is a necessary complement to all of the expert reviews done previously. Test data from candidates answering the questions can provide valuable information not available during committee reviews.

    Pretesting is done on operational test forms. Some of the questions that each candidate takes are pretest questions and do not contribute to the candidate’s score. This provides the best possible data for pretesting because the candidates taking the pretest questions are representative of actual candidates in terms of ability and motivation. In this way the pretest data is representative as possible of actual candidate performance.

    Pretest data are analyzed and statistics are used to identify questions for further review before they can be used as operational questions and contribute to candidate scores. A number of statistics are reviewed, including the following.

    • Item difficulty – This is the percent of candidates answering the question correctly.
    • Distribution of responses – This is the percent of candidates choosing each response option.
    • Item to test correlation – This is the correlation of performance on each question to performance on the other questions.

    Test questions that do not meet established statistical criteria are identified for review. Those questions are then reviewed by CPAs to determine if they should be deleted, revised, or accepted for future use.

    Test Construction

    After questions are approved for use based on pretesting, tests are assembled for administration. Test forms are assembled to meet specifications on a number of dimensions, with the goal being that test forms are comparable to each other. It is important that the test be fair to all candidates, regardless of which form of the test they take.

    The content specifications describe how many questions of each type will be included (multiple-choice, simulations, writing questions) as well as the distribution of questions by content area.

    Statistical specifications relate to the difficulty and other statistical properties of test questions.

    Once the test forms are assembled and checked to ensure that they meet the specifications they are reviewed again in final preparation for test administration.

    Candidate Comments on Test Questions

    When candidates take the exam they have an opportunity to provide comments about specific test questions. Candidate comments on test questions are reviewed by content experts to ensure that questions are accurate and appropriate for scoring.

    Conclusion

    The process used to develop the Uniform CPA Examination is a comprehensive one, incorporating many steps and quality checks by experts in accounting and in testing. It is a process designed to provide a fair and appropriate test for all candidates.




    A A A


     
    Copyright © 2006-2014 American Institute of CPAs.