Here on the blog we’ve talked a lot about the International Testing Commission’s ITC Guidelines for Translating and Adapting Tests (both first and second edition), which for years have informed our work translating K-12 educational assessments. But just like any good recipe also depends on the quality of its ingredients, it’s time to discuss best practices for the test development process.
Today I’d like to start by taking a closer look at The Standards for Educational and Psychological Testing. Are you familiar with them? If not, what do you use to help ensure a high-quality test development process? Please let us know in the comments below.
Published in 2014, The Standards for Educational and Psychological Testing offers a best practice framework for the development, administration and interpretation of tests used in educational and psychological settings. These standards were created by the American Educational Research Association (AERA), the American Psychological Association (APA) and the National Council on Measurement in Education (NCME) to help ensure reliability, validity and fairness in testing.
According to the framework, test construction should begin with the establishment of a clear and well-defined purpose. This involves specifying what is to be measured, such as reading comprehension or mathematical ability. Test developers should also consider the target population for which the test is intended, ensuring that the content and format of the test are appropriate for test takers.
Once the test purpose is established, a test blueprint is created. The blueprint outlines the content areas or domains to be covered by the test, as well as the relative importance of each domain. This helps ensure that the test adequately samples the content domain and that the test items are related to what is being measured.
Test items are then developed based on the test blueprint. Test developers use various item formats, such as multiple choice, short answer or essay question, depending on the test construct and the intended uses of the test. The items are written to be clear, unambiguous and free from bias.
After initial item development, a process of item review and revision takes place. Subject matter experts review the items for content accuracy, relevance and clarity. Items may be revised or eliminated based on the feedback received during this review process.
Once the item pool is finalized, pilot testing is conducted. This involves administering the test items to a small sample of test takers who are similar to the intended population. Pilot testing helps identify any issues with the items, such as poor item performance or ambiguity. The data collected during pilot testing is analyzed to evaluate the psychometric properties of the items and make necessary revisions.
After pilot testing, the test is ready for large-scale administration. Test administration involves following standardized procedures for test delivery, including instructions provided to test takers, time limits and test administration conditions. Test administrators are trained to ensure consistency in test administration across different test centers or exam sessions.
Once the test has been administered to a large number of test takers, the data is collected and analyzed. Statistical techniques are used to evaluate the reliability and validity of the test. Test evaluation also involves examining a test’s fairness. Differential item functioning analysis is often conducted to examine whether the test items function differently for different groups of test takers.
Based on test evaluation results, test developers make revisions to the test as necessary. This may involve modifying or eliminating problematic items, adjusting scoring procedures or refining test instructions. The revised test then undergoes further evaluation to ensure that the changes have improved its psychometric properties.
Throughout the test development process, documentation and transparency are crucial. Test developers should document all stages of test development, including item development, item review and data analysis. This documentation serves as evidence of the test’s quality and helps establish the test’s validity and defensibility.
Responsive Translation knows that the test development process is a collaborative one. And we’d like to help.
We specialize in the translation, adaptation, validation and review of educational assessments. To learn more about how we can support your organization, please get in touch at 646-847-3309 or [email protected].