Introduction

Significant conceptual and empirical work over the last 10 years highlight the importance of understanding and specifying the knowledge, skills, and other abilities (KSAs) that achievement test items elicit from examinees how those KSAs align along a latent proficiency continuum. This work includes evidence centered test design (e.g,. Mislevy & Haertel, 2006), item difficulty modeling (Gorin & Svetina, 2010), task modeling (Alves, Gierl, & Lai, 2010), cognitive diagnostic modeling (e.g., DiBello & Stout, 2007), aligning items and proficiency level descriptors (e.g., Schneider, Huff, Egan, Tully, & Ferrara, 2010), and in other areas. Further, this work suggests the need for coherent and comprehensive understandings of how examinees interact with test items and the controllable features of those items that elicit predictable interactions. In this study we examine several item coding frameworks to identify item response demands that may elicit targeted examinee knowledge, skills, cognitive processes, and other abilities. We define item response demands as knowledge, understanding, skills, and processes that an item requires a student to call on in order to respond fully or partially successfully to an item (Ferrara, Svetina, Skucha, & Murphy, 2009).

We first describe each coding framework and then describe how we used the frameworks to code items in an assessment task from the Maryland School Performance Assessment Program (MSPAP) 2001 grade 8 science assessment. We then synthesize results from each of the coding frameworks into a coherent picture of the response demands of the items in the assessment task. In the discussion and conclusions section, we propose a broad framework for forming hypotheses about item response demands and evaluating and validating items; specifying items (including accurately predicting their empirical difficulty) and training item writers; and supporting inferences about examinee proficiency based on a collection of test items to identify proficiency status, determine achievement growth, and provide cognitive-diagnostic information.

Continue Reading

View .PDF of the complete research paper »