Embedded Standard Setting Bibliography

Brice, A. (2021, June). Principled PLD Design and Embedded Standard Setting
     for an In-Flight Assessment Program.  In D. Lewis (Organizer). Embedded
     Standard Setting: Analytics and Practical Applications to Item 
     Development and Item Alignment. Session at the Council of Chief State
     School Officers National Conference on Student Assessment, Virtual.
    
Chen, J., Lewis, D. & Cook, R. (2021, May). The Coherence of Item Alignment
     and Empirical Data as a Criterion for the Efficacy of Embedded Standard
     Setting. Paper presented at for the 2021 annual meeting of the National
     Council on Measurement in Education, virtual.

Gaertner, M. & Powers, S. (2021, June). How to Win at Standard Setting with
     Imperfect Data. In Davis, L. (Organizer), Creating Coherence: Integrating
     Principled Assessment Design, PLDs, and Standard Setting. Organized
     session at the 2021 annual meeting of the National Council on
     Measurement in Education, Virtual.

Kachchaf, R. & Lewis, D. (2021, May). Practical applications of Embedded                 Standard Setting In a mature testing program. In D. Lewis (Organizer),
     Embedded Standard Setting: Research & Advances. Symposium at the 2021
     annual meeting of the National Council on Measurement in Education,
     virtual.

Lewis, D. & Cook, R. (2020). Embedded Standard Setting: Aligning standard           setting methodology with contemporary assessment design principles.             Educational Measurement: Issues and Practices, 39(1), 8-21.

Lewis, D., Lee, S., & Choi, S. (2021, May). An Investigation of Two Embedded             Standard Setting Cut Score Estimation Algorithms: ESS-Count & ESS-                 Weight. In D. Lewis (Organizer), Embedded Standard Setting: Research &         Advances. Symposium at the 2021 annual meeting of the National Council       on Measurement in Education, virtual.

Lewis, D. (2021, May). The Embedded Methods Validity Framework in Support       of Assessment System Coherence. In D. Lewis (Organizer), Developing               an Alternate English Language Proficiency Assessment within a Principled       Design Framework Symposium at the 2021 annual meeting of the National       Council on Measurement in Education, virtual.

Lewis, D. (2021, May). Analytic Methods and Tools Supporting the Coherence         of PLDs, Item alignment, and Cut Scores in a PAD Framework.
     In L. Davis (Organizer), Creating Coherence: Integrating Principled
     Assessment Design, PLDs, and Standard Setting. Symposium at the 2021
     annual meeting of the National Council on Measurement in Education.

Lewis, D. & Cook, R. (2020, August). Embedded Standard Setting: Standard
     setting as a resolution of the alignment hypothesis. Paper presented at for
     the 2020 annual meeting of the National Council on Measurement in
     Education, virtual. 

Schneider, M. C. & Lewis, D. (2021, May) Embedded ID Matching: ESS-                       Enhancements to ID Matching to Reduce Panelists’ Cognitive Load. In D.           Lewis (Organizer), Embedded Standard Setting: Research & Advances.               Symposium at the 2021 annual meeting of the National Council on                     Measurement in Education, virtual.

ESS NCME Training Session
Training session attendees included university faculty, state DoE personnel, and test publisher psychometric staff.

       Embedded Standard Setting in Practice
       Thursday, April 12 from 1-5 PM, with
        Daniel Lewis, Creative Measurement Solutions LLC
        Ellen Forte, edCount, LLC
        Amanda Brice, Curriculum Associates

Embedded Standard Setting Partners

In 2021–22, Creative Measurement Solutions LLC (CMS) collaborated with the following partners:

Curriculum Associates (CA): ESS was successfully implemented to establish cut scores for CA’s Assessment of Spanish Reading, and studies are underway to explore ESS for other CA assessment programs.

ESS is being implemented for assessments developed by the following consortia:

• Collaborative for the Alternate Assessment of English Language Proficiency (CAAELP; www.altelpa.org), a consortium of 10 states
• Stackable, Instructionally-Embedded, Portable Science (SIPS) Assessments project (www.sipsassessments.org), a consortium of 7 states funded by a Competitive Grant for Statewide Assessments from the U.S. Department of Education

These consortia ESS implementations began with the development of well-articulated Performance Level Descriptors (PLDs) and the alignment of each newly developed test item or task to a specific performance level. After pilot studies, the hypothesized alignments will be evaluated relative to empirical data. Item-PLD Alignments inconsistent with empirical data will be reviewed and resolved by subject matter experts, including qualified educators. Cut scores will emerge organically and analytically following ESS-Inconsistent item review and resolution using EmStanS, CMS’s proprietary ESS software.

WestEd: ESS has been successfully implemented for a publisher’s diagnostic assessment program in collaboration with WestEd, and a second ESS implementation is currently underway.

Smarter Balanced: CMS is collaborating with Smarter Balanced on a series of pilot studies to support the ongoing alignment of test items to the Smarter Balanced Range Achievement Level Descriptors. Currently, we are exploring the alignment of test items to the higher Mathematics claims (e.g., Problem Solving, Communicating Reasoning, and Modeling and Data Analysis).

National Restaurant Association: A successful ESS Pilot Study was conducted with the National Restaurant Association in the first ESS pilot implementation within the professional credentialing industry. The results are being used to determine how ESS might support subsequent standard setting efforts for this organization.

NWEA: Creative Measurement Solutions has supported NWEA in developing their internal systems to support ESS methods and applications.​

Contact:      Dan.Lewis@CreativeMeasurement.com
                  
                       831-601-8555