Learn more about the EdExchange  

Search and Filter

Submit a paper

Not yet affiliated? Have a paper you wish to post? Check out the EdWorkingPapers' scope and FAQs, and then submit your manuscript here.

Mechanisms of Effect Size Differences Between Researcher Developed and Independently Developed Outcomes: A Meta-Analysis of Item-Level Data

Joshua B. Gilbert,
James Soland

Differences in effect sizes between researcher developed (RD) and independently developed (ID) outcome measures are widely documented but poorly understood in education research. We conduct a meta-analysis using item-level outcome data to test potential mechanisms that explain differences in effects by RD or ID outcome type. Our analysis of 45 effect sizes from 30 studies shows that both greater standard deviations of item-specific treatment effects and lower correlations between item-specific effects and item easiness predict larger effect sizes and reduce the observed difference between RD and ID measures from .24 SDs to .15 SDs. The findings advance our understanding of how item properties predict educational intervention outcomes and underscore the affordances of analyzing item-level data for building theory in education research.

Keywords
meta-analysis, moderation, psychometrics, item analysis, randomized controlled trials
Education level
K-12 Education
Topics
Methods
Tags
Assessment,
Mathematics education,
Reading and literacy education
Download 11/2024
Document Object Identifier (DOI)
10.26300/8axs-y713
EdWorkingPaper suggested citation:
Gilbert, Joshua B., and James Soland. (). Mechanisms of Effect Size Differences Between Researcher Developed and Independently Developed Outcomes: A Meta-Analysis of Item-Level Data. (EdWorkingPaper: -1082). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/8axs-y713

Machine-readable bibliographic record: RIS, BibTeX