Mechanisms of Effect Size Differences Between Researcher Developed and Independently Developed Outcomes: An Item-Level Meta-Analysis

July 2026

Differences in effect sizes between researcher developed (RD) and independently developed (ID) outcome measures are widely documented but poorly understood in education research. We conduct a meta-analysis using item-level outcome data to test potential mechanisms that explain differences in effects by RD or ID outcome type. Our analysis of 54 effect sizes from 32 studies shows that greater standard deviations of item-specific treatment effects predict larger effect sizes and reduce the observed difference between RD and ID measures from .14 SDs to .12 SDs, leaving the remainder attributable to other factors. The findings advance our understanding of how item properties predict educational intervention outcomes and underscore the importance of considering item-level data for building theory in education research.

Keywords

meta-analysis, moderation, psychometrics, item analysis, randomized controlled trials

Education level

K-12 Education

Topics

Methods