Validation Theory

The central move in 20th-century validation theory was a shift in what validity is a property of. Early frameworks treated validity as a feature of the measurement instrument — a test was valid if it measured what it said it measured. Samuel Messick's 1989 essay "Validity" — the definitive treatment in the field — argued instead that validity is a property of the interpretations made from scores and the uses to which those interpretations are put. A score that validly supports one inference may not support another. A test valid in one population may not be valid in another. Validation is therefore never finished: it is the ongoing process of accumulating and evaluating evidence for the interpretations and uses a test is put to.

Lee Cronbach and Paul Meehl's 1955 paper introduced the concept of construct validity: to claim that a measurement instrument is valid for a particular construct requires demonstrating that the instrument is embedded in a nomological network — a web of theoretical and empirical relationships connecting the construct to other constructs and to observable variables. The instrument must behave as theory predicts: correlating with what it should correlate with, not correlating with what it should not, and responding to interventions in the predicted direction. This is not a one-time demonstration but an ongoing programme of empirical inquiry.

Messick (1989) unified the previously fragmented validity concepts — content validity, criterion validity, construct validity — into a single construct validity framework with six aspects: content (does the instrument sample the domain appropriately?), substantive (does it engage the processes it claims to?), structural (does its internal structure reflect the construct's structure?), generalisability (do results hold across groups, settings, and tasks?), external (does it relate to other variables as expected?), and consequential (are the social consequences of its use acceptable?). The consequential aspect was controversial when introduced but is now recognised as essential — a test that produces valid scores but has discriminatory consequences fails on validity grounds.

Michael Kane's argument-based approach (2006, 2013) provides the most operationally useful framework for validation in practice. The interpretive argument — the chain of inferences from observed performance to a construct claim, through scoring, generalisation, extrapolation, and decision — should be stated explicitly and then subjected to systematic scrutiny. Each inference link is a claim that can be challenged, and validation research is organised around gathering evidence for or against each link. This makes validation a structured programme of inquiry rather than a property to be asserted once.