Robustness? Range Tests for Equality and Equivalence Across Specifications

CEPR Discussion Paper 21802, July 2026; IZA Discussion Paper 18851, July 2026

Author

David A. Jaeger

This paper develops bootstrap-based tests of equality and equivalence across multiple specifications while accounting for their joint sampling distribution. These tests are used to assess whether the results are robust.

Paper

Abstract

Applied economists routinely compare estimates across specifications, observe that they are “similar,” and conclude that their results are “robust.” This common procedure makes an implicit inferential claim about the range of estimates, but usually does not account for their joint sampling distribution when the estimates are generated from the same data. I formalize informal practice with two bootstrap statistics that account for the dependence among estimates. The minimum equivalence bound, \(R^*_{1-\alpha}\), is the smallest tolerance within which the estimates can be judged equivalent. The range-based equality \(p\)-value, \(p_R\), tests whether the estimates are statistically distinguishable. Together they distinguish failure to detect differences from affirmative evidence of agreement. Simulations across a range of designs show approximately correct size and demonstrate that common visual rules often provide false comfort. Applications to five prominent papers using common identification strategies validate some conventional robustness claims while revealing cases in which apparent agreement reflects imprecision rather than stability. In a pre-registered survey of CEPR and NBER affiliates, expert judgments align with the framework in obvious cases but diverge in intermediate cases, where the verdict depends on the joint sampling distribution rather than on the visible spread of point estimates alone. I suggest that \(R^*_{.95}\) and \(p_R\) be reported whenever multiple specifications are presented as evidence of robustness.

Software

Replication Materials

Replication materials will be posted here.

Survey Pre-registration

Citation

If you use this paper, please cite it as:

Jaeger, David A. (2026). “Robustness? Range Tests for Equivalence and Equality Across Specifications.” Working paper.

@article{jaeger2026robustness,
  author = {Jaeger, David A.},
  title = {Robustness? Range Tests for Equality and Equivalence Across Multiple Specifications},
  year = {2026},
  note = {CEPR Discussion Paper 21802}
}