Transportability Fallacy

Findings from one place get assumed to always work somewhere else too. Results are treated as if they'd move unchanged between settings, even when local differences actually matter a lot.

Transportability fallacy is incorrectly assuming that causal inferences or performance metrics generalize unchanged across different environments or populations. The error overlooks heterogeneity, confounding shifts, and context-specific moderators that alter external validity.

A school district reads that a homework-reduction program dramatically improved test scores in a wealthy suburban district and immediately rolls it out across its low-income urban schools, without checking whether differences in home environments, after-school support, or resource availability would change the outcome. The program fails to replicate its original results, because the student populations are fundamentally different.

A clinical prediction model trained on electronic health records from a large academic medical center in a high-income country achieves an AUROC of 0.87 for 30-day sepsis readmission. It gets deployed without recalibration in a rural district hospital in a lower-income setting, one with a different case-mix severity, antibiotic availability, and coding practices. Because the covariate distributions of lab values, comorbidity indices, and treatment protocols diverge substantially from the source population, conditional exchangeability is violated, and the model's learned decision boundary produces systematically biased risk scores. A formal transportability analysis using do-calculus or weighting on measured selection variables would have revealed the causal effect's nonidentifiability under the target distribution, flagging the need for domain adaptation or local re-estimation of effect modifiers before deployment.

When data or rules from one setting get reused elsewhere, hidden differences quietly change the outcomes. Those unseen differences are exactly what leads to the wrong conclusions in the new place.

Causation fails to transport because of asymmetric covariate distributions and constrained overlap between the source and target populations, with selection mechanisms weighting the evidence nonuniformly. Structural causal elements like conditional exchangeability and directed edges show how that weighting asymmetry and constrained supports bias the inference.

Check whether the new setting actually matches the original one before trusting the results. Testing the rule against local data, and adjusting as needed, catches what got missed.

Estimating transportability through reweighting, covariate adjustment, or domain adaptation techniques, then validating externally, addresses this directly. Explicitly modeling effect modifiers and running sensitivity analyses for unobserved differences rounds out the check.

covariate_distribution_shift; unmeasured_effect_modifiers; selection_bias_in_source

An adversarial actor can selectively cite findings from a highly favorable source population to justify a policy or product deployment in a target population where key moderators differ, manufacturing false confidence in generalizability. Regulatory or procurement contexts are especially vulnerable: a vendor can present benchmark performance metrics derived from curated datasets to claim universal efficacy, suppressing covariate distribution shift evidence. In geopolitical or military planning, intelligence products derived from one theater can be deliberately reframed as universally applicable doctrine, masking context-specific failure conditions.

Require explicit documentation of source-population covariate distributions and pre-registration of target-domain applicability criteria before results are acted upon. Apply formal transportability analysis—including reweighting via inverse probability weighting or covariate adjustment—and mandate external validation on held-out target-domain data. Institutionalize sensitivity analyses for unmeasured effect modifiers as a standard deliverable in any cross-context generalization claim.