Transportability Fallacy

Transportability fallacy is the wrong idea that findings from one place always work in another place. People assume results move unchanged between settings when local differences matter.

Transportability fallacy denotes incorrectly assuming causal inferences or performance metrics generalize unchanged across different environments or populations. This error overlooks heterogeneity, confounding shifts, and context-specific moderators that alter external validity.

A school district reads that a homework-reduction program dramatically improved test scores in a wealthy suburban district and immediately rolls it out across its low-income urban schools—without checking whether differences in home environments, after-school support, or resource availability would change the outcome. The program fails to replicate its original results because the student populations are fundamentally different.

A clinical prediction model trained on electronic health records from a large academic medical center in a high-income country achieves an AUROC of 0.87 for 30-day sepsis readmission. The model is deployed without recalibration in a rural district hospital in a lower-income setting with different case-mix severity, antibiotic availability, and coding practices. Because the covariate distributions of lab values, comorbidity indices, and treatment protocols diverge substantially from the source population, conditional exchangeability is violated: the model's learned decision boundary produces systematically biased risk scores. A formal transportability analysis using do-calculus or weighting on measured selection variables would have revealed nonidentifiability of the causal effect under the target distribution, flagging the need for domain adaptation or local re-estimation of effect modifiers before deployment.

When data or rules from one setting are reused, hidden differences change outcomes. Those unseen differences cause wrong conclusions in the new place.

Causation fails to transport due to asymmetric covariate distributions and constrained overlap between source and target populations, with selection mechanisms weighting evidence nonuniformly. Structural causal elements like conditional exchangeability and directed edges reveal how weighting_asymmetry and constrained supports bias inference.

Check if the new setting matches the original one before using the results. Test the rule with local data and adjust if needed.

Estimate transportability using reweighting, covariate adjustment, or domain adaptation techniques and validate externally. Explicitly model effect modifiers and perform sensitivity analyses for unobserved differences.

covariate_distribution_shift; unmeasured_effect_modifiers; selection_bias_in_source

An adversarial actor can selectively cite findings from a highly favorable source population to justify a policy or product deployment in a target population where key moderators differ, manufacturing false confidence in generalizability. Regulatory or procurement contexts are especially vulnerable: a vendor can present benchmark performance metrics derived from curated datasets to claim universal efficacy, suppressing covariate distribution shift evidence. In geopolitical or military planning, intelligence products derived from one theater can be deliberately reframed as universally applicable doctrine, masking context-specific failure conditions.

Require explicit documentation of source-population covariate distributions and pre-registration of target-domain applicability criteria before results are acted upon. Apply formal transportability analysis—including reweighting via inverse probability weighting or covariate adjustment—and mandate external validation on held-out target-domain data. Institutionalize sensitivity analyses for unmeasured effect modifiers as a standard deliverable in any cross-context generalization claim.