Uncovering the Test Pilots' Risk Management Framework

This study explores how experimental flight test professionals manage catastrophic risk in complex socio-technical systems. Using an ethnographic, mixed-methods approach with survey (n = 49; validation n = 21), interviews, and observation of flight test practitioners, the research found that flight test teams maintained both statistical (Bayesian, ISO 31000-based) and non-statistical (Precautionary, heuristic) methods in parallel. Statistical tools were applied where repeatable, deterministic elements permitted probabilistic reasoning. While resource-intensive Precautionary controls and experienced-practitioner heuristics were used where novelty, emergence, and an absence of prior knowledge precluded calculation of the probability of a hazard. Practitioners commonly reported corporate mandates to probability approaches but treated those outputs with caution or disregard when clearly unsuitable. The dual approach traded efficiency for effectiveness, ensuring identified risks were managed even when statistical measures were unreliable. Aligning statistical tools with deterministic system elements, and precautionary, heuristic approaches with complex system elements offers a pragmatic, empirically grounded template for managing catastrophic risks in complex systems. This case study invites managers and researchers to reconsider reliance on probability where uncertainty is irreducible.