In the late 1980s, around the time the Airbus A340 was introduced (1991), those of us working in software engineering/safety used to exchange a (probably apocryphal) story. The story was about how the fly-by-wire avionics software on major commercial airliners was tested.

According to the story, Airbus engineers employed the latest and greatest formal methods, and provided model checking and formal proofs of all of their avionics code. Meanwhile, according to the story, Boeing performed extensive design review and testing, and made all their software engineers fly on the first test flights. The general upshot of the story was that most of us (it seemed) felt more comfortable flying on Boeing aircraft. (It would be interesting to see if that would still be the majority opinion in the software engineering community.)

Today, in a workshop, I was reminded of this story. I realized how poor a security choice that second approach would be even if it might be a reasonable software test. All it would take is one engineer (or test pilot) willing to sacrifice himself/herself, or a well-concealed attack, or someone near the test field with an air to ground missile, and it would be possible to destroy the entire pool of engineers in one fell swoop…as well as the prototype, and possibly (eventually) the company.

Related to recent events, I would also suggest that pen-testing at the wrong time, with insufficient overall knowledge (or evil intent) could lead to consequences with some similar characteristics. Testing on live systems needs to be carefully considered if catastrophic failures are possibly realized.

No grand conclusions here, other than to think about how testing interacts with security. The threat to the design organization needs to be part of the landscape — not simply testing the deployed product to protect the end-users.