Of bullet holes, bombers and survival bias

It was World War II and Europe was under Nazi control. The American military and the Allies were trying to destroy the German industrial might in a bid to hasten the end of the war so bombers struck targets from the air.

But they couldn't fly into German controlled airspace unopposed. German gunners shot to kill and enemy fighters, closer to their supply lines and able to fly multiple sorties against the same Allied bombing run, closed with them in the air. With bombing missions pushing further and further into Germany, they exceeded the ability of Allied fighters to escort them in and back out. This left only the guns they had on board to defend them.

As a result, these bombers were shot down at high rates. With some missions, it was as high as forty percent.

This was day of Strategic Daylight Bombing and it was dangerous. In its beginnings, in 1942, it quickly became apparent that it was statistically impossible for flight crews to complete twenty-five missions over Europe before being shot down.

That was too much so the military had a problem: They had to better protect these bombers.

But how could they do it?

A long range fighter would be a solution. But they wouldn't have one until later in the war with the introduction of the P-51.

So that was out.

That left only one other option: Armor.

Armor the planes.

This was a simple sounding fix. But it wasn't simple at all. It had problems.

First of all, you couldn't just armor the whole airplane. That made it more vulnerable. Full armor made the planes heavy and heavy meant they couldn't maneuver as well to take evasive action. And, since that amount of armor wouldn't make the bombers bulletproof or flak proof anyway, only more bullet and flak resistant, that meant that enemy gunners and fighter planes could get a bead on them, keep them in their sights for much longer, and concentrate their fire. The result would be more downed planes anyway.

The second problem was that these heavier bombers would use more fuel which meant less range. That increased the risk of ditching the plane before they made it home. Or there would be more of a trade off with the third problem.

Payload. Heavier bombers couldn't carry as many bombs. Less bombs carried were less bombs dropped on the targets so more planes would have to be flown or more missions sent to get a particular job done. And if payload had to be sacrificed to take on more fuel, even less bombs would be available.

The end result would be more planes sent out and more planes shot down.

So armoring the whole plane wasn't going to do it.

But what about armoring parts of the planes? What if they could find out which parts were taking the most hits and armor those? They could armor only those parts that needed it, the parts that were sustaining the most damage, the most vulnerable parts. Doing this would protect the plane and not add all that much weight so they wouldn't need to sacrifice maneuverability, payload or fuel consumption.

But which parts?

They didn't know but they had some data.

They had the planes that returned.

They could examine the damage on the planes that came back and see where they took the most hits. Those would be the spots to armor.

That's what the military researchers went out and did. They examined the returning planes and found that the greatest number of bullet holes and shrapnel damage were to parts of the fuselage and to the wings and tail.

That was where they'd add the armor.

They took this assessment to the top brass. But the top brass wanted confirmation. They wanted the Statistical Research Group (SRG) to weigh in on it.

So the researchers took their data and their conclusions to the Statistical Research Group, to the man many considered to be the smartest of the smart people in that group.

That man was Abraham Wald.

Abraham Wald was a Jewish mathematician who was born in what is now Romania of the European Union but what was then the Austro-Hungarian Empire. He went to the University of Vienna for his academic studies and earned a degree in mathematics.

But it was the 1930s and Austria, like the rest of the world, was in economic distress. Things were tough all around and jobs were hard to come by. But what made it worse for him was that he was both a foreigner and a Jew.

German ideas of racial superiority and purity marching hand in hand with German nationalism “uber alles” were making an appearance then and anti-Semitism had had some respectability since at least Wagner in the 1800s. And maybe even before that.

There were some in Austria at the time who looked to Germany for their inspiration and many of them aspired to unify the country with the German fatherland so anti-Semitism was rife.

That made things difficult for Abraham Wald. But a friend who worked in the Austrian Institute for Economic Research, intervened. This friend, Oskar Morgenstern, who would later immigrate to the United States and help develop game theory, put in a good word for him and the Institute hired Wald.

In 1938, when the Germans invaded, Wald fled to the U.S. to escape Nazi persecution, a persecution that would begin in the ghettos and end for millions of Jews in the concentration camps with the Nazis' unspeakable Final Solution.

He took a position first at an economic institute in Colorado Springs and then left there for a position at Columbia University in New York.

When World War II broke out, Wald became a member of the Statistical Research Group.

The Statistical Research Group was a cluster of very smart people gathered together to help solve the problems of war fighting. According to Jordan Ellenburg, the SRG “where Wald spent much of World War II, was a classified program that yoked the assembled might of American statisticians to the war effort —something like the Manhattan Project, except the weapons being developed were equations, not explosives.”

A number of other groups were housed in the same building with the SRG. They worked on things like the optimum maneuvers for fighter pilots in a dogfight, protocols for strategic bombing, Columbia's part of the atom bomb project, among other things.

Pretty important stuff in their own right.

But, according to Ellenburg: “[T]he SRG was the most high-powered, and ultimately the most influential, of any of these groups...

“It was the 'the most extraordinary group of statisticians ever organized, taking into account both number and quality...' This was a group where Milton Friedman, the future Nobelist in economics, was often the fourth-smartest person in the room.

“The smartest person in the room was usually Abraham Wald. Wald...functioned as a kind of mathematical eminence to the group. Still an 'enemy alien,' he was not technically allowed to see the classified reports he was producing; the joke around SRG was that the secretaries were required to pull each sheet of notepaper out of his hands as soon as he was finished writing on it” because he wasn't cleared to read it.

It was to this Abraham Wald that the military took their conclusions and supporting data.

Wald looked at the data and came back with his own recommendation.

It was not the recommendation the military was looking for.

Wald told them they had got it wrong, that they were going to armor the wrong parts of the bombers. They were going to armor where the bullet holes were. But, he said, they should do the opposite; they should armor the places where the bullets holes weren't.

The places where the bullets holes weren't? This was a stupid answer. These military men had the data and knew which parts of the airplanes were sustaining the most damage—how could they possibly be wrong about that?

They were wrong, Wald insisted, because they were operating under an assumption that was wrong. They were assuming that the returning airplanes were a representative sample of all the planes that had sustained damage. But that just wasn't true. What they actually had was only a sample of those planes that had returned, the planes that had survived.

These were the planes that had been hit but were still able to make it back home. The data they had came only from those planes.

But the planes that had not survived, the ones that had been so damaged by enemy fire that they could not return?

According to Wald, they had sustained damage in the places where the bullet holes were not found on the returning planes. If you could sample these planes, the downed planes, he argued, you'd find that the damage would be on the other parts of the plane, the cockpit and the engines.

The military had in fact made an assumption when they thought they were relying on facts and that assumption wasn't reasonable. The enemy wasn't shooting at and hitting only the wings, fuselage and tail of these bombers. This was especially true of the anti-aircraft fire which was designed to explode and rip the plane open with shrapnel. They were shooting at the planes themselves, at the whole plane, not specific parts of it. That fact would have created a more even distribution of bullet holes and shrapnel damage across the whole airplane if the downed planes could be examined.

Armor the places where the bullet holes weren't. That was Wald's recommendation and he ended up convincing the military. They armored where the bullets weren't, the engines and cockpit, and the purported result was that more planes survived.

What Wald saw was what is called survival bias. Survival bias is the error that occurs in thinking when attention is focused only on the people or things that have crossed a certain selection threshold; everything else is simply ignored. Only those that cross that threshold are considered relevant to the issue, not those that do not cross it, that do not survive.

Survival bias is a cognitive bias, a bias that can skew the results of critical thinking. It not only affects the military but it can bias any thinking in any area. That means it's a potential problem for business, manufacturing, finance (including personal finance), economics, investments, marketing, the study of history, medicine, and the fields of architecture and construction, as well as any attempts to enlighten you about the habits of highly successful people, for instance.

But notice what Wald did here. He saw the survival bias but to find a solution to the military's problem he had to make some assumptions himself. The military's assumption wasn't any good but he himself couldn't get away without making assumptions either.

His assumptions, however, were more reasonable.

Let's look at them.

The first assumption he made was that the greater number of planes were being downed by enemy fire rather than by mechanical failure or pilot error. In a war where the enemy was shooting back this was a reasonable assumption. It was more likely that enemy fire was the cause of the downed airplanes and not a failure of that plane's systems or pilot mistakes.

Wald's second assumption was that the bullet holes would be more evenly distributed along all parts of the plane if all the planes could be examined.

Again this was a reasonable assumption. It's like I said before, the gunners were shooting at the planes not at specific parts of the planes. That would mean that bullet holes would be found all over if the planes were considered in the aggregate.

Without both of these assumptions, Wald couldn't have come up with an answer. This just goes to show that assumptions can be useful as long as they are made knowingly and they are made with the best approximation of reality that you can come up with at the time. But assumptions are often rejected as something that will only make an ass out of you and me.

Don't believe it. But we'll talk about this more later.

For now, though, count the bullet holes. But make sure you don't just count the ones that survived.

I'm Scott Clark. And this has been On Thinking.