This story is closely related with that of the man who led the efforts to bring statistical approaches and methods to the industry: William Edwards Deming. Born in Sioux City, Iowa in 1900, he would devote a big part of his life convincing managers, engineers and CEO’s to start adopting statistical thinking in their everyday businesses [1][2]. Deming was widely successful at his task, even when many of the men he advised had never even heard of statistics before.

Deming’s career started far away from the manufacturing industry, as a mathematical physicist in the Bureau of Chemistry and Soils at the U.S. Department of Agriculture [1]. At the time, most of Deming’s career was devoted to the field of chemistry, where he and his wife published a couple of papers about the study and behaviour of compressed gases [2] [3]. During his spare time at work, Deming arranged, within the Department, a series of lectures from notorious statisticians of the time: Cochran lectured at the department, Wishart as well; Ronald Fisher attended in 1936 and Jerzy Neyman in 1937. Neyman’s lectures were in fact published as a book, thanks in good part to Deming’s efforts. Neyman’s Lectures and Conferences on Mathematical Statistics would become one of the earliest and most popular text-books on Statistics in the United States. It was also during this time that Deming met his mentor, friend and collaborator, a statistician who developed the principles of statistical quality control while working for the Bell telephone company.

Walter A. Shewhart was the Head of the Physics Department at the Wisconsin Normal School. A PhD in Physics, he spent the earlier years of his career lecturing, till he was offered a job within the Western Electric Company. He stayed in there 7 years, until he was transferred to the Bell Telephone Laboratories in 1925: he would spend the rest of his career at Bell Labs [1]. Shewhart is commonly referred as the father of statistical quality control.

Statistical quality control was born from a political decision made by the Engineering Department management at Western Electric. At some point, they decided to define a scientific method for ensuring a new global and updated view of quality assurance. To achieve this, a team of specialists, that included Shewhart, worked on the project [1]. For him, manufacturing work was equivalent to a researcher’s task of continually searching for knowledge: “mass production viewed in this way constitutes a continuing and self-corrective method for making the most efficient use of raw and fabricated material”, he wrote [1]. It was during this time that Shewhart developed what is usually regarded as the first control chart (though some argue William Gosset or Yule thought of it first [4]).

To provide a theoretical foundation for the use of control charts, Shewhart defined the concept of a “constant system of chance causes” [1]. As he saw it, every process or system is subject to random deviations, variance that is inherent to the system and attributable to random, unknown causes. A process that is in control follows such a pattern, which is predictable and can be modeled via probability distributions. But according to Shewhart, few manufacturing processes follow such patterns, as in reality, significant causes of perturbation occur, disturbances that do not fit with the expected distribution associated with the process. These external deviations are called “assignable causes”, as they can be pointed to specific situations which can be identified and eliminated, in order to improve the reliability of a process, therefore improving its quality. The control chart was merely a graphical tool for detecting such assignable causes. Shewhart also proposed the use of designed, controlled experiments to understand the variation and behaviour of a process, as a means to identify and test for the existence of assignable causes.

Deming found great inspiration in the work of Shewhart. Deming saw that his ideas could be applied not only to manufacturing processes, but also to other operations by which enterprises are led and managed. He saw the potential to improve not only manufacturing processes, but also tasks at the managerial level. Shewhart’s ideas led directly to what would become Deming’s theory of management.

Even after meeting Shewhart, it took a while before Deming would start his brilliant career bringing businesses closer to statistics. From 1939 to 1945, Deming became Head Mathematician and sampling advisor for the U.S. Bureau of Census and later on he would continue to provide sampling advice, this time for the Bureau of the Budget [1].

As a sampling specialist, Deming built a worthy career and reputation. He is attributed as one of the creators of “raking” an algorithm for calculating sampling weights [1]. In the 1940s, the Census Bureau implemented their first sample long form, in which they interviewed not all but a fraction of the U.S. population. But since they had the information of the complete census available, it was important for them that the data from the sample was consistent with the marginal estimations of selected variables in the population. Deming, along with Frederick Stephan, published a paper in 1940 describing the Iterative proportions method for weighting, which would also be known as the Deming method, or more popularly as raking.

Call it randomness of faith, but, in late 1947, Deming was assigned to General Douglas MacArthur’s Supreme Command of the Allied Powers in Tokyo, as Advisor in Sampling Techniques for the realization of the 1951 census in Japan. At the time, Japanese territory was still being occupied by the allied forces, with General MacArthur acting as commander in chief. The allied occupation of Japan would not end until 1952 [5].

Although his main task in Japan was planning the upcoming census, Deming found an eager audience for his knowledge on the Shewhart methodology. After he finished his appointment in the country, he remained to offer consulting to the Union of Japanese Scientists and Engineers (JUSE) for two more years. It wouldn’t be uncharted territory for Deming. He spent some time teaching Shewhart’s methods of statistical quality control during wartime at the U.S., with the task of improving production. The demand boom American products experienced when the war ended, caused them to forget much of what Deming had taught them, but post-war Japan by 1950 had a reputation of such poor quality products that it eagerly embraced Deming’s teachings on statistical control methods. Deming was so influential and respected in Japan even in those early years, that by 1950 the JUSE established the Deming Prize for excellence in quality control, a recognition that has been awarded annually since 1951.

Although Deming still held some teaching positions after that, most of his work now involved private consulting. He conducted frequent multi-day seminars in the U.S. and around the world, focusing on production and managerial methods. He always insisted upon the involvement of top-level management in his programs.

Deming’s name and ideas were fairly recognized and popular in Japan and in the east. He wouldn’t return to consult U.S. companies until the 1980’s, after the airing of an NBC broadcast titled “If Japan Can… Why Can’t We?”. This T.V. show spotlighted Deming’s contribution to the industrial boom the country experienced in the preceding decades [2]. The then-current fame of Japan as a manufacturer power-house was a great promotion for Deming’s work within North America. It was only then, until the latest part of his life, that he would spend some time consulting and building a name within his home nation.

Deming’s contribution to statistical quality control cannot be underestimated. Although it was Shewhart who devised the principles and mathematical notions, his work had one major flaw: it was complicated. Deming himself declared that Shewhart had an “uncanny ability to make things difficult” [6]. As such, Deming’s contribution lied not in developing new concepts (which he did, yet later on), but in figuring out ways of sharing and teaching Shewhart’s concepts to a statistically illiterate audience.

One of the first tasks for Deming was to teach the basic notions of statistics, in order to convince everyone of the role this science had in their everyday work. This was a rather complicated task if I may add: convincing managers, engineers and CEO’s on the idea of choosing science over their lifelong experience. For such a herculean challenge, Deming devised a series of experiments and carefully crafted exercises that showed people how uncertainty is inherently affecting every aspect of our lives and companies.

The funnel experiment is one of the most recognized of Deming’s exercises. Concepts like mean, deviation, distribution, all fundamentals in statistical theory, could be studied. In this experiment, Deming exemplified the concept of variation, the idea that, no matter how good you are at your task, the final result will almost certainly be different at each try, i.e., subject to variation due to “chance causes”. For the funnel experiment, Deming usually threw a few marbles through a funnel, marbles that would land a few centimetres below the funnel on to a flat surface with a target painted on it [7]. Ideally, the marbles should land and stop on the target: in reality, the marbles usually fell far from the target and rolled away a bit. Some of them stayed close enough, yet some didn’t. Deming used this experiment to teach two things. First, the outcome of a process usually has a probabilistic component: the outcome is uncertain and no matter how much we try, we can never be sure of where the marble will stop. For many companies back in the days, this was an outstanding result: most managers, operators and engineers had never before thought of their work in probabilistic terms. For manufacturers, whenever a faulty piece was built, it meant an error, a horrible flaw in the system, which required adjustments. But Deming also used the experiment to show people how tampering with a process without actually understanding it can be worse than doing nothing. After dropping a few marbles that fall left of the mark, he used to move the funnel a bit to the left, in order to “fix” the skew of the marbles falling, only to prove how the results worsened after that. Believe or not, it was very common back in those days for businesses to spend too many time and resources unnecessarily changing how they did their everyday tasks, because of overreacting to the last result they got [7]. 23 years after Deming passed away, some companies are still making the exact same mistake: how many retailers overreact when a customer has a minor problem, or how many restaurants change their service based on what the last customer survey answered (you know, those little sheets with the cute faces on them, where the only three options are amazing, great and good). That is the kind of behaviour Deming critiqued so many years ago: don’t react before scientifically studying your process. The funnel experiment was a clever activity to show, in a simple and easy to understand way, all these concepts.

The Red Bead experiment is another example of a simple exercise that illustrated basic statistical concepts in a simple way. In this exercise, a group of “workers” were assigned a nearly impossible task: randomly select beads from a container but avoiding the red beads, which account for about 20% of the total [8]. Along with notions of probability, distributions and expected value (as well as some lessons on human psychology), the experiment was very useful to show how the lack of proper statistical methods can cause the management team to focus on irrelevant factors for the process. With this experiment, Deming showed how you can get lost within the noise of the process while missing the signal.

Thanks to his didactic exercises, Deming was able to introduce statistics to a whole new audience: concepts such as variance, experiment or sampling, were now being used by top Japanese industrialists of the likes of Akio Morita, the cofounder of Sony Corp [9]. Thanks to Deming’s teachings, Japanese manufacturers achieved never before seen levels of quality and productivity, becoming the top producers of the world at the time. I find many parallels between Deming’s work on spreading statistical notions and today’s surge of interest in analytical methods within companies. I just hope we, current statisticians, can be half as good in our task of teaching and spreading statistical science, as Deming was in his time.

References

[1] Heyde, C.C., Seneta, E., Crepel, P., Fienberg, S.E., Gani, J. Statisticians of the Centuries. Springer-Verlag New York. (2001) ISBN 978-1-4613-0179-0
[2] Deming The Man. The Deming Institute Website. (2016)
https://www.deming.org/theman/overview
[3] Deming, Edwards W. & Shupe, Lola E. The constants of the Beattie-Bridgeman equation of state with Bartlett’s p-v-t data on nitrogen. Journal of American Chemistry Society 1930, 52 (4), pp 1382–1389
[4] Henderson, Robin. Control charts – tools for understanding variation. StatisticsViews (Dec., 2013)
http://www.statisticsviews.com/details/feature/5663861/Control-charts–tools-for-understanding-variation.html
[5] Occupation of Japan. Wikipedia The Free Encyclopedia. (Last modified on 8 Sept 2016)
https://en.wikipedia.org/wiki/Occupation_of_Japan#End_of_the_occupation
[6] Tortorella, Michael J. The Three Careers of W. Edwards Deming . Siam News (July 16, 1995)
https://www.deming.org/content/three-careers-w-edwards-deming
[7] The Funnel experiment. The Deming Institute Website. (2016)
https://www.deming.org/theman/theories/funnelexperiment
[8] The Red Bead experiment. The Deming Institute Website. (2016)
https://www.deming.org/theman/theories/redbeadexperiment
[9] Noguchi, Junji. The Legacy of W. Edwards Deming. Quality Progress. 28 (12): 35–38. (Oct 1995)
http://asq.org/qic/display-item/?item=13030