Providing data analysis for the highest chain of command: An interview with Christine Fox

At this year’s Joint Statistical Meetings held in Seattle, USA, the President’s Invited Address Speaker was Ms Christine Fox. Ms Fox is Assistant Director for Policy and Analysis at the John Hopkins University Applied Physics Laboratory. She holds a distinction of having been the highest-ranking civilian woman ever to serve in the Department of Defense, an honour earned whilst in the role of Acting Deputy Secretary of Defense for the Pentagon. She was thus a Presidential appointee confirmed by the Senate and thus served as the principal staff assistant for the Secretary of Defense for analysing and evaluating plans, programs and budgets in relation to US defense objectives.

In her current position at John Hopkins University Applied Physics Laboratory, Ms Fox leads efforts to analyse critical challenges for the US – for decision-makers at the Department of Defense; intelligence community, NASA and other federal agencies. She possesses three decades of experience as an analyst and research manager, focussing on defense issue with an emphasis on operations.

Ms Fox formerly served as President of the Center for Naval Analysis, a federally funded research and development center and as the scientific analyst, to the Admiral in charge of the Navy, the Chief of Naval Operations. Prior to leaving the Center, she was Vice President and Director of its Operations and Evaluations group, where she was responsible for approximately 85 field representatives focussed on helping operational commanders execute missions. In this position, she oversaw analysis of real world operations, including those in Bosnia and Kosovo in the 1990s, in Afghanistan immediately after 9/11 and then in Iraq in early 2003.

She also served as a member of NASA’s Return to Flight Task Group, a panel chartered by the NASA administrator to certify the recommendations made by the Columbia Accident Investigation Board. She is a member of the Council on Foreign Relations and she serves as a trustee for Woods Hole Oceanographic Institute. In addition to her outstanding career, Ms Fox also has a connection to the world of cinema. During preparation for the 1986 smash hit film Top Gun starring Tom Cruise, the producers were trying to decide on a female protagonist to star opposite Cruise, considering a gymnast as his love interest. But upon meeting Ms Fox at Miramar Naval Air Station in San Diego during research for the film and the role was changed to be an astrophysicist and civilian Top Gun instructor and offered to actress Kelly McGillis.

Ms Fox possesses an outstanding and diverse background in decision-making analytics at the highest levels of government and the private sector. Statistics Views talks to Ms Fox about her extensive experience in these senior roles from how analysis she helped to work on helped change the way the US Air Force now train their new recruits, to working for different Secretaries and accomodating her statistical skills to each, to her proudest moment working for the Department of Defense to the transition of her role at John Hopkins University Applied Physics Laboratory, all the while offering us statisticans advice every step of the way.

1. From your many diverse experiences, what advice would you offer to a statistician?

I’ve come to believe that helping people make better decisions is the most important contribution the statistical community can make but I also want to tell statisticians that in order to make that contribution, we are going to have to try and develop some different skills, approach our work with a slightly different attitude and take on the challenge of successful communication with new effort and zeal. If we can do this, there is a hunger out there from senior people facing incredibly complex challenges in whatever walk of life. My own experience is in national security but in whatever aspect of government, industry, etc., people are hungry to make better decisions and statisticians can help them do this. The onus is to figure out the best way forward and the best way to make an impact.

2. What are some of the tendencies or pitfalls that a statistician or data analyst can experience in your opinion that we need to overcome?

I think that we have a tendency towards criticism. I think we are critical for good reasons – it is our critical thinking of each other and decisions that are made that pushes us to do the best analysis that we can. But when you are a senior decision-maker and you meet with a really smart statistician and all you feel in the room is criticism, you don’t really feel that the statistician is part of the team. Boy, are you all good at what you do! But the more senior you are, the harder the decisions you face and the aura that there is a perfect solution to an issue is getting smaller and smaller the higher up you go.

I used to give my analysis to the senior decision-maker and off I went afterwards but I wasn’t really engaging with him. If you are on the team, you look for the responses – have I actually helped? You listen, and think about the issues raised, and you realise that you can help solve a problem.

Another fact is that the more senior you are, the less time you have to make a decision. Recognise that your senior leader does not like his or her timeline any more than you do – they have to make it whether you help them or not.

I have to tell you that from my time in government that I have been the victim of bad, skewed, self-serving and poorly communicated analysis. That is what we have to be on alert for and avoid.

3. You served as president of the Center for Naval Analyses, the Department of the Navy’s federally funded research and development center. Obviously your work was extremely confidential working at the highest authority. Is there anything you can tell us about a time statistics and data analysis made a significant contribution?

Whilst I was at the Center for Naval Analyses, when conducting operations analysis, we were trying to enforce sanctions against Saddam Hussein. This was before the 2003 Iraq war. We were trying to increase the pressure on him through a series of strikes which were part of an operation called Desert Fox, which was as about as cleanly packaged and planned operation as can be. The analytic team found that for the first night of the strikes, the success of the pilots in hitting their intended targets was 30%. They were using the same precision guided weapons where we had expected the outcome to be more like 80-90%. 30% was an outstandingly disappointing number and we were asked to help them figure out why.

There was a group of analysts on the team that noticed that there was a small subset of pilots who achieved much higher hit rates. Without being asked, they took it upon themselves to dig into the training records of those individual pilots and what they found is that the pilots who had good hit rates, had had very basic experience early on in a training cycle. The other pilots had missed this early experience as they had come into the programme at a later date.

So, we only had 100 data points but the result was so strong. So we showed our results to the Commander. It was a bitter pill to swallow and they kept us on board the carrier a couple of days more to work on it which seemed to indicate how much they hated the result. But eventually they had to let us go, and once again we rechecked and rechecked the analysis over and over again. But we stuck with it as it kept coming out the same way.

CNA has a tradition of briefing real world operational results all the way up the chain in command to the senior most people and we hence briefed the most senior navy admiral, the Chief of Naval Operations in charge of the entire Navy. At the time, that Chief was an aviator. Given the response that we had received from the Air Force, we were needless to say anxious that day before we went in to show him the result of our 100 data points. But the Chief saw the power in this result and on the spot, decided to change the way in which pilots are assigned to squadrons such that they would all arrive in time for that basic training event.

But that was going to take some time and the Navy was about to embark on strikes in Kosovo, so he also turned to the analyst who had completed the original work and said ‘But I need to make sure that you are right, so I expect you to do exactly the same analysis for the Kosovo operation, and if you are wrong, you’d better get back here right away and tell me.” So of course, we did the analysis for Kosovo and now with 1000 data points, we found that the pilots who had received that basic training were on an entirely different learning curve over the course of a three month operation rather than a three day operation. This work has now actually changed the way the navy thinks about training aviators. They now focus on critical skills that are necessary for them to build proficiency in their pilots. I think this analysis illustrates the importance of creativity, rigour, perseverance and the desire to help.

4. Have you seen data analysts becoming too excited over their work and not focussing on the issues at hand?

Absolutely but I understand completely. I have done the same, I have been there! If there is one thing I would like anyone to take away from reading this interview it is that we need to listen. We are not very good at listening to the decision-maker. We are too excited about our analysis but the key to helping senior leaders make a better decision is to help them. I have seen analysts come in and talk about what they want to do and what they could do, rather than talk about what the decision-maker wants. I have seen analysts use the cover of data analysis to advocate a position to get the decision-maker to make the decision that they want. I’m sorry to say that this can be the norm, which then makes decision-makers jittery about taking meetings with analysts. This is a challenge for us as a community as you have to listen to get the question and understand the symptoms that can make the decision-maker wary. Desert Fox is a perfect example. The symptom was 30%. It is up to us to get the question right and make clear the analysis that helps to solve the underlying problem, and also help them with what they are truly interested in, not just the symptom.

If there is one thing I would like anyone to take away from reading this interview it is that we need to listen. We are not very good at listening to the decision-maker. We are too excited about our analysis but the key to helping senior leaders make a better decision is to help them…It is up to us to get the question right and make clear the analysis that helps to solve the underlying problem, and also help them with what they are truly interested in…

5. You served as director of Cost Assessment and Program Evaluation (CAPE) for the Office of the Secretary of Defense. While at CAPE, Fox led several Department-wide reviews related to operations, technology and, especially, the budget. Later you served as Acting Defense Secretary for five months. You were therefore working with senior leaders on a daily basis. What were the most important lessons that you learnt that could be helpful to statisticians and data analysts?

It is extremely important to communicate. I really do think that the point about whether senior leaders want analysis to help them make better decisions can get conflated with whether or not the senior leader wants to hear a lot of PowerPoint! Don’t get me wrong, PowerPoint is a very useful tool but it can be distracting. You understandably love the analysis you are working on, all the ins and outs, and are proud of showing all your slides. But by helping the senior leader understand what the key points are, he or she then knows with confidence that you know what you are talking about. It is the hardest type of analysis to complete – to pull out the most important points that you know will appeal to that senior leader, which can take as long as the analysis itself did!

The senior leader lives in a complex world, having to think about everything from political constraints to international constraints to stockholder expectations and an expectant Board waiting for good results. If you can help them understand a solution to make a better decision according to the analysis you have made, you are helping them make the best decision considering their own constraints.

I used to use the same analyses over and over again, but that was wrong. You have to give a presentation bearing in mind your audience. The more senior the person, the more important it is to tailor your story. I supported three very different Secretaries of Defense and was responsible for taking to them analyses to help them make decisions in the defense budget. This could be very complicated with a limited amount of time and very big stakes at hand. It was very hard and I hold up my hand and admit I made mistakes.

6. Are you are able to share any of those mistakes with us?

Firstly, I was very lucky when I first started to be working with Secretary Gates – he was formerly a CIA analyst and liked to talk about the analysis. He often gave me thoughts and insights as to how he would have handled it which was very beneficial as well as extraordinarily humbling. But what I knew about him that I had not factored into my communications was that he has a near photographic memory for facts and figures. Soon after I joined the department, I was providing an update and was feeling good as we had had a great first meeting. I started the briefing and he stopped me immediately, and said, “Those aren’t the same numbers you showed me last time.” I didn’t even realise that they had changed. They were really close and the analysis was going in the right direction. But to Secretary Gates, I was toast. My credibility was shot. Fortunately, I had another chance and learnt that you never start an update with Secretary Gates where you didn’t go back to the last numbers you showed him. He was fine if the numbers had changed, which they did and sometimes a lot, as long as you could start where we had left off. He knew that he had analysts to support him and took on every decision with incredible zeal, no matter how politically challenging and he would fight like crazy to make the right decision. And he won over and over again, and it was just a thrill to see him use analysis to make the right decision.

After Gates retired, Leon Panetta became the new secretary. Panetta was a fantastic leader but he wasn’t an analyst. He didn’t like graphs, charts or PowerPoint of any kind. Now I knew this as I had been warned by his Chief of Staff. He called me in early on to brief him on the budget and I had a GREAT chart, just one and it showed, beautifully, the challenge for the department. I took it with me despite the warnings and briefed it. During one of these briefings, it is not just you and the Secretary – there is the Deputy Secretary, all the other Secretaries and the chairman of the Joint Chiefs, etc. Panetta was tolerant when I showed the chart but when we got to the Q&A session, he asked a question that was clear as day on the chart. I put that chart in front of him and guess what came back? Some very interesting four-letter words said in a very loud tone of voice about what he would do if he ever saw that chart again. After that meeting, the entire department thanked me for saving them the anxiety of ever showing Secretary Panetta a chart! What had I done? I had not listened, I had listened to myself. Secretary Panetta’s style was very interesting – he liked conversations. He wanted the key points of your analysis on one page with bullet points and then we would have a conversation, almost like a family dinner conversation. He would ask you questions that surrounded the topic and you had to be prepared to answer any question. I found it just fascinating to be engaged with him in this way and watch him have one of these conversations which would then zip back to the bullet point on my one page every time. It was remarkable. Not only did he take away the bullet points from the key analysis I wanted him to, but he received it in a whole context in which he could explore his decision. Just like Gates, he was able to take that information and also fight like crazy for the right thing for the department, which he did very effectively. He had a very different style to Gates but there was a very similar outcome.

I created a killer chart. I called it the Orange Wedge…I knew that it had worked when I started hearing the Department talk to each other about dealing with the challenge of the Orange Wedge. There’s another piece of advice to statisticians when working with senior leaders – think about making one killer chart that just gets it right. I knew it had been a killer chart when I started doing Hill calls, and Congressmen were telling me that they were concerned about the Department’s ability to handle the Orange Wedge.

7. Was there a moment whilst working for the Office of the Secretary of Defense of which you are proudest?

After Panetta, Secretary Chuck Hagel took up the post and he gave me the opportunity to lead his strategic choices and management review, which he was working on to save costs but still meet strategic objectives. We worked on making cuts to the civilian work force and boiled it down to a handful of charts to communicate to the White House and Congress that didn’t compete with the points we were trying to make. We found we could cut several things out of the Department to save money, like the number of forces, army people, ships, etc. We didn’t plan for this to occur quickly, the savings would occur gradually over time but the budget cut would be immediate and I created a chart that reflected this – a killer chart. I called it the Orange Wedge to indicate the gap between what the department needed (after cuts) and the budget. I knew that it had worked when I started hearing the department talk to each other about dealing with the challenge of the Orange Wedge. There’s another piece of advice to statisticians when working with senior leaders – think about making one killer chart that just gets it right. I knew it had been a killer chart when I started doing Hill calls, and Congressmen were telling me that they were concerned about the Department’s ability to handle the Orange Wedge. Woohoo, this was communication! It was a huge win and I was so happy to have come up with that with my team.

8. Finally, with these three decades of experience as an analyst and research manager, what last piece of advice would you offer to statisticians to help them “make better decisions”, which was the theme for this year’s Joint Statistical Meetings?

It’s about them, not about us. Concentrate on what your senior leader wants to achieve. You can only make those decisions better, by finding that your analysis is not only valuable but vital. When you hit that sweet spot and see a senior leader take your work to make a better decision and to implement that decision, there is really no greater reward. And can I add that I’ve loved every minute of what I do.


Copyright: Image appears courtesy of Ms Fox