The statistics of speech: from Corn Laws to the Presidential debates
- Date: 22 Oct 2012
- Copyright: Photograph appears courtesy of Dr. Schonhardt-Bailey. Facebook images appear courtesy of iStockPhoto.
Dr. Cheryl Schonhardt-Bailey is Reader in Political Science in the Government Department of the London School of Economics and Political Science, where she teaches courses in the politics of economic policy and legislative politics.
Her research interests are in political economy and quantitative textual analysis. By measuring the words, arguments and deliberation of politicians and policy makers, she aims to gauge the extent to which ideas, interests and institutions shape political behaviour.
Here, Statistics Views Editor Alison Oliver talks to Dr. Schonhardt-Bailey on how statistics has played an influential part in her research and career. Her article, ‘Yes, Ronald Reagan’s rhetoric was unique – but statistically, how unique?’ from Presidential Studies Quarterly, Volume 42(3) is available at http://onlinelibrary.wiley.com/doi/10.1111/j.1741-5705.2012.03990.x/abstract
1. Your recent paper in Presidential Studies Quarterly uses automated textual analysis to compare Ronald Reagan's rhetoric with that of presidents Woodrow Wilson through to Barack Obama, using their State of the Union speeches. Can you tell us how you came to use statistical analysis in this context?
I started working on how to systematically capture the role of ideas and ideological arguments by 19th century politicians after having written my PhD dissertation (UCLA, 1991). My research on Britain’s shift to free trade—and particularly the role of ideas, interests and institutions--gradually evolved into my book From the Corn Laws to Free Trade (MIT Press, 2006). Much of this research was based on statistical analysis, such as regression analysis, but it also included the ways to measure the effect of economic theories (e.g., Ricardian theory of rent), ideology and ideas (e.g., religion, morality, prosperity) on the thinking and behavior of political actors.
2. What tools did you use for your research at the time?
Unfortunately the tools I required were not available at the time that I wrote my PhD dissertation. My early research provided depth and insight about the Corn Laws but not within the systematic framework I wished for. In the later 1990s, I was looking into the various options of manual content analysis and I was not finding answers on ways to identify the characteristics of politicians, until I came across a statistical package called ‘Alceste’ which did everything I wanted. The key task that I required was the ability to capture systematically the themes and arguments articulated by a particular speaker (or set of speakers), with statistical significance. Moreover, it allowed me to make linkages between characteristics of the speakers (e.g., party affiliation, constituency characteristics) or the timing of the speech, and the sorts of arguments that were made. So, I could assess the types of arguments made specifically by Conservatives, and how these changed from, say, the first to the third readings, or over other time periods.
I received funding from the Nuffield Foundation for digitizing 19th Century Parliamentary debates on trade policy. With this funding, I was then able to use Alceste in order to weigh the relative influences of ideas, interests and institutions, with a degree of statistical significance that had previously not been possible.
3. Have you then continued to use Alceste in your research since?
Indeed I have continued to use Alceste on a variety of different projects (e.g., the speeches by George Bush and John Kerry on national security during the Presidential elections of 2004 and Senate debates on abortion). I’ve also used it to analyze transcripts from the Federal Open Market Committee of the Federal Reserve and congressional hearings on monetary policy oversight. The latter has evolved into a book—Deliberating American Monetary Policy: A Textual Analysis—which is due to be published by MIT Press in 2013. This is co-authored with Andrew Bailey (my husband). For this book, we were interested in gauging the motivations of both central bankers (as they formulate monetary policy) and politicians (as they conduct oversight of the Federal Reserve). Interestingly, our analysis shows that during much of the time, Federal officials and politicians in Congress talk past one another, rather than to one another. We cover the period from 1976 to 2008, which has been quite a critical period in US monetary policy history.
The project actually began with our dissatisfaction with many of the existing empirical analyses of monetary policy, which often use reaction function models to gauge the decisions of Federal officials. Unfortunately, these approaches discounted the role for deliberation within the monetary policy committee setting (policy makers may as well have mailed in their votes and foregone the committee meeting entirely). We felt that there was (and is) a role for deliberation, persuasion, and ideas more generally in the policy making process. Hence our task was to measure these effects statistically.
The project later evolved to include an analysis of the deliberations by senators and representatives in the congressional banking committees, as they conduct oversight of monetary policy. The contrast between the deliberations of central bankers and those of politicians on monetary policy is a fascinating one.
4. How then was the use of statistics important to your research?
What started out as an effort to quantify the role of ideas has evolved into a focus on quantifying deliberation in real world policy settings (e.g. the Federal Reserve and Congress). The use of textual analysis has grown enormously in the past ten years, as well as the availability of data and software. In the late 1990s, neither the software nor the availability of textual data were as advanced as they are now. (For instance, the 19th Century parliamentary debates that I analyzed for the Corn Laws book had to be scanned from very thin hard copies, where the print from one side of the paper was shadowed on the other side. A vast amount of correcting needed to be done before these could be analyzed as text. Since 2005, the debates are now all available freely on-line. Much has changed in a relatively short period of time).
The advantage of having a multitude of textual analysis software is that there are a variety of approaches to choose from. However this can also be a drawback, since it leads to uncertainty and confusion as to what might be the right approach for any given project. More technical support and more people to teach textual analysis are both certainly needed.
I have also started to use Iramuteq, which uses the R interface to replicate the thematic results from Alceste. While Iramuteq is free software (and Alceste is proprietary), neither is without technical problems. Ideally, there would be a robust, reliable, and free textual analysis software that could capture the role of themes, arguments and deliberations. But we are not there yet. It’s my hope that one will be created in the not too distant future. Without a doubt it would be used by students and statisticians worldwide in ample quantities!
5. The Royal Statistical Society has set up the getstats campaign to increase the awareness of the use of statistics in every-day life and in our jobs. They recently sent MPs a short statistical test and 3 out of 5 MPs answered incorrectly. When MPs were questioned as whether or not they use official statistics when preparing their policies and speeches, 'only 17% of Conservative respondents agreed, as did 30% of the Labour members who took part.' Do these results surprise you?
It may be discouraging but in reality, when you think of the current Presidential debates between Obama and Romney, they use empirical ideas to make points. It’s about how persuasive you can make your ideas and arguments, not about the significance of the statistics. In Deliberating American Monetary Policy, we discuss the various aspects of monetary policy deliberations and how Members of Congress seek to avoid engaging in the technical details (some statistically based) of monetary policy. They know more about other policies, and tend to shift the discussion to these other areas (where, invariably, they are also more likely to gain political points). Most members of Congress are lawyers, not trained economists, and so they don’t understand monetary policy and don’t want to appear stupid discussing the details of monetary policy in these highly publicized hearings
To the extent that I’ve compared the expertise of MPs in the Treasury Select Committee with that of their counterparts in the US Congress, MPs on the TSC are actually more comfortable with and happy to engage directly on the details of monetary policy. More generally, I think that the getstats campaign has a great focus to make MPs, as well as others more aware of the role of statistics in their work. However, the people generating the statistics (e.g., central bankers, bureaucrats, researchers, and so on) also have a responsibility to make the information more digestible and accessible to non-expert audiences (like MPs in parliamentary committees). One way might be to find more ways to make statistical data visually appealing (with graphs and so on). But I’m sure that there are many other ways to accomplish this.
An interesting point about increasing public understanding of statistics is with regards to how weather forecasts are given in the UK and the US. In the US, the public are given percentages, “Today, we have a 20% chance of precipitation” or “60% chance of snow”. In the UK, this information is not given like this at all, instead it’s “there is the odd chance of a shower” or “it will be mostly dry with patchy rain”. There seems to be a supposition by persons unknown that the British public do not comprehend probabilities. So, there are actually simple ways that statistics can be brought into everyday life, if there were a will to do so.
6. As a university professor, what do you think the future of teaching economic policy will be? Is there a role of statistics in there?
Absolutely. I focus on teaching trade and monetary policy and the use of statistics is especially useful in conveying how policy makers grapple with uncertainty. For instance, the Bank of England has coined the term “fan chart” in its inflation reports to convey future inflation. The Bank uses this graphical device to help the public understand inflation forecasting.
But there are many examples of the use of statistics in trade policy, agricultural policy and elsewhere. There is no lack of usage, but the difficulty always rests in conveying complex relationships as simply as possible. Not an easy task.
7. Do you find that your students typically have a good grasp of basic statistics or not? If not, what do you think can be done to improve this?
Students who are studying for a joint degree in politics and economics have a good grasp of basic statistics. With Masters level students, the level is more variable, since they come from many different undergraduate backgrounds. I think that the quality of the teaching really matters all the way from primary school to university. Teachers need to be able to explain quantitative methods in statistics and use examples that are relevant to students. Since my first teaching role at the London School of Economics (LSE) in the early 1990s, the teaching of statistics has widely improved. Back then, I was one of a few political scientists who taught statistics whereas now there are many in our own department, as well as far more extensive teaching of statistics offered to political scientists.
8. Over the years, how have your teaching, consulting, and research motivated and influenced each other?
My real research interests are political economy and legislative studies and my teaching has certainly influenced them. In my view, some of the best teachers are those who are “experts” in their field, because they can then bring in examples and puzzles from their own research when relevant, and use material that they are very familiar with and passionate about. Merging research and teaching also affords opportunities to then offer advice and support to others embarking on similar research.
9. Where do you get inspiration for your research projects and books?
Over the years, I have been attracted to intellectual challenges and debates on important issues such as the war on terror, national security and abortion and how the moral issues that underpin these issues evoke emotive responses from politicians and voters. In my article published for Presidential Studies Quarterly, I examine Reagan’s religious rhetoric and how it differed from previous and subsequent presidents. I have also been fascinated by the financial crisis and its aftermath. As a political economist, the past five years is a little like having lived through something like the Great Depression—a treasure trove of opportunities for future study and research.
10. Which people or events that have been most influential in your career? How/why?
In terms of political science, it would have to be Mancur Olson. In terms of empirical thinking, people like Ken Shepsle and Jeffry Frieden (both from Harvard University) have been influential. Barbara Geddes taught me at UCLA the importance of research design before starting on a project and I still use her 10 question template to this day! William O. Aydelotte was the pioneer of examining statistical data on the British Parliament of 1841-47. During the 1960s, he put together a massive dataset on the individual characteristics of MPs.
11. Do you have any advice for students starting out on a career path in political science?
Don’t worry about where you might end up—i.e. don’t make all your selections based upon future job prospects. The most important thing is to do what you enjoy and to be able to remain curious and intellectually stimulated. I made just about every wrong career move one could make (e.g. I didn’t declare a major in political science until late in my undergraduate program; I started graduate school on a pathway to becoming a “Sovietologist”; I never studied for my GRE (an entrance exam for graduate school) since I thought it was basically an IQ test and so either you had it or you didn’t; and I could go on with many more devastating and embarrassing examples). But, as long as you work hard and you enjoy what you do, then all else falls into place (usually, eventually). And when it doesn’t you persevere, and don’t give up.
12. Are there specific books, resources, blogs that you consider invaluable in your subject field?
Confession time: I don’t read blogs, since life is too short. As a political economist, the one publication that I read with absolute regularity is The Economist. (I also enjoy the fact that it began as part of the lobbying effort against the Corn Laws in 1843, and 19th Century trade policy is clearly a subject near and dear to my heart, as I’ve written extensively on that!). But, more generally, I love being at the LSE, where academia and the real world of policymaking are so interwoven. I suppose that I can’t really provide a template for “here are the key books to read in political science or political economy”. Rather, it is a matter of allowing my curiosity to lead me to books, articles, software, websites, videos, whatever. I may even run across a blog or two of interest… maybe.