In the New Year’s Honours List for 2013, Professor David J. Hand was awarded an OBE for services to research and innovation. Earlier this year he was also appointed as a non-executive director of the UK Statistics Authority for a period of three years.

Professor Hand is currently Emeritus Professor of Mathematics at Imperial College, London. He was previously Professor of Statistics at Imperial College (1999-2011) during which time he was also Head of the Mathematics in Banking and Finance Programme (2005-09); before that he was Professor of Statistics at the Open University (1988-99). Since 2010, he has served as Chief Scientific Adviser to Winton Capital Management.

1. What are you focusing on currently and what do you hope to achieve through your research?

I tend to be working on a few projects at a time but one in particular that I am working on at the moment is data sets which have tiny signals or anomalies. This is becoming increasingly important as we get larger data sets. For simple problems with big signals you don’t need large datasets, but nowadays, inevitably since the obvious signals have been spotted, we are looking for smaller signals and effects. Such problems bring deep and often unexpected inferential challenges.

2. You have served on a number of research councils over the years including the Office for National Statistics Methodology Advisory Committee, UK Strategic Forum for the Social Sciences, and currently the ONS Measuring National Wellbeing Technical Advisory Group. Prime Minister David Cameron asked the ONS in 2010 to somehow measure how happy the UK was as part of a year wellbeing project costing £2 million. How is the project going so far?

It is a very tough problem and there has been some public scepticism about it but it is going well. Measurement is something I’ve long been fascinated by and is indeed one of my major areas of research. When I was an undergraduate, I studied mathematical physics and after my PhD in statistics, I worked at the Institute of Psychiatry. Both physicists and psychologists use the word ‘measurement’ but measuring IQ and measuring weight are very different notions; measuring pain and measuring height are also different. But in all these cases the same word is used.

I got very interested in whether the same word should be used and whether it was appropriate to use the same statistical tools in these different measurement situations. I eventually wrote a book, Measurement Theory and Practice: The World Through Quantification. ‘Pragmatic measurement’ is where you define what you are measuring and describe how to measure it simultaneously. This is illustrated by a look at the wellbeing literature, where you will see that the concept spans many aspects – health, social interaction, security, and so on – and you have to consider what aspects are relevant and how to combine them. You define the concept and describe how to measure it at the same time. Not everyone will produce the same definition and way of measuring the concept, but by being explicit you know what you are talking about, and you can explore the relationship between your concept of wellbeing and other variables.

‘Pragmatic measurement’ is where you define what you are measuring and describe how to measure it simultaneously. This is illustrated by a look at the wellbeing literature, where you will see that the concept spans many aspects – health, social interaction, security, and so on – and you have to consider what aspects are relevant and how to combine them.

It will be interesting to see how the public react to the initiative when the results do come out!

3. You have authored many publications and to your credit, you were asked to write Statistics: A Very Short Introduction for Oxford University Press and you include humour when writing about statistics with the 30kg greater mouse-eared bat in Information General: How Data Rule Our World. In a similar way to the getstats campaign, in your own way, you appear to be reaching out through your written work to show how statistics can be widely used and what an enjoyable subject it can be. Is this the case? Also, is there a particular article or book that you are most proud of?

That’s exactly right. I was trying to reach out to people with Information Generation and I’d like to convey to people that statistics is not the dry and dusty subject it is perceived to be but is an incredibly exciting discipline. The point to remember is that statisticians are there when the discoveries are made because the statisticians are, almost by defintion, the ones analysing the data and seeing the patterns. Often they are patterns no one else has seen before. Statisticians are the modern day version of the 19th century explorer.

In terms of books, I am proudest of the measurement book as it was original and unified lots of areas. The great Duncan Luce described it as ‘a very fine book’! But when I’m interviewing people for university positions and ask them what research of theirs they are most excited about, the most common answer is, quite naturally, the work they are currently doing. I have a book coming out in 2014, currently titled The Improbability Principle about highly improbable events and that’s what I’m currently most excited about.

4. Do you have any advice for students considering a university degree in statistics?

The first thing I tell them is that they’ve got it right! Choosing to study statistics is a great choice because of the intrinsic excitement of the discipline (you’re making discoveries) because of its importance (what you do matters – you could literally be saving lives in medical statistics for example), it’s cutting edge (think of internet companies based on statistics), and because it attracts excellent remuneration (very substantial in some sectors, such as finance). There are so many opportunities for statisticians. More generally, if they are thinking of doing a PhD, I advise them to think carefully about the problem they are going to tackle. It should be something people care about, but not a problem that too many people care about in case someone else cracks it first. And it should also be a problem they have a chance at solving.

You can’t go wrong with a career in statistics. Most statisticians start out with a degree in maths but some of the best I’ve known started off as biochemists or economists or in some other area, and then have retrained as statisticians. Since they have an understanding of a particular problem domain they can have an advantage in analysing data in that area.

You can’t go wrong with a career in statistics. Most statisticians start out with a degree in maths but some of the best I’ve known started off as biochemists or economists or in some other area, and then have retrained as statisticians. Since they have an understanding of a particular problem domain they can have an advantage in analysing data in that area.

But it is important to remember that statistics is not simply a branch of mathematics. You get 13 year old prodigies in mathematics but not in statistics because to be a good statistician you need to understand more than the mathematics. I know many first class mathematicians who simply cannot do statistics.

5. Over the years, how has your teaching, consulting, and research motivated and influenced each other? Do you continue to get research ideas from statistics and incorporate your ideas into your teaching? Where do you get inspiration for your research projects and books?

Very much so. I often use illustrations from my work in my teaching because it helps keep it alive. I recall one time when teaching a statistics class where I suspected that some of the students had been cheating and copying each other. Coincidentally, I had developed tools for fraud detection and I went into the class and said, ‘Here is some of my research’ and put up a slide on statistical methods for fraud detection. You could hear the room go quiet. I said, ‘I’ve been applying these tools to the answers you gave’. I won’t take that example further (!) but I think it’s important to take living statistics into your teaching. The worst thing in statistics books is where people make up numerical examples: ‘in the Kingdom of Randomania…’. Using real examples helps to show that statistics is relevant and matters – if all your examples are made up numbers, why should your students think it has any relevance? Many of my research ideas come from consulting projects.

6. What do you think the most important recent developments in the field have been? What do you think will be the most exciting and productive areas of research in statistics during the next few years?

The computer has totally revolutionised the subject. It’s arguable that statistics is now more of a computational discipline than a mathematical one. You cannot do modern statistics without a computer. The computer has enabled us not merely to work more quickly but also to do things we would never have thought of before. Bootstrap methods are an example. It scarcely needs saying that Bayesian methodology has undergone a revolution in recent years, again because of the computer.

Big data is of course another important development and very large data sets are involved in all sorts of areas from genomics to astrostatistics. Data mining and machine learning are subdisciplines of statistics, driven by the computer.

In terms of the future, the impact of the computer has not stopped. More powerful computers are developing all the time. People think we have come a long way in past 30 years but I think the next 30 years will be even more impressive.

7. What do you see as the greatest challenges facing the profession of statistics in the coming years?

One is insufficient statisticians being trained but for a long time, there have not been many statisticians. Recent articles in Amstat News suggest otherwise. The discipline is so important. Modern societies are built on the infrastructure of statistics. We need statisticians to cope with the data for medical research, government policies, engineering and there have not been enough. Society would benefit tremendously from statisticians and there is a danger that there aren’t going to be enough.

Data quality is another – as you correct big datasets, faults may occur and for example, a wrong medical diagnosis may be made. The importance of data quality is a big concern for the future.

My period of Presidency of the Royal Statistical Society changed my perception of statistics and I think it does for all Presidents.

8. Are there people or events that have been influential in your career?

David Cox and Brad Efron have had a big influence on the ways I’ve thought about statistics. Fred Smith at Southampton influenced me whilst I was studying for my MSc.

My period of Presidency of the Royal Statistical Society changed my perception of statistics and I think it does for all Presidents. Presidents have a responsibility for the whole of the discipline, not just their own narrow part of it. It forced me to take a step back and gain a much broader perspective.

Copyright: Image appears courtesy of Professor Hand