1. Congratulations on being appointed Co-Editor of Research Synthesis Methods. How did you first become involved with the Journal?

I have been an Associate Editor since the journal was founded and we’re now on Volume 4. The journal is an outgrowth of the Society and the rationale for the journal was that the Society is about bringing together people from different disciplines with different approaches to research synthesis in one place, so that we can learn from each other, share our perspectives and try and tease out the degree to which our disciplinary differences affect the way we go about synthesis.

We realised that there was no journal that really crosses disciplines and specifically addresses the methodology of research synthesis. There is Statistics in Medicine but that’s for medicine and Psychological Methods for psychology and so on. We wanted to have one place where people can write to a broader audience on questions that might be of more general interest. I’m on the Social Science end but I have always been interested how people work in other areas. I was following natural interests of my own and when Mark Lipsey decided to step down, I was asked to run for Editor and I did, and here we are.

2. What makes Research Synthesis Methods different from other journals in the field?

The Journal is about methods specifically in the area of meta-analysis and other types of synthesis but we’re not limited to a specific approach to this issue. Any tools and techniques for putting together the results of different studies in a scientific fashion would be of interest to our Journal; also we are not only interested in the quantitative statistics area. If an author has a paper about the visual presentation of data or how you formulate your problem which affects the results or information retrieval – these would all be welcome to the Journal.

We want to get people away from the idea that synthesis is primarily about crunching the numbers because it is not and it does the field an injustice to focus only on that part of it. That being said, I think many submissions are from statistically-oriented people but as I said before, I would like people to look at the big picture so that when a statistician or an information-retrieval specialist reads an article on graphical presentation, it enriches their knowledge and gives them more of a context in which to understand where their role in the synthesis fits in.

We want to get people away from the idea that synthesis is primarily about crunching the numbers because it is not and it does the field an injustice to focus only on that part.

3. Who should be reading the Journal and why?

Anybody who is interested in understanding more about how to do synthesis and what the different options are, and what the different consequences for these options are across disciplines should read the Journal. Definitely not just statisticians should read this Journal. I know that one of our most highly cited papers has been the one that I was even fortunate to be a co-author on (!) – ‘The differences between fixed and random effects meta-analysis’ which is a conceptually oriented paper on a question almost everybody starting their first synthesis wants to know. There are definitely more things for more methodologically or statistically sophisticated folks but we also have tutorials, software reviews and a special issue coming out on synthesis across non-randomised studies which has its own set of concerns. So anybody who is going to do a synthesis and isn’t just going through it mechanically or anybody who is interested on how these things work should read the journal. We have published an article on particle physics and synthesis, articles on psychology, education and ecology so it’s not just for medical statisticians. We did not call it the ‘Journal of Research Synthesis Methods in Medicine’ for a reason!

4. There is current debate about the mechanics of peer review. What can a new author expect from the Journal’s policy?

We try to do quick turnaround. We don’t always succeed because it is harder than you can imagine in getting qualified individuals to do a review. Unless you are on the Editorial Board or an Associate Editor, a peer reviewer does not receive recognition other than thanks from the Editor. In the current industrialised academic atmosphere, people are not interested in reviewing and the people who are interested start being over-used and over-worked, and at some point, they have to say ‘no.’ In terms of my own reviewing, I really like to say ‘yes’, especially if it’s a journal outside my area, so that I get to read about something new (I am usually asked to do a methodological review) and I find it fun but at some point, I myself have to say ‘no’. I am on several editorial boards, I am now the Editor of this Journal and I cannot do any other reviewing because my plate is too full.

In my dreams, journals get together and establish a policy where anybody who wants to publish in a journal has to be willing to review for that journal. You can’t do it on a journal by journal basis as then that person will go to a journal which doesn’t have those policies, but if most journals have uniform standards of policy, I don’t think it’s an unreasonable expectation to say people have to reciprocate. Everyone wants their paper reviewed quickly so the other side of the coin is that they should be required somehow to do reviews. For the most part, authors who are submitting papers on a recurring basis are capable of carrying out decent reviews and we have to find a way of inviting more people into that arena. We also need to work with them to ensure everything is constructive and developmental. In my opinion, science over the past couple of 100 years has shifted from being more about the discovery and being part of the discovery process to being more about authorship and ownership. I think current problems with peer review are a reflection of this and we need to find a solution – a better way of working in the system. Some want to do away with peer review entirely; I don’t although it is an imperfect system. Although I teach meta-analysis, I remind students you have to have your own criteria for exclusion and inclusion in your synthesis and not use peer review as a substitute for your own assessment of study quality. The peer review system works on the whole, but it has some flaws like any other system and we need to attend to these before the system is irretrievably broken.

5. What are you enjoying most about being Editor so far?

I most enjoy getting to read a wider variety of papers that otherwise I would not have had an opportunity to read. Being an Editor makes me read more broadly.

6. What are your main priorities/objectives for the Journal in the year ahead?

I would like to ensure that we attract more authors with quality submissions across the whole breadth of research synthesis; to work with Wiley to ensure we are getting indexed now that we have passed the three year mark and bring more attention to the journal; and hopefully improve the turnaround time.

I most enjoy getting to read a wider variety of papers that otherwise I would not have had an opportunity to read. Being an Editor makes me read more broadly.

7. The Journal is the official Journal of the Society for Research Synthesis Methodology. As Journal Co-Editor, how does the relationship between Journal and Society work?

There is a lot of overlap. The Editors are members of the Society as are most of the Associate Editors but this is not true of the reviewers. We want to emphasise that the Journal is not the exclusive province of the Society but again, it arose out of the feeling the Society had that there was no journal dedicated to research synthesis methods. We are trying to publish a literature that up until now was scattered and allow it be more cohesive.

8. With an educational background in psychology and political science at Brooklyn College, Hebrew University of Jerusalem and University of Maryland, when and how did you first become aware of statistics as a discipline? How can it help in terms of research in psychology and business/management?

Statistics is very important in modern life, and you can see that now more than ever, for example, with Nate Silver correctly predicting the 2012 US presidential election results with amazing accuracy and even better than the predictions from the typical polls. I think that we really depend so heavily on statistical tools and there is a large part of business and management where statistics is essential – heavily quantitative areas and finance. It is no good for either the substantive expert or the statistician to work in isolation from each other – when they work together from the initial design of the project on to its completion, they produce a better product. It can be too late for the substantive expert to take the data analysis over to the statistician for help towards the end of the project; I think that statistics should be used as a means of thinking about a substantive problem. Experts can get hung up on the technical and finer points and they lose sight of the big picture – what is the central question being asked here? I would like to see people working together more. The most interesting statistical developments historically, as well as those developed for research synthesis, came from practical problems in the real world that working researchers needed to solve.

9. You have also published two books with Wiley on meta-analysis. What is it about this subject that fascinates you?

I’ve always been interested in what criteria people use to support their work and their evidence-based decisions – what is evidence, according to them? I used to joke that I had made up the term “I am an applied epistemologist” – to describe what I do – I try to figure out the basis from which people use to say they know things and I try to see how that applies to specific sets of problems. It turns out there is a field called applied epistemology, so I’ve stopped using that term.

Meta-analysis basically says that we cannot get a full picture of what we know from any single study. So let’s take all of the studies that have been done in a particular area and let’s put them together in a way that allows us to understand what is going on. It’s a very interesting and more scientific way of putting studies together than the traditional way of an expert reading material, considering it and giving his/her opinion. That is so subject to idiosyncrasies and subjectivity; it is an opaque process. One of the great things about research synthesis is that it makes the whole process of combining the results of studies much more transparent. I think transparency is a critical part of the scientific process because science is what we know, and can defend publically, not what we know personally.

It is no good for either the substantive expert or the statistician to work in isolation from each other – when they work together from the initial design of the project on to its completion, they produce a better product.

10. From your experience, are there specific challenges in conveying or teaching statistics concepts within the social sciences?

To undergraduate students, there are challenges as many are still afraid of mathematics or statistics. I often hear “I’m not very good at maths” or “statistics scare me” and if you can stop young people speaking so much negative self-talk, get them to calm down, think of things more conceptually and then just use the formulas as a means of communicating ideas in a more succinct way, they are then more confident.

I figured this out when I had children. Girls tend to have more maths anxiety and I’m very proud to say that my daughter took Advanced Placement Statistics in high school. She enjoys statistics and understands how it underlies real life phenomena—like how Las Vegas was built based on probability theory expressed through gambling. My husband and I are very pleased that we have been able to demystify maths for her and hopefully for our students as well.

11. What has been the most exciting development that you have worked on in teaching business management during your career?

I have to include my daughter there! I’ve been fortunate to start my career by working with one of the founders of meta-analysis (Frank Schmidt) – right after I got my PhD, I got a job in the same place where he was working. For me, it’s been exciting just to watch the emergence of a whole new paradigm over the past 30 years or so, and to see how it has gone from being ridiculed to being probably the most utilized method of combining research results, at least in the social, biomedical and animal sciences, economics and evolutionary biology. The ability to feel that you are living at the time of a paradigm shift and a little scientific revolution, and to see that it has been so widely adopted and watch how it has continues to develop to become more sophisticated and the quality standards become more refined. It has been great fun!

Girls tend to have more maths anxiety…my husband and I are very pleased that we have been able to demystify maths for our daughter and hopefully for our students as well.

12. Are there people or events that have been influential in your career?

Absolutely! The first person was a person during my PhD who shall remain nameless but he said to me, “Rothstein, you will never be a methodologist”. So, of course, I had to prove him wrong! When I got my PhD, I had an infant and a toddler and I did not want to work full-time but I did want to work at my professional level. I was very fortunate to obtain a job in the US government doing personnel selection research that was a part-time job but required someone at doctorate level. I then met Frank Schmidt as he and Jack Hunter were developing meta-analysis in the field of industrial psychology. Jack and Frank were very influential – wonderful mentors and very kind and generous in sharing their time and knowledge, working with young researchers.

Since this was at the very beginnings of meta-analysis, I got to meet at least all the others in social sciences, so I met Larry Hedges, who was one of the absolutely pre-eminent meta-analysts who was also very kind and generous. The third person would have to be Harris Cooper who was involved in starting the whole process of thinking not just about statistics but also about synthesis as a much broader enterprise of reviewing and integrating literatures. Later of course, I benefitted a great deal from meeting and working with my co-authors on our Wiley books, Michael Borenstein and Julian Higgins. Each of these men were very collegial and generous and I learnt so much from all of them. I am tremendously indebted.