The Tomorrow’s Giants conference, jointly hosted by Nature and the Royal Society, was held on 1 July 2010 and focussed on the future of UK science. The meeting was composed of three strands: careers, measurement and assessment, and the future of data. In this final part, I summarise the main discussion points from the Measurement strand. All quotes are paraphrased rather than direct (I was typing as fast as I could, but may have mislaid the odd word). I’ve also combined views from various session (preparatory, feedback and panel debate) into one narrative.

David Sweeney, Director of Research, Innovation and Skills at the Higher Education Funding Council for England, was in combative mood opening the discussion session on measuring scientific success. In a lively session, the Sweeney repeatedly accused the audience of ‘peddling myths’ surrounding UK research.

“It’s a complete myth that we’re tremendous at research but poor at application. We are ahead of the US on many of the measures for these sort of things, and the US wish they could be more like us. It’s a far more complex story than people make out. It is just tosh if you look at the evidence.”

‘Why are you doing science?,’ he then asked the assembled crowd of talented young researchers.

The answers were varied.

‘Curiosity.’

‘Passion.’

‘It seemed like a good idea at the time.’

‘An extension of what we do as children, exploring the world.’

“Yes, I often think that academics are children,” quipped Sweeney, asking us not to tweet that (he didn’t say anything about blogging it). A final suggestion, that we do science because it’s fun, bristled Sweeney into a retort: “It’s all very well saying you do it because it’s fun, but think who pays your salary. Not scientists, but taxpayers. Why should the taxpayer fund you to have fun?”. To which a riled audience member responded that scientists are tax payers too, and wouldn’t it be much better for everyone if people wouldd stop referring to researchers as though they were not themselves part of society.

Next question: ‘How, as a scientist, do you know if you’re a success?’

‘You change the world.’

‘Finishing the job.’

‘Making a difference.’

‘Peer recognition.’

‘Citations.’

‘Just staying alive.’

“Very few research careers change the world,” observed Sweeney. “We have to support career scientists while also supporting the occasional individual who excels. Our international reputation depends on just 1% of our research. How tempting for the treasury to see that bottom line and make cuts? But that 1% relies upon a broad base of basic, necessary research.”

The topic then strayed somewhat off course, moving into a discussion about the freedom of academics to pursue the research lines they want to, and how much funders should direct this. A balance needs to be struck. One audience member noted that some private funding sources are more willing to be flexible with research and are less bureaucratic than grant bodies drawing on public money. Could we do more to attract such funders to UK science? “Believe me,” countered Sweeney, “the Bill & Melissa Gates Foundation are just as bureaucratic”. And back to the bureaucracy involved with public money: “There’s an old in-joke: ‘If you win a £1,000,000 grant from the EU, the best thing you can do is pay them £2 million to go away’”.

Sweeney then brought the discussion back to topic, “Generally, it’s not individuals who are funded, but institutions. Universities may have a different agenda and objectives to the funders and to the individuals. So it’s really complex to assess performance. Any single metric, or indeed limited basket of numbers, is poison. We’ve got to involve the judgement of peers, contribution to society, and quantitative information, all together.”

The audience then asked why we should care about the UK specifically, when research is so international. “You’re dead right,” replied Sweeney. "Research is, of course, a global operation. Nonetheless, most of the funding for research done here is from the UK, not the EU. Therefore the tax payers want to see a result from UK science. It’s a balance.

The next point from the floor concerned the ‘research impact’ section on grant forms – how the proposed research will directly impact society. “Filling in this section is futile and bunkum,” claimed the audience member. “Numerical, evidence is better. I can write complete waffle and get money for it.” Sweeney wasn’t moved. “The research council won’t stand for PR waffle, and you’d be penalised. I see no objection at all to asking academics to think about how they engage with society. The biggest danger of assessment is to treat it as a game. Because the rules of the games change all the time. You must not take it as a box-ticking exercise, but pursue excellence.”

A discussion in a later session moved on to the subject of peer review, and the current issue of post-publication review – for example, a comments system beneath a paper for readers to leave remarks and suggestions. “Who reviews the post-review reviewers?,” smirked one audience member. “You migth end up with tribes of scientists fighting in the comments,” worried another. “Who is going to write these things, given the huge existing load. I can’t be bothered.” “Another post-peer review is not needed. I don’t want to plough through ‘dross’ trying to decide what’s worth reading.” So, not many voices in support of such mechanisms, however inevitable they seem.

In the afternoon panel discussion, someone questioned the effectiveness of the Research Assessment Exercise (which ranks higher education institutions according to their research excellence), something I’d expected to be at the core of the day’s discussion on measurements, but mostly absent until now. Sir John Beringer, who chaired the panel behind the RAE, had this to say:

“The assessment itself is not the issue, it’s what we do with the results. It’s usual to say we’ll change the allocation of resources, not change the departments that are not doing so well…As a young person you have to work your socks off, under local continuous assessment, but that assessment pressure drops off when you’re lucky enough to become a professor. The RAE can act as a way of weeding out poor academics at the senior level.”

As a final note, I was heartened by the words of David Willetts concerning the effectiveness of the new Con-Lib government. "We’ve shown in the way we’ve revised ministerial code and the way we’ve sustained commitment to science projects that we care about science and research. One of the good things about the coalition is that, because you are negotiating with another party, it forces you to appeal to evidence. The scientific way of thinking has, perhaps unexpectedly, proved to be one of the beneficiaries of this form of government.