Guest blog by Jonathan Adams, Chief Scientist at Digital Science.

Do you know what good research looks like? One of the great successes of research management in leading research economies is that it is usually equitable and fair in judgements about who gets the jobs, who gets the grants and who gets the recognition. That is a powerful incentive to the people who deliver research outcomes of benefit to society and the economy. Lose their confidence and enthusiasm and you have lost everything.

But not all is quite as it should be.  Our recent report for Digital Science shows that when researchers put on their best show for the UK’s Research Assessment Exercise, they think journal articles are the best bet – even where surveys say that monographs or conference proceedings are key to their subject. And they think that articles from high impact journals provide a better signal than other highly-cited work. The evidence suggests that the ‘right’ signal has become more important than the real substance.

Of course, we can’t tell whether this second-guessing works: we don’t know whether the peer panels actually find the material compelling! Signal over substance could be a delusional UK phenomenon. But another recent article points to a similar outcome in a different area.

What benefits your career progress? A report in Current Biology analyses the association between research achievement and the likelihood of becoming an independent investigator. One factor that stands out is publishing in a high impact journal, and this has a higher benefit to career success than the impact of the individual papers.

Publishing in a high impact journal is a real signal of achievement. It’s tough in a highly competitive environment to get past those editorial and refereeing decisions. So, it’s great to get into Nature or Science or the leading international journal in your field. But the point here, in the Current Biology study and in our own analyses, is that this seems to be more influential than the actual paper. Some of the UK researchers actually had more highly cited papers available.

Game-playing is part of any evaluation. Students know this when they apply to university and flesh out their personal statements and when they apply for work experience with an optimistic positivism to every claim for their ability. It works because those who sift through applications need shortcuts to create a shortlist. It would be no surprise if proxies were used by peer reviewers faced with thousands of research outputs.

The task for stakeholders in the research system is to think how game-playing might be offset by parallel tactics to identify and foreground real achievement. HEFCE has set up a review group to look at this. They have no easy task, but it is worth avoiding shortcuts and putting in enough resource to make sure that any evaluation system continues to retain the respect of those who are evaluated.