There were some surprising results when the Higher Education Funding Council for England (HEFCE) released the details of its first attempt at measuring the impact of research in English universities (Times Higher Education).

HEFCE had already announced in November 2010 that the pilot was broadly successful (see Nature’s story). But now that the data from the pilot have come out, it appears that a number of departments that had performed well in the 2008 Research Assessment Exercise did unexpectedly badly on the impact assessment, while many lower-rated departments performed quite well. For example, in english language and literature, the University of Manchester, rated fourth best in 2008, had 80% of its impact submission rated just 1*. The scale ranged from 1* (good), to 4* (exceptional). Meanwhile Lancaster University, with a middling english department according to the RAE, had 35% of its impact submission rated 4* and 50% 3*. Similarly unexpected results were seen in physics.

HEFCE has cautioned against reading too much into the results. After all, this was a pilot exercise, and both the universities and those reviewing the submissions are still figuring out how the new system will work. But it is also a warning to the biggest universities that they cannot coast on their reputation for doing excellent research – they will have to do better at generating impact from that research, and find a way to explain and prove that impact.

Impact will be worth up to 25% of the total marks in the Research Excellence Framework, which will replace the RAE from 2014. Universities will be spending the next three years studying these result very closely indeed.