Engineering Professors’ Council
Research Excellence Framework – HEFCE
The Engineering Professors’ Council has had extensive discussions within the membership of our 1600 UK based Professors of Engineering and our view has not changed substantially from that submitted in September 2006 in response to the earlier consultation by RCUK.
Ninety seven per cent of the responses from our membership support the retention of peer review and reject a bibliometric basis of review.
Our objections to using metrics as the sole component of Research Assessment are based on the principle that primarily input metrics (research income, numbers of research students registered etc.) cannot be used to judge standards of output quality. They may, at some level of aggregation be correlated, but the two are not necessarily linked. There needs to be some measure of “efficiency” linking the input to output if input factors are to be used. In addition, we do not believe that the proposed Citation Analysis (CA) metric has been adequately evaluated for engineering and consequently it seems to us that insufficient thought has been given to the effect that the proposals will have on the future of engineering research and teaching in UK universities
We feel strongly that Engineering, like mathematics, should be assessed by a combination of light-touch peer review and metrics.
If we now turn to bibliometrics and in particular the use of citations, the consultation document itself admits that there are serious problems for engineering in using citation data from the Web of Science (WoS). Engineering Journal coverage is very low (less than 50%) and this primarily relates to traditional ‘blue skies’ research. However, we note that engineers currently carry out a great deal of applied research for government, the military, private and public companies which is not covered by WoS as well as the traditional ‘blue skies’ research which is.
Good industrial design or military research is often not published in journals available publicly for security or commercial reasons. Equally industry often prefers engineers to publish in international conferences which are not included in WoS. This work therefore cannot be credited by citations. Finally highly cited publications may not actually be of great value to the community that engineers are supposed to serve. For example papers with mistakes may be cited for that very reason. We believe, therefore that the use of metrics removes the incentive for engineering academics to work with industry and this runs counter to other governmental thinking.
In summary, we are concerned that: -
There is poor citation data for Engineering
Citations give no indication of the work’s worth to industry
Design work will rarely gain citations
Metrics cannot easily measure quality
The obvious way for departments to improve bibliometric counts is to stop doing applied research with industry and instead do fundamental science. We do not believe that this is really what the UK wants or needs. We conclude that the case for metrics being able to measure engineering research ‘impact’ has not yet been well made.
Our discussions show that the vast majority of engineering academics believe that the exclusive use of metrics will serve neither the engineering community nor the UK well and many believe that the amount of work involved by HEIs in checking and sourcing data, could well equal that of the current RAE.
Evidence of how different engineering is from other medical, biological and physical science can be seen in Annex C of the HEFCE report where the percentage of papers not cited is almost twice as high for every engineering subject as all the other subjects. We believe that this difference is sufficiently important to merit peer review for engineering subjects.
We therefore believe that Engineering should be assessed using
Engineering Professors’ Council
4th February 2008