Annex A

Consultation response form for the Research Excellence Framework

• Respondents should complete the form below.

• Responses should be e-mailed to refconsultation@hefce.ac.uk by Thursday 14 February 2008. HEIs in Northern Ireland should send a copy of their response to research.branch@delni.gov.uk

• Institutions wishing to express an interest in taking part in the pilot of the bibliometrics indicator should e-mail their details to refconsultation@hefce.ac.uk by Thursday 31 January 2008.

• We will publish an analysis of responses to the consultation. Additionally, all responses may be disclosed on request, under the terms of the Freedom of Information Act. The Act gives a public right of access to any information held by a public authority, in this case HEFCE. This includes information provided in response to a consultation. We have a responsibility to decide whether any responses, including information about your identity, should be made public or treated as confidential. We can refuse to disclose information only in exceptional circumstances. This means responses to this consultation are unlikely to be treated as confidential except in very particular circumstances. Further information about the Act is available at www.informationcommissioner.gov.uk.

Respondent’s details

Consultation questions

(Boxes for responses can be expanded to the desired length.)

Consultation question 1a: Do you endorse our proposals for defining the broad group of science-based disciplines, and for dividing this into six main subject groups, in the context of our new approach to assessment and funding?

Separate departments of Engineering in many Universities are now part of Schools, Faculties or Colleges and so at the level of Institutional funding the proposed broad groups are sensible. However in terms of individual departments within these Schools distribution of QR will then be a matter for the Heads of those Schools etc. Some Departmental heads are concerned that this distribution methodology will lose the authority of being determined externally.

On balance though we quite understand that internal distribution is a matter for each HEI and thus we can accept the new larger groupings.

Consultation question 1b: Are there issues in relation to specific disciplines within this framework that we should consider?

In many Universities there are more than 3 separate departments of Engineering (e.g. Mechanical, Electronic, Civil etc) and often these are rated rather differently. By having the larger units the profiles will inevitably become averaged thereby losing the appearance of top flight research in some centres. This is applicable to physical science subjects also.

Consultation question 2a: Do you agree that bibliometric indicators produced on the basis that we propose can provide a robust quality indicator in the context of our framework?

Bibliometric indicators cannot produce a robust quality indicator in applied research and particularly not in Engineering. HEFCE accepts that less than 50% of Engineering journals are covered by Thompson Web of Science which means that citation data will be anything but robust. When viewed alongside the data in your Annex C it is clear that the Engineering subjects all have high levels of papers with no citations and very low citation rates compared with science. This is a feature of applied research which cannot be totally accounted for by a normalisation process.

Even your own commissioned study (CWTS) states in several places that bibliometric data should be used alongside peer review in assessing research quality. Section 3.1 states “Therefore, as a general principle we state that optimal research evaluation is realised through a combination of metrics and peer review…. Metrics and peer review both have their strengths and limits. The challenge is to combine the two methodologies in such a way that the strengths of one compensates for the limitations of the other”.

In section 3.2 of the CWTS report there is much discussion on peer review and it is stated that “Peer review is and has to remain the principal procedure for judgement of quality”. Whilst the report does highlight some problems of peer review also, the UK RAE system has a good deal of experience and expertise in peer review which ought to reduce any problems of bias and inexperience mentioned.

In Engineering the impact of research on industrial practice is considered by industry to be very important. However fundamental research is also carried out in Engineering Departments and its quality should be recognised alongside the impact of more applied research. The sole use of citation data will encourage academic engineers to publish in scientific journals and ultimately to move towards less applied research.

We believe that the CWTS study was right to dismiss journal impact factors just as RAE procedures have done in recent times. Impact factors do not guarantee high quality publications and they can be manipulated.

CWTS address the issue of non-citation index publications (non-CI) and expect that these will play an important role for a range of subjects including engineering. Bibliometric analysis of non-CI publications is quite complicated but is possible (Butler and Visser, 2006 and van Leeuwen et al 2007). However the HEFCE proposals neglect this important aspect from an engineering standpoint.

The HEFCE document states (para 52) that it would welcome views on how to recognise non-CI output and we therefore propose that before any metrics system be considered for introduction, HEFCE address this issue as a matter of urgency.

Consultation question 2b: Are there particular issues of significance needing to be resolved that we have not highlighted?

Citation counting will change the nature of publishing not only as stated above but also, in order to gain higher citations it would be sensible to publish in “special issues” of journals or publish “review articles” since PhD students read these and cite them regularly in their own publications. There is no doubt that this change of behaviour will occur just as the other examples cited in the CWTS report have already occurred in response to the existing RAE.

Our membership is already reporting that some universities are holding meetings to show academics how to maximise their citations by changing the modes of publishing and type of research carried out. This has also been encountered in places like the Netherlands where citations have been introduced.

Exclusion of self-citations can be a deterrent to collaboration. If a multi authored paper is cited by one of the multi-authors then are the other authors automatically excluded from using the citations? If they are it will deter collaboration and if they are not it is a potential loophole.

The CWTS report argues that US academics do not favour citing US authors but it is clear that they do favour citing US journals. This could mean that UK researchers will move their publication preferences to US rather than European journals in order to achieve greater citation rates. This will have an impact on European publishing.

Consultation question 3a: What are the key issues that we should consider in developing light touch peer review for the non science-based disciplines?

Peer review should be extended to Engineering subjects and this is an overwhelming view held by academics we represent (97% of the respondents from our membership oppose bibliometrics) and wish to retain a light touch Peer Review.

Consultation question 3b: What are the main options for the form and conduct of this review?

EPC would like to see peer review continue in a similar way to the way it is now with the inclusion of engineering. We also think that many physical science communities would also wish to be included in peer review.

Consultation question 4: Is there additional quantitative information that we should use in the assessment and funding framework to capture user value or the quality of applied research, or other key aspects of research excellence? Please be specific in terms of what the information is, what essential element of research it casts light on, how it may be found or collected, and where and how it might be used within the framework.

In engineering, patents, confidential reports and technology transfer are important aspects of academic research which are strongly encouraged by Government. Design work also falls into this category yet none of these will be captured by a bibliometrics based system. In previous RAEs it was quite usual to submit confidential reports for assessment and this could be done perfectly well using peer review.

Quantitative and qualitative data should be collected on research esteem (especially international prizes and honours) as this indicates the recognition of research quality by the wider community.

QAA recognised the fundamental difference between engineering and science in its definition “Realisation encapsulates the whole range of creative abilities which distinguish the engineer from the scientist; to conceive, make and actually bring to fruition something which has never existed before. This creativity and innovation to develop economically viable and ethically sound sustainable solutions is an essential and distinguishing characteristic of engineering….”

It is therefore vital that this is captured in a robust way.

Consultation question 5: Are our proposals for the role of expert panels workable within the framework? Are there other key issues on which we might take their advice?

As we understand it, expert panels will not be involved in any form of peer review but will be taking an overview and interpretative view of the data. We believe that a proper peer review panel could do the same role whilst also providing proper consideration of the quality of the papers.

Consultation question 6: Are there significant implications for the burden on the sector of implementing our new framework that we have not identified? What more can we do to minimise the burden as we introduce the new arrangements?

We believe that the checking of data by HEIs will impose a large burden on them and the fact that it is a new system will inevitably lead to over activity on this until sensitivities and working practices have been established. Thus it is by no means certain that the proposed new system will reduce the administrative burden.

Consultation question 7: Do you consider that the proposals in this document are likely to have any negative impact on equal opportunities? What issues will we need to pay particular attention to?

One good point about the RAE2008 is the consideration which can be shown for early career researchers and people who have taken career breaks (the majority of whom are women). This is not obviously the case in the proposed system.

Consultation question 8: Do you have any other comments about our proposals, which are not covered by the above questions?

There are likely to be major changes to the publication patterns of academics. We can see engineers moving away from working with industry and moving towards fundamental science publications to boost their citations. Also the type of publication will change. Fewer and larger (review type) papers are likely to be forthcoming to maximise citations to the detriment of engineering research and innovation that government seeks to help drive industry forward and kick start the UK economy again.

Indeed it is already clear that in anticipation of a metrics based system being introduced, some engineering departments are now recruiting physicists, chemists and mathematicians to teach engineering rather than engineers because of their publication patterns in high impact factor journals. This may well have serious consequences for the education of our future engineers.