Are you responding:
(Delete one)

• On behalf of an organisation
• As an individual
  

Name of responding organisation/individual

Council of University Heads of Pharmacy (CUHOP)

Contact name

Dr Robert Dewdney

Position within organisation (if applicable)

Secretary

Contact telephone number

029 208 79086/7

Contact e-mail address

DewdneyRS@Cardiff.ac.uk

With respect to our own discipline, Pharmacy is a highly multidisciplinary subject, involving chemistry, biology, pharmacology, clinical science, material science and engineering.  It is difficult to see how bibliometrics for Pharmacy might be correctly normalized within one of the six subject groups.  As has been trailed within the consultation, it is probable that the discipline will be dispersed among at least two science subject groups.  Pharmacy research also covers areas of clinical/professional practice and education which might well be classed as part of arts, humanities and social sciences.  In the current RAE these areas of research have been encompassed in UOA 13 (Pharmacy).  If some practice and education areas were to be removed to arts, humanities and social sciences, then these parts of Pharmacy may remain un-assessed until 2013/14.

This is not though fundamentally to object to the proposals from HEFCE, rather it is to flag up that we wish to be assured that there will be no damage to multidisciplinary subjects such as Pharmacy.  At the moment there is a worry, possibly unwarranted but real nonetheless, that applied science disciplines will be judged by bibliometric criteria developed for basic research.  The RAE 2008 UoAs were of course familiar and understood, with defined boundaries and easily identified and accepted expert peers and advisers.  Much will have to be done to bring understanding and confidence in the proposed new system, especially for multidisciplinary subjects such as Pharmacy.

Pharmacy presents obvious issues.  The discipline straddles Health Sciences, Biological Sciences and Physical Sciences. In addition, certain social and clinical research dimensions in Pharmacy (mentioned under 1a, above) may fit alongside Nursing or even outside of science altogether.

There are fears that need to be calmed; where would Pharmacy lie? If, as seems likely, it is split among different major groups, on what basis might an institution return QR monies generated?

In principle at least, bibliometric assessment of research quality based on independent citation rates seems reasonable.  To be fully convinced, we need to see as much evidence as possible, including from pilot exercises, that numbers of citations correlate well with research quality as assessed at RAE for all of the proposed broad subject groupings.

We wonder if there needs to be a peer-review filter of the bibliometric data.  Without such a filter there might be distortions due to the niche nature of some research.  Just because a large percentage of the research community is not engaged (and therefore citing) work in an area does not necessarily indicate that it is of less value than mainstream research.   If there is insufficient care taken here the result will be a reduction in innovation and risk-taking.  Occasional leaps in scientific knowledge and understanding might be impaired by inappropriate, 'blunt instrument', judgements of worth favouring publication in the mainstream of the incremental, iterative advance of science.

It has been well-trailed but warrants repeating that if a published article states a point which is subsequently disproved or retracted, this will require discussion and prompt citations.  Contentious (although not necessarily excellent) papers may result in letters/editorials in journals and many citations.  A useful but unremarkable modification to a standard experimental method can create many citations.  All may skew citation results.

All of the relevant issues appear to have been covered in the consultation document but it is worth highlighting some that might be later overlooked in the dash to early implementation.  This leads to the important side observation and viewpoint that the timescale for implementation is too short; there are issues to be fully sorted out before the exercise can be robust.

There are some potential practical problems with the Thompson ISI Web of Science (WoS) which are not insurmountable but may entail considerable work for HEIs, especially with regard to the accurate and reliable collection of bibliometric data.  At present, the entire proposal is based on the use of this single commercial database.  The accurate identification of individuals and their work in this database can be difficult and may require very careful sifting/manipulation of the database by trained operatives, as has been acknowledged by the authors of the consultation document.

Recent articles, which might include the output of Early Career Researchers (ECRs), are likely to have attracted few citations.  There will therefore be a whole tranche of recent work, particularly by ECRs, which cannot be assessed by bibliographic means, putting at risk the overall analysis of quality for the subject under assessment.

Research is a dynamic activity, with researchers moving into new fields and disciplines taking on a new relevance. There will need to be a mechanism for reselection and de-selection of staff submitted for assessment.

Multi-authorship may mean substantial sharing of papers within or even across subject groupings.  This may have an influence on the average citation rate.

If the Research Excellence Framework (REF) is intended to assess the research quality of a subject area in an HEI, then the published work under assessment should be associated with employees at the census date.

Normalisation will be critical and will require widespread sector agreement in order for the REF to be credible.

There is a strong likelihood that citation strategies will evolve.  It will be important to identify these and differentiate them from standard practices within discipline areas (which might not, of course, be consistent with the consolidated behaviour of a whole subject group).

A review of WoS classification of journals is proposed in paragraph 41.  The panel of UK-based experts will need to be drawn widely from across the RAE 2008 units of assessment in order to ensure credibility at discipline level for this review.

The pilot exercise must cover multidisciplinary subjects such as Pharmacy in order to be credible and reassuring.

A decision will need to be made or be made clearly known, as to whether all papers or selected papers will be subject to bibliometrics.  If all papers are included then this may affect publishing patterns, perhaps to the detriment of dissemination to the research user community.

We support the use of data relating to research income and postgraduate research training alongside bibliometric data.  As for bibliometric data, these will need to be normalised by discipline.

For applied disciplines such as Pharmacy, patent numbers and details, realized products and licensing arrangements would be obvious candidates for inclusion in the assessment and funding framework to capture user value.  Also to capture user value, engagement with industry, in our case pharmaceutical companies, might be a powerful indicator of research worth.  Income generated in this way, that can have restrictions on publication, might be judged separately.

There is a real risk that emphasis on quantitative measures will lead to a loss of engagement of academic staff in presently esteemed activities; serving on advisory committees, research panels, etc.  John Denham, in a keynote speech in January at the Royal Society of Arts, expressed concern that there “may be disincentives in the system that emphasize published and peer-reviewed work over public policy advice”. He went on to say, “to my mind, a scientist who produces fewer papers but produces excellent evidence and advice in the national interest on behalf of the Government should not feel that they disadvantage themselves, their research colleagues or their institution when research funds are distributed.”

As yet, not much detail of the proposed roles of the expert panels is provided.  Some expert oversight of a metrics-based approach will be essential to prevent distortions of UK scientific endeavour in response to numerical targets.

The members of the expert panels must adequately cover the disciplines (some of which will have unique features) within each broad subject grouping.  It is therefore likely that each membership will be large.  The features of smaller disciplines must not be obliterated by the character of larger disciplines.

It is unclear how the REF will address the objective of "promoting equality and diversity"; however it is equally difficult to envisage it fundamentally running counter to this objective.  We assume that there will be adjustments to take account of aspects like maternity leave and ECR status.

In order for the REF to promote “excellence and dynamism and encourage research that benefits the economy and society” (paragraph 14d), indicators for these characteristics may need to be established for use in the assessment process.

There might need to be some form of self-evaluation for subject groups (or even disciplines) within HEIs; otherwise there will be no evidence of strategic potential.

A too-frequent review might work against changing research strategies – particularly within smaller institutions – if there is a risk of a low-output developmental period.

Without any assessment of development in research environment and research strategy (as in RAE 2008) there will be a substantial lag before there is any manifestation of the effect of these improvements on citation indices (by virtue of the intrinsic delay in citation).  This will have a tendency to negate, or at best delay the impact of internal investments on QR funding allocation.

The proposed timescale of commencement 2009 is too close to RAE 2008; the bibliographic analysis will be looking back over a period which was embraced by RAE 2008.  Indeed, with citation analysis only giving an accurate picture of activity five or more years ago, if implemented as early as 2010, the REF would likely reflect published research from about 2002 to 2004 and RAE 2008 would be giving a more up-to-date picture.

We are surprised at the proposal for institutions to nominate researchers as early as January 2009; only a few working days after receiving and beginning to digest the outcomes of RAE 2008.

As the main subject representative group for Pharmacy in UK HE, CUHOP stands ready for any discussions which HEFCE might wish to engage in to achieve Pharmacy’s appropriate positioning within the REF.