Beyond the RAE – Panel session

This session being moderated by Alan Thomson from the THE – not sure how this is going to work, it’s a big panel (all the speakers from today = 10).

Alan comparing research assessment to a high stakes poker game…

So, Alan is going to take questions, and then put them to the panel for discussion…

Q: (Ruth Jenkins, Nottingham) Are the funding councils going to a deal to get access to underlying data from Web of Science and/or Scopus

A: HEFCE would like to do this, although complicated – more than one version of the data (from the publishers directly, or with intermediaries such as Evidence Ltd or Leiden who have customised versions of the data)

In Australia, there have been some consortium deals, and some national deals for data.

Thomson Reuters are seeing the need for access to some of the underlying data, and are looking at how they can support this (using APIs, via system integration etc.) – want to make the right financial and technical arrangements to make a system work.

Follow up question about the pros and cons of using Google Scholar to generate citation data – directed at Les Carr from Southampton, as they have done some work on this. As part of his depts response to the RAE they looked at citation data and how this could be used to show that their dept was the ‘best’. They believe they need to take information from many difference sources – including Google Scholar.

Charles Oppenheim comments that there have been studies on the quality of data from Google Scholar vs other systems, and they generally find that the quality of data from Google Scholar is not good enough for it to give reliable information.

Q: What is the scope of the HEFCE pilot (what disciplines? all staff?)

A: HEFCE want to start off with the widest possible data set – so envisage pilot will be as wide a scope as possible. For the pilot they want institutions to tell them about all staff. However, if not enough institutions can manage this, they will look at different approaches.

Q: (Rosarie Coughlan, NUI Galway) What work has been done in bibliometrics in social sciences and humanities?

A: Anthony from Leiden says, you need to look at detail – for example psychology can really be regarded as a ‘science’ from the point of view of bibliometrics, but not sociology. Anthony from Leiden suggests that as long as the coverage is above zero, you should try bibliometrics, as they have found that there can be some.

Jonathan from Evidence Ltd less sanguine about this – the key thing for him is that researchers need to be involved in defining the measures that are relevant to them. Evidence Ltd. did some work in this area – they asked those assessing bids etc. how they assessed whether a piece of research was ‘excellent’ or not. They found the factors weren’t that different to natural sciences. However, Evidence Ltd. found that this did not mean the same measures were applicable.

Jonathan is concerned that the DIUS announcement starts ‘metrics creep’. Although Jonathan believes that at a high level there will be good correlations, you need a great deal of confidence in the system. If you undermine confidence in the judgements, then you get serious problems.

Linda from Australia they found that introducing book and book chapter citation to historians and political scientists they suddenly had a picture that they recognised, so they had more confidence.

Q: (Me!) What behavioural effects will the introduction of bibliometrics have?

A: Anthony from Leiden says, there will be behavioural effects, but we don’t know what – this needs careful assessment. There are likely to be different behavioural effects in short term and long term behaviours.

Anthony believes that changes will only be significant enough to impact on the validity of bibliometrics in the longer term, and believes that researchers won’t let it go that far as it impacts so much on the fundamentals of scholarly communication.

A comment from Hywel from Leicester (and backed up from Kings), that more junior researchers leave their names off early papers, so that they can cite later (presumably to avoid self-citation? I didn’t understand this). Anthony from Leiden defends the adjustment for self citation again, but maintains you should keep the self-citation as a separate factor, so you can see when it is low or high and needs further investigation. However, again the issue here is not that it makes a difference, but that people are changing their behaviour (already!) because they think it will have an effect.

I added to the question mentioning the research indicating possible link between Open Access availability and increased citation. James from Thomson says that his personal research (not Thomson view) shows that OA leads to accelerated citation rate, but not increased over the longterm – but research on this is in early stages, and only just seeing high quality research in this area.

Q: Question about why there has been no mention of the ‘h factor’ today?

A: Anthony makes point that there is no discipline specific factor in the h factor, and there is a strong correlation between career age and h factor, so not a good measure to use.

Q: (THE) Will Arts, Humanities and Social Sciences be included in pilot?

A: HEFCE want to include all disciplines where there is moderate citation coverage. They will be asking institutions if they would like to include those areas with less coverage.

Q: (Research Fortnight) The USA rejected bringing bibliometrics into assessment after resistance from the scientific community. Is there still the possibility that the REF could not use bibliometrics?

A: Graeme from HEFCE slightly avoiding the question, but feels that the sector accepts that a system with a  metrics based component can work (I think that ‘metrics based’ gives quite a lot of wriggle room here)

Q: How is negative citation going to work? How do you ‘attack’ a piece of work without adding to it’s citation rate?

A: Linda says that the effects by these edge cases disappear in large amounts of data. Also a panel of expertise is needed to make sure that any exceptions are caught.

Jonathan noting that citation analysis has been used to assess ‘Impact’ rather than ‘Quality’ for a reason. Impact is an apposite word.

Another comment that if we are making a selection of what work is being put forward for the REF, then universities are very unlikely to put forward papers that are being cited for all the wrong reasons.

Q: I didn’t quite get this one, but about the ‘centralised database’ vs ‘local’ (departmental based) databases.

A: Have to persuade via quality of central offering

Follow on comment about costs – it’s OK for research geared institutions like King’s (and of course, of that matter my own, Imperial) – but what about institutions that have not focused on research or don’t have the same level of resources?

A comment from the floor – a university cannot afford not to have a centralised system as part of their basic infrastructure.

Q: A question about inter-disciplinary areas of work – how are these affected?

A: Linda saying that essentially all subjects turn out to be inter-disciplinary in terms of where they publish. You need to analyse this (looking at individual articles, and where they have been published) and then aggregate at the unit level – and it’s not that difficult to do.

Collecting and Verifying Research Output

Case Study 1: King’s College London

Mary Davies (Deputy Direction of Information Services and Systems at KCL) is presenting this

Context

  • In 2004, preparations for RAE 2008
  • RAE 2001 had highlighted
    • Lack of linkage between systems
    • Data quality issues – discrepancies between central data and local (Dept) data
    • Time spent of data verification and manual data input
  • Decided to build a ‘Research Gateway’ to collage information from our corporate systems for submission to RAE – regular data feeds from HR, student, grants and finance database

Mary showed a couple of screenshots from the Research Gateway.

The Research Gateway was meant to avoid duplication of effort and promote effective data management. Kings didn’t have a publications database, and found that searches by institutional name on Web of Knowledge revealed at least 25% of Kings research output not attributed to Kings – often because where institutions had merged with Kings, the researchers continued to use the name of the original institution, not KCL. Also confusion where staff work for both Kings and an NHS Trust, and where research was attributed.

Kings decided to have a College Citation Policy advising staff on the importance of citing KCL as the institution.

Methodology

Project setup for the RAE 2008. Eventually submitted 4897 outputs from 1589 staff, across 37 UoAs (units of assessment).

Survey of Schools, Depts etc. found publications stored in a wide variety of systems – they decided to capture electronically back to 2001 to provide initial database contents. Only two of the locally developed databases were mapped to the Research Gateway.

Methodology was to search databases with address/affiliation fields using variations of institutional name + postcodes – e.g.: Web of Science, ISI Proceedings, Medline, Cinahl, etc.

Much easier in Health schools that other departments – especially for book, book chapter and other non-journal article information. Lots of data extracted from Worldcat and other sources, imported into Reference Manager database, big de-duplication exercise.

Los of work to clean up data

Challenges/Problems

Interestingly Mary has identified as an problem that the UoAs were substituting selected publications up to the last minute – we had this problem as well, and in the library this meant we were checking the accuracy of records that weren’t submitted, and ever shifting target in terms of how many submissions that needed checking.

Plans for REF

  • Develop the Research Gateway into a ‘Virtual Research Environment’, including integration with the Institutional Repository – attuned to requirements of REF (when these are clear)
  • Continue to work with supplier concerning database indexing policies, where relevant
  • Investigate research IDs and identifiers and institutional IDs developments
  • Develop intern expertise in bibliometrics – sent staff on the Leiden course
  • Relates to developing organisation culture of knowledge and comfort with bibliometrics
  • Concerned about Source data issue and appropriate coverage
  • Need to work on verification of citation information, especially if self citation excluded
  • Need clarification of how open access forms of publication will be taken into account
  • Capitalise on team work between academic and administrative staff
  • Develop HE forum for sharing best practice

 

Case Study 2: University of Leicester

After RAE 2001, Leicester found many depts didn’t have good records, and were often trying to collect 6+ years of data in a few months. When they came to RAE 2008 decided to invest, in software etc. to collect data.

RAE 2008 submission was 2700+ outputs including 260 books, from c.800 staff

Leicester bought GENIUS from InfoEd to use as their database – which they called Research Expertise Database (RED) – similar to King’s Research Gateway. This has a web based interface, with users empowered to update records, and integration to the staff records from HR etc.

Journals selected from pick list to prevent mistypes and use of abbreviations. Allows you to indicate a publication should be used for RAE.

They then could export data from RED to that RAE software. The RAE software was not seen as usable by the administrators for the Units of Assessment – they just had to export from RED to Excel, do some ‘simple’ data manipulation, and import to RAE software.

I have to admit from the description of what the administrators had to do once the data was in Excel, it sounds quite complicated – lots of things to update. However, they say it worked, and that all the UoA administrators were able to follow the system OK.

Looking forward to REF, Leicester concerned about the amount of work involved in excluding self-citation. The Library is biding for a new post to cover both the management of the Institutional Repository, and to be a expert on bibliometrics.

Case Study 3: University of Southampton

Leslie Carr from Southampton presenting… (at last, some nice looking slides – nothing startling – but elegant and clean and a nice font!)

Leslie saying that ongoing research evaluation isn’t peculiar to HEFCE – essentially it is something that all those involved in research do – asking Why am I doing this research? Why are we funding you to do this research? etc.

The answer? Leslie suggests – because it’s important, it has impact, it is changing the world, it is saving lives, it is changing policy, it is saving money etc. – the point Leslie makes is that beyond the ‘measures’, there is a ‘story’ behind a piece of research.

At Southampton, they have used the research/institutional repository to generate data for the RAE.

  • A repository helps an institution to collect an maintain its data, its impact measures and its story
  • The library is a key agency in helping to collect and curate the research outputs, analyses and interpretation

(at this point Leslie’s mobile goes off!)

OK – back to the presentation

The RAE at Southampton made use of inter-service expertise – the management group drew from expertise across the University.

They used the repository to

  • Store bibliographic detail of research outputs
  • Store scanned PDFs (with appropriate levels of access to lock down RAE material)
  • Store information to help write RA5 (don’t know what this is, probably should)
  • Store RA2 (don’t know what this is either) descriptions for submission
  • Output select and de-selection functionality
  • Reporting functionality

The software they used was based on the JISC IRRA project, with additional development (1fte plus a 6 month secondment).

The repository collected the evidence, and handled as much Quality Assurance as possible, this evidence was then handed over to another system to add in finance data etc.

Again, had to bring together a number of existing databases in their ‘Schools’. Some were sophisticated – some were not… varying amounts of effort to import records and bring up to a reasonable standard of quality.

Core staffing was 1 repository editorial manager, 0.6 editorial assistant, 3 staff contributing a few hours a week, and at peak 7fte additional temp staff – this additional effort varied over 18 months.

They had issues with

  • Recruiting and retaining staff with relevant experience for short term spells
  • Staff training and supervision
  • Dealing with high volume of legacy data (next time with ongoing use of the repository this should be less of an issue)
  • Verification of publication status

There was a tension between the HEFCE preference for the DOI, and the remit of the Southampton respository storing PDFs of full-text. However, it was a good route to get researchers engaged with the repository, they got lots of feedback, and won a reputation for reliability of support. There were no RAE specifi
c posts – rather they were repository posts.

This involvement raised the profile for services like Library and Planning, giving them a voice in ongoing discussions about research management – broadly, not just REF focused.

Future issues:

  • Liaise earlier to lobby HEFCE about any difficult requirements e.g. month for journal articles
  • Some over-reliance on individual staff expertise to be addressed (certainly from the library perspective, we had this issue at Imperial as well)
  • Further development of the research management and open access roles of the repository in the REF era
  • Further development of the research management system to pull data together from core databases, provide analytical tools and re-presentation options

Beyond the RAE – the afternoon sessions

Well, before lunch, my battery went, so I missed the final Q&A. However, over lunch I found a power socket, and got a bit more charge into the battery (although not fully charged, so won’t last the afternoon).

The afternoon of the session is mainly about case studies. However the first session is about a questionnaire all delegates filled out before the conference. The results are:

  • 75 Universities represented
  • About a third from library and information services
  • 41 have ‘Research’ in their job titles
  • Almost all delegates were involved in the collection of publications metadata for the RAE

Delegates identified the following challenges:

  • Verifying individual researchers publications lists (60%)
  • Collecting the source data (46%)
  • Engaging researchers support (36%)
  • Ensuring accuracy of data (6%)
  • DOIs (3%)
  • Meeting RAE requirements

More than half the delegates believed they would be involved in the REF. The main concerns were:

Getting accurate and verifiable data

  • Publications – knowing whether complete
  • Citation data
    • replicate citation counts
    • verfiy and challenge if necessary
  • Getting corrections in Web of Science

Institutional Infratstructure

  • Institutional repository or Publications database
  • Linking output to individuals
  • Difficulty of including citation information
  • Perhaps – National infrastructure
  • Perhaps – Centralised data collection
  • Systems Architecture – flexible
  • More sophisticated – inform research strategy

Knowing what is required

  • Hopefully today will help
  • What tools will be available to help?

Time and workflow

  • Too short timescale
  • Not just 4 publications per researcher
  • If citation window is 10 years (or more) – even more data to collect
  • Stability – perhaps become more integrated to institutional workflow, rather than special event every few years

Bibliometrics

Subject Coverage

  • Interdisciplinary areas – e.g. Nursing/Health

Selection of individuals

Gamesmanship

Using Web of Science

  • Comprehensiveness

Using other sources

  • Publications not in WoS

Scholarly Communications

Beyond the RAE – the afternoon sessions

Well, before lunch, my battery went, so I missed the final Q&A. However, over lunch I found a power socket, and got a bit more charge into the battery (although not fully charged, so won’t last the afternoon).

The afternoon of the session is mainly about case studies. However the first session is about a questionnaire all delegates filled out before the conference. The results are:

  • 75 Universities represented
  • About a third from library and information services
  • 41 have ‘Research’ in their job titles
  • Almost all delegates were involved in the collection of publications metadata for the RAE

Delegates identified the following challenges:

  • Verifying individual researchers publications lists (60%)
  • Collecting the source data (46%)
  • Engaging researchers support (36%)
  • Ensuring accuracy of data (6%)
  • DOIs (3%)
  • Meeting RAE requirements

More than half the delegates believed they would be involved in the REF. The main concerns were:

Getting accurate and verifiable data

  • Publications – knowing whether complete
  • Citation data
    • replicate citation counts
    • verfiy and challenge if necessary
  • Getting corrections in Web of Science

Institutional Infratstructure

  • Institutional repository or Publications database
  • Linking output to individuals
  • Difficulty of including citation information
  • Perhaps – National infrastructure
  • Perhaps – Centralised data collection
  • Systems Architecture – flexible
  • More sophisticated – inform research strategy

Knowing what is required

  • Hopefully today will help
  • What tools will be available to help?

Time and workflow

  • Too short timescale
  • Not just 4 publications per researcher
  • If citation window is 10 years (or more) – even more data to collect
  • Stability – perhaps become more integrated to institutional workflow, rather than special event every few years

Bibliometrics

Subject Coverage

  • Interdisciplinary areas – e.g. Nursing/Health

Selection of individuals

Gamesmanship

Using Web of Science

  • Comprehensiveness

Using other sources

  • Publications not in WoS

Scholarly Communications

Beyond the RAE – Q and A

Wow – this morning has been a real race through the issues – now the Q and A session, if my battery survives…

Q: (Charles Oppenheim, Loughborough) A lot of misunderstanding of citation analysis – Charles has a particular question about ‘self-citation’ – why exclude self citation, as evidence indicates that this makes very little difference to outcome.

A: Leiden CTWS use self-citation as one of their indicators. Self citation tends to drop off quickly over time. They have found that in some cases 50-60% of citation is self-citation – some groups over self-citate. From HEFCE point of view, they don’t want to fund ‘self-citation’ – if there is any direct opportunity to ‘game play’ the system, then self-citation is the most obvious area that can be abused. However, HEFCE happy to look at again in the next set of work. Linda indicates that at a high level of aggregation, she doesn’t beleive it has much impact.

Jonathan indicating that it is perhaps dangerous to cut out self-citation, as cutting edge research groups are bound to be citing themselves – and that a message from HEFCE that says ‘self-citation is bad’ (I noted Graeme from HEFCE shaking his head at this phrase!), you start to really mess around with the building blocks which the concept of bibliometrics is built on (being quite scathing about HEFCE here…)

Q: Leiden have looked at links between age and any bias in bibliometrics (and found there was no link). However, shouldn’t they have been looking at ‘career age’ not physical age?

A: Yes. But there was a high correlation in the group they looked at. As part of pilot HEFCE want to link the researchers submitted to their HESA record so they can look at this as a factor.

Q: Is there any thought of how to reward sharing and re-analysis of data sets?

A: James, from Thomson, issue of data sets is at very early stage of development with publishers – should they be peer reviewed? How could they be cited? etc. Important, but no answers yet, and a long way to go. HEFCE – the question is what’s really important to the subject community – need to balance complexity vs

Q: (from THE) Why did HEFCE use Web of Science over Scopus, and how did they reach that decision

A: When HEFCE first looked at the issue, consultancy advised that Web of Science was the only real option. However, now further work has suggested that Scopus is viable as well. HEFCE hasn’t made a decision yet. They may possibly look at both for the pilot, and that would inform what was used in the REF itself (which could be one or the other, or both)

Jonathan noting that recent (significant) changes in the THE global rankings were linked to switch from using Web of Science to Scopus, and we might anticipate a similar effect on REF.

Q: What software was used in Australia for bringing together data mentioned in Linda’s presentation (staff data, student data, publications data)

A: No direct answer – but 37 universities in Australia have been looking at these issues – go talk to them!

Q: Question about what bibliometrics actual measure, and a comment to say that we could expect to see increased citations as a result of a bibliometrics based exercise (mentioning some specific things, e.g. negative citations – citing work to show it’s wrong)

A: Negative citations – not a big impact, only happens in very specific areas (e.g. cold fusion) – peer review and journal system tends to filter out bad papers, so issue doesn’t arise. However, outside science more of an issue, as you get different ‘schools of thoughts’ who simply disagree with each other – so more likelihood that you would cite something you didn’t see as ‘excellent’.

Re: Increased citations – there is a general trend towards increased citation. But you don’t have a lot of control over how this happens – so very difficult to coordinate an exercise to affect the outcome.

I personally think this misses the point a bit – the issue is not that people will actually coordinate to effect outcome, but they will act in a way that they think will affect the outcome – obviously I don’t know if this is going to be significant, but it seems to me it is likely to change the rules of the game to some extent – which comes back to the point that Jonathan made from the Banking Industry example – such measures are only relevant for a certain time frame before the action of those you are measuring renders them invalid.

Out of battery now…

Beyond the RAE – Q and A

Wow – this morning has been a real race through the issues – now the Q and A session, if my battery survives…

Q: (Charles Oppenheim, Loughborough) A lot of misunderstanding of citation analysis – Charles has a particular question about ‘self-citation’ – why exclude self citation, as evidence indicates that this makes very little difference to outcome.

A: Leiden CTWS use self-citation as one of their indicators. Self citation tends to drop off quickly over time. They have found that in some cases 50-60% of citation is self-citation – some groups over self-citate. From HEFCE point of view, they don’t want to fund ‘self-citation’ – if there is any direct opportunity to ‘game play’ the system, then self-citation is the most obvious area that can be abused. However, HEFCE happy to look at again in the next set of work. Linda indicates that at a high level of aggregation, she doesn’t beleive it has much impact.

Jonathan indicating that it is perhaps dangerous to cut out self-citation, as cutting edge research groups are bound to be citing themselves – and that a message from HEFCE that says ‘self-citation is bad’ (I noted Graeme from HEFCE shaking his head at this phrase!), you start to really mess around with the building blocks which the concept of bibliometrics is built on (being quite scathing about HEFCE here…)

Q: Leiden have looked at links between age and any bias in bibliometrics (and found there was no link). However, shouldn’t they have been looking at ‘career age’ not physical age?

A: Yes. But there was a high correlation in the group they looked at. As part of pilot HEFCE want to link the researchers submitted to their HESA record so they can look at this as a factor.

Q: Is there any thought of how to reward sharing and re-analysis of data sets?

A: James, from Thomson, issue of data sets is at very early stage of development with publishers – should they be peer reviewed? How could they be cited? etc. Important, but no answers yet, and a long way to go. HEFCE – the question is what’s really important to the subject community – need to balance complexity vs

Q: (from THE) Why did HEFCE use Web of Science over Scopus, and how did they reach that decision

A: When HEFCE first looked at the issue, consultancy advised that Web of Science was the only real option. However, now further work has suggested that Scopus is viable as well. HEFCE hasn’t made a decision yet. They may possibly look at both for the pilot, and that would inform what was used in the REF itself (which could be one or the other, or both)

Jonathan noting that recent (significant) changes in the THE global rankings were linked to switch from using Web of Science to Scopus, and we might anticipate a similar effect on REF.

Q: What software was used in Australia for bringing together data mentioned in Linda’s presentation (staff data, student data, publications data)

A: No direct answer – but 37 universities in Australia have been looking at these issues – go talk to them!

Q: Question about what bibliometrics actual measure, and a comment to say that we could expect to see increased citations as a result of a bibliometrics based exercise (mentioning some specific things, e.g. negative citations – citing work to show it’s wrong)

A: Negative citations – not a big impact, only happens in very specific areas (e.g. cold fusion) – peer review and journal system tends to filter out bad papers, so issue doesn’t arise. However, outside science more of an issue, as you get different ‘schools of thoughts’ who simply disagree with each other – so more likelihood that you would cite something you didn’t see as ‘excellent’.

Re: Increased citations – there is a general trend towards increased citation. But you don’t have a lot of control over how this happens – so very difficult to coordinate an exercise to affect the outcome.

I personally think this misses the point a bit – the issue is not that people will actually coordinate to effect outcome, but they will act in a way that they think will affect the outcome – obviously I don’t know if this is going to be significant, but it seems to me it is likely to change the rules of the game to some extent – which comes back to the point that Jonathan made from the Banking Industry example – such measures are only relevant for a certain time frame before the action of those you are measuring renders them invalid.

Out of battery now…

An Australian perspective on metrics-based assessment systems

This by Linda Butler, from the Research Evaluation and Policy Project in Australia.

In Australia:

  • 1990+ Research Quantum (RQ) formual funding
  • 1993: Refinement of RQ formula – Universities establish publications collections (which were audited several times – although early audits found error rates of up to 60%)
  • 2004: Research Quality Framework (RQF) announced
  • 2006: Details of RQF process start to emerge – with metrics taking a leading role
    • Most Universities refine their MIS
    • Many import Thomson data directly into their publications databases
    • Link publication databases to those covering staff, student and grants data (either via proprietary software or through ‘home grown’ systems – that latter being rare, and seen as risky by Linda)
  • 2007:  The RQF abandoned by new government, being replaced by new metrics based system – going in a very similar direction to the REF, and Linda believes they will be very similar

The data that is available already in the system that might be used for metrics in Australia is:

  • Research Income – amount and source (but done at very broad level of aggregation)
  • Publication counts – books, chapters, journal articles, conference papers
  • Formulas also use separately collected Research student data – number of students; completions

Recommended RQF metrics were:

  • Publication outlets ranked into tiers – journals, publishers, conferences, venues (for performing arts)
    • Journal ranking process has been ongoing for 6 months

I would have thought this really makes the likelihood of ‘game playing’ very high – already we can see the dominance of certain publications in some areas, and this will surely just reinforce this?

  • External income – differentiate between types of income
    • However, this was dropped as universities could not supply the data at an appropriate level of detail
  • Citation data – 1. Citations per publication; 2. Distribution across percentiles
    • standard methodology for disciplines with >50% coverage – include most science (inc. Maths) and Engineering asked to be included, although the coverage was closer to 40%

Linda is saying that the methodology chose directly affects costs (to the universities) and the transparency of the process – the type of thing you need to look at is:

  • Level of aggregation – Institution, or discipline, or group (in order of ascending difficulty)
  • Who a university can claim – on staff at census date (complex, time consuming), university appears on publication (straightforward)
  • Who a university can submit – all staff (low cost), a subset of ‘research active staff’ (more complex and expensive
  • Complexity of measures – treatment of collaborative papers, self-citations, etc.

What is the degree of risk that different methodologies will produce different outcomes?

Linda is expressing her own opinion here – only some of this currently backed up with empirical data. Talking about a high level of aggregation only – University or Discipline level:

  • Level of aggregation – not much impact
  • Who a university can claim – moderate impact – mostly not much impact, but perhaps an impact for a small number of universities. Generally you will gain some and lose some researchers
  • Who a University can submit – low to moderate (depending on funding algorithm)
  • Complexity of measures – low (this assessment has some empirical data behind it)

Essentially Linda seems to be saying – you may as well chose the cheapest approach, as she doesn’t believe it will impact significantly.

Linda draws the following lessons for the REF:

  • The development of institutional information systems is complex and takes a long time
  • Choice between simplicity and complexity has huge implication for the costs of the exercise
  • Often those lobbying for a particular methodology have no real understanding of the cost implication of their preferred choice

Those ‘at the coalface’ need to communicate to others in the institution the impact of what they are asking for in terms of cost to the institution.

The use of bibliometrics to measure research quality in UK HEIs

This session by Jonathan Adams (from Evidence Ltd)

Evidence Ltd have been working in this area for some years. Evidence Ltd believe that the UK’s international research status has increased over the last 20 years, and the REF needs to support this.

Jonathan is showing a graph of output from various European countries which shows that over the last 20 years, output from each country has been reasonably stable, except the Netherlands, which has experienced a large growth. He believes we need to avoid a system that leads to this disproportionate change (and is saying that the adoption of bibliometrics in the Netherlands caused their increased output)

Jonathan raising questions – will the introduction of metrics lead to manipulation, that invalidate the assumptions on which metrics have been judged and introduced.

Jonathan making the point that the recent DIUS announcement on peer review does not mean metrics are not to be used – in fact, any peer review in the REF must use the metrics – the panels will not be able sensibly to disregard what the metrics say.

Measurement of Impact (normalised) 1996-2000 shows high correlation to RAE2001 grade – however, he says there is a huge variation is residual data (I don’t understand what he means here).

One of the key questions (says Jonathan) is whether we assess total activity or selected papers – this makes a significant difference to the outcome of metrics you apply (although I would have thought the same was bound to be true of any measure of quality – allowing ‘nomination’ of papers is a filter for quality surely?)

Jonathan making the point (that Anthony did as well), that averages do not describe underlying profiles – the data is highly skewed (lots of low values, few very high values). So – we need to do profiling for different disciplines – we cannot rely on averages, as the data is so skewed. Comparing distributions is much better than averages, which is what these profiles are meant to allow us to do.

Another issue – there are many papers published that are never cited – we need to be clear how these will be dealt with.

It is clear that the Normalisation strategy adopted will have a significant impact on the outcome – Jonathan showing how three different normalisation strategies can change the outcome.

Any clustering model needs to fit a UK research model – need to be sure that a discipline is reviewed alongside similar disciplines from the bibliometric viewpoint. For example, Chemical engineering publications from the UK are published in physics journals – so need to think about it in these terms, not just lump it in with the rest of chemistry when deciding how to normalise etc.

Evidence Ltd have produced a RAE/REF Predictor (‘You are the REF’) which allows anyone (? or anyone from a UK HEI?) to decide on different weighting factors etc., and see what happens – this is apparently going to be available soon at http://RAE2008.com

Goodhart’s Law – once you use a metric for matters of public policy, it loses its effectiveness overtime (based on experience in the banking sector)

Best practice in the use of citation data

As This talk by Anthony van Raan from the University of Leiden (who was a consultant to HEFCE on the use of bibliometrics I think)

Anthony is at the Centre for Science and Technology Studies (CWTS), who have been working on bibliometrics for many years. CWTS have a licence agreement with Thomson for the use of Web of Science raw data, but has also created its own bibliometric data system using algorithms.

Apparently citation isn’t as complicated as quantum mechanics! That’s a relief…

What do citations measure:

  • There is a positive correlation between citation based studies and qualitaty judgements by peers

Anthony is going through this at quite a pace, so these notes may be sketchy…

There is a set of total publications, and various systems list subsets of this ‘universe’ – Web of Science is a big subset, but also LNCS, Medline, arXiv. Now Scopus is another big subset, and CWTS are now comparing Web of Science to Scopus.

Anthony is now going of key indicators and what they help measure – however, he is going through it so fast I can’t keep up – it might not be quantum mechanics, but as this speed I can’t capture it all.

As Web of Science covers (as you might expect) mainly science – and it is the ‘science’ based disciplines that use journal publication as the major route for scholarly communication. In Humanities and Arts, books are used more, and CWTS is experimenting with a new kind of ranking – which measures if ‘your’ book is available in the top 250 university libraries (he doesn’t say how top 250 university libraries are measured) – if this gets adopted, will we start to see an increased demand for libraries to buy specific books to increase ranking?

Anthony demonstrating how Chemistry researchers publish across journals from many areas (e.g. Rheumatology, Cell Biology, Applied Physics) – so even an established discipline like Chemistry turns out to be very interdisciplinary – at least when it comes to publication.

Anthony indicating that bibliometric indicators can be manipulated and/or they discriminate certain persons/types of research – but establishing evidence for some of this needs work – for example CWTS have found that the assertion that bibliometrics discriminate against younger academics is not supported by the evidence. However, they have found that journal editors behave in such a way that maniuplates impact factor (CWTS published an article on this in the Journal of Documentation)

Principles of citation-based evaluation

The session is by James Pringle from Thomson Scientific (now Thomson Reuters) (a.k.a. ISI)

I have to admit to some slight cynicism around Thomson’s involvement, as they clearly have one of the richest sets of citation based data, and so I feel that they have an innately biased approach – however, James has said he is speaking (as far as he can) not as a vendor in this context (but he’s hardly likely to say bibliometrics aren’t very helpful is he?)

James is indicating that Thomson believe that citation data need to be used wisely – to inform discussion rather than to replace discussions.

James is describing citations as “User-generated content” of formal scholarly communications (ok – but using a buzzword here – but all the published papers are ‘user-generated’ as well. I would think it would be more accurate to regard it as the ‘link’ of scholarly communications, which makes using it to generate bibliometrics akin to Google’s pagerank)

However, although citation-based evaluation is fundamental to effective evaluation and management, say James, it is frequently mis-used and misunderstood in an era of democratisation – the sector needs ‘best practices’ to avoid this (apparently  “Using bibliometrics in evaluating research” by David Pendlebury lists 10 rules for using citation-based evaluation). The availability of citation information leads researchers to make their own assessments – which may not be accurate – or at the least, not uniform. It also has lead to researchers submitting ‘corrections’ to Thomson, which challenges them to respond quickly and provide more accurate information.

James outlines how use is made of citation data for research evaluation at the level of Individuals, University Departments, University Management and External Entities (such as funding bodies). These different levels have different needs and requirements, so we need to producing citation data that can serve needs across this range.

Thomson have looked at Institutional Research Evaluation workflows:

  • Identification of researchers and their work
  • Data validation and metadata management
  • Populating local systems with ongoing updates
  • Enabling access to data via internal systems and respositories
  • Data and metrics to create reports for external agencies

These are not unique to the UK – they are being tackled by institutions across the globe, although the detail is effected by national policies etc.

So, how do Thomson support research evaluation – they have a number of principals:

  • First, do no harm – encourage and develop use of best practices
  • Integrate and partner – work with institutions to define and develop integrated solutions
  • Enhance content to meet emerging needs
    • Right Journal content
      • Careful increase journals in Web of Science
    • Recognize expanding scholarly communications
      • Integrate proceedings citation content into Web of Science
    • Engage researcher community
      • ResearcherID.com and Author disambiguation
    • Engage Libraries and Research Management
      • Institutional Identifiers and Institutional Disambiguation/Unification

All this sounds OK – but I’m worried about how it joins up with activity in the sector, and by other organisations. How does ‘ResearcherID.com’ link to OCLC Identities work? It would be great to see some joined up thinking across the library/information sector on this, as otherwise we will end up with multiple methods of identification.