Wow – this morning has been a real race through the issues – now the Q and A session, if my battery survives…
Q: (Charles Oppenheim, Loughborough) A lot of misunderstanding of citation analysis – Charles has a particular question about ‘self-citation’ – why exclude self citation, as evidence indicates that this makes very little difference to outcome.
A: Leiden CTWS use self-citation as one of their indicators. Self citation tends to drop off quickly over time. They have found that in some cases 50-60% of citation is self-citation – some groups over self-citate. From HEFCE point of view, they don’t want to fund ‘self-citation’ – if there is any direct opportunity to ‘game play’ the system, then self-citation is the most obvious area that can be abused. However, HEFCE happy to look at again in the next set of work. Linda indicates that at a high level of aggregation, she doesn’t beleive it has much impact.
Jonathan indicating that it is perhaps dangerous to cut out self-citation, as cutting edge research groups are bound to be citing themselves – and that a message from HEFCE that says ‘self-citation is bad’ (I noted Graeme from HEFCE shaking his head at this phrase!), you start to really mess around with the building blocks which the concept of bibliometrics is built on (being quite scathing about HEFCE here…)
Q: Leiden have looked at links between age and any bias in bibliometrics (and found there was no link). However, shouldn’t they have been looking at ‘career age’ not physical age?
A: Yes. But there was a high correlation in the group they looked at. As part of pilot HEFCE want to link the researchers submitted to their HESA record so they can look at this as a factor.
Q: Is there any thought of how to reward sharing and re-analysis of data sets?
A: James, from Thomson, issue of data sets is at very early stage of development with publishers – should they be peer reviewed? How could they be cited? etc. Important, but no answers yet, and a long way to go. HEFCE – the question is what’s really important to the subject community – need to balance complexity vs
Q: (from THE) Why did HEFCE use Web of Science over Scopus, and how did they reach that decision
A: When HEFCE first looked at the issue, consultancy advised that Web of Science was the only real option. However, now further work has suggested that Scopus is viable as well. HEFCE hasn’t made a decision yet. They may possibly look at both for the pilot, and that would inform what was used in the REF itself (which could be one or the other, or both)
Jonathan noting that recent (significant) changes in the THE global rankings were linked to switch from using Web of Science to Scopus, and we might anticipate a similar effect on REF.
Q: What software was used in Australia for bringing together data mentioned in Linda’s presentation (staff data, student data, publications data)
A: No direct answer – but 37 universities in Australia have been looking at these issues – go talk to them!
Q: Question about what bibliometrics actual measure, and a comment to say that we could expect to see increased citations as a result of a bibliometrics based exercise (mentioning some specific things, e.g. negative citations – citing work to show it’s wrong)
A: Negative citations – not a big impact, only happens in very specific areas (e.g. cold fusion) – peer review and journal system tends to filter out bad papers, so issue doesn’t arise. However, outside science more of an issue, as you get different ‘schools of thoughts’ who simply disagree with each other – so more likelihood that you would cite something you didn’t see as ‘excellent’.
Re: Increased citations – there is a general trend towards increased citation. But you don’t have a lot of control over how this happens – so very difficult to coordinate an exercise to affect the outcome.
I personally think this misses the point a bit – the issue is not that people will actually coordinate to effect outcome, but they will act in a way that they think will affect the outcome – obviously I don’t know if this is going to be significant, but it seems to me it is likely to change the rules of the game to some extent – which comes back to the point that Jonathan made from the Banking Industry example – such measures are only relevant for a certain time frame before the action of those you are measuring renders them invalid.
Out of battery now…