Case Study 1: King’s College London

Mary Davies (Deputy Direction of Information Services and Systems at KCL) is presenting this

Context

• In 2004, preparations for RAE 2008
• RAE 2001 had highlighted
  • Lack of linkage between systems
  • Data quality issues – discrepancies between central data and local (Dept) data
  • Time spent of data verification and manual data input
• Decided to build a ‘Research Gateway’ to collage information from our corporate systems for submission to RAE – regular data feeds from HR, student, grants and finance database

Mary showed a couple of screenshots from the Research Gateway.

The Research Gateway was meant to avoid duplication of effort and promote effective data management. Kings didn’t have a publications database, and found that searches by institutional name on Web of Knowledge revealed at least 25% of Kings research output not attributed to Kings – often because where institutions had merged with Kings, the researchers continued to use the name of the original institution, not KCL. Also confusion where staff work for both Kings and an NHS Trust, and where research was attributed.

Kings decided to have a College Citation Policy advising staff on the importance of citing KCL as the institution.

Methodology

Project setup for the RAE 2008. Eventually submitted 4897 outputs from 1589 staff, across 37 UoAs (units of assessment).

Survey of Schools, Depts etc. found publications stored in a wide variety of systems – they decided to capture electronically back to 2001 to provide initial database contents. Only two of the locally developed databases were mapped to the Research Gateway.

Methodology was to search databases with address/affiliation fields using variations of institutional name + postcodes – e.g.: Web of Science, ISI Proceedings, Medline, Cinahl, etc.

Much easier in Health schools that other departments – especially for book, book chapter and other non-journal article information. Lots of data extracted from Worldcat and other sources, imported into Reference Manager database, big de-duplication exercise.

Los of work to clean up data

Challenges/Problems

Interestingly Mary has identified as an problem that the UoAs were substituting selected publications up to the last minute – we had this problem as well, and in the library this meant we were checking the accuracy of records that weren’t submitted, and ever shifting target in terms of how many submissions that needed checking.

Plans for REF

• Develop the Research Gateway into a ‘Virtual Research Environment’, including integration with the Institutional Repository – attuned to requirements of REF (when these are clear)
• Continue to work with supplier concerning database indexing policies, where relevant
• Investigate research IDs and identifiers and institutional IDs developments
• Develop intern expertise in bibliometrics – sent staff on the Leiden course
• Relates to developing organisation culture of knowledge and comfort with bibliometrics
• Concerned about Source data issue and appropriate coverage
• Need to work on verification of citation information, especially if self citation excluded
• Need clarification of how open access forms of publication will be taken into account
• Capitalise on team work between academic and administrative staff
• Develop HE forum for sharing best practice

 

Case Study 2: University of Leicester

After RAE 2001, Leicester found many depts didn’t have good records, and were often trying to collect 6+ years of data in a few months. When they came to RAE 2008 decided to invest, in software etc. to collect data.

RAE 2008 submission was 2700+ outputs including 260 books, from c.800 staff

Leicester bought GENIUS from InfoEd to use as their database – which they called Research Expertise Database (RED) – similar to King’s Research Gateway. This has a web based interface, with users empowered to update records, and integration to the staff records from HR etc.

Journals selected from pick list to prevent mistypes and use of abbreviations. Allows you to indicate a publication should be used for RAE.

They then could export data from RED to that RAE software. The RAE software was not seen as usable by the administrators for the Units of Assessment – they just had to export from RED to Excel, do some ‘simple’ data manipulation, and import to RAE software.

I have to admit from the description of what the administrators had to do once the data was in Excel, it sounds quite complicated – lots of things to update. However, they say it worked, and that all the UoA administrators were able to follow the system OK.

Looking forward to REF, Leicester concerned about the amount of work involved in excluding self-citation. The Library is biding for a new post to cover both the management of the Institutional Repository, and to be a expert on bibliometrics.

Case Study 3: University of Southampton

Leslie Carr from Southampton presenting… (at last, some nice looking slides – nothing startling – but elegant and clean and a nice font!)

Leslie saying that ongoing research evaluation isn’t peculiar to HEFCE – essentially it is something that all those involved in research do – asking Why am I doing this research? Why are we funding you to do this research? etc.

The answer? Leslie suggests – because it’s important, it has impact, it is changing the world, it is saving lives, it is changing policy, it is saving money etc. – the point Leslie makes is that beyond the ‘measures’, there is a ‘story’ behind a piece of research.

At Southampton, they have used the research/institutional repository to generate data for the RAE.

• A repository helps an institution to collect an maintain its data, its impact measures and its story
• The library is a key agency in helping to collect and curate the research outputs, analyses and interpretation

(at this point Leslie’s mobile goes off!)

OK – back to the presentation

The RAE at Southampton made use of inter-service expertise – the management group drew from expertise across the University.

They used the repository to

• Store bibliographic detail of research outputs
• Store scanned PDFs (with appropriate levels of access to lock down RAE material)
• Store information to help write RA5 (don’t know what this is, probably should)
• Store RA2 (don’t know what this is either) descriptions for submission
• Output select and de-selection functionality
• Reporting functionality

The software they used was based on the JISC IRRA project, with additional development (1fte plus a 6 month secondment).

The repository collected the evidence, and handled as much Quality Assurance as possible, this evidence was then handed over to another system to add in finance data etc.

Again, had to bring together a number of existing databases in their ‘Schools’. Some were sophisticated – some were not… varying amounts of effort to import records and bring up to a reasonable standard of quality.

Core staffing was 1 repository editorial manager, 0.6 editorial assistant, 3 staff contributing a few hours a week, and at peak 7fte additional temp staff – this additional effort varied over 18 months.

They had issues with

• Recruiting and retaining staff with relevant experience for short term spells
• Staff training and supervision
• Dealing with high volume of legacy data (next time with ongoing use of the repository this should be less of an issue)
• Verification of publication status

There was a tension between the HEFCE preference for the DOI, and the remit of the Southampton respository storing PDFs of full-text. However, it was a good route to get researchers engaged with the repository, they got lots of feedback, and won a reputation for reliability of support. There were no RAE specifi
c posts – rather they were repository posts.

This involvement raised the profile for services like Library and Planning, giving them a voice in ongoing discussions about research management – broadly, not just REF focused.

Future issues:

• Liaise earlier to lobby HEFCE about any difficult requirements e.g. month for journal articles
• Some over-reliance on individual staff expertise to be addressed (certainly from the library perspective, we had this issue at Imperial as well)
• Further development of the research management and open access roles of the repository in the REF era
• Further development of the research management system to pull data together from core databases, provide analytical tools and re-presentation options