{"id":135,"date":"2008-04-30T17:42:06","date_gmt":"2008-05-01T00:42:06","guid":{"rendered":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/?p=135"},"modified":"2008-04-30T17:42:06","modified_gmt":"2008-05-01T00:42:06","slug":"an-australian-perspective-on-metrics-based-assessment-systems","status":"publish","type":"post","link":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/2008\/04\/an-australian-perspective-on-metrics-based-assessment-systems\/","title":{"rendered":"An Australian perspective on metrics-based assessment systems"},"content":{"rendered":"<p>This by Linda Butler, from the Research Evaluation and Policy Project in Australia.<\/p>\n<p>In Australia:<\/p>\n<ul>\n<li>1990+ Research Quantum (RQ) formual funding\n<li>1993: Refinement of RQ formula &#8211; Universities establish publications collections (which were audited several times &#8211; although early audits found error rates of up to 60%)\n<li>2004: Research Quality Framework (RQF) announced\n<li>2006: Details of RQF process start to emerge &#8211; with metrics taking a leading role\n<ul>\n<li>Most Universities refine their MIS\n<li>Many import Thomson data directly into their publications databases\n<li>Link publication databases to those covering staff, student and grants data (either via proprietary software or through &#8216;home grown&#8217; systems &#8211; that latter being rare, and seen as risky by Linda)<\/li>\n<\/ul>\n<li>2007:&nbsp; The RQF abandoned by new government, being replaced by new metrics based system &#8211; going in a very similar direction to the REF, and Linda believes they will be very similar<\/li>\n<\/ul>\n<p>The data that is available already in the system that might be used for metrics in Australia is:<\/p>\n<ul>\n<li>Research Income &#8211; amount and source (but done at very broad level of aggregation)\n<li>Publication counts &#8211; books, chapters, journal articles, conference papers\n<li>Formulas also use separately collected Research student data &#8211; number of students; completions<\/li>\n<\/ul>\n<p>Recommended RQF metrics were:<\/p>\n<ul>\n<li>Publication outlets ranked into tiers &#8211; journals, publishers, conferences, venues (for performing arts)\n<ul>\n<li>Journal ranking process has been ongoing for 6 months<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>I would have thought this really makes the likelihood of &#8216;game playing&#8217; very high &#8211; already we can see the dominance of certain publications in some areas, and this will surely just reinforce this?<\/p>\n<ul>\n<li>External income &#8211; differentiate between types of income\n<ul>\n<li>However, this was dropped as universities could not supply the data at an appropriate level of detail<\/li>\n<\/ul>\n<li>Citation data &#8211; 1. Citations per publication; 2. Distribution across percentiles\n<ul>\n<li>standard methodology for disciplines with &gt;50% coverage &#8211; include most science (inc. Maths) and Engineering asked to be included, although the coverage was closer to 40%<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Linda is saying that the methodology chose directly affects costs (to the universities) and the transparency of the process &#8211; the type of thing you need to look at is:<\/p>\n<ul>\n<li>Level of aggregation &#8211; Institution, or discipline, or group (in order of ascending difficulty)\n<li>Who a university can claim &#8211; on staff at census date (complex, time consuming), university appears on publication (straightforward)\n<li>Who a university can submit &#8211; all staff (low cost), a subset of &#8216;research active staff&#8217; (more complex and expensive\n<li>Complexity of measures &#8211; treatment of collaborative papers, self-citations, etc.<\/li>\n<\/ul>\n<p>What is the degree of risk that different methodologies will produce different outcomes?<\/p>\n<p>Linda is expressing her own opinion here &#8211; only some of this currently backed up with empirical data. Talking about a high level of aggregation only &#8211; University or Discipline level:<\/p>\n<ul>\n<li>Level of aggregation &#8211; not much impact\n<li>Who a university can claim &#8211; moderate impact &#8211; mostly not much impact, but perhaps an impact for a small number of universities. Generally you will gain some and lose some researchers\n<li>Who a University can submit &#8211; low to moderate (depending on funding algorithm)\n<li>Complexity of measures &#8211; low (this assessment has some empirical data behind it)<\/li>\n<\/ul>\n<p>Essentially Linda seems to be saying &#8211; you may as well chose the cheapest approach, as she doesn&#8217;t believe it will impact significantly.<\/p>\n<p>Linda draws the following lessons for the REF:<\/p>\n<ul>\n<li>The development of institutional information systems is complex and takes a long time\n<li>Choice between simplicity and complexity has huge implication for the costs of the exercise\n<li>Often those lobbying for a particular methodology have no real understanding of the cost implication of their preferred choice<\/li>\n<\/ul>\n<p>Those &#8216;at the coalface&#8217; need to communicate to others in the institution the impact of what they are asking for in terms of cost to the institution.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This by Linda Butler, from the Research Evaluation and Policy Project in Australia. In Australia: 1990+ Research Quantum (RQ) formual funding 1993: Refinement of RQ formula &#8211; Universities establish publications collections (which were audited several times &#8211; although early audits found error rates of up to 60%) 2004: Research Quality Framework (RQF) announced 2006: Details [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[12,13,14],"class_list":["post-135","post","type-post","status-publish","format-standard","hentry","tag-beyondtherae","tag-bibliometrics","tag-ref"],"_links":{"self":[{"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/posts\/135","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/comments?post=135"}],"version-history":[{"count":0,"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/posts\/135\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/media?parent=135"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/categories?post=135"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.meanboyfriend.com\/overdue_ideas\/wp-json\/wp\/v2\/tags?post=135"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}