- 20
- 3 720
Research on Research Institute
Приєднався 16 жов 2019
The Research on Research Institute (RoRI) is an international consortium of research funders, academics and technologists working to champion transformative and translational research on research systems, cultures and decision-making.
We want to change the way research is done so that it works for everybody and solves the challenges we face.
We do this by turning the tools of research back on itself - applying an R&D mindset to the inner workings of the research system by gathering data, testing ideas and reiterating to make improvements. We translate ideas, theories and evidence into practical, real world solutions to improve research culture and systems. We bring together people and organisations that care about research and develop tools to inform and improve how research is funded, practised, communicated and evaluated.
Find out more about our projects and tools at www.researchonresearch.org
Follow us @RoRInstitute
We want to change the way research is done so that it works for everybody and solves the challenges we face.
We do this by turning the tools of research back on itself - applying an R&D mindset to the inner workings of the research system by gathering data, testing ideas and reiterating to make improvements. We translate ideas, theories and evidence into practical, real world solutions to improve research culture and systems. We bring together people and organisations that care about research and develop tools to inform and improve how research is funded, practised, communicated and evaluated.
Find out more about our projects and tools at www.researchonresearch.org
Follow us @RoRInstitute
Enhancing EDI in application processes: Evaluating an anonymisation-deanonymisation review process
The CES Transformation Fund is a grant scheme run by Rachel Kendal and team, on behalf of the Cultural Evolution Society (CES) thanks to the generosity of the John Templeton Foundation (Grant #61913: Transforming the field of cultural evolution and its application to global human futures).
The vision for the grant scheme was to boost, strengthen and extend scientific discoveries of cultural evolution. To achieve this the funding scheme explicitly aimed to encourage diversity through tackling (i) early career obstacles, (ii) western centrism, (iii) traditional disciplinary divides, and (iv) division of academics and policy makers.
These aims were embedded in the design of our application and review process which contained many measures to enhance equity diversity and inclusion (EDI) both in selecting who was funded and in the research conducted by our awardees.
This talk will provide an overview of the EDI measures we employed, including sequential anonymized-deanonymized review, a 2-stage application process incorporating mentoring, a diversity scoring system, an ‘ethics box’, and diverse review panels (see blog summary). The success of these measures in achieving our EDI objectives will be outlined as well as the practicalities of employing these measures. Survey responses from applicants and reviewers regarding the measures will also be discussed.
About the Speaker
Prof Rachel Kendal received her PhD in Zoology from the University of Cambridge (UK) and has been based in the Anthropology Department at Durham University (UK) for most of her academic career. She developed an interest in EDI in funding through roles involving running funding competitions as (i) Director of Postgraduate Admissions for her department, (ii) founder of the Research Working Party for the Primate Society of Great Britian, and (iii) President of the Cultural Evolution Society when she applied for funding to run the CES Transformation Fund with scientific rigour and equity at its core.
Recorded on 18 December 2024
This talk was part of our AFIRE Funders’ Forum Event series, a series of talks and discussions for funders interested in formal experiments and evaluations of innovations in the research and innovation funding system, part of RoRI’s AFIRE programme: an Accelerator For Innovation & Research Funding Experimentation.
The vision for the grant scheme was to boost, strengthen and extend scientific discoveries of cultural evolution. To achieve this the funding scheme explicitly aimed to encourage diversity through tackling (i) early career obstacles, (ii) western centrism, (iii) traditional disciplinary divides, and (iv) division of academics and policy makers.
These aims were embedded in the design of our application and review process which contained many measures to enhance equity diversity and inclusion (EDI) both in selecting who was funded and in the research conducted by our awardees.
This talk will provide an overview of the EDI measures we employed, including sequential anonymized-deanonymized review, a 2-stage application process incorporating mentoring, a diversity scoring system, an ‘ethics box’, and diverse review panels (see blog summary). The success of these measures in achieving our EDI objectives will be outlined as well as the practicalities of employing these measures. Survey responses from applicants and reviewers regarding the measures will also be discussed.
About the Speaker
Prof Rachel Kendal received her PhD in Zoology from the University of Cambridge (UK) and has been based in the Anthropology Department at Durham University (UK) for most of her academic career. She developed an interest in EDI in funding through roles involving running funding competitions as (i) Director of Postgraduate Admissions for her department, (ii) founder of the Research Working Party for the Primate Society of Great Britian, and (iii) President of the Cultural Evolution Society when she applied for funding to run the CES Transformation Fund with scientific rigour and equity at its core.
Recorded on 18 December 2024
This talk was part of our AFIRE Funders’ Forum Event series, a series of talks and discussions for funders interested in formal experiments and evaluations of innovations in the research and innovation funding system, part of RoRI’s AFIRE programme: an Accelerator For Innovation & Research Funding Experimentation.
Переглядів: 94
Відео
Predicting Progress: A Pilot of Expected Utility Forecasting in Funding | AFIRE Funders’ Forum
Переглядів 3128 днів тому
The conventional peer review process for reviewing grant proposals is known to be biased against riskier proposals. Expected utility - a formal quantitative measure of predicted success and impact - has been proposed by researchers Chiara Franzoni and Paula Stephan as a better metric for assessing the risk and reward profile of science proposals than traditional rubric evaluations. Inspired by ...
Utility of Large Language Models (LLMs) in identifying and assessing academic genres
Переглядів 10028 днів тому
We discuss how large language models (LLMs) can support assessment of various academic genres. INTRODUCTION Research assessment as intertextual reading: Opportunities and challenges in the use of Artificial intelligence in evaluation of SSH Special advisor Dr Jon Holm, Research Council of Norway SPEAKERS Evaluating social science, arts and humanities journal article quality with ChatGPT Profess...
Using Natural Language Processing in classification of SSH research and societal impact
Переглядів 12428 днів тому
We explore AI in research assessment by presenting two classification exercises using Natural Language Processing (NLP). MODERATOR Special advisor Dr Jon Holm, Research Council of Norway SPEAKERS Topic modeling of SSH publications from the VABB publication database Senior researcher Dr Raf Guns, Flemish Centre for Research & Development Monitoring (ECOOM), Antwerp University Classifying REF imp...
Villum Experiment: A Grant Scheme with Double-blind Evaluation | AFIRE Funders’ Forum
Переглядів 512 місяці тому
Villum Foundation is a philanthropic foundation that supports technical and scientific research and education as well as environmental, social, and cultural projects in Denmark and internationally. In 2023, Villum Foundation allocated EUR 153 million in grants. This talk provided an overview of the Villum Experiment grant programme, which targeted research proposals out of the ordinary that cha...
Metaresearch in Canada | Trends currently driving research culture
Переглядів 588 місяців тому
A panel discussion on metaresearch in Canada, with James Wilsdon (RoRI Executive Director); Kelly Cobey (Professor at University of Ottawa and Co-Chair of DORA); Karim Khan (Professor at University of British Columbia and Scientific Director of the CIHR Institute of Musculoskeletal Health and Arthritis); Vincent Larivière (Professor at University of Montreal and UNESCO Chair for Open Science); ...
Experimenting with research and innovation funding | AFIRE Project Launch
Переглядів 848 місяців тому
On 13 May 2024, 31 research funders from 17 countries participated in the launch of RoRI’s project AFIRE (Accelerator For Innovation & Research Funding Experimentation): researchonresearch.org/project/a-f-i-r-e/ Across research systems worldwide, interest in trialing novel methods of research funding and evaluation is growing fast, but funders are at different points in their engagement and rea...
Sir Peter Gluckman - Introductory Remarks - Making transdisciplinarity real: the role of funders
Переглядів 70Рік тому
Sir Peter Gluckman delivers his introductory remarks for the opening of the 'Making transdisciplinarity real: the role of funders' session at the Research on Research Institute's (RoRI) UNDISCIPLINED project workshop. The workshop was part of a two-day RoRI consortium meeting at the Volkswagen Foundation in Hannover, Germany on 4th and 5th December 2023. Find out more about the UNDISCIPLINED pr...
Can AI predict research impacts? A RoRI seminar with Amy Nelson & Parashkev Nachev, 15 June 2023
Переглядів 162Рік тому
The success or failure of medical research is judged by patient outcomes far downstream of the strategic decisions that initiate it. Optimising translational impact therefore relies on long range forecasting, for which no established framework exists. The evaluation of research proposals by expert appraisal of their content is undermined by difficulties with scaling, reproducibility, generalisa...
'Invert the order! Government's role in shaping a science superpower.' A RoRI talk by James Phillips
Переглядів 180Рік тому
What is required for the UK to stay at the cutting edge of science and technology and make harnessing its benefits our national purpose? And what role does the government have in that? In the first of a new Research on Research Institute (RoRI) seminar series on the theme of creative destruction, former special adviser on S&T to the UK Prime Minister, James Phillips, reflects on his experiences...
Machine learning, metrics and merit: the future of research assessment
Переглядів 5992 роки тому
The use of quantitative indicators and metrics in research assessment continues to generate a mix of enthusiasm, hostility and critique. To these we can now add growing interest in uses of machine learning and artificial intelligence (AI) to automate assessment processes, and reduce the bureaucracy of conventional methods of peer and panel-based review. Novel methods also bring potential pitfal...
'When priorities don't align with needs: the case of mental health research.' Ismael Rafols
Переглядів 1852 роки тому
The ways in which researchers approach, understand, and engage with mental health need to be broad, ranging from the biophysiological mechanisms underpinning brain function, to the societal determinants which alter it. In this second seminar in the Research on Research Institute's "Culture Shift" series, Ismael Rafols, senior researcher at Centre for Science and Technology studies (CWTS, Leiden...
'The Quantified Scholar' by Juan Pablo Pardo-Guerra. RoRI Culture Shift seminar, 16 June 2022
Переглядів 4242 роки тому
The new RoRI 'Culture Shift' seminar series aims to spotlight some of the most exciting thinkers, practitioners and social entrepreneurs who are at the forefront of analysing, pioneering and propelling culture shifts across science and research. We kicked off the series on 16 June with Juan Pablo Pardo-Guerra, associate professor of sociology at the University of California, San Diego and autho...
Introducing the Research on Research Institute
Переглядів 5453 роки тому
The Research on Research Institute (RoRI) is an international consortium of research funders, academics and technologists working to champion transformative and translational research on research systems, cultures and decision-making. We want to change the way research is done so that it works for everybody and solves the challenges we face. We do this by turning the tools of research back on i...
New Frontiers for Research on Research Session 2: PRIORITIES
Переглядів 1385 років тому
Session 2: PRIORITIES: from data to deliberation and decision-making. How can RoR support prioritisation & allocation by governments and funders? Chair: Anne-Marie Coriat, Head of UK & Europe Research Landscape, Wellcome Panorama: K. VijayRaghavan, Principal Science Adviser, Government of India (video link) Snapshots: Justin Parkhurst, Assistant Professor, Global Health Policy, London School of...
New Frontiers for Research on Research Session 1: FRONTIERS
Переглядів 4055 років тому
New Frontiers for Research on Research Session 1: FRONTIERS
New Frontiers for Research on Research Session 6: PARTNERSHIPS
Переглядів 785 років тому
New Frontiers for Research on Research Session 6: PARTNERSHIPS
New Frontiers for Research on Research Session 4: CULTURES
Переглядів 1505 років тому
New Frontiers for Research on Research Session 4: CULTURES
New Frontiers for Research on Research Session 5: DECISIONS
Переглядів 1065 років тому
New Frontiers for Research on Research Session 5: DECISIONS
New Frontiers for Research on Research Session 3: CAREERS
Переглядів 1375 років тому
New Frontiers for Research on Research Session 3: CAREERS
P r o m o s m
*promosm* 🌈
Great overview of exciting developments in this field. Have watched with great interest. Sorry I couldn't be there. Dorothy, thanks for being a voice of reason (as ever). I agree with every single word you said (from 1:19:06 onwards). Still don't understand why we are calling REF review "peer review" . 😏😏 @ Helen/Lizzie/David: interesting reflections on "do we need to evaluate at all"? Loved Lizzie's caution on incorporating the emotional costs of REF, which as an UoA-co-chair responsible for our output selection and research environment statement I have certainly felt. Then there is also the simple opportunity cost, I have probably spent at least 6 month full-time on the REF over the years and I could have done so much more *productive* work in creating positive research cultures in that time. Multiply that by the around 2-3K academics involved as UoA-chairs or co-chairs and the 1K academics involved in the assessment that makes about two thousand *years* of productive academic time lost that could have been used to provide *positive* leadership and promote developments such as Open Science. Ultimately, this is my biggest concern about the REF, it is such a poor use of the time and skills of some of our the UK's best academics AND research managers as I argued in my original white paper: harzing.com/publications/white-papers/running-the-ref-on-a-rainy-sunday-afternoon-do-metrics-match-peer-review#positive.