Scientific method Essay Example for Free

Scientific method EssayPolicy and utilization come tos of look for funded by the scotch and Social interrogation Council A case case of the Future of bring in curriculum, approach and analysis Steven Wooding, Edward Nason, Lisa Klautzer, Jennifer Rubin, Stephen Hanney, Jonathan Grant Policy and work stirs of look funded by the Economic and Social look Council A case body of work of the Future of draw design, approach and analysis Steven Wooding, Edward Nason, Lisa Klautzer, Jennifer Rubin, Stephen Hanney, Jonathan Grant Prep bed for the Economic and Social explore Council. The interrogation described in this motif was prepargond for the Economic and Social look for Council. The RAND Corporation is a nonprofit question organization providing objective analysis and effective solutions that address the challenges lining the populace and private sectors around the world. RANDs realityations do not necessarily reflect the opinions of its study clients and spon sors. R is a registered trademark. Copyright 2007 RAND Corporation All rights reserved.No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or info storage and retrieval) with emerge permission in writing from RAND. Published 2007 by the RAND Corporation 1776 Main Street, P. O. lash 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 Westbrook Centre, Milton Road, Cambridge CB4 1YG, United Kingdom RAND universal resource locator http//www.rand. org/ RAND europium URL http//www. rand. org/randeurope.To assign RAND documents or to obtain additional teaching, contact Distribution Services Teleph champion (310) 451-7002 Fax (310) 451-6915 telecommunicate emailprotected org Preface This report, prepared for the Economic and Social Research Council (ESRC), learns the impact of the ESRCs Future of Work program on form _or_ system of government makers, professional practitioners and other stems outside academia.It to a fault explores the pertinency of the retribution good example, a conceptual manakin for investigate paygrade, to societal science. The Future of Work programme was an initiative that aimed to bring together leading look intoers in the United Kingdom in an investigation of the future prospects for paid and unpaid work. The scratch line phase of the programme started in October 1998, followed by a second phase in January 2001. The report is presented in two volumes. This volume presents the conclusions of the search and summarises the methods and results.The second volume includes a brief literature go over of the military rank of complaisant science and the influence of investigate on insurance an overview of the Future of Work programme exact analysis of a survey of Future of Work PIs (Principal Investigators) and four complete case study narratives of visualizes fro m the programme. The report will be of interest to the ESRC and constitution makers in the wider social science and constitution community who are interested in how social science informs indemnity and practice. It will overly be of interest to those development methods to esteem search.The research was led by RAND Europe in collaboration with the Health Economics Research Group (HERG). RAND Europe is an independent not-for-profit think tank and research organisation that serves the public interest by providing evidence for policy making and public debate. HERG, a Specialist Research Institute of Brunel University, has as one of its main research themes, methodo system of logical and empirical studies of the impact of research. This report has been accomplice check overed in accordance with RANDs quality assurance standards ( observe http//www. rand. org/about/standards/) and at that placefore may be represented as a RAND Europe intersection point.For more information abo ut RAND Europe or this document, please contact Steven Wooding Senior Policy analyst Tel +44 1223 273897 Email emailprotected org RAND Europe Westbrook Centre, Milton Road Cambridge. CB4 1YG, United Kingdom Jonathan Grant Deputy to the President Tel +44 1223 293 893 Email emailprotected org emailprotected org iii Contents Preface iii Overview of impact vii Executive summary ix Acknowledgments.. xiii CHAPTER 1 Introduction 1 CHAPTER 2 Methodology and project structure .. 3 2. 1 The analytical good example .3 2. 2 sign tasks .. 5 2. 2. 1 Brief review of social science impacts literature . 5 2. 2. 2 Review of FoW documentation 5 2. 2. 3 Key seed interviews. 5 2. 2. 4 Output of initial tasks 5 2.3 requital survey .. 6 2. 4 Interim report 7 2. 5 Case studies. 7 2. 6 Analysis workshop 9 CHAPTER 3 Results .11 3. 1 Key findings from the literature review and key informant interviews.. 11 3. 2 Summary of results from survey.. 13 3. 3 User interviews 26 3. 4 Case study summaries.. 2 7 3. 5 Case study A. 27 3.6 Case study B. 30 3. 7 Case study C 33 3. 8 Case study D 35 3. 9 Concluding comments 37 CHAPTER 4 Discussion.. 39 4.1 Impact of the FoW programme .. 39 4. 1. 1 The FoW programme has had significant wider impacts on policy and practice 39 v Policy and practice impacts of ESRC funded research RAND Europe 4. 2 4. 3 4. 4 4. 1. 2 What impact has the FoW programme had? 39 4. 1. 3 Why has the FoW programme had an impact? 40 4. 1. 4 What affects the impact of projects ..41 Applying the retribution Model to wider impacts of social science .. 42 4. 2. 1 The Payback fabric can be applied to social science .. 42 4. 2. 2 Generalisation of categories.. 42 4. 2. 3 General points about assessing impacts of social science . 43 Further research.. 45 Concluding comments.46 REFERENCES. 47 Reference list.. 49 vi Overview of impact The Future of Work (FoW) programme succeeded in speech together an interdisciplinary group of academics, bear upon constructive discussions an d providing ingress to policy makers to agree the wider impact of the research. The FoW programme had significant academic and wider impacts. in that respect was genuine output from the programme in the area of knowledge production. To experience this has included 11 books and 69 book chapters four journal special issues and over 100 peer reviewed articles, alongside over 200 conference presentations. Six of the researchers felt they had changed the style of their research field. In call of capacity development, involvement in the FoW programme was seen as a moderate or considerable ratifier to 20 academic promotions, and the most common benefit of the programme cited by researchers were the opportunities to meet other researchers and in providing fora for discussion.In our survey of Principal Investigators (PIs) they reported 50 policy impacts, across a domain of organisations including internal government, policy-making parties, employers and unions. Contributions to the policy debate included more than 60 working papers and official reports seminars for the DTI, piteous Pay Commission and Cabinet Office. More directly there were nine secondments, which placed the researchers in a policy environment, including a senior role in the DTI Women and Equality unit, where the researcher was able to influence strategy and policy decisions relating to equality.Further specific examples of policy impact were the chairmanship of the TUC Partnership Institute by a researcher, allowing him to impact on employer/union relationships the drafting of guidance notes on complying with employment legislation for the DTI by a research group direct input into the Work and Families Bill (2003), which introduced new legislation on maternity and fatherhood bring out and citation in a House of Lords judgement on pay and conditions, specifically looking at unfair dismissal.Outside the government sectors our case studies identified a number of impacts on employers, includi ng changes in workload policies and career structure effects on maternity and family friendly working practices in a large consulting organisation and the negotiation of union-employer partnership deals. vii Policy and practice impacts of ESRC funded research RAND Europe public exposure Almost half of the PIs felt the extensive networks of the political platform Director, and steering committee, had reard them with direct access to policy makers.These policy makers included those in the Work Foundation and a government agency. In addition, the FoW Media Fellow enhance the impact of the research on more distant policy makers. He achieved this by producing research summaries gear up in the context of latest policy and other research findings. Crucially, he worked to timescales suitable for policy makers quite a than those of researchers. rating methodology This evaluation utilize the Payback Framework as a conceptual structure and showed that the framework is useful for evalua ting the wider impacts of social science.We used a number of techniques to collect data for the study document review key informant interviews an on-line survey and case studies. However, we found that some impacts are inaccessible to evaluation, because of political sensitivity or anonymity guarantees. viii Executive summary We get windd how the ESRC Future of Work (FoW) programme influenced policy and professional practice. While doing so we reflected on the methods used to assess and identify impacts. Specifically, we considered whether the Payback Framework, a conceptual model for research evaluation, was appropriate for social science.Here we summarise the key findings. The FoW brought together an interdisciplinary group of academics, stimulated constructive discussion and provided access to policy makers. This may be specially significant devoted the view expressed by key informants that employment policy and oversight practices may be especially hard to reach with evidenc e as they are heavily contextual and apt to be influenced by fashion and ideology. Impacts The FoW programme had significant impacts on knowledge and research.This was evident in the numerous publications and conference presentations placed to the programme. near Principal Investigators (PIs) attributed incremental changes in their field of research to their projects, and some attributed a clear change of direction in their field of research to their projects. Most of the projects also influenced other researchers. The FoW programme had significant impacts on public policy. Although some PIs could identify specific impacts of their research, many found it difficult to identify actual policies they had influenced.PIs generally vox populi they had influenced policy in an incremental way and informed the policy debate. PIs also gave many presentations of FoW research to policy audiences. The FoW programme had significant impacts on career development. More than 75% of PIs idea the FoW programme had helped them to form networks with researchers, policy makers and practitioners n early(a) half of PIs attributed career development for researchers to their FoW projects, including nine secondments to government. The FoW programme impacted on the policies and practice of organisations.There were many presentations given in organisations PIs thought organisational practices were influenced by the research, but only some were easily identifiable. The policy environment determines policy impact. In one case the heightened awareness among policy makers of issues around maternity leave and women returning to work ix Policy and practice impacts of ESRC funded research RAND Europe provided fertile ground for research on how women make these decisions. In a second case the waning interest in union-employer partnerships was thought to bind reduced the impact of a TUC institute chaired by a designer FoW researcher.In general, the FoW research seldom caused major changes in policy but often resulted in impacts such as stimulating debate, fine-tuning policy, dispelling myths and providing confirmatory patronise. Dissemination The FoW programme provided access to policy makers. It effectively combined the networks of the Director and steering committee, and provided the researchers access to these networks which included key policy makers in the DTI, Low Pay Commission (LPC) and Cabinet Office. The FoW Media Fellow enhanced the impact on policy makers.This was achieved largely for two reasons. First, because his summaries of the FoW research were produced to a timescale suitable for policy makers, rather than researchers. And second because they were accessible to policy makers setting the FoW research in the context of other research and current policy discussions. Researchers and policy makers differed in their views on how best to disseminate to policy makers. The two groups consider contrastive channels to be important researchers favouring academ ic publications, policy makers favouring the Media Fellows publications.The Payback Framework is a useful model for evaluating social science research. The Payback Framework provides a structure for research evaluation. It comprises a logic model of the research and dissemination process and a classification scheme for the immediate and wider impacts of research. This consists of cinque categories Knowledge Impacts on future research Impacts on policy Impacts on practice and Wider social and stinting impacts. Both the literature review and fieldwork showed that the Framework could be effectively applied to social science research.Impacts and attributionSome impacts may be inaccessible to evaluation, for example some impacts were politically sensitive, so participants requested that they were not discussed. Also, subjects of the original research may have been influenced by their participation in that research, but their identity could not be revealed to the researchers in this eva luation. A confluence of inputs and incremental knowledge creep make it difficult to attribute policy change to a given input. The Payback Framework provides a structure in which to explore the context indoors which projects are developed.However, the incremental nature of policy remains a difficulty in assessing impact at the project level. There are few mechanisms in social science to systematise and synthesise research. In contrast to biomedical science, in the fields covered by the FoW programme there are fewer formal mechanisms to systematically review research these mechanisms can offer tracers of policy influence. x RAND Europe Executive summary Timing Research on impacts may happen too early or too late. If research on impacts occurs too early, some impacts may not yet have occurred.If it occurs too late, certain impacts may have already come and gone. This possibility of transience makes it harder to investigate the impacts, as they may not be captured by a current snapsh ot of policies and policy debates. In order to provide a comprehensive view of the wider impacts of research this project suggests it would be important to warn researchers at the start of the project about likely evaluations provide researchers with a mechanism to capture early impacts and thus evaluate research after further impacts have had time to develop, probably 5-10 years after culmination of the research.The literature suggests that for research relating to hot topics in policy, initial impact is likely to occur earlier and that 2 years post completion may provide the best time frame for evaluation. Implementation of evaluation There was widespread cooperation in the evaluation. The majority of PIs (including all case study PIs), 80% of projectd research users, and others nominated by PIs, agreed to participate in the research when approached. Researchers and users may prefer structured interviews to scripted surveys. Our experience also suggests that such interviews woul d provide more useful information for evaluation.Our on-line survey required significantly more of most participants time than predicted. xi Acknowledgments This study would have been impossible without the generous support of those involved with the Future of Work programme we thank them for their constructive criticism and for sparing their time for interviews and to action in our survey. We would particularly like to thank those researchers whose grants were selected as case studies. We would also like to thank veronica Littlewood of the Economic and Social Research Council for her help and advice, including at the analysis workshop.Finally, we would like to thank prof Martin Buxton and Stijn Hoorens who acted as the quality assurance reviewers. xiii CHAPTER 1 Introduction This report explores the wider impacts of social science research how research affects policy, practitioner behaviour and public opinion. It does this by examining the Economic and Social Research Councils ( ESRC) Future of Work (FoW) research programme. The ESRC is the UKs largest research funder and training agency addressing stinting and social concerns. As such it aims to provide high quality research on issues of importance to business, the public sector and government.The FoW programme set out to bring together leading UK researchers, across a wide range of disciplines, in order to investigate the future prospects for paid and unpaid work. The programme was shaped by a consultation exercise involving one hundred forty policy makers, academics and practitioners, carried out by Professor Peter Nolan in 1997. The first phase of the programme started in October 1998, followed by a second phase in January 2001. The total funding of the programme amounted to ? 4 one million million million and attracted 221 applications for the first phase, of which 19 were supported. A further eight projects were supported in the second phase.The aims of the programme are shown in Box 1. To crea te the evidence base that would and so ground theories of work To enhance public understanding of the critical developments most likely to impact on peoples working lives To sharpen accounts of the future of work by systematic mapping of past and present shifts and continuities To foster interdisciplinary and proportional perspectives To use innovative methods to engage with research users To act as a focus for debate within and between the academic, practitioner and policy-making communities Box 1. Aims of the FoW programme.Peter Nolan, Montague Burton Chair of Industrial Relations at Leeds, went on to direct the programme. He encouraged interaction between the research groups, promoted dissemination of programme findings and increase the programmes profile. The Director was assisted by a programme advisory committee of senior representatives from government, the Trades married couple Congress (TUC), business and academia. The advisory group attended meetings, participated in site visits to meet researchers, and provided access to policy networks. The written dissemination activities were led by Robert Taylor, a 1.Policy and practice impacts of ESRC funded research RAND Europe former journalist at the Financial Times, who was appointed as the programmes Media Fellow. He wrote a serial of seven booklets, aimed at policy makers, that described FoW research and set it in context. The output of the programme has been substantial its outputs so far include 11 books, 69 book chapters, over 100 refereed articles and over four hundred media mentions. This study explores the wider impacts of the programme in more detail. Over the past decade there has been an increasing floriculture of accountability affecting government spending.This climate has led ESRC to investigate the most effective ways to evaluate social science research, and to demonstrate the wider impact of its research on society. This report builds on experience of evaluating research in the welln ess services and biomedical settings and seeks to apply it to social science. In this work we take the Payback Framework, originally developed by the Health Economics Research Group (HERG) at Brunel University, and test its applicability to social science.The Payback Framework was initially developed to examine the payback of health services research (Buxton et al., 1994 Buxton and Hanney, 1994 Buxton and Hanney, 1996). It was further developed in an earlier ESRC analysis of non-academic impact from research (Cave and Hanney, 1996) and subsequently extended to examine basic and clinical biomedical research (Wooding et al. , 2005 Wooding et al. , 2004).This study tested whether the Framework could be applied to examine the payback of social science research. To do this we briefly reviewed the literature on social science evaluation and the common models for examining the impact of evidence on policy, and concluded that they could be aligned with the Payback Framework.We then used t he Payback Framework to examine the research projects in the FoW programme. We used the Payback Framework to structure a programme-wide questionnaire and a series of four case studies. Finally, we used these three streams of evidence to summarise the wider impacts of the FoW study and to see what can be learnt from the programme. We also used the evidence to develop a refined Payback Framework and consider its applicability for evaluating the wider impacts of social science research. 2 CHAPTER 2 Methodology and project structure.This project set out to examine the wider impacts of the FoW programme and to test the applicability of the Payback Framework to social science. It used a number of data collection methods (shown establishedally in Figure 1). First we conducted a brief review of the literature concerning the ways in which social science affects policy and how the impacts of social science can be assessed. To develop our understanding of the FoW programme we reviewed documen ts from the ESRC and interviewed key individuals.We then surveyed all the Principal Investigators (PIs) who held grants from the FoW programmeto investigate the wider impacts of their grants, and asked them to nominate a user of their research for follow up via a telephone interview. To examine the pathways to impact in more detail we carried out four case studies of FoW PIs. The data collected through and throughout the study were then analysed in a one-day workshop. More detail on each of these stages is provided in the sections that follow. Figure 1. Project schematic 2. 1 The analytical framework The analytical framework for the study was based on the Payback Framework.The Payback Framework consists of two elements a logic model representation of the complete research process (for the purposes of research evaluation), and a series of categories to classify the individual paybacks from research. The logic model and categories of the Payback Framework that served as the jump poin t for this study are presented below in Figure 2 and Box 2 respectively. The logic model provides a framework for analysing the story of a research idea from initial inception 3 Policy and practice impacts of ESRC funded research RAND Europe.(Stage 0) through the research process (Stage 2) into dissemination (Interface B) and on towards its impact on people and society (Stage 6). The model is meant as a research tool to facilitate cross-case analysis. It does this by providing a common structure for each case study thereby ensuring cognate information for each study is recorded in the same place. The model is not meant to imply that the research process itself is linear. If necessary, individual pieces of information can be recorded in more than one place in the Framework to ensure they are picked up in the relevant cross-case comparisons.Figure 2. The version of the Payback Framework used as a starting point for the study1 Knowledge production Journal articles conference presentati ons books book chapters research reports Research targeting and capacity mental synthesis Better targeting of future research development of research skills, personnel and overall research capacity staff development and educational benefits Informing policy and product development Improved information bases for political and executive decisions development of pharmaceutical products and therapeutic techniques Health and health sector benefits.Improved health cost reduction in speech communication of existing services qualitative improvements in the process of delivery improved equity in service delivery Broader economic benefits Wider economic benefits from commercial exploitation of innovations arising from RD economic benefits from a healthy men and reduction in working days lost Box 2. The payback categories of the Payback Framework used as a starting point for the study (Source Hanney et al. , 2004) The categories of the Payback Framework are considered in the Discussion sec tion of this report where the applicability of the model to social science research is discussed.1 Source Hanney et al. , 2004 4 RAND Europe Methodology and project structure 2. 2 2. 2. 1 Initial tasks Brief review of social science impacts literature A brief review of the literature, presented in mass II, examines frameworks of evaluation previously used to examine the impact of social science research as sanitary as models of research impact. The review was intended to identify lessons from the literature that would inform the current study, but was not intended to be comprehensive. 2. 2.2 Review of FoW documentation At the outset of the project we also reviewed the ESRC records coating the FoW programme. This informed our interview protocol for the key informant interviews and provided the basis of our overview of the FoW programme, which was used as background information throughout the study. This overview is presented in Volume II. 2. 2. 3 Key informant interviews To gain a deeper understanding of the overall context and impact of the FoW programme we carried out interviews with six key informants.Suitable key informants were identified by ESRC and by the Director of the FoW programme o o o o o o Professor Peter Nolan (FoW Programme Director) John Hougham (Chair of Advisory Board) Professor Toby Wall (Member of panel that appointed the Programme Director and Member of Advisory Panel) Bill Callaghan (Member of Research Priorities Board and Chair of Commissioning Panel) Professor William cook (PI on two grants, one in each phase of the programme) Robert Taylor (Programme Media Fellow). These interviews were written up and examined for themes relating to the wider impact of social science and the FoW programme.These themes were then clustered and used to inform the refinement of the Payback Framework. We also carried out a second follow-up interview with Peter Nolan late in the project to examine various issues that had been raised in the survey and cas e study phases of the project. 2. 2. 4 Output of initial tasks The findings of the initial tasks suggested that the logic model aspect of the Payback Framework was generally appropriate for the social sciences. However, the categories needed some generalisation our initial revision of these categories was presented in our interim report to the ESRC and are shown in Box 3.5 Policy and practice impacts of ESRC funded research RAND Europe Knowledge production Journal articles conference presentations books + chapters research reports Research targeting and capacity building Sparking new research proposals providing research training supporting career advancement Informing policy and product development Raising the profile/awareness of existing research among policy/practitioners makers dispelling/resisting myths providing policy options prioritising areas designing management assessment tools developing benchmarking protocols Employment sector benefits.Improved working conditions highe r participation in workforce more effective regulation Societal and broader economic benefits Lower stress among workers improved public health improved mental health through decreased unemployment greater productivity improved equity Box 3 Draft Payback Categories for the Social Sciences afer initial tasks. 2. 3 Payback survey To examine the range and types of payback produced across the FoW programme we invited all the PIs to complete an online survey. The survey concentrated on the wider impacts of the projects, but also asked some questions about the initiation of the research.The survey questions were based on those used in previous payback studies and modified in light of the key informant interviews and literature review2. PIs were invited to participate in the survey exploitation personalised emails which contained a direct hyperlink to their questionnaire. PIs who had grants in both phases of the FoW programme received two emails linking to two speciate surveys. The surve y was implemented using MMIC web questionnaire software. 3 Data were downloaded from MMIC and analysed using SPSS version 14 and Microsoft outperform version 2000. 4 The questionnaire was originally drafted on paper.The paper draft was reviewed by the ESRC and by both of the projects quality assurance reviewers. After incorporating their comments it was converted into a web questionnaire and again reviewed by the ESRC. We also asked a RAND researcher from outside the project team to test the questionnaire by talking us through their thoughts as they filled it in. This helped us to identify misunderstandings and confusing questions. 2 Payback questionnaires first used in Buxton et al. , 2000 and subsequently refined for payback analysis of the NHS Research Implementation Methods Programme and the Dutch and UK Health.