Evaluation of the MONA system (Microdata Online Access)

Transkript

1 Evaluation of the MONA system (Microdata Online Access) VETENSKAPSRÅDETS RAPPORTER 2014

2 EVALUATION OF THE MONA SYSTEM (MICRODATA ONLINE ACCESS) SWEDISH RESEARCH COUNCIL VETENSKAPSRÅDET Box 1035 SE Stockholm, SWEDEN Swedish Research Council ISBN DIARIENUMMER

3 Evaluation of the MONA system (Microdata Online Access)

4 PREFACE The Swedish Research Council (VR) is an authority within the Ministry of Education and Research. VR has a leading role in developing Swedish research of the highest scientific quality, thereby contributing to the development of society. Besides research funding, the agency advises the government on research-related issues and actively participates in the discussions to create an understanding of the long-term benefits of research. The mandate of the Council for Research Infrastructure (RFI), within VR, is to ensure that research infrastructures of the highest quality are being set up and used. One of the tasks for RFI is to facilitate conditions and support infrastructures for register-based research. The MONA (Microdata Online Access) system is Statistics Sweden s standard tool for the release of microdata. VR contributes via RFI to the financing of MONA in order to provide researchers in Sweden with register data more efficiently and at lower costs. In 2013, VR received a government commission to improve the accessibility and facilitate the use of register data for research purposes. One part of this task is to explore the possibilities to improve the coordination of register data for research purposes between data owners. The objective of this evaluation is twofold: 1) To assess the value of the VR investment in MONA including its functionality; and 2) To assess possibilities to coordinate register data for research purposes between different data owners. Focus has been on Statistics Sweden and the National Board of Health and Welfare. This report constitutes an independent assessment, with conclusions and recommendations from a Nordic expert panel. The report will serve as a basis for further discussions regarding future directions for how to improve the accessibility to and facilitate the use of register data for research purposes. On behalf of VR and RFI, I would like to express my sincere gratitude to the members of the panel for devoting their time and expertise to this important task. VR would also like to thank the representatives from Statistics Sweden and the National Board of Health and Welfare for providing the necessary background material for the evaluation and for fruitful discussions at the hearing. Thanks also to the researchers who expressed their valuable opinions in user surveys and at the hearing with the panel. Stockholm November 2014 Juni Palmgren Secretary General Council for Research Infrastructures Swedish Research Council EVALUATION OF THE MONA SYSTEM 3

5 INNEHÅLL PREFACE... 3 SAMMANFATTNING... 5 SUMMARY INTRODUCTION Background The evaluation process EVALUATION OF THE MONA SYSTEM Data availability and documentation Data security User satisfaction Costs Formal agreements between the Swedish Research Council and Statistics Sweden Panel s conclusions and recommendations regarding MONA, from a researcher s perspective COMPARISON BETWEEN STATISTICS SWEDEN AND THE NATIONAL BOARD OF HEALTH AND WELFARE Data availability and documentation Data security User satisfaction Costs The panel s conclusions in light of the comparison between Statistics Sweden and the National Board of Health and Welfare A FUTURE COORDINATED SYSTEM OF REMOTE ACCESS TO USE REGISTER DATA FOR RESEARCH PURPOSES? The suitability of a remote access system for use of registers at the National Board of Health and Welfare views of the National Board of Health and Welfare and Statistics Sweden Panel s conclusions and recommendations regarding a future coordinated system APPENDIX 1 SHORT PRESENTATION OF PANEL MEMBERS APPENDIX 2 TERMS OF REFERENCES APPENDIX 3 SCHEDULE AND PARTICIPANTS AT HEARINGS APPENDIX 4 INSTRUCTIONS FOR SELF-EVALUATION STATISTICS SWEDEN APPENDIX 5 INSTRUCTIONS FOR SELF-EVALUATION THE NATIONAL BOARD OF HEALTH AND WELFARE EVALUATION OF THE MONA SYSTEM 4

6 SAMMANFATTNING MONA-systemet (Microdata Online Access) är Statistiska Centralbyråns (SCB) standardverktyg för att tillgängliggöra mikrodata för forsknings- och statistikändamål. MONA etablerades Sedan 2006 har Vetenskapsrådet (VR) täckt en del av utvecklingskostnaderna och forskarnas kostnader för att använda MONA, i syfte att förse forskare i Sverige med väldokumenterade och kvalitetssäkrade registerdata på ett effektivt sätt och till lägre kostnader fick VR i uppdrag av regeringen att förbättra tillgängligheten till och underlätta användningen av registerdata för forskningsändamål. Som ett led i detta undersöks möjligheterna till förbättrad samordning av registerdata för forskningsändamål genom ökad tillgänglighet, i ett första steg data från SCB och Socialstyrelsen. Syftet med föreliggande utvärderingen är tvåfaldigt: dels att utvärdera värdet av Vetenskapsrådets investering i MONA-systemet hittills och dels att bedöma möjligheterna till samordning mellan SCB:s och Socialstyrelsens registerdata för forskningsändamål. Utvärderingen genomfördes av en nordisk expertpanel. Panelen baserade sin bedömning på självvärderingar från SCB och Socialstyrelsen, användarundersökningar samt utfrågningar av representanter från SCB och Socialstyrelsen samt samtal med forskare som använder registerdata. Panelens huvudsakliga slutsatser om det befintliga MONA-systemet är att det är ett säkert sätt att tillhandahålla registerdata på och att det också kan öka tillgängligheten av data till vissa forskare. VR:s stöd till MONA minskar forskarnas avgifter för att komma åt data och ökar därmed tillgängligheten. Men panelen identifierar också ett antal betydande problem med MONA, så som bristande kapacitet på SCB vad gäller bearbetning och lagring av data, begränsad tillgång till mjukvara, ingen möjlighet för forskare att använda sina egna servrar, svårigheter att använda MONA från andra länder och begränsade möjligheter för studenter att använda systemet. Därutöver är väntetiden från ansökan till att forskare får tillgång till data från SCB ofta mycket lång. Panelen rekommenderar Vetenskapsrådet att: Fortsätta stödja MONA systemet vid SCB, men att i kommande överenskommelser ta upp de ovan nämnda problemen och finna vägar för att överbrygga dem. MONA-systemet måste uppgraderas och förbättras för att möta forskarnas krav och behov. Uppmuntra SCB att fortsatt prioritera arbetet med att förbättra dokumentation, inklusive metadata. Uppmuntra SCB att etablera en särskild enhet för forskarservice. Panelens huvudsakliga slutsatser om ett framtida samordnat system är att fjärråtkomstsystem har många fördelar jämfört med att leverera data via fysiska lagringsmedia, så som CD-skivor, inte minst från säkerhetssynpunkt. Lika viktigt är att ett fjärråtkomstsystem kan öka tillgängligheten för många forskare, uppmuntra till användning av data och serva forskare på ett effektivt och säkert sätt. Ett fjärråtkomstsystem innebär högre kostnader för den som tillhandahåller data, men å andra sidan kan kostnaderna troligtvis reduceras i forskningsmiljöerna. De legala problem som Socialstyrelsen uppfattar sig ha med att överföra data till SCB måste lösas. Detta kan göras antingen genom en överenskommelse mellan myndigheterna eller, om nödvändigt, genom en lagändring. Panelen rekommenderar: Att det redan etablerade MONA-systemet används för att skapa tillgång även till Socialstyrelsens data, snarare än att etablera ett nytt fjärråtkomstsystem vid Socialstyrelsen. Att MONA systemet ges en mer oberoende status som en nationell infrastruktur för svensk forskning, enligt en modell med ett myndighetsgemensamt ägande av infrastrukturen och med SCB som värdorganisation. Att MONA-systemet uppgraderas och dess kapacitet förbättras, oavsett framtida organisation och uppgifter. Panelen föreslår att VR stödjer detta finansiellt. EVALUATION OF THE MONA SYSTEM 5

7 Att VR träffar en överenskommelse med Socialstyrelsen om att förbättra dokumentation och metadata i syfte att underlätta för forskare att använda dess register. VR bör bidra till finansieringen av detta. Att VR initierar och finansierar ett pilotprojekt, som drivs av Socialstyrelsen och SCB och där data från Socialstyrelsen överförs till SCB och, efter matchning med t.ex. SCB-data, görs tillgängliga för forskare genom MONA. EVALUATION OF THE MONA SYSTEM 6

8 SUMMARY The MONA (Microdata Online Access) system is Statistics Sweden s (SCB) standard tool to make microdata available for research and statistical purposes. MONA was put into operation in Since 2006, parts of MONA development and all access costs for researchers have been covered by the Swedish Research Council (VR) in order to provide researchers in Sweden with well documented and quality assured register data efficiently and at lower costs. In 2013 VR got the government s commission to improve the accessibility to and facilitate the use of register data for research purposes. One part of this task is to explore the possibilities of improving the coordination of register data for research purposes to increase accessibility, as a first step between SCB and the National Board of Health and Welfare (NBHW). The objective of the current evaluation is twofold: to assess the value of VR s investments so far in the MONA system and to assess the possibilities of coordination of register data for research purposes between SCB and NBHW. A Nordic expert panel has been appointed to perform the evaluation. The panel has based its assessment upon self-evaluations by SCB and NBHW, user surveys and hearings with representatives from SCB and NBHW, and with researchers using register data. The panel s main conclusions regarding the existing MONA system is that it is a secure way to provide register data and that it can also increase availability to data for many researchers. The VR s support to MONA reduces the fees for data paid by the researchers, thereby increasing the availability. However, the panel identifies a number of considerable problems, such as lack of capacity for processing and storage, limited software availability, no possibility for researchers to use their own servers, difficulties in using MONA from abroad and restrictions on its use by students. In addition, the waiting time for researchers between leaving an application and getting access to data from SCB is often very long. The panel recommends that VR: Continue to support the MONA system at SCB, but in the coming agreements raise the above-mentioned problems and find ways to overcome these. The MONA system must be upgraded and improved to meet the demands and needs of researchers. Encourage SCB to give high priority to continuing the process to improve documentation, including metadata. Encourage SCB to establish a special unit for research services. The panel s main conclusions regarding a future coordinated system is that remote access systems have many advantages in comparison with data delivery in physical storage media, such as CD, not least from a security perspective. Equally important, a remote access system can improve accessibility for many researchers, encourage use of data, and in many cases serve researchers in a more efficient and safe way. A remote access system means higher costs for the data provider, but on the other hand costs can probably be reduced in the research environments. The legal problems, perceived by NBHW in transferring data from NBHW to SCB, must be solved. This could be done either through an agreement between the agencies or, if necessary, by changing the law. The panel recommends: That the already established MONA system is used to provide access to NBHW s data, rather than establishing a new remote access system at NBHW. That the MONA system is given a more independent status as a national infrastructure for Swedish research. This could be done by an organisational model with joint ownership of the infrastructure while it is hosted by SCB. EVALUATION OF THE MONA SYSTEM 7

9 That, irrespective of future organisation and tasks, the MONA system is upgraded and the capacity improved. The panel suggests that VR should support that financially. That VR makes an agreement with NBHW to improve documentation and metadata in order to facilitate the use of registers for research. VR should contribute to the funding. That VR initiates and funds a pilot project, run by NBHW and SCB, where data from NBHW are transferred to SCB and, after matching with e.g. SCB data, are made accessible to researchers through MONA. EVALUATION OF THE MONA SYSTEM 8

10 1. INTRODUCTION 1.1 Background The MONA (Microdata Online Access) system is Statistics Sweden s (SCB) standard tool to make microdata available for research and statistical purposes. MONA was taken into operation in 2004, and provides secure access to microdata at SCB from an internet connection. In 2006, the Swedish Research Council (VR) and SCB concluded an agreement in order to provide researchers in Sweden with well documented and quality assured register data efficiently and at lower costs. Since then, parts of MONA development, and all access costs for researchers, have been covered by VR, with an annual contribution of about MSEK 9. An evaluation of the MONA system has become important in relation to a recent commission given to VR by the Swedish government. Since 2013 it has been stated in the appropriation direction for VR that VR should improve the accessibility to, and facilitate the use of, register data for research purposes. VR should also assist researchers with information about registers and relevant legal regulations. To implement this government commission a project Infrastructures for register-based research has been initiated. One part of the project is to explore the possibilities of improving the coordination of register data for research to increase accessibility, as a first step between SCB and the National Board of Health and Welfare (NBHW). The result of this evaluation will serve as input and guidance into that project. 1.2 The evaluation process The objective of the evaluation is twofold. The first objective of the evaluation is to assess the value of VR s investments so far in the MONA system, and to find out if MONA is a suitable way to finance register-based research. The other objective is to lay a ground for considerations regarding an expansion of the existing MONA system, and for the possible future introduction of a remote access system at NBHW. The evaluation panel has been asked to consider both the research and the authority perspectives. The terms of reference for the evaluation have been set by the Council for Research Infrastructure at VR (see Appendix 2). In the longer perspective the evaluation should contribute to strengthening the conditions for high-qualitative register-based research in Sweden. A Nordic expert panel has been appointed for the evaluation, comprised by Bengt Westerberg (Sweden, panel chair), Päivi Hämäläinen (Finland), Vigdis Namtvedt Kvalheim (Norway), Elsebeth Lynge (Denmark) and Antti Pursula (Finland). In Appendix 1, the panel members are presented. The expert panel met in Stockholm on September, 2014, for hearings with representatives of SCB and NBHW as well as with three different groups of researchers (five persons in each group): a) MONA users b) users of SCB register data but not through MONA, and c) users of NBHW s register data. The schedule of the meeting and the participants of the hearings are presented in Appendix 3. The panel also had a preparatory telephone meeting on 16 September, The panel has done its evaluation on the basis of the hearings and six documents: A self-evaluation made by SCB, including annual reports from MONA for the years 2011, 2012 and 2013, and a survey made in 2012 with MONA users (See Appendix 4 for Instructions for the self-evaluation) A self-evaluation made by NBHW (See Appendix 5 for Instructions for the self-evaluation) Written comments from SCB upon the self-evaluation of NBHW Written comments from NBHW upon the self-evaluation of SCB Results from a web-based user survey directed to 662 current MONA users (194 individuals responded). The survey was performed by SCB and designed in collaboration between SCB and VR. Results from a qualitative client survey with users of NBHW s register data (interviews with 27 current users). The survey was performed by Govorno on behalf of NBHW and the interview-guide was designed in a collaboration between NBHW and VR. In addition, the panel has had access to the Framework Agreement for and the Production Agreement for 2013, between VR and SCB. EVALUATION OF THE MONA SYSTEM 9

11 The panel s assessment, conclusions, and recommendations are presented in this report (chapter 2 to 4). The evaluation has been planned and coordinated by Maud Quist (project manager, VR) and Sten Söderberg (VR). Juni Palmgren (Secretary General for Council for Research Infrastructures, VR), Maria Nilsson (Project Manager, Infrastructures for register-based research, VR) and Susanna Bylin (VR) have constituted a reference group. EVALUATION OF THE MONA SYSTEM 10

12 2. EVALUATION OF THE MONA SYSTEM 2.1 Data availability and documentation Individual data collected for administrative or statistical purposes are often protected by secrecy rules. Such data can, however, under certain conditions be used for research. Before the remote access system MONA (Microdata Online Access) was introduced, data from Statistics Sweden (SCB) were delivered on physical storage media. However, SCB claims that they experienced that data provided to researchers were used for other purposes or by other users than those authorised. The MONA system was developed to stop these kinds of unauthorised use. It was put into operation in Register data that SCB is directly responsible for, or produce on behalf of other statistical agencies, may be delivered through the MONA platform. Most of these registers are stored in the MONA warehouse. The data in the warehouse can be used in different projects, which means that they do not have to be transferred to MONA for each project. At least one year s version of the registers in the data warehouse has also been documented in a consistent manner. Data produced by other agencies can also be accessed through MONA. In such cases the other agencies send copies of their data files to SCB, who will match data from different registers and make them accessible in deidentified form to the researcher. SCB has offered other agencies, which produce their own official statistics, to store data in the MONA warehouse, but without success. With the MONA system users are given access to data from practically any place in Sweden that has internet access. Data are processed and analysed through a set of applications, and the outcome of the analysis is automatically sent to the users via predefined addresses. Providing data to researchers is a fee-funded activity for SCB. The fees for such activities are expected to cover the total costs. Since 2006, thanks to funding from the Swedish Research Council (VR), all the costs for researchers use of MONA have been covered. In 2007 the Director General of SCB decided that the MONA platform shall be the standard tool for the release of microdata. However, not all data deliveries by SCB are made through MONA. SCB also provides data on other media, for example when a researcher wants to match data from SCB with data from the National Board of Health and Welfare (NBHW). A condition for statistics to be accessible and useful is adequate documentation. For all official statistics, a description of the statistics is presented on SCB s website together with the statistical results. Documentation of statistical registers in the form of metadata, and a detailed description of how the statistics have been produced, should also be available. According to SCB, at the end of 2013 this was true for 92 per cent of the surveys/registers. Efforts are currently made to document the remaining surveys/registers. The site has been developed partially with funding of VR. Since 2013 measures have also been taken to improve the quality of documentation. For a number of standardised variables, the documentation must now be made in a similar way. Several new controls have been added to the scrutiny carried out before the documentation of a register is published. SCB has emphasised that, according to its instruction, it is not among its main tasks to carry out assignments to researchers. This might affect waiting times and priorities. A commission set up by the Swedish government (Statistikutredningen, SOU 2012:83) has proposed that service to researchers should be added as a prioritised task for SCB. According to the Ministry of Finance such an amendment will be made as from 1 January EVALUATION OF THE MONA SYSTEM 11

13 The time it will take from request to delivery varies. The more complex the request is, the longer it will take. The process will also be delayed when the request is incomplete. According to SCB, the current average time between when the request is received by SCB and the last invoice is sent is about five months. 2.2 Data security As mentioned, data collected for administrative and statistical purposes can be provided for research. A request for disclosure of data for research purposes must be written and contain a detailed description of the project, and in some cases a supportive judgment of an Ethical Review Board is required. All requests will undergo a harm test by SCB, including those that have been approved by an Ethical Review Board. The harm test is carried out in accordance with chapter 24, 8 Public Access to Information and Secrecy Act (offentlighets- och sekretesslagen). In order to be able to disclose data, the authority must make sure that individuals, or someone close to them, will not suffer injury or harm. Disclosure from SCB for research purposes will normally occur according to the secrecy policy adopted by the agency. It means that the data provided will usually be de-identified. According to the Act on the Official Statistics (lagen om den officiella statistiken), SCB can however, on a researcher s request, save a code key that makes it possible to later update or supplement the data. Longitudinal studies are considered as having a particular need for this. If a release of data is made to a state university or government authority, the statistical secrecy is passed on to it. If the receiver is a private organisation SCB will make a secrecy agreement with it, including restrictions that are illegal to break. The Swedish legislation cannot protect data that has been released to another country. Therefore the Personal Data Act (personuppgiftslagen) forbids releasing individual data to a country outside the EU or EEA, if the country does not have an adequate level of protection for the data. In the MONA system external users can get access to microdata via the internet. This requires security measures that guarantee that data are not to be used by unauthorised persons. According to SCB a number of tests and evaluations of the security of the MONA platform have been performed over the years by a wellknown security company, and the overall judgments have been that the security is high. Even if the security of the MONA platform has been judged to be safe, the threats to the internet are constantly changing and becoming more sophisticated. Therefore, the safety of the MONA platform is regularly evaluated and tested. Any researcher who needs access to MONA must sign a user certificate where he or she commits not to copy microdata from the MONA system to his or her own computer, not to use MONA from another place than specified in the decision, not to reveal his or her username or password (login information) to anyone, and to immediately report any misuse of login credentials to the MONA support at SCB. MONA is a closed system where the user cannot insert his or her own data or software without the help of SCB. 2.3 User satisfaction In 2009 an evaluation panel commissioned by VR to evaluate the Database Infrastructure Committee (DISC) also assessed the MONA system. 1 The panel found, from interviews with and a special survey to MONA users, that the services provided by the SCB staff to researchers had worked reasonably well. 1 Final report from the first evaluation of the Database Infrastructure Committee (DISC) Dnr EVALUATION OF THE MONA SYSTEM 12

14 Based on remarks made by that panel a reference group with users has been set up by SCB, and the agreement between SCB and VR has been adjusted with requirements for annual reports. Through SCB has noticed an increase in requests of microdata releases to researchers and for statistical purposes. In mid-2014 approximately 1,200 people were registered as MONA users, of which about 700 were researchers. The University of Stockholm was the university, and the Employment Service (Arbetsförmedlingen) the government agency, with the most users. From year to year the number of users increases. In 2012 SCB carried out a user survey. It showed that a number of users found it hard to get started, quite frequently experienced performance problems, lacked some desired software, or found the provided documentation insufficient. However, almost all of the users said that they were satisfied with the support provided by the MONA support. Another web-based user survey directed to 662 researchers using MONA has been carried out in June 2014 on behalf of VR and SCB. 192 individuals responded; those with more experience from MONA were overrepresented among them. The main results were: 41 per cent of the respondents claimed that their ability to analyse data increases with MONA while 23 per cent claimed the opposite. 55 per cent claimed that they would prefer getting data delivered directly in data files for further processing, while 29 per cent preferred MONA. 35 per cent said that it took longer than expected to get data via MONA while 5 per cent found that it was faster. More than 40 per cent claimed that they had had to wait longer, at least occasionally, than expected for their jobs to be finished. About one third found the documentation sufficient and one third not sufficient. As in the 2012 survey a large majority thought that MONA support helps them in a timely fashion. The panel has met with three groups of researchers who regularly use data from SCB: one group with people who get data delivered through the MONA system, one with people who get them through other media and one with people who use data from NBHW. The group of MONA users claimed four advantages with MONA: Data security is high. Data security is taken care of by MONA, which means that the researchers themselves and their research institutions can limit their security arrangements. The data are available from different places. The data provided are normally kept up-to-date. However, the participants also experienced the following problems: The waiting times to get requested data available in MONA is often very long, not less than six months and often 1-2 years (thus perceived as longer than SCB claims in its self-evaluation). The software that can be used is limited. There are often capacity problems of different kinds: the system runs out of memory or other resources, processing capacity is often scarce, not least for big data, and service is limited to regular working hours (Monday-Friday, 8-17). Researchers are not allowed to use MONA from abroad without special permission. Sometimes the administrators of SCB show little understanding about research complexity and time frames. The time limits for access to stored datasets in MONA are often too short for research tasks. It is unclear how and if research data stored in MONA are archived and under which conditions they are available for future research. MONA cannot be accessed from LINUX and Mac operating systems. EVALUATION OF THE MONA SYSTEM 13

15 Students cannot get access to MONA. Lacks in documentation (metadata and variable descriptions). These opinions correspond quite well with those appearing in the survey. According to the researchers of the group using SCB data but not MONA, there are two main reasons, besides matching SCB data with data from NBHW, to allow researchers to get access to data through other media other than MONA. One reason is that the software used for analysis is not available or cannot be used in MONA. The other is that researchers are sometimes allowed to build and update large databases outside MONA, for example for longitudinal studies. The possibility for some research groups to get SCB data delivered directly for building their own databases and research infrastructures are perceived by other researchers who get access to these data only through MONA as unequal working conditions (VIP access). 2.4 Costs The costs of the development of the MONA platform in were entirely financed by SCB. Since the start of the MONA system in 2004 the operational costs per project and user have decreased, especially during the first years. In 2013 the costs amounted to about SEK 40,000 per project and SEK 10,000 per user. The total number of research projects using data through MONA amounted in the end of 2013 to 234, 45 of which were new for that year. MONA activities include costs for staff at the MONA support, IT managers who manage and develop the platform, and costs for hardware and software. SCB has estimated that the number of hours spent by SCB staff on the MONA platform in 2013 was about 7,600 (corresponding to 4-5 full-time employees). The total fixed costs for MONA funded by VR in 2013 amounted to MSEK 9.6 (about 70 per cent of the total fixed costs for the system). SCB estimates that the costs paid directly by researchers to get access to microdata through MONA amounted to about MSEK 4. This indicates that VR is funding about two thirds of the total costs for register microdata made accessible to researchers through MONA. It must be underscored, however, that the figures are very approximate and that it is not possible, from the accounts of SCB, to find out the exact figures. 2.5 Formal agreements between the Swedish Research Council and Statistics Sweden The Framework and Production Agreements between VR and SCB are not very detailed. According to the Framework Agreement the aim is to provide researchers in Sweden with more efficient and affordable access to well-documented and quality assured register data via the MONA platform. The cooperation shall include participation in the development of an agency-wide system of register research governed by a separate agreement. According to the Production Agreement for 2013 SCB shall guarantee access to the data warehouse that has been built up for register research on behalf of VR. The data warehouse must include documentation regarding at least one annual volume per register. The aims stated in the agreements are to some extent fulfilled, but as mentioned above there are some complaints about efficiency. No separate agreement of an agency-wide system has been initiated. EVALUATION OF THE MONA SYSTEM 14

16 2.6 Panel s conclusions and recommendations regarding MONA, from a researcher s perspective MONA was established for security reasons, not to improve availability for researchers. SCB claims that they had experienced unauthorised use of data when these were delivered in physical storage media to researchers. The panel s assessment is that SCB had identified a potential threat that motivated measures to improve the security. The panel assesses MONA to provide a safe environment. This will hopefully contribute to public trust, which is important to get public support for register research. MONA is, however, only one way to provide data from SCB registers. Traditionally data were delivered in physical storage media and this technique is still used by SCB. The Director General of SCB has decided that in the future delivery through MONA shall be the preferred option, regardless of what individual researchers might wish. The number of deliveries through MONA is also increasing. The panel supports this priority. Although MONA was established for security reasons, the panel judges that it can also be used to increase availability to data for many researchers. The panel s opinion is that the system is valuable not least for researchers in environments too small to safeguard data security, which often requires employed experts and investments in advanced technical systems. Furthermore, some of the researchers the panel has met with assess that MONA can also be economically competitive for larger research environments. The panel agrees that the MONA system can probably save data management resources in the environments of many research institutions. However, the panel has identified a number of considerable problems for researchers with the MONA system, such as a lack of capacity for processing and storage, limited software availability, no possibility to use own servers, difficulties in using MONA from abroad in EU and EEA, and restrictions for its use by students. In addition, the waiting time from leaving an application to getting access to data from SCB is very long. Therefore the panel s opinion is that the MONA system, that was created more than ten years ago, must be upgraded and improved to meet the demands and needs of researchers. The panel has learnt from the Ministry of Finance that the instruction for SCB will be changed as from 1 January 2015, meaning that research will be a new prioritised task for the agency. In NBHW, and in for example Statistics Denmark, special units for research services have already been established. The panel urges VR to encourage SCB to do that as well, in order to reduce the time needed for internal administration in the process to grant access to data. SCB has already started a process to improve documentation including metadata. The panel urges VR to encourage SCB to make it a high priority to continue this work. Since 2006 VR has contributed to the financing of MONA. The funding has speeded up the development of the system but in particular it has reduced the fees for data paid by the researchers. As VR pays about two thirds of the total costs of research projects it is reasonable to believe that the price per project would roughly triple without that contribution. With high probability this would reduce the number of projects using register data even if these projects were partly compensated through higher contributions from VR. The projects that could not afford data access without VR financing of MONA would likely be particularly from smaller institutions and research groups, and the panel finds that this might limit scientific innovation. The aim in the present agreement between VR and SCB seems to be fulfilled to some extent. The panel urges VR to continue supporting the MONA system. However, it also urges VR to raise the above-mentioned issues in the future agreements between VR and SCB and thereby include an upgrading of the system, including reduced time to get access to data. It is the panel s opinion that many of the research groups currently working on SCB data outside the MONA system could have their research needs met within the MONA system if the above-mentioned bottleneck problems were solved. EVALUATION OF THE MONA SYSTEM 15

17 3. COMPARISON BETWEEN STATISTICS SWEDEN AND THE NATIONAL BOARD OF HEALTH AND WELFARE 3.1 Data availability and documentation The legal conditions for providing data from NBHW are the same as for SCB. NBHW has set up a special Register Service Unit. Among its tasks are to support register researchers and contribute to higher quality in the national quality registers (held by County and Regional Councils). The aim with the unit is to make it easier for external parties to order data by gathering all external orders in a single unit. Each year the unit processes orders for individual-based data for research purposes. Contrary to the MONA system, in NBHW its data to researchers are regularly delivered on a physical storage medium, generally a CD that is sent to the researcher by registered postal mail. The CD is encrypted and the encryption password is sent to the researcher separately. After that the data is copied from the CD to the researcher s own data environment. It is quite common that the researchers want to match data from NBHW s registers with data from other agencies, e.g. SCB or the Swedish Social Insurance Agency (Försäkringskassan). In these cases the case officers of the different agencies make direct contacts and agree how to handle the case at each agency. A harm test is carried out by each agency. It happens that their decisions differ, but usually they correspond. When data are provided from several agencies they agree on a code key so that data can be delivered to the researcher with identical but unidentifiable serial numbers that can be used for the matching. The code key also makes it possible to later supplement or update the data. As mentioned above, also the data from SCB are in these cases delivered on a physical storage medium, not through MONA. An application to NBHW that is well documented and does not require joint processing of various registers can usually be fully processed in less than a month. This is much shorter than the time used by SCB, which, according to researchers, amounts to several months, often a year or more. In addition to the differences in organisational structures, probable differences between the project portfolios of the two agencies should also be taken into account. 3.2 Data security Again, the legal conditions are in principle the same for NBHW and SCB. Most data provided by NBHW are of a sensitive character and can be released only with supportive judgments from an Ethical Review Board. Also, most data from NBHW are delivered de-identified. If there are special reasons, identifiable data are supplied by NBHW, for example for validation studies or the examination of patient records. These studies are usually quite limited regarding the number of objects. The main weakness with the current NBHW system is that many copies of the data will be stored in the various institutions and NBHW cannot monitor how the data are used once they have been delivered. It is simple to copy files from a CD and use them in a way that is not consistent with the purpose for which they were disclosed. The system used by NBHW relies on the presumption that the data are handled correctly. The research body responsible signs an agreement to do so and states that they will inform all data users about the EVALUATION OF THE MONA SYSTEM 16

18 regulations. The CD is encrypted and NBHW assesses the risk that an unauthorised person will get access to the content of the CD to be small. 3.3 User satisfaction According to a user survey carried out in the spring of 2014 with NBHW users per cent seem to be satisfied with the service provided by the Register Service Unit. On behalf of VR and NBHW a qualitative client interview survey, comprising of 27 respondents, has been carried out among users of NBHW data by a consultancy firm, Governo. It showed that: The majority of respondents described the accessibility of the registers of NBHW as good. There was a consistent and overall positive picture of the processes for ordering and accessing register data where the service offered by administrators at the Register Service Unit was perceived to be of decisive importance. The application procedure was considered demanding but reasonable and the majority deemed the procedure to be relatively smooth. Several respondents believed, however, that it could be easier and more efficient. The time it takes was perceived as reasonable in most cases. The quality of documentation varies and should be developed. The coordination between the various registers in different agencies should be improved. There were respondents who perceived differences in administration depending on who the applicant is. The perception of these researchers was that individual researchers or smaller institutions may find it more difficult to be issued with data than larger institutions. The panel has met with a group of researchers using data from NBHW. In addition to the opinions reflected in the survey they raised problems with the quality of data, inadequacies in quality control and changing interpretations of the law over time when NBHW is asked to deliver data. The researchers also expressed that the personnel at NBHW do not always have sufficient insight into the data. In the Governo survey the users were also asked about their opinion on a remote access system. The majority of respondents, irrespective of personal practical experiences of the system, see such a system as an important tool in the future for facilitating the use of data in a secure manner with quality-assured and updated data. Some respondents are more sceptical and all fear that the system will restrict the management and processing of data, especially large amounts of data requiring high capacity and joint processing of data. Although these opinions were reflected in the group of researchers that the panel met with, these researchers generally expressed a willingness to transfer from their own research environments to the MONA system if the bottlenecks in the system were removed. 3.4 Costs The total cost per project in NBHW on average amounts to SEK 20,000-30,000 which is about half the cost of SCB. Many researchers also perceive that that service of SCB is more expensive than that of NBHW. It has not been possible for the panel to analyse this difference in perceptions and whether it is based on facts or not. EVALUATION OF THE MONA SYSTEM 17

19 3.5 The panel s conclusions in light of the comparison between Statistics Sweden and the National Board of Health and Welfare Unlike SCB, NBHW says that they have not experienced unauthorised use of data. We cannot, however, disregard the possibility that the same kind of unauthorised use as SCB has experienced has also occurred in health research. NBHW has little control of what happens with the data when they have been delivered to the researcher. This problem is taken care of within a remote access system, such as MONA, where the data are more secure. It should also be noted that NBHW serves fewer research environments than SCB. Most of the research with health data is carried out at the medical faculties, of which there are only seven in Sweden. They are probably all well-equipped regarding security arrangements. The total number of universities in Sweden carrying out research exceeds 30, so the potential number of researchers using SCB data is considerably higher. In principle NBHW sees advantages with a remote access system, but they also identify some problems, both legal and technical. As they have not experienced unauthorised use their incentive to invest in higher security is limited. NBHW also seems to believe that a remote access system might cause problems for some researchers, an opinion also expressed by some of the researchers the panel met with. We agree with the assessment made by SCB that there are risks related to delivering data on physical storage media, like CD. The panel s overall judgment is that a remote access system, such as MONA, must be regarded as safer than the system used by NBHW. Equally important, a remote access system can improve accessibility for many researchers, encourage use of data, and in many cases serve researchers in a more efficient and safe way. EVALUATION OF THE MONA SYSTEM 18

20 4. A FUTURE COORDINATED SYSTEM OF REMOTE ACCESS TO USE REGISTER DATA FOR RESEARCH PURPOSES? In the 2009 evaluation of DISC it was noted that SCB, in agreement with VR, had undertaken to coordinate data from other bodies that keep registers and be in charge of collaboration with these bodies. This has not worked out as expected. SCB has had meetings with several other agencies to identify their interest in providing data to the MONA warehouse but with no result. At the same time all these agencies, except NBHW, can make data accessible through the MONA system. Other agencies send copies of their registers for specific projects to SCB who match data from different registers and deliver them de-identified through MONA to the researcher. This is not the case, however, with NBHW who perceives there are legal obstacles to transfer data to SCB. 4.1 The suitability of a remote access system for use of registers at the National Board of Health and Welfare views of the National Board of Health and Welfare and Statistics Sweden NBHW has in its self-evaluation report to VR discussed advantages and disadvantages of a remote access system. According to NBHW one advantage of a remote access system is that the agency has control of how the data are used. For example, there is less risk that data is used in other research projects. It is also possible to control that the data is not used abroad where Swedish secrecy legislation is not applicable. The agency can also deprive the researcher of access to data when the study has been completed or if data are misused. Furthermore, it is easy for the agency to update the researcher s dataset. A remote access system, however, means high demands on accessibility. Researchers must always have access to their data, which puts high pressure on IT-support and system reliability. The IT-support must not only make sure that the relevant technical systems are up and running, but also help researchers with questions on how to use the system in an effective way. Given today s large-scale register-based studies it is essential that a remote access system can meet the high technical demands required for efficient data access and data storage. Many users must be able to analyse data at the same time without experiencing the system as slow. The demands for storage space will continue to increase, for example in the case of genome sequencing. There must also be a system to make back-up copies of the material at frequent intervals. All these technical solutions are readily available but entail high costs and require long-term budgeting. These requirements are the same as what the panel has identified in the MONA system. They constitute challenges for the system but do not seem to be unsolvable. According to NBHW and researchers that the panel has met with, in the MONA system researchers can only use the statistical analysis software that the agency chooses to install in the system. Hence, researchers cannot use the specific software that they are used to, or use custom-made software that has been created to carry out calculations in a way that has not been done previously. On the other hand, for some researchers the access to statistical analysis software through a remote access tool means that they do not have to buy their own licences for this software. Furthermore, NBHW thinks that a comprehensive investigation regarding a number of legal issues needs to be performed before it can introduce a remote access system. For example, they find that the legal EVALUATION OF THE MONA SYSTEM 19

21 responsibility for data stored in a remote access system needs to be clarified before a system can be implemented. The investigation must make clear whether data from other agencies stored on one of the NBHW servers for several years are still to be regarded as being held by the agencies, and what the position then is if, for example, a third party requests that the data be released. They find it to be of primary importance that all legal issues are completely resolved since the legal aspects of data disclosure weigh heavily for NBHW. According to NBHW another issue that needs to be resolved before introducing a remote access system is its long-term financing. A remote access system is unlikely to save money for the agency since the bulk of the applications will still require large amounts of manual handling by case officers, legal staff and administrators, and are not affected by the introduction of a remote access system. The cost of sending a CD containing data is tiny in comparison. When this cost is compared with building up and administering a remote access system it appears to NBHW that it would be very difficult to demonstrate the cost-effectiveness of a remote access system, at least if NBHW has to build it on its own. In the panel s meeting with NBHW representatives it found that the self-evaluation and the user survey had given rise to internal reconsiderations and discussions concerning the future modes of operation. SCB, on its part, sees that it would be a great advantage if deliveries from SCB could be coordinated with deliveries from NBHW and data be made accessible through platforms like MONA. According to SCB, providing a common infrastructure with shared resources, like software, metadata systems etc., with data from several different agencies should mean increased efficiency and more registerbased research. If different platforms could exchange data and metadata with each other it would facilitate interdisciplinary research. It would make it easier for researchers to get information about different registers if metadata were provided in similar formats. SCB admits, however, that the MONA platform is not properly designed to fit the deliveries currently made to researchers for matching with NBHW data. This needs supplementing computing power and software. In this context, the panel wants to add that both NBHW and SCB currently report very good and prestigefree cooperation with well-established contacts that are maintained through annual meetings and regular dialogues on phone and Panel s conclusions and recommendations regarding a future coordinated system It is our assessment that remote access systems have many advantages, not least from a security perspective, but for many researchers such systems may also increase the accessibility of data available at the state administrative agencies, particularly SCB and NBHW. A remote access system means higher costs for the data provider, but on the other hand costs can probably be reduced in the research environments as they, with such a system, can lower their costs for data storage, data processing, technical personnel etc. According to the terms of reference for the evaluation of MONA, VR has initiated a dialogue with SCB and NBHW that aim to improve access to registers for research purposes. One option discussed is establishing a remote access system like MONA at NBHW. If established, such a system must be able to communicate with MONA. The panel thinks that it may be difficult to show that such a solution is cost-efficient if the costs must be borne by an agency such as NBHW with other main tasks than delivering data for statistics and research. Another approach would be to use the already established MONA system to also provide access to NBHW s data. This is what the panel recommends. As stated above NBHW sees some legal problems with transferring data from NBHW to SCB. We think this problem must be solved, either through an agreement between the agencies or, if necessary, by changing the law. The collaboration between Statistics Denmark and the State Serum Institute in Denmark regarding such transfers has recently been established and can serve as a role model. EVALUATION OF THE MONA SYSTEM 20