EVALUATION OF THE NUTEK-VINNOVA PROGRAMME IN COMPLEX TECHNICAL SYSTEMS 1997-2001

V I N N O V A R E P O R T V R 2 0 0 4 : 0 3 EVALUATION OF THE NUTEK-VINNOVA PROGRAMME IN COMPLEX TECHNICAL SYSTEMS 1997-2001 Utvärdering av ett FoU-program i Komplexa Tekniska System 1997-2001 E R I K A R N O L D E T A L, T E C H N O P O L I S L T D

Titel / Title: Evaluation of the NUTEK-VINNOVA Programme in Complex Technical Systems 1997-2001 - Utvärdering av ett FoU-program i Komplexa Tekniska System 1997-2001 Författare / Author: Erik Arnold et al, Technopolis Ltd Serie / Series: VINNOVA Rapport VR 2004:03 ISBN: 91-85084-05-0 ISSN: 1650-3104 Utgiven/ Published: April 2004 Utgivare / Publisher: VINNOVA - Verket för Innovatonssystem / Swedish Agency for Innovation Systems VINNOVA Diarienr / Case No: 2002-00241 REFERAT (syfte, metod, resultat): 1991 deklarerade den svenska regeringen i en rapport att kunskap i att hantera stora system saknas i svenska företag trots att kompetens i systemtänkande är mycket viktig för den svenska industrins konkurrenskraft. Både forskning och undervisning måste etableras. För att komma till rätta med denna brist föreslogs en större satsning på forskning och utbildning inom systemteknologi. Resultatet av ett antal seminarier arrangerade av NUTEK och IVA blev ett förslag till ett pilotprogram Systemteknologi och metodik för utveckling av komplexta tekniska system. Programmet startade 1994. Det var från allra första början en gemensam satsning mellan akademin och industrin där regeringen via NUTEK finansierade hälften av kostnaderna och industrin den andra hälften, huvudsakligen genom deltagande i projekten med egen arbetskraft. En första utvärdering betecknade programmet som klart framgångsrikt och rekommenderade en fortsättning i större skala efter dess planerade slut 1997. Programmet fortsattes därför med ökade resurser under namnet Komplexa Tekniska System och de första projekten startades 1998. Den aktuella utvärderingen initierades efter programmet slut 2001. Målet var att bedöma resultaten och effekten av programmet, speciellt från ett industriellt perspektiv, liksom att bedöma de akademiska impulserna till att öppna nya forskningsområden och nya kurser. Avsikten var också att fånga in synpunkter från de som deltagit i projekten. Utvärderarna ombads också att ge sina åsikter om framtida forskningsinriktningar och industriella insatser inom området i Sverige. För utvärderingen kontrakterades Dr. Erik Arnold från Technololis Ltd som projektledare. Utvärderingsrapporten kompletteras med en mer detaljerad redogörelse av bakgrunden till programmet, dess historia inklusive styrgruppens aktiviteter samt en statistiksummering av resultaten. ABSTRACT (sim, method, results): In 1991, a programme in the area of Systems Technology was suggested in a Swedish government report, which stated as a conclusion: Knowledge how to handle large systems is lacking in Swedish companies although competence in systems thinking is very important for Swedish industry competitiveness. Both research and education need to be established. Following a series of seminars, organised by the Swedish Agency for Technology Development (NUTEK) and The Royal Academy of Engineering Sciences (IVA), a decision was taken to initiate a pilot programme to be named Systems technology and methodology for Development of Complex Technical Systems. The programme started to operate in 1994. From its initiation, the programme was a cooperative effort between academia and industry, with government financing one half of the cost and industry the other half, primarily through own work. A first evaluation designated the programme as clearly successful and recommended a continuation of the programme on a larger scale after the end of term in 1997. This was in fact done and the first projects in the continued and enlarged programme Complex Technical Systems started in 1998. The present evaluation was initiated after the end of the programme 2001. The objective was to assess the outcomes and impact of the programme, especially from an industrial point of view, but also to address the academic achievements, opening up of new scientific fields, new curricula etc, and finally the overall views of those who had participated in the projects. The evaluators were asked, also, to give their views on future directions for research and industrial efforts in the area of Complex Technical Systems in Sweden. For this evaluation, Technopolis Ltd, was contracted, with dr Erik Arnold as team leader. The evaluation report is supplemented (in Swedish) by a more detailed account of the background and history of the programme, a summary of the activities of the steering group as well as some statistics on the programme. I VINNOVAs Verket för Innovationssystem publikationsserier redovisar bl a forskare, utredare och analytiker sina projekt. Publiceringen innebär inte att VINNOVA tar ställning till framförda åsikter, slutsatser och resultat. Undantag är publikationsserien VINNOVA Policy som återger VINNOVAs synpunkter och ställningstaganden. VINNOVAs publikationer finns att beställa, läsa eller ladda ner via www.vinnova.se. Tryckta utgåvor av VINNOVA Analys, Forum och Rapport säljs via Fritzes Offentliga Publikationer, www.fritzes.se, tel 08-690 91 90, fax 08-690 91 91 eller order.fritzes@liber.se VINNOVA Swedish Agency for Innovation Systems publications are published at www.vinnova.se

Evaluation of the NUTEK-VINNOVA programme in Complex Technical Systems 1997-2001 Utvärdering av ett FoU-program i Komplexa Tekniska System 1997-2001 Erik Arnold et al, Technopolis Ltd 1

Introduction The present evaluation report concludes NUTEK/VINNOVAs programme in the area of Complex Technical Systems. The programme itself was closed in August/September 2001 after being in operation for almost eight years. Already in 1991, a programme in the area of Systems Technology was suggested in a government report IT2000 (Dep of Industry Ds 1991:63) which stated as a conclusion: Knowledge how to handle large systems is lacking in Swedish companies although competence in systems thinking is very important for Swedish industry competitiveness. Both research and education need to be established. Following a series of seminars, organised by NUTEK and The Royal Academy of Engineering Sciences (IVA), on Complex Systems in 1992, a decision was taken to initiate a pilot programme to be named Systems technology and methodology for Development of Complex Technical Systems. The programme started to operate in 1994. From its initiation, the programme was a cooperative effort between academia and industry, with NUTEK/VINNOVA financing one half of the cost and industry the other half, primarily through own work. A first evaluation was concluded in 1996. It designated the programme as clearly successful and recommended a continuation of the programme on a larger scale after the end of term in 1997. The first call for proposals within this new/continued programme was made public in the autumn 1997 and the first projects started in 1998. A second evaluation was concluded in May 2000, where 44 projects were evaluated by an international panel of 10 peers. This second evaluation primarily aimed at assessing the scientific quality of the projects. It was based on written submissions and individual presentations to the panel by the project leaders and, in some cases, also by industrial partners. The present evaluation, the third one, was initiated after the end of the programme. The objective was to assess the outcomes and impact of the programme, especially from an industrial point of view, but also to address the academic achievements, opening up of new scientific fields, new curricula etc, and finally the overall views of those who had participated in the projects. The evaluators were asked, also, to give their views on future directions for research and industrial efforts in the area of Complex Technical Systems in Sweden. For this evaluation, Technopolis Ltd, was contracted, with dr Erik Arnold as team leader. The evaluation report is supplemented (in Swedish) by a more detailed account of the background and history of the programme, a summary of the activities of the

The Programme Board and VINNOVA staff lined up after their last meeting in 2001. From left to right. First row: Bengt Larsson - VINNOVA, Lena Mårtensson - KTH, Anders Hedin & Barbro Atlestam - VINNOVA. Second row: Jonas Bjarne & John Graffman - VINNOVA, Ove Borgström - Ericsson, Lars-Göran Rosengren - VOLVO TU, Arne Otteblad, Ingemar Karlsson - FMV, Bengt Sjögren - Kompetens Lagman AB, Ulf Holmgren - VINNOVA. Absent were: Hans Skoog - ABB, Gustaf Olsson - Lunds Universitet, Karl-Einar Sjödin & Ingvar Åkerstedt - VINNOVA.

Contents The Complex Technical Systems Programme at NUTEK and VINNOVA: An Evaluation by Technopolis Ltd, June 2003 Appendix 1/Bilaga 1: Om Komplexa tekniska system - programmens bakgrund, syften och genomförande (in Swedish) Appendix 2/Bilaga 2: Styrgruppens metoder för verksamhetsstyrning och intervenering (in Swedish) Appendix 3/Bilaga 3: Statistics from the programme

The Complex Technical Systems Programme at NUTEK and VINNOVA: An Evaluation Report to VINNOVA Erik Arnold John Bessant Enrico Deiaco James Stroyan Ben Thuriaux Shaun Whitehouse Rapela Zaman June 2003

Contents Summary 1 Introduction and Methods 1 1.1 What are Complex Technical Systems? 1 1.2 Scope of this Evaluation 2 1.3 Methods and Limitations 2 1.4 Acknowledgements 3 2 The Complex Technical Systems Programme (KTS) 5 2.1 The First KTS Programme, 1993-1996 5 2.2 Transition 7 2.3 The Second KTS Programme, 1997 1999/2000 7 2.4 Evaluation of Projects in 2000 11 3 What the Programme Did 13 3.1 Programme Composition 13 3.2 What the Participants Did 22 4 Results of the Programme 29 4.1 Questionnaire Analysis 29 4.2 Results Reported by Projects 40 5 Management and Administration 41 5.1 The Programme Board 41 5.2 Interviews with Board Members 42 5.3 NUTEK Support and Procedures Questionnaire and Interview Evidence 43 5.4 Programme Administration An Evaluation Perspective 45 6 Project Experience and Case Studies 46 6.1 Project Participant Views 46 6.2 Case Studies 49 7 Complex Technical Systems in an Agency for Innovation Systems 63 7.1 Knowledge Production and Innovation 64 7.2 The Theoretical Problem 67 7.3 The Human and Social Dimension of Technology 68 7.4 Interdisciplinarity 71 7.5 Competitiveness and Innovation Systems 72 8 Conclusions and Recommendations 76 8.1 Conclusions on KTS as an Advanced Technology Programme 76 8.2 Implications for VINNOVA as an Innovation Systems Agency 78 8.3 Next Steps 84 Appendix A Questionnaires to Project Participants 88 Appendix B Questionnaire Non-Response Analysis 103 B.1 Additional Questionnaire Analyses 105 i

Summary This evaluation report to VINNOVA tackles two somewhat different questions. First, we evaluate the Complex Technical Systems (KTS) programme. We find it to be a strong programme, well tuned to industrial needs and producing many useful outputs, including both re-usable knowledge and results of more direct use in industry. Second, we consider what VINNOVA can learn on a more general level from the programme for its own role in the innovation system and about how to run significant technology and innovation activities. Technical complexity and what we might call systemicity are important themes in modern innovation systems. This is partly because we face more technological opportunities than previous societies and partly because our society operates at unprecedented scale. Mastering complex technical systems, and especially their design is therefore an important agenda. The first KTS programme ran originally from 1993 1996. This evaluation covers the second KTS programme, which ran after a gap of a year from 1997 to 2000. The issue of complex technical systems was much discussed at the start of the 1990s, when it was generally recognised that this was an economically important area. NUTEK launched an R&D programme in 1993 to find out how better to design such systems, and thereby to support the competitiveness of Swedish industry. A larger successor programme was launched in 1997, which is the subject of this evaluation. This aimed to support Swedish competitiveness by reinforcing knowledge about systems development and devising general methods that could be used to tackle a range of types of complexity. The programme was later cut back by 4 months owing to budgetary restrictions. A key strategic change during the life of the programme was the introduction of horizontal projects, aiming to generalise and transfer knowledge between the other, more domain-oriented sub-programmes. The KTS programme generated well over 265 MSEK of activity across 130 organisations. Half the effort went on the broad field of systems development. Most of the effort went on this and other specific complex technical domains. The smallest amount went to horizontal projects, aiming to synthesise and develop more generic knowledge an aspect which industry was not inclined to co-finance. Most of the projects in the programme were rather small and long, with 18 (a quarter) having a budget above 5 MSEK across a three year period. Project leaders were predominantly university personnel, who were positioned as potentially key nodes in the wider national network of the programme. Projects were strategically important and close to the core business of those who performed them. They were often incremental, reflecting the strong industrial influence in project definition. While this influence appears to have been more important than is usual, in many respects the KTS projects had similar characteristics to those carried out in other advanced technology programmes, with knowledge outputs being central. These tend to produce intermediate products, useful in doing further development and research, rather than final products or processes. ii

Our questionnaire analysis suggests that participants did well in satisfying their own goals through the programme, but is less clearly positive about the projects effectiveness in reaching programme goals. Most of the projects were additional and there appears to have been little free riding in the programme. The main outputs were knowledge and networking. Despite the long term nature of the projects, some short term commercial exploitation had already been achieved. New knowledge was generated but externalities from the projects, in the form of knowledge and capability transfers, had been limited. Both industrial and academic participants felt that the benefits of being involved in the KTS programme had very significantly outweighed the costs. The overall picture, then, is consistent with the impression we have already formed of projects that have strong industrial steering but a weaker link to wider social and technical goals. NUTEK had well-established routines for programme definition and management, which in general performed well. These were documented in a benchmarking report 1 commissioned by NUTEK in the mid-1990s, and have subsequently been compared with other R&D programme management practices, especially in the Nordic area. 2 We chose, therefore, in this evaluation not to attempt an exhaustive review of the programme management processes involved in KTS. Instead, we used our questionnaire, interviews and discussions with programme management to look for signs of trouble, and to get an overall assessment of programme management from the user perspective. The programme board comprised experienced and influential stakeholders and provided helpful continuity with the past, a powerful knowledge base and a resource for mobilising projects. It took measures that resulted in high industrial influence over the programme and its projects. However, the great size of this second KTS programme meant that both the board and the administration at NUTEK/VINNOVA were stretched. The board stressed programme conferences as a way to bring the programme together. This was partly successful, though the wide scope of the programme proved to be a barrier to bringing together people with very different interests. The rather bottom-up way the programme had been populated with projects led to fragmentation. Generic complexity questions did not arise spontaneously from industry or from academics. Participants were generally very positive about the quality and timeliness of administration. From our evaluation perspective, and indeed in terms of programme management s need for basic management information and some kind of statistical process control, the programme did not have adequate Information Systems support. Discussions with project participants confirmed that many of them worked in rather long-standing networks with industry, though new relationships were also being formed. University people were key in establishing many projects, while end users were rarely involved in project definition or execution. Projects generally claimed to be developing tools and methods for complex technical systems development and 1 2 Erik Arnold and Paul Simmonds, Programme Management Benchmark Study; Final Report to NUTEK, Brighton: Technopolis 1997 James Stroyan and Erik Arnold, Comparative Study on Administrative Burdens and Rules fo Procedure between the EU Research Programmes and those of the Individual Member States, project IV/98/06, Luxembourg: European Parliament, 1998 iii

many to be raising systems design capabilities in universities and industry, including through postgraduate education. Project leaders found the KTS programme rather broad and, in some cases, hard to identify with. NUTEK administration got high marks from the project leaders. The case studies showed that there is a wide range of project types. Some projects had succeeded in integrating human factors, others had not. Those that worked well had produced useful knowledge and applicable results in the form of inputs to development and changed working practices. Horizontal project cases aiming to synthesise domain-specific work in the programme in order to develop generic systems understanding suffered from the lack of generalisable results and tended to become (useful) state of the art reviews rather than true syntheses. KTS tried, as the test pilots say. to push the envelope : that is, to see how far it could go beyond what is normal in this kind of technology programme. At the very least, we can conclude that KTS tackled some of the more difficult aspects of modern technological development. These difficult aspects include the desire to integrate human beings and human factors into thinking about complex technical systems and the programmes explicit aim to be interdisciplinary. An overall conclusion of this evaluation is that the KTS programme was interesting, useful and well executed. It was additional, it raised capabilities, developed new knowledge and techniques and led to important benefits for participants. Our evaluation of the programme in its context and in its time is positive. However, we do evaluation not only in order to make judgements about whether programmes reach useful results or whether they are well run. Evaluation is an important way to learn, and KTS offers some interesting and, we think, useful lessons. The be useful, we need to consider these lessons using current theory, since VINNOVA as an agency has been conceptualised on the basis of innovation theory that was only just beginning to be formed in the early 1990s, when the KTS programme was conceived. Also, in trying to make practical as well as intellectual progress in tackling generic complexity issues in industrial design practice, KTS bit off more than it could chew. This is not a normal type of conclusion in an evaluation of NUTEK or VINNOVA programmes, but neither is KTS a normal programme. Like other NUTEK/VINNOVA programmes, KTS is a Mode 2 programme, involving industry, university and other R&D workers, as opposed to a Mode 1 disciplinary activity. This type of research is becoming increasingly important as new technologies evolve at the boundaries between disciplines and as the capacity to do R&D increases in all parts of the economy. In some cases, simultaneous R&D and more fundamental work may be helpful in supporting and creating opportunities for innovation. Researchers may also move back and forth between the two Modes, according to need. Knowledge needs are constantly changing. Programmes can help create communities of Mode 2 practice, in order to spread generic knowledge and also to help achieve links with longer term work that can break bottlenecks in the innovation process, such as those encountered in KTS. A key bottleneck was the lack of an adequate generic systems theory, or a hypotheses within the KTS programme about what such a theory might look like. The stress laid iv

in the programme on industrial participation tended to squeeze out the more fundamental work needed, and the KTS programme lacked incentives and instruments that could overcome this market failure. Different project structures, for example involving more end users, could help improve the ability of the research funded to integrate human factors. Other configurations (such as larger, multidepartment projects) might have been useful for increasing the amount of interdisciplinary work in the programme. Not least, the very technical focus of the programme has tended to push these aspects down the agenda. In an innovation systems agency context, it would make sense to take account of the different ways that low volume complex product systems and high-volume complex products are produced, because these appear to require different hard and soft technological approaches. It seems that VINNOVA has inherited an inadequate range of funding instruments from its predecessors. The problems KTS tried to tackle could not be solved without a combination of instruments to involve industrial co-operation but also to fund more strategic or fundamental research, for which industry simply will not pay because of the well known market failures associated with longer term research. The structure of the Swedish knowledge infrastructure continues to be an impediment to tackling many of the more important challenges in knowledge production and use, such as interdisciplinarity, scale and the shift in the locus of knowledge production to being primarily outside the universities. There is a continuing need for a change agency here. The governance and incentive systems used need to be tuned to the activities of individual programmes. Those used with KTS tended to lock the programme into the shorter-term concerns of industry and to lock out the work that should have been present to meet its longer term needs. VINNOVA needs to counterbalance the excellent tradition of stakeholder involvement in programme design that it has inherited from NUTEK with both economic and technological analysis capabilities, in order to balance short- and longer-term needs. VINNOVA is grappling with the question: What, if anything, should follow on from the KTS programme? It would be possible to focus a new programme design by soliciting these kinds of longer-term business needs driven questions and confronting them with a set of more research-driven ones. The additional element required is analysis. Before trying to launch a programme, it is necessary to Understand the state of the art, to make sure that the answers to the programme s research questions are not already available or that the proposals do not introduce some new mission impossible to the agenda Scope the potential areas of social and economic impact. This is not to say that an economic impact or cost-benefit analysis should be attempted, for this is methodologically untenable. Rather, it is important simply to confirm that the intended action is likely to affect large or critical parts of the Swedish innovation system rather than small or unimportant ones It would make sense to complement these analyses with an explicit programme activity that maintains a technology watch on foreign developments. In the short term, VINNOVA would have at least three options in terms of the type of programme to put in place v

1 A combined co-funded industrial programme and a 100%-funded more researchintensive strand, somewhat in the style of TEKES 2 A call for tenders to establish two or three new competence centres or, perhaps, a single, much larger multi-site competence centre 3 An invitation to construct an industry consortium or consortia, working with long term research questions in the area of complex technical systems. The MediaLab at MIT could provide a model, where companies buy into research programmes for extended periods. IMEC, the major microelectronics research institute in Flanders, works in a similar way The main criteria for choosing among these forms is the extent to which the problems identified are likely to persist, the extent to which tackling them involves cutting across existing departmental lines in the universities and the willingness of industry to become involved as a long-term partner. In the longer term, VINNOVA has an opportunity to use the complexity agenda as a platform for negotiating a pattern of working together with research councils and foundations that goes beyond today s rather minimal level of co-ordination. vi

1 Introduction and Methods 1.1 What are Complex Technical Systems? Technical complexity and what we might call systemicity are important themes in modern innovation systems. This is partly because we face more technological opportunities than previous societies and partly because our society operates at unprecedented scale. The two things are strongly interconnected. Equally, since electronics matured into microelectronics and gave us the power to work with millions of bits of information and processing elements at a time, information technology has opened the door to huge possibilities. Today, the average Swedish household probably has more information processing power than did most countries a few decades ago. The technology lets us address more complex situations than before and it allows us to invent more complex situations to address. Complexity in this simple sense of having many parts is not necessarily hard to handle. Imagine driving from Borås to Gothenburg. For simplicity, let s do it after meeting our grandparents in church on Christmas Eve, when there is almost no traffic. We navigate the streets of Borås, take the motorway past Landvetter Airport and soon arrive in Gothenburg, where we navigate to our destination. If we do not know the way we may use a map. But, despite the fact that we are using a small corner of an immense continental road network, the problem is not much more difficult than the one our grandparents face in walking home from church to their house in Borås or would have faced, doing the same 50 years ago. Shift the perspective now to that of the Swedish National Roads Administration (Vägverket) and the local councils who between them design, build and manage these same roads. They have to consider not one car but many, and crucially the interactions among them. What happens when the shift change at Torslanda meets the cross-town morning traffic? When the morning commuters from Borås and the newly-landed Stockholmers in their taxis from Landvetter all converge on the motorway exit, where the traffic has slowed to a crawl because of the sheer number of vehicles interacting? To answer these questions, we first have to understand the roads not just as a network but as a system, whose parts interact. Because of these interactions, properties emerge that are not obvious from looking at the network. The planners find they need complex computer models with feedback loops to account for these emergent properties such as the traffic jams. They fit the road network with sensors, link the sensors to computers and the computers to the traffic lights. They provide real-time traffic information to the car drivers to influence their behaviour. They struggle to manage the highway engineers, electronics specialists, social and behavioural scientists, economists and mathematical modellers they bring in, but who hardly understand each other. They argue about whether the SNRA should modify the highways or the local council should change the layout of the one-way system. Just when things seem to be getting better, a tram breaks down at a central junction. Drivers take to the back streets to avoid it, and unexpectedly bring most of the 1

downtown area into gridlock, showing how vulnerable the new street layout is to disruption. These elements big networks, complex interactions with feedback loops and emergent properties, interdisciplinarity, the influence of human behaviour recur with increasing frequency in our societies, which is why complex technical systems are so important. A significant part of OECD industry, and especially that of Sweden, builds operates and makes components for complex technical systems, so knowledge and skills about these systems are economically very important. The cars in the Gothenburg traffic are themselves complex technical systems, and the production systems that make them even more so. The problems when complex technical systems go wrong are economically significant, too whether we think of the time wasted by all the people sitting in the traffic jam looking at Liseberg go past in slow motion, or the bill the tax payers have to foot when the latest big administrative systems development project goes five times over budget and still does not work. 1.2 Scope of this Evaluation Mastering complex technical systems, and especially their design since most problems are designed in from the start is therefore an important agenda, not only for Sweden but also in other countries (one reason why we think this evaluation will have international interest). The importance of complex technical systems was recognised in Swedish policy debate already in the early 1990s. NUTEK (the Swedish National Board for Industrial and Technological Development) launched an R&D support programme to tackle the question. This evaluation covers the second Complex Technical Systems (KTS) programme run by NUTEK. The first ran originally from 1993 1996, though some projects lived on beyond the formal end of the programme, and several were incorporated (sometimes in modified form) into the second programme, which ran after a gap of a year from 1997 to 2000. 1.3 Methods and Limitations Evaluation of this type of activity is by no means an exact science. The methods at our disposal are individually not completely reliable and are often subject to bias. To reach conclusions in which we can believe, therefore, it is important to combine different ways of looking at the entity to be evaluated. We can have confidence in our findings and conclusions to the extent that the indications from these different tools and techniques converge. To tackle the KTS programme, we Reviewed programme documentation Reviewed other relevant literature on complexity Interviewed programme management Interviewed six of the eight members of the programme Board Sent a questionnaire to 183 participants in the programme, asking about their motives and goals in participating, their achievements and their experience of participating Interviewed a selection of 21 project participants to deepen our understanding about participation 2

Participated in four meetings with VINNOVA personnel and other interested parties to discuss our methods and the emerging conclusions of the evaluation We also conducted a number of illustrative case studies, intended to help both the reader and ourselves understand in a more qualitative way how the programme operated. While we naturally try to be as rigorous as we can, an evaluation such as this finally represents a number of judgements. We have tried to make clear distinctions between what we regard as data and our judgements, since the reader may want to interpret the data differently from us. The evaluation team has collectively conducted upwards of one hundred R&D programme evaluations, so our judgements are clearly informed by experience, but also by research and theory about research, innovation and innovation systems. Finally, however, this kind of report can never be Truth. We see it rather as one contribution among others to a debate about research and innovation policy, and hope that it is a valuable input. As concerns the questionnaire, in the middle of August 2002, a total of 183 questionnaires were e-mailed to participants in the KTS programme. Almost half (43%) of the questionnaires have been completed, though this figure rises to 49% if we discount from the population those recipients who were either not able to be contacted or who indicated that their involvement / recollection was insufficient to permit a detailed response. We made efforts to contact all non-respondents, both by email and by telephone. Exhibit 1 sets out the responses obtained by 18 th October 2002, at which point we froze 3 the analysis. Exhibit 1 Questionnaire Responses Research base Industry Total Share of total Completed 41 37 78 43% Undeliverable 1 5 6 3% Refused 3 22 25 14% Not responded 26 48 74 40% Total (mailed) 71 112 183 100% As the Exhibit indicates, the responses rate is much higher from academics (58%) than from industry (33%). However, the responses of the two populations are surprisingly similar. (Where they differ in an important way, we show separately the responses from the two populations.) We explored the responding and nonresponding populations to see whether there were other important differences. Two potential sources of bias emerge. One is that non-respondents were more likely to come from the Systems Development sub-programme. The other is that people from single-organisation projects were somewhat less likely than others to respond. 1.4 Acknowledgements The evaluation team is grateful to the large number of people who helped us in this work: interviewees; the many people who filled in our questionnaires; and not least 3 The 2 questionnaires since received have been disregarded 3

the steering committee, programme management and evaluation team at VINNOVA, the agency which inherited the KTS programme and a great deal more from NUTEK. The responsibility for mistakes, misunderstanding or misinterpretations, of course, lies wholly with the ourselves. 4

2 The Complex Technical Systems Programme (KTS) The issue of complex technical systems was much discussed in Swedish R&D policy circles at the start of the 1990s, when it was generally recognised that this was an economically important area. NUTEK launched an R&D programme in 1993 to find out how better to design such systems, and thereby to support the competitiveness of Swedish industry. Following a positive evaluation and a gap of a year during which there we negotiations about financing, a larger successor programme was launched in 1997, which is the subject of this evaluation. The programme aimed to support Swedish competitiveness by reinforcing knowledge about systems development, devising general methods that could be used to tackle a range of types of complexity. The programme was executed, but cut back by 4 months owing to budgetary restrictions. A key strategic change during the life of the programme was the introduction of horizontal projects, aiming to generalise and transfer knowledge about complex systems design between the other, more applications-oriented subprogrammes. 2.1 The First KTS Programme, 1993-1996 Technical complexity appears to have been much debated in Swedish R&D policy circles at the start of the 1990s. One symptom of this was evident in the IT2000 report, which was produced by a national committee in 1991, considering the industrial and research policy implications of information technology (IT). The committee observed that the capability to deal with large systems is in short supply in all large firms, while it is exactly this systems capability that is Swedish companies most important competitive resource. The idea of a complex technical systems programme was considered inside NUTEK and in the short-lived Swedish Research Council for Engineering Sciences (TFR), both of which had been set up in 1991. The idea of an R&D programme on complex technical systems found support both in NUTEK s routine researchers surveys and in a joint seminar between NUTEK and the Royal Swedish Academy of Engineering (IVA) in 1992. A group convened by TFR suggested in 1992 that there should be an R&D programme on the subject. This did not find favour with the council, but the following year after a considerable amount of preparation NUTEK decided to run a KTS programme, which was actually entitled Systems technology and development methodologies for complex technical systems. A paper 4 from a joint IEEE and IFAC (Institute of Electrical and Electronic Engineering and International Federation for Automatic Control) summarises a number of challenges, which were in the air at the time Growing combinatorial complexity in engineering systems being designed. Systems are made up of an increasing number of sub-systems and components. As the number of system elements increases, the number of interactions among elements rises even faster 4 A Beneviste, K J Åström, P E Caines, G Cohen and L Ljung, Facing the challenge of computer science in the industrial applications of control, October 10, 1991 5

The growing need for interdisciplinary approaches, as systems bring together different technologies (such as electrical, mechanical, hydraulic, and so on) Barriers to the movement of knowledge among engineering disciplines. Most important, the growing use of digital circuitry in control not being accompanied by a take-up of mathematically based software engineering techniques and modularity, used in computer science to manage complexity and assure quality. The key barrier here was seen as a compartmentalised education system Insufficient academic/industrial interaction on these problems, leading to insufficiently global understanding of the issues in industry and a research tradition insufficiently grounded in experience NUTEK launched its first KTS programme, based on a plan 5 devised with the assistance of a group of industrial and research institute people. These extremely competent stakeholders provided both their best assessment of the situation and a crucial link between the KTS programme and the Swedish communities who should be involved. However, there was no more formal state of the art or literature review in what was to prove a theoretical minefield. The programme s goals were to find general methods, tools, descriptive models etc to develop, manage, produce and use complex technical systems and master the negative aspects associated with complexity. It was to involve industry, research institutes and universities, and was to focus on problems of relevance to Swedish industry. Implicitly, it was understood that an important proportion of Swedish industrial activity is in the production of such systems. Complexity was seen as positive in that it provided a means to increase systems performance. However, the programme plan also viewed complexity as having a number of negative aspects many of which are cognitive, rather than being in the nature of the technologies themselves Non-deterministic (or poorly understood) relations among system components and sub-systems The need to integrate across multiple systems and technologies (and therefore disciplines) Inability to communicate among disciplines, causing difficulties in understanding individual complex systems Combinatorial complexity Complexity in the sense of the complicated or difficult was explicitly excluded from the programme. Thus, for example, complex flows were out of scope. Rather, the programme focused on complexity in relation to systems. The programme s strategy was to start with problem areas or applications areas, where complex technical systems are produced, and to develop projects, which would tackle problems specific to these. One of the criteria for project selection was intended to be the use of scientifically based methods that could be used in several 5 NUTEK, with the co-operation of Ove Borgström Ericsson Telecom, Ingemar Carlsson FMV, Sven Gunnar Edlund STFI, Krister Gerdin SMHI, Ulf Rehme Saab Scania, Lars Göran Rosengren Volvo, Programplan för systemteknik och utvecklingsmetodik för komplexa tekniska system, NUTEK, 1993-04-19 6