Annexes to COM(2001)276 - Final evaluations of the multi-annual programme to stimulate the development of a European multimedia content industry and to encourage the use of multimedia content in the emerging information society (INFO2000) and of the multi-annual programme to promote the linguistic diversity of the EC in the information society (MLIS)

Please note

This page contains a limited version of this dossier in the EU Monitor.

dossier COM(2001)276 - Final evaluations of the multi-annual programme to stimulate the development of a European multimedia content industry and ...
document COM(2001)276 EN
date May 28, 2001
annexed to this Communication [2]. Interim reports during the course of the two evaluations were used by the Commission in formulating its proposal for a follow-up programme, eContent [3].

[1] For INFO2000, Technopolis Ltd (lead contractor), Databank Consulting SpA, IDATE, LENTIC; for MLIS, ECOTEC Research & Consulting Ltd.

[2] Final Evaluation of the INFO2000 Programme: Final Report, June 2000. Technopolis Ltd, Databank Consulting SpA, IDATE, LENTIC . http://europa.eu.int/comm/information_society/evaluation/index_en.htm

info2000

[3] COM(2000) 323 of 24.5.2000, Proposal for a Council Decision adopting a Multi-annual Community programme to stimulate the development and use of European digital content on global networks and to promote linguistic diversity in the Information Society.

INFO2000: a programme to stimulate the development and use of multimedia content

Background

The objectives of INFO2000, as specified in its Council Decision [4], were to:

[4] Council Decision 96/339/EC of 20 May 1996 (OJ L 129 , 30.5.1996, p.24).

- Create favourable conditions for the development of the European multimedia content industry.

- Stimulate demand for, and use of, multimedia content.

- Contribute to the professional, social and cultural development of citizens.

- Promote the exchange of knowledge between users and suppliers.

INFO2000 covered the four-year period from January 1996 to December 1999 with a budget of 65 million euro. The programme was implemented through co-financing around 200 projects, involving over 700 organisations, through four action lines:

- Stimulating demand and raising awareness;

- Exploiting public sector information;

- Triggering Europe's multimedia potential;

- Support actions.

Evaluation objectives

The INFO2000 evaluation assessed the extent to which the programme was effective in meeting its objectives as laid down in its Council Decision; the effectiveness of the programme's theory of action (intervention logic); the efficiency of the programme's organisation and management; issues related to rationale, synergy and sustainability; the extent to which the programme met the needs of its target audience; and whether the recommendations of the intermediate evaluation had been implemented. In assessing impact, the evaluators drew on a concurrent study [5].

[5] Impact Assessment Review of INFO2000 Programme, BIPE for DG INFSO, April 2000.

Key evaluation findings

The evaluators found that the benefits of the programme overall were significant, and that it raised awareness of the opportunities offered by multimedia (especially amongst SMEs) via a network of organisations located in every Member State; acted as a catalyst to the European multimedia industry by funding projects across a variety of areas; and stimulated the debate on exploiting Europe's public sector information, leading to the creation and adoption of a Green Paper [6].

[6] COM(1998) 585 of 20.1.1999, Public Sector Information: A Key Resource for Europe -- A Green Paper on Public Sector Information in the Information Society.

The evaluators determined that the programme also provided a stimulating environment for public and private sector actors to work together effectively, and contributed to the progress on issues concerning European multimedia rights trading and other legal and regulatory topics. The evaluation report stresses the positive effect that extra funding might have had in the support of multimedia projects, and notes that the budget reduction (from an initial 100 million euro requested by the Commission to the 65 million euro allocated by the Council) led to a significant decrease in activities with a corresponding decrease of impact on the marketplace. The evaluators found, however, that in line with its declared objectives, the programme had succeeded in addressing the SME target group [7].

[7] Some 43% of participants in actions lines 2 and 3 of INFO2000 were SMEs.

The evaluators found that problems encountered during the programme's implementation were relatively minor: there were some delays in awarding contracts and making payments, and networking and information exchange within and between projects could have been more effective.

Main recommendations and Commission comments

The INFO2000 evaluation report contains three main recommendations [8]:

[8] Page 12 (with arguments developed on pages 10-11) of attached INFO2000 Final Evaluation Report.

1. The selection and direction of the themes for the follow-on programme, eContent, should be accepted.

Comment: The Commission welcomes this recommendation, and notes that it is based on a comprehensive evaluation of INFO2000 involving a sectoral analysis [9], case-studies [10], a participant survey, and the incorporation of the findings of a complementary impact study [11].

[9] Appended separately to the INFO2000 Final Evaluation Report.

[10] Attached as an appendix to the INFO2000 Final Evaluation Report.

[11] Impact Assessment Review of INFO2000 Programme, BIPE for DG INFSO, April 2000.

2. Legislation, regulation and standardisation should be addressed in the follow-on programme.

Comment: The Commission shares the view expressed in the evaluation report that these issues remain significant barriers to the construction of a single European market in this sector. The proposal for the follow-on programme, eContent, foresees an action line to tackle these issues and support standardisation. The main action lines of programme foresee measures to encourage standardisation (e.g. metadata in the use of Public Sector Information and linguistic infrastructure for the Linguistic and Cultural customisation). In addition a number of horizontal and support measures targeting these problem areas, including the creation of several digital content rights clearing centres, funding strategic studies to monitor and analyse developments, the establishment of a group of industrialists to facilitate debate on content related issues will contribute to keep the programme action near the needs of the market place. Legislative and regulatory matters do not belong to the scope of the eContent programme. However they may be initiated as appropriate in case actions undertaken under eContent would show such need.

3. The future programme should address, with adequate funds, issues of inter-project and inter-programme transference of information, knowledge and skills.

Comment: In its proposal for eContent, the follow-on programme, the Commission has taken into account the need to provide an environment conducive to project clustering and the exchange of best practice between projects. Specific measures to this end will be proposed, at the level of the workprogramme, relating to both contractual arrangements and the size of individual projects. In addition -- because of the expected high level of SME participation, as witnessed in the case of the preparatory actions for the year 2000 [12] -- the Commission will propose specific measures to lighten the administrative burden involved in taking part in and implementing the programme. Furthermore, the Commission takes full note of the need to draw up and execute a proactive, effective information dissemination and communications policy as a key means of addressing the target groups specified in the eContent programme proposal.

[12] Budget item B5-334, Budget of the European Communities year 2000

MLIS: Multilingual Information Society Programme

Background

The objectives of MLIS, as specified in its Council Decision [13], were to:

[13] Council Decision 96/664/EC of 21 November 1996 (OJ L306 , 28.11.1996, p.40).

- Raise awareness of and stimulate the provision of multilingual services in the Community, utilising language technologies, resources and standards.

- Create favourable conditions for the development of the language industries.

- Reduce the costs of information transfer among languages, particularly for SMEs, and contribute to the promotion of the linguistic diversity of the Community.

MLIS covered the three-year period from November 1996 to November 1999 with a budget of 15 million euro. The programme was implemented through co-financing 35 projects, involving nearly 190 organisations, through three action lines:

- Supporting the creation of a framework of services for European language resources.

- Encouraging the use of modern language technologies, resources and standards.

- Promoting the use of advanced language tools in the Community and Member States' public sector.

MLIS also included several dissemination actions and accompanying measures.

Evaluation objectives

The MLIS evaluation assessed the impact and effectiveness of the programme in reaching the objectives laid down in its Council Decision and subsequent workplan; the effectiveness of the programme's theory of action (intervention logic); the efficiency of the programme's organisation and management; and issues related to synergy, complementarity and sustainability.

Key evaluation findings

In analysing the programme's intervention logic and the policy formulation process leading up to the Council Decision, the evaluators found that the attempt to incorporate many diverse socio-economic rationales into a single programme had led to problems in accomplishing the ambitious objectives set with the limited budget available. In assessing organisation and management issues, the evaluators found that implementation was slower than expected, with a number of weaknesses caused by procedural and structural factors. Notable among these was the relative rigidity of programme mechanisms largely copied from RTD programmes, and the lack of adequate human resources. The evaluation report states that the final impacts of MLIS relative to its objectives at the Community level are modest and so far incomplete, with many projects still underway. However, the report notes that most activities funded within the programme are highly pertinent, with progress made relative to all programme objectives.

Main recommendations and Commission comments

The MLIS evaluation report contains nine main recommendations [14]:

[14] Listed on p. 23 of attached MLIS Final Evaluation Report. The recommendations are here slightly regrouped to enable comments to be thematically clustered.

1. An EU language policy framework should be established.

2. The Commission should establish an Inter-Service Group on information and communications technologies (ICT) and language issues, and should publish, via a crosscutting Web site, a guide to EU-supported activities in this area.

Comment: The Commission will actively consider these two recommendations when further designing and planning the proposed eContent programme, and in preparing its contribution to the 2001 "European year of languages". All language-related EU projects managed by the Information Society DG are already presented via a single website [15].

[15] http://www.hltcentral.org

3. In considering future programmes, emphasis should be given to stimulating demand for and the take-up of language services.

Comment: The Commission accepts this recommendation. The proposal for a follow-on programme to MLIS and INFO2000, eContent, includes as one of its four specific objectives "Promotion of multilingualism in digital content on global networks and increase in export opportunities for European content firms (particularly SMEs) through take-up of linguistic and cultural customisation". The means envisaged is to foster partnerships between digital content and language industry players to encourage the widespread adoption of multi-language product development strategies and solutions by supporting collaborative, cost-shared projects.

4. Greater emphasis should be given to measures concerning dissemination, awareness-raising and good practice, stimulating demand and the catalytic role of the EU.

Comment: The Commission accepts this recommendation and will implement it via the accompanying measures planned as part of the support actions of the proposed eContent programme. Steps in this direction, including print and electronic publications, have already been taken during the last year of implementation of the MLIS programme. Their positive effects can be seen in the mobilisation of a much broader range of market players responding to the call for eContent preparatory actions launched in April 2000.

5. Any follow-up programme should include as part of the preparation work ex ante assessments concerned with the problematique and the rationale for EU intervention, together with more closely defined and measurable objectives and appropriate indicators of progress.

6. Funding needs to be determined before details of any future Council Decision so that ambitions are commensurate with resources.

Comment: The Commission will give particular attention to these recommendations, which echo key aspects of the recent Commission Communication on strengthening the evaluation of Commission activities [16] and are in line with the Commission's progressive introduction of activity-based budgeting (ABB) as part of the ongoing reform of the Commission. In the particular case of the eContent proposal, several studies [17] were undertaken in order to derive a problem statement and generate an appropriate intervention logic, and a number of quantifiable objectives and corresponding indicators have been derived on the basis of this analysis [18].

[16] SEC(2000) 1051 of 26.7.2000, Focus on Results: Strengthening Evaluation of Commission Activities.

[17] See notably "Multilingual Digital Content: Export Potential and Linguistic Customisation of Digital Products and Services", Equipe Consortium, July 2000, and "Evaluation of the Economic and Social Impact of Multilinguality in Europe", Bureau van Dijk Ingénieurs Conseils, July 2000.

[18] See section 9, "Elements of Cost-Effectiveness Evaluation", of Commission's proposal for a Council Decision on eContent.

7. Given that many of the MLIS projects are not yet complete and are likely to generate interesting results, mechanisms should be put in place to ensure that lessons can be derived and disseminated.

Comment: These projects will be carefully monitored as part of the preparations for the proposed eContent programme. Dissemination mechanisms will include: HLTCentral, the Web-based observatory of human language technologies; presentations and showcases at appropriate events; and regular workshops and information exchange sessions with other relevant projects.

8. There would be benefit in a more flexible project-funding model oriented towards the take-up of language services (demand), and that would allow for more emphasis on outputs rather than inputs at project level.

Comment: The Commission welcomes this recommendation. In the context of realising the European Research Area, serious consideration is being given to opening up the funding models currently used in EU programmes and to switching to forms of contract based on the submission and acceptance of deliverables.

9. Benchmarks should be established for assessing value for money at the project level.

Comment: Any project in the proposed follow-up programme, eContent, will be required to derive specific objectives from its goal and to specify these in a measurable and verifiable form. A project will also be required to establish benchmarks by describing the current situation in its area of intervention with some basic baseline data. Furthermore, on-going projects will continue to undergo regular peer reviews until their completion.

CONCLUSION

The Commission takes full note of the findings and recommendations of the final evaluation reports on the INFO2000 and MLIS programmes. In the light of the Commission's responses to these reports, it invites the European Parliament, the Council, the Economic and Social Committee and the Committee of the Regions to:

(1) Support the Commission in its approach of including digital content as part of the eEurope 2002 Action Plan in the light of the key role of digital content in supporting the economic and social development of European businesses and citizens.

(2) Support the Commission in positioning eContent programme as a key element of the eEurope 2002 Action Plan [19] endorsed by the European Council of Feira.

[19] http://europa.eu.int/comm/information_society/eeurope/actionplan/index_en.htm

(3) Support the Commission in its drive to simplify project administration procedures and thereby further encourage the participation of SMEs in the proposed eContent programme.


ANNEX

Final Evaluation of the INFO2000 Programme

Conducted for the European Commission

Executive Summary

Technopolis Ltd, UK

Databank Consulting SpA, Italy

IDATE, France

LENTIC, Belgium

July 17th 2000


Contents Page

Executive Summary

1 An abstract

2 Introduction and Methodology

3 INFO2000 in Context

3.1 Stimulating Demand and Raising Awareness

3.2 Exploiting Public Sector Information

3.3 Triggering Europe's Multimedia Potential

3.4 Support Actions

4 Evaluation Issues

4.1 Has INFO2000 Met its Objectives-

4.2 Was the 'Intervention Logic' behind the programme effective-

4.3 Was the programme managed and organised efficiently-

4.4 Rationale, synergy and sustainability

4.5 Did the programme meet the needs of its target audience-

4.6 Were the Interim Evaluation recommendations implemented-

5 The future and the e-Content programme

6 Recommendations


Executive Summary

This executive summary contains both an abstract for those readers wanting an exceptionally concise précis of the evaluation, and a fuller executive summary. The latter contains a description of the programme, followed by the responses to the questions posed by the evaluation.

1. An abstract

INFO2000 was a broad European programme of activities designed to stimulate the supply and use of multimedia. It had a budget of 65 MEURO, and ran for four years from 1996 to 1999.

The programme had four action lines:

- Stimulating demand and raising awareness

- Exploiting Europe's public sector information

- Triggering European multimedia potential

- Support actions

A network of centres was set up to disseminate information and co-ordinate activities and events. Known as MIDAS-NET, this network was active in 17 countries and collectively it conducted over 2,500 activities. The network nodes had varying degrees of impact, whilst some nodes provided an excellent level of service, others were less effective. The Commission supported the work of the nodes with the provision of a dedicated multimedia webserver - I*M EUROPE. In parallel, a competition - EuroPrix - was created to showcase the work of good quality multimedia companies. This proved to be a popular and high profile event, attracting large numbers of applications.

Obtaining improved exploitation of under-utilised public sector information was a key element of the programme. As a pan-European institution, the Commission was well-placed to foster debate on this issue, and to progress a European strategy. The formal outcome of the process was the publication and adoption of a Green Paper 'Public Sector Information: A Key Resource for Europe' in 1999. The creation of a number of projects to bring together public sector 'content owners', and private sector 'content developers' to work in partnership was a practical realisation of some of the conceptual frameworks contained in the Green Paper. In many cases, these partnerships were a significant achievement given the combination of weakly expressed demand, historical antipathies between some actors and the fragmented nature of the market.

The 'triggering European multimedia potential' action line provided an environment in which multimedia firms - particularly SMEs - could start to develop products and services, as well as their internal and networking and knowledge transfer capabilities. Participation in the programme was clearly seen as a step in the developmental process, rather than an occasion for the direct realisation of end products Nonetheless, it is also clear that the majority of projects were significant to the participating organisations, and without the programme they would not have been progressed at all. Overall project quality was good, with the two-phase application process enabling weaker projects to be terminated. A companion activity was the launch of projects designed as experimental 'testbeds' in which different approaches to the problem of trading multimedia clearance rights (ensuring that copyright holders are identified and recompensed for the re-use of their material) were trialled.

The support actions were varied They included a series of comparative studies on multimedia in each Member State, a number of global reference studies into multimedia and a number of pilot training and educational projects. There was also significant support for activities such as the Open Information Interchange which monitored global standards, the Legal Advisory Board which tracked legislative issues. These activities - addressing the framework issues underpinning the development of a European multimedia market - were key.

The benefits of the programme overall were significant. It raised awareness of multimedia via a network located in every Member State. It acted as a catalyst to the European multimedia industry by funding projects across a variety of sectors. It stimulated the debate on exploiting Europe's public sector information, leading to the creation and adoption of a Green Paper 'Public Sector Information: A Key Resource for Europe'. It also provided an environment for public and private sector actors to work together effectively, and contributed to progress on issues concerning European multimedia rights trading, and other legal and regulatory topics.

Problems encountered during the programme were relatively minor. They included payment delays and delays in awarding contracts. Networking and information exchange within and between projects could have been more effective.

Despite the progress made under INFO2000, further programme support needs to be given to developing a European multimedia capability, particularly in exploiting public sector information and helping firms overcome capital market barriers.

2. Introduction and Methodology

INFO2000 was a European Commission programme (running from January 1996 to December 1999 [20]) designed to stimulate the use of multimedia in Europe, and to support the development of a European multimedia industry capable of competing on a global scale. It sought to help create a multimedia industry able to satisfy the needs of Europe's enterprises and citizens for digital content accessed over electronic networks. The creation and supply of such content would lead to economic growth, competitiveness and employment and to individual professional, social and cultural development. There were four main Action Lines:

[20] Although some projects were started in 1999, and are still ongoing. The programme is therefore still 'live', which presents some problems in attempting to analyse the final quality of all of the outputs. However, as most projects are complete, it is still possible to gain a representative picture at this stage.

- Stimulating demand and raising awareness

- Exploiting Europe's public sector information

- Triggering European multimedia potential

- Support actions

This evaluation has been carried out in response to a requirement laid down in the Council Decision 96/339/EC adopting the programme. The terms of reference were to understand the extent to which:

- INFO 2000 was effective in meeting its objectives as outlined in the Council Decision 96/339/EC

- the 'Intervention Logic' behind the programme was effective

- the programme was managed and organised efficiently

- the programme's rationale, synergy and sustainability were aligned

- the programme met the needs of its target audience

- the recommendations in the Interim Evaluation have been implemented.

The evaluation was carried out over a 6 month period (January to June 2000) by a trans-national consortium led by Technopolis. It involved analysis of the documentation created during the life of the programme, structured interviews with Commission staff, project participants and industry observers, and a questionnaire sent to over 500 project participants.

3. INFO2000 in Context

When INFO2000 started there were few comparable national programmes in existence to stimulate the use of multimedia. The US led the world in the creation, use and distribution of electronic content and the Internet was in its infancy. At the European level action had already been taken to try to minimise the technology gap between the US and Europe - in the form of a number of political actions and communications [21], partially translated into activities via the predecessor programme to INFO2000 called IMPACT II. However, as the gap between the US and Europe was not closing, more activity was needed in order to develop the marketplace.

[21] Examples include the publication in 1994 of the European Commission's 'Europe's Way to the Information Society - an Action Plan', and the Council Resolution of 4th April 1995 on Culture and Media, outlining the importance of multimedia for facilitating the development of the content industry. There were also several high level discussions of the topic at the 1993 Brussels Summit and at the 1994 Corfu and Essen Summits.

The objectives of INFO200 were therefore to:

- Create favourable conditions for the development of the European multimedia content industry

- Stimulate demand for, and use of, multimedia content

- Contribute to the professional, social and cultural development of citizens

- Promote the exchange of knowledge between users and suppliers.

INFO2000 had a very broad remit. Its objectives required very different approaches to their resolution, hence the choice of four different Action Lines.

3.1 Stimulating Demand and Raising Awareness

The prime target audiences for INFO2000's awareness raising activities were industry (particularly Small and Medium size Enterprises, known as SMEs), and citizens accessed through libraries. The key mechanism used to achieve this was a network of Multimedia Information Demonstration and Support Nodes (MIDAS-NET). Each node comprised a consortium of organisations, which represented a cluster of multimedia specific knowledge and expertise. Their task was to create and to run a series of activities - hosting websites, Information Days, seminars, exhibitions, newsletters, roadshows - to inform audiences of the potentialities of multimedia. Over 2,500 actions were launched. In parallel, a competition known as Europrix was launched to showcase the work of good quality multimedia firms.

The rationale behind the creation of a network of awareness raising bodies was solid, although the performance of the MIDAS-NET nodes was too variable; some were excellent, others were poor. The EuroPrix event was well organised and effective.

3.2 Exploiting Public Sector Information

Public sector content holders were fragmented, and demand for the information they held, weakly expressed. The Commission was in a uniquely strong position of being a pan-European organisation with a remit to stimulate social and economic cohesion, able to address this market failure by developing policies to access and exploit public sector information in a systematic way.

Consequently, under the auspices of INFO2000, a Green Paper entitled Public Sector Information: A Key Resource for Europe [22] was adopted. This document set out the challenges for Europe - to overcome the fragmented nature of the market, where information is held in different formats, by different bodies, constrained by different national legislative approaches concerning its exploitation and use, whilst retaining confidentiality. The combined effect was to preclude users from making the best use of public sector information, and from allowing administrations to make policy and communicate as effectively as they might wish.

[22] COM (1998) 585 Public Sector Information: A Key Resource for Europe. A Green Paper on Public Sector Information in the Information Society.

To demonstrate how Public Sector Information might be used, INFO2000 also funded 26 projects. These included digitising photographic archives, facilitating access to information on business taxation legislation, providing multimedia information on air pollution and water regulation, promoting good health and providing tools to help map Europe geographically. The projects brought together content holders (public sector bodies) and content developers (usually either in the private sector or academics) to work together to develop solutions to common problems of inter-operability, harmonising data access policies, networking resources, taking account of cultural, linguistic and legislative differences and understanding marketplace demands.

Despite delays, this action line provided a tangible output to assist in policy making - the Green Paper - as well as funding a number of projects which enabled its concepts to be translated into practical applications. Bringing together hitherto disparate actors was a significant achievement, as was the creation of widespread public awareness of the potentialities of public sector data.

3.3 Triggering Europe's Multimedia Potential

A successful European multimedia industry can only be attained by the creation of an environment capable of stimulating sustainable creativity and technological development within firms. In order to do this INFO2000 encouraged participating organisations to form trans-national collaborations, involving academia, business and the public sector, from the whole of the supply chain from information providers to distributors. It was vital to attract SMEs - which the programme did very successfully - as the 'drivers' of the European economy.

Budgetary constraints meant that only four sectors could be chosen as the focus for the programme - cultural heritage, business information, geographic information, and scientific, medical or technological information. The outputs from the projects concentrated on CD-ROM as the delivery platform - the prevailing medium of the time. Today, most projects would routinely utilise the Internet; and e-commerce solutions would also be common.

In parallel, progress was also made in encouraging firms and other organisations to work together to find potential solutions to one of multimedia's regulatory problems - trading multimedia rights.

The main thrust of this action line concentrated on stimulating multimedia developments. It became clear from participants that the programme was significant in this respect - without INFO2000 the majority of projects (despite their strategic importance) would not have progressed. The informed support from the Commission was valued, and the impact on the participants own technological, employment and competitive standing was clear. It was also evident, however, that payment procedures and delays were unacceptable.

In terms of trading multimedia clearance rights the programme achieved its goals of contributing to the alleviation of the problem, by enabling different approaches to be piloted across a number of sectors including music, audio-visual and book publishing.

3.4 Support Actions

The final activity of the programme was to continue the development of the supporting framework around the multimedia industry. Detailed studies were conducted into the status of the European multimedia industry in terms of technological and societal developments. The Open Information Interchange (OII) was created to collect data on global standards and inter-operability activities and bodies. The Legal Advisory Board Work contributed to regulatory issues, in particular the Internet Action Plan. Designed to provide a safe environment for citizens and businesses, it was formally adopted on 25th January 1999. Work was also carried out into computer crime, in preparation for a future Commission Communication. There were also projects aimed at developing educational and training programmes that could be transferred to the wider environment via other Community programmes such as SOCRATES and LEONARDO.

These 'framework issues' which underpinned the development of the European multimedia industry during a time of great change were extremely important - indeed they continue to be so. The wide ranging goals of these activities were mainly achieved - with the partial exception of the delayed studies on multimedia developments in the Member States.

4. Evaluation Issues

4.1 Has INFO2000 Met its Objectives-

The programme largely reached its goals, making a contribution to reaching the overall objective of fostering a competitive European multimedia industry.

The construction of the programme created favourable conditions for the development of the European multimedia industry. By covering a wide range of issues from building technological capabilities and skills capacities within firms to launching a European debate on the future of Public Sector Information and investigating the complexities of trading multimedia rights, the programme illuminated all the key structural elements of the industry. The only issue which was not covered was the perennial problem of access to finance - and it is proposed that this should be tackled in the follow-on programme to INFO2000.

The programme stimulated demand by informing a wide range of actors. By using a multiplier approach (the MIDAS-NET nodes) INFO2000 reached a wide audience - building a 'network of networks'. It is not possible to draw a causal relationship between the existence of the MIDAS-NET network and increased demand for, and use of, multimedia. However, the fact that over 2,500 actions were begun as a result of the network indicates that many actors did become aware of the potentialities of multimedia. The EuroPrix events also reached large audiences, showcasing the best multimedia products available in Europe.

Recognising that demand was likely to be weak in Less Favoured Regions, the Commission offered increased project funding levels for Objective 1 Status regions. This had a patchy effect, and overall we cannot conclude that the programme particularly stimulated demand in the Less Favoured Regions of Europe.

By offering increased funding levels the Commission also sought to encourage SMEs to participate in the programme, as they are the drivers of multimedia expansion in Europe. Here the programme was effective and successful, with high participation rates amongst SMEs -43% of participants in Action Lines 2 and 3 were SMEs.

The programme contributed to the professional, social and cultural development of citizens. Individual projects have addressed these needs, but a greater influence of this type will be felt as a result of the publication of the Green Paper Public Sector Information: A Key Resource for Europe. This has the potential to influence the quality of life of a much greater proportion of the European population.

The programme also addressed the needs of new media professionals with a series of projects aimed at developing the multimedia experts of the future. By targeting efforts on schools, colleges and Universities the programme sought to shape and deepen the pool of intellectual capital of the future. And by creating a mechanism whereby pilot activities could be transferred to larger framework programmes such as LEONARDO or SOCRATES, the impacts will be felt by larger numbers of future employees and entrepreneurs. This will to help improve an inherent structural weakness in the market - skills shortages.

The programme promoted the exchange of knowledge between users and suppliers. Participation in projects brought together public and private sector actors, encouraging them to work together and to share expertise. Networking goals were very important to participants and the benefits were widely felt. Participants also acknowledged that INFO2000 represented an excellent opportunity to enhance their knowledge base, investigate new technologies and develop, evaluate and improve their tools and techniques. New partnerships were forged, and existing ones strengthened. In addition, INFO2000 was successful at bringing together disparate groups within the same sector, encouraging them to work together rather than working in opposition to each other - this was particularly noticeable in projects employing Geographical Information Systems (GIS).

4.2 Was the 'Intervention Logic' behind the programme effective-

'Intervention Logic' is a term which is used to describe the relationship between the programme and the resources it uses, and the outputs and impacts which it achieved.

INFO2000 suffered significant budgetary cuts right at the start of the programme, losing approximately one third of its funding - the initial budget requested was 100 million Euro, the final allocated budget was 65 million Euro. It had always been an ambitious programme, attempting to cover a wide range of issues and problems; the reduction in funds made the attempt to fulfil these expectations even more difficult.

There is little doubt that had the budget been larger, the impact of parts of the programme - particularly Action Line 3 - would have been greater. Although good quality outputs were produced from the projects, these were on too small a scale to achieve critical mass or to catalyse the development of the market significantly, especially as the projects were spread across heterogeneous market segments. Increased funding for the public sector projects, particularly those involving Geographic Information Systems, with their acknowledged deep seated problems in terms of metadata collection, could have resulted in some major European demonstrator projects - the types of achievement which could not be realised by national agencies alone.

It is less clear that greater funding would have benefited the awareness raising or support activities to the same extent, without greater specialisation and focus.

Overall, the choice of the four Action Lines was sensible at the time, given the breadth of issues any content-specific programme had to address - the newly emerging nature of the market, the technologies, the regulatory issues, the newness of the concepts, and even the players themselves. It was appropriate to devote the majority of funds to the projects to trigger European multimedia potential, despite the problems introduced by heterogeneity in the marketplace. This was the action line where most effect was going to be felt on industry players, who were the catalysts for increased take-up and demand in the field.

4.3 Was the programme managed and organised efficiently-

Overall the programme has been efficiently implemented. Project participants appreciated the level of knowledge and informed support provided by the experienced Project Officers and other staff. Procedures for proposal selection, evaluation, project management and information dissemination were transparent, rigorous, uniformly applied, and generally well accepted. There was one area of inefficiency, however, concerning the Commission payment procedures. This needs to be improved, especially in a market oriented programme which depends on attracting good quality firms for its success.

The two stage project funding mechanisms used in the selection of many of the projects was not the most efficient way of funding projects (there were increased costs in terms of evaluators fees and time), but it was effective. It enabled many more projects to be funded in the first instance (as the funding ceiling was quite low, 80 of them could receive funding), and then provided a mechanism to channel extra funds into a smaller set (29) of the best quality projects. This enabled experimental and pilot phase ideas to be 'market tested' first, and only those with the right mix of technological feasibility and market attraction to be developed further.

4.4 Rationale, synergy and sustainability

The rationale for the programme was proven - there were few comparable programmes in existence at the time of the creation of INFO2000, and the 'technology gap' between the US and Europe showed no signs of closing.

During the lifetime of INFO2000 there was a 'sister programme' called the Multi-Lingual Information Society (MLIS) and a number of other more distantly related, albeit complementary programmes. Although there has been no area of overlap or duplication of effort, neither has there been much meaningful interaction between them, especially at the project level. Synergy has not, therefore, been maximised.

There is still a need to continue the work begun by INFO2000. Many of the underlying aspects of the Council Decision setting out the objectives for INFO2000 have not changed. There is still a need to drive forward the creation and exploitation of the electronic content industries as a prime creator of jobs and economic wealth and as a way of supporting cohesion and social inclusion goals.

To achieve this there is still a need to invest time, expertise, and resources in developing frameworks to cope with IPR and trading rights issues, problems which are difficult to solve at a European level as national priorities are not aligned.

There is also still a need to stimulate the public sector, which is only now beginning to respond to increasing market demands. These are often motivated by national priorities for social inclusion and cohesion (perhaps using Structural Funds) which set out to promote easy access to public authority services. These often then have a demonstrator or multiplier effect into other areas of public life, such as education, libraries or leisure services.

Other areas of priority focus have changed. As well as skilled employees, new media firms need adequate financing - for the purchase of professional quality capital equipment, premises, marketing and start-up costs. Despite the recent upturn in interest in New Technology Based Firms by venture capitalists, new media funds, and business angels (led by the example of the US and the NASDAQ market) some institutional investors, and traditional finance providers remain sceptical. It can still be difficult to convince them to participate in what remains a risky marketplace.

4.5 Did the programme meet the needs of its target audience-

Given that the target audience was exceptionally broad - the public sector, citizens, SMEs and policymakers - we can only realistically say that the programme has partially met the needs of the constituent audience. There is still work to be done; and there is a need for a follow-on programme.

Successful awareness raising was variable amongst the MIDAS-NET nodes, although EuroPrix was a very targeted event, with a clear remit. It was widely publicised and can seen to have met the needs of participating organisations and its sponsors. The publication of the Green Paper was widely commented upon, and raised awareness amongst all target groups, particularly the public sector and policymakers. The support activities helped inform policymakers primarily, although businesses have also benefited. The projects have raised awareness and developed competencies amongst their participants. In some cases they have also developed a common approach amongst some constituent groups - most prominently amongst GIS actors.

Dissemination of results and information has been exemplary at one level - the publication of INFO2000 data on the I*M webserver, has been full and transparent, including the results of evaluations. There has also been publication of numerous publicity documents and study materials. However, there has been less effective dissemination of project results and very little sharing of results, methodologies and solutions to common problems between projects.

4.6. Were the Interim Evaluation recommendations implemented-

There were six strategic recommendations, and they have all been implemented to a satisfactory or higher standard:

- Creating a vision for the future

The Commission has shown leadership in the hitherto under-developed area of exploiting public sector information, and has begun to achieve some vision as a key supporter of the development of a European multimedia market by bringing some of the market actors together. It has also led discussion and experimentation concerning the framework conditions - standards, regulations, inter-operability, and skills - necessary to stimulate the marketplace in the longer term.

- Addressing convergence (technological and programme level)

There was some criticism by project evaluators that the projects failed to capitalise on the opportunities offered by the Internet and e-commerce. However, these platforms and applications were still emerging during the life of INFO2000. Bearing in mind the complexity of the other issues facing the projects, this is perhaps an unfair criticism. During the next programming period multiple delivery platforms will be the norm, multimedia actors will be more 'mature' and e-commerce will have become more widely used. Provided that the requirements to use advanced applications are clearly signposted in the Terms of Reference for new projects, then the likelihood of them being successfully implemented is much greater.

In terms of greater convergence and synergy with other Commission programmes, it is proposed that the follow-on programme will bring together the two most closely aligned programmes - MLIS and INFO2000.

- Creating the knowledgeable digital society

In terms of creating 'balanced networks' of participants in projects INFO2000 has been successful. It has attracted both new participants, and more experienced ones. Public and private partnerships have been built and knowledge and technology transfers have been key features of the programme.

- Easier access to seed and compounded funding

During the evaluation, multimedia firms cited lack of access to capital as a serious barrier to the development of their activities. INFO2000 sought to alleviate this by virtue of its very existence, and by the fact that participants have formed alliances and relationships which will enable them to pursue other funding in the future. The follow-on programme will seek to do this in more applied and focused ways - as it is proposed that facilitating access to capital should become one of its central themes.

- Better user targeting and awareness

Increased targeting of user communities did occur during the programme, but it was problematic. Although the better quality MIDAS-NET nodes attempted to network with existing information providers (particularly business support organisations) and to identify particularly important groups, it sometimes proved difficult to prioritise or group messages according to user needs.

As there is no awareness raising proposed in the follow on programme, this activity will not be continued there. The rationale for this is that most people are 'already aware of multimedia'. Whilst this is not strictly true - disadvantaged groups such as the elderly or the unemployed, and schoolchildren are not particularly aware for example - we accept that the Commission is seeking to prioritise its resources to develop the multimedia industry.

5. The future and the e-Content programme

The Commission's recently announced e-Europe: An Information Society for All initiative sets out the current and future European agenda for the further stimulation and growth of the Information Society. Its main action lines target faster, cheaper and secure Internet access, greater investment in people and skills, and measures to stimulate Internet use, including digital content, e-commerce, access to electronic government on-line and public sector services, including healthcare and transport. Therefore, there is no doubt that there will continue to be a huge requirement for high quality content, and it is entirely appropriate that the Commission continues to sponsor a programme to support and stimulate its production.

However, as the concept of electronic content and the use of multimedia tools become more accepted and exploited, it is no longer appropriate for the Commission to effectively 'subsidise' a small number of projects in four particular domains. What is appropriate, however, is for the Commission to take a lead in supporting issues of significant European-level importance, or issues where it is not clear there is going to be a market driven solution.

The follow-on programme to INFO2000, entitled e-Content, has three proposed action lines:

1. Exploiting public sector information

2. Enhancing linguistic and cultural customisation

3. Supporting market enablers

Continuing the exploitation of public sector information is an entirely appropriate, indeed essential, activity. This is a clear example of some form of market failure as although there is latent demand for public sector services, the current active demand is still relatively weak. This is compounded by the fact that information is held in different formats by different institutions, which have different policy agendas and different regulatory and legislative conditions. The key to unlocking the power of this information will be partially determined by the cultural attitudes of the public sector information holders and partly by the market awareness and skills of the private sector. The former is concerned with privacy and security issues, the latter needs to make money out of exploiting the data.

A homogenous, partnership-based approach to solving these problems has only just begun to occur at the European level. Whilst these cultural and cross-border issues will not be totally solved by the follow-up programme to INFO2000, even their partial resolution will represent a forward move, and will show some level of demonstrable benefit to European businesses and citizens.

The second proposed action line - stimulating the production of multi-lingual content is somewhat outside the scope of this evaluation. Nevertheless there are clearly issues which need to be resolved (both technical and cultural) in order to give European citizens and business access to sufficient volumes of good quality multi-lingual data, so that they are able to take full advantage of the opportunities offered by the Information Society.

The market enablers proposed in this case are access to finance and trading rights. Industry observers, programme participants themselves, and our own evaluative work on multimedia clusters all indicate that access to finance is a major barrier to developing firms. It is also clear that continuing to develop solutions to the trading of multimedia rights is appropriate.

However, the Commission must not lose sight of the fact that an equally important consideration is the capabilities of the multimedia firms themselves. These are not solely concentrated on the technical abilities of firms, but encompass equally marketing, management and financial abilities. Bearing in mind the majority of multimedia firms are microfirms (with less than 10 staff) or small firms many of them are unlikely to possess the needed breadth and depth of skill or capacity, and will find it correspondingly difficult to impress potential investors.

6. Recommendations

We endorse the selection and directions of the themes for the follow-up programme and recommend their acceptance. In addition, there are two other areas worthy of consideration. One has potentially global repercussions, the second is issue which is consistently under-addressed in EU programmes.

The first area which ought to be addressed in the follow-up programme is legislation, regulation and standardisation. These problems are not confined to Europe, nor are they going to be solved quickly or totally by a follow-on programme to INFO2000. However, they remain a significant barrier to the construction of a single European marketplace, and have not been markedly diminished yet. There is more useful work for the Commission to do in facilitating the debate by funding studies or organising High Level Groups or other networks. It can also usefully continue to provide assistance for experimentation and piloting of different approaches, and for monitoring developments. There should, therefore, be a continuation and development of the current programme activities in this area.

Finally, the future programme should address, with adequate funds, issues of inter-project and inter-programme transference of information, knowledge and skills. Project participants frequently face common issues and problems, but mechanisms allowing them to learn from each other, and to avoid duplication of effort, are all too infrequent.


Final Evaluation of the Multi-Lingual Information Society Programme (MLIS)

Final Report, June 2000

// ECOTEC

Research & Consulting Limited

13B Avenue de Tervuren

B-1040 Brussels

BELGIUM

Tel: +32 (02) 743 89 49

Fax: +44 (02) 732 71 11

Web address www.ecotec.com


Executive Summary

The Evaluation

This report presents the findings of the final evaluation of the Multilingual Information Society Programme (MLIS). The evaluation was undertaken between December 1999 and June 2000 by ECOTEC Research and Consulting Ltd on behalf of the European Commission and under the guidance of a Steering Committee appointed by the Commission.

The evaluation assessed:

- the impact and effectiveness of MLIS in reaching its objectives as outlined in the Council Decision of 1996 and subsequent Workplan;

- the effectiveness of the programme's 'Intervention Logic';

- the efficiency of the programme's organisation and management ;

- issues related to synergy, complementarity and sustainability.

The evaluation has involved a review of programme documentation, contact with all projects funded under the programme, case studies of completed projects and stakeholder interviews with those from European Institutions, the European Commission, the language industries, national organisations and information and communication technology (ICT) companies. The methodology is detailed in Section 2 of the report.

The MLIS Programme

The objectives of MLIS as specified in the 1996 Council Decision were:

- To raise awareness of and stimulate the provision of multilingual services in the Community, utilising language technologies, resources and standards.

- To create favourable conditions for the development of the language industries;

- To reduce the costs of information transfer among languages, in particular for the sake of Small and Medium sized Enterprises (SME) and to contribute to the promotion of the linguistic diversity of the Community.

MLIS was implemented through co-financing pilot projects and stimulating concertation and commissioning studies to improve understanding of the issues for the actors concerned. The budget for the MLIS was 15 million Euro. The programme was implemented over the period 1996 to 1999. In practice the progress of implementation accelerated towards the end of this period.

MLIS had three Action Lines:

Action Line 1 which supported the creation of a framework of services for European language resources through two sub-actions:

- Developing and electronic market place for language resources

- Developing a forum/network concerned with terminology

Fifteen projects have been supported under Action Line1 with EU funding of 3.9 million Euro.

Action Line 2 which encouraged the use of modern language technologies, resources and standards, through two sub-actions:

- Promoting multi-lingual services in businesses

- Demonstrating good practice in translation and interpretation

Fourteen projects with EU funding of 4.4 million Euro have been supported under Action Line 2.

Action Line 3 which promoted the use of advanced language tools in the Community and Member States' public sector, through two sub-actions:

- Advanced language tools showcase

- Co-operative multi-lingual public administration projects

Five projects with EU funding of 3.8 million Euro have been supported under Action Line 3.

In addition, the programme included a number of accompanying measures, specifically;

- Studies to assess the economic and social impact of multilingualism in Europe

- Production of demonstration materials, publications, and awareness raising activities

- International co-operation

These activities have received EU funding of 1.5 million Euro.

Overall, 35 projects received funding from MLIS. Of these, 13 projects had been completed at the start of the evaluation. A total of 189 organisations participated in MLIS projects. Of these; 37% were private enterprises; 26% were research/academic organisations in the public sector; 20% were not-for-profit organisations in the private sector and 9% were government bodies. The participation of private companies was highest under Action line 2, which primarily addressed the needs of the language industry. The average EU funding was 300 000 Euro per project. The average EU resources available to each partner was approximately 50 000 Euro.

The Impact and Effectiveness of MLIS

The results of the survey of all projects indicate that nearly all projects consider that they have worked (or are working) towards supporting multilingual diversity, promoting networks and co-operation and reducing the costs of information transfer among languages. However, given the broad objectives, the small number of projects completed and the resources available for each Action Line, the final impacts of MLIS relative to its objectives at the EU level are modest and so far incomplete. Also many of the major players involved with ICT developments that are affecting the language industries were not directly involved in the MLIS. The impacts have been considered with respect to each Action Line.

Action Line 1: Supporting the creation of a framework of services for European languages resources.

The Action Line has generated outputs that have increased the availability of language resources and enabled language resource holders to upgrade and digitise their language resources. These outputs include dictionaries and other language resources made available over the Internet. Two issues have however, constrained outputs and impact. First, technical aspects of implementation were often more complex and took more time than originally anticipated. Second, due to concerns over intellectual property rights (IPR), commercial entities such as publishers participating in MLIS projects were reluctant to progress from digital upgrade to online access. To some extent access remains limited to project participants.

Action Line 1 enabled co-operation at a European level on standardisation both for the production and the delivery of terminology. This has been achieved through the relevant projects bringing together terminology creators and users in order to develop a more user-oriented approach in both production and delivery. However, the weight of impact of the programme is small relative to the scale of activity in this field. There is a need for a terminology standards compatible with and applicable to language engineering solutions and further ICT developments. A consensus between terminology experts would help the successful adoption of such standards by the ICT industry worldwide. Such consensus was not achieved at the end of the MLIS programme, but the programme did stimulate co-operation between terminologists.

Action Line 1 also supported projects dealing with non-core languages, which have been successfully implemented. Those projects with a cultural and public policy rather than commercial rationale have been less constrained by IPR concerns.

Action Line 2: Encouraging the use of modern language technologies, resources and standards

The MLIS Workplan envisaged pilot projects that "...demonstrated successful practice in overcoming language barriers in trade and business". Although there are positive outcomes at the project level, Action Line 2 has generated few outputs and has only advanced the relevant objectives to a small extent. The Action Line has had little effect in influencing business strategies addressing multilingual markets in Europe and only a small impact on SME in particular. The Action Line was constrained by a number of factors:

- There was a lack of clarity over who were to be and the level of involvement of end-users of the demonstration products.

- There were a number of projects that were not clearly focussed on multilingualism per se, but on wider cultural differences corresponding poorly with MLIS overall objectives.

- Many of the projects were focussed on developing, rather than promoting multilingual services.

- The projects lacked sufficient user involvement from companies with successful experience in cross-language business practices.

- There was a shortfall of technical skills in some projects. Those projects that had strong technical partners and a good understanding of end-users were more likely to lead to a successful demonstration project and potential commercialisation.

- Few of the projects were sufficiently 'catalytic' in nature, and the resources were inadequate to ensure that demonstration effects could be disseminated.

With the benefit of hindsight a more specific sectoral focus for this Action Line may have increased its overall effectiveness.

Action Line 3: Promoting the Use of Advanced Language Tools in the Community and the Member States Public Sector

This Action Line contributed to part of the costs of the development of the system used by the Commission's Translation Service to aid translation. This was done in order that the benefits might also be made more widely available. The system has the potential, were it used more widely, to markedly decrease the costs of translation and hence increase the volume of texts translated into EU languages. The translation memory resources and the ease of use are especially important in this respect.

Progress in widening the system's application is anticipated through the four other projects within this Action Line which have begun during 2000. The project model being followed is to bring together a public administration and a technology provider. The emphasis is on languages where the supporting tools are not well developed. No impacts have so far arisen from these projects but they are likely to assist in extending the capabilities of the public sector to generate information transfer amongst languages at lower costs than hitherto.

There were delays in implementing Action Line 3 because of the lack of project proposals coming forward from the member states. There were two reasons for this; firstly, the project proposers had to be national administrations; and secondly there were delays in projects finding technical partners, as this process had to adhere to public procurement procedures.

Dissemination and Accompanying Measures

The MIDAS-NET network of nodes in each member state was established to serve as a dissemination service to the INFO2000 programme and was shared with MLIS in order to take advantage of potential economies and synergies. A number of the nodes produced dissemination material, held events and signposted language resources on the Internet. As a result they were effective in raising awareness of language issues and in encouraging businesses to adopt a language strategies. The MIDAS-NET did not focus on the language industries, there remains a need for improving awareness within parts of the industry of key ICT developments and the activities of key players in the field.

Trans-national co-ordination between the nodes was limited, and the small level of resources available constrained the impact of MIDAS-NET with respect to language issues.

The accompanying measures included an assessment of the economic and social impact of multilingualism. A study (referred to as ASSIM) was commissioned. This will provide data and insights into the key dynamics of the language industries. The work is due for completion in October 2000.

A website for MLIS was established in the later stages of the programme period. In the order of 10% of the programme resources were spent on these activities. Given the strong promotional objectives of MLIS there would have been merit in spending a greater proportion of the available resources in this way and in reducing the number of and budget allocation to projects.

The intervention logic of MLIS

The research revealed that several important and varied undercurrents influenced the policy formulation process in the period leading up to the MLIS Council Decision of November 1996:

- The developments and major investments made by the European Commission in machine translation had failed to deliver the expected results.

- There were economic costs associated with a multi-lingual EU and the strong prospect that the effective application of ICT could reduce these.

- There was concern over the threat to the language industries arising from ICT and at the same time some prospects that it could be a source of employment.

- There was concern that the increasing use of English in international communication would undermine the integrity of other major EU languages and impact on the use and availability of information in less widely spoken languages.

- The application of ICT was seen as having potential for improving access to information held by the public sector in languages other than those in which it already existed.

- There was considered to be benefit in the EU becoming involved with the application (rather than development) of ICT in this sphere so that a more direct impact on users could be attained.

- Resources were required to realise and spread the benefits from the previous EU investments in the application of ICT to language issues.

Thus there existed a number of rationales (economic, social cohesion, public service/ citizenship/cultural and political). These rationales remain valid but since the Council Decision there have been significant developments in the application of ICT to languages issues. The main basis for intervention at the EU level was in order to 'achieve substantial economies of scale and cohesion between the various language areas'.

The incorporation of such diverse concerns and interests into a single programme was a complex process. There was a debate over whether the appropriate legal basis should be economic or cultural and high level political involvement. On reflection it would have been better to have established the budget and then drafted the decision in light of the actual (very limited budget). This would have enabled more precise (and perhaps fewer) rationales, a stronger focus on what was best to do at EU level and the definition of objectives that could have been operationalised. A rigorous analysis of baseline conditions would have enabled a thorough ex ante appraisal, indicators etc.

Efficiency of Organisation and Management of MLIS

MLIS was pioneering and visionary and its aims were ambitious. Most activities funded within the programme were highly pertinent and progress has been made relative to all programme objectives. However implementation was slower than anticipated and there were a number of structural and procedural weaknesses.

First, the time and resources available to MLIS were constraints on the programme being able to generate its own good practice and demonstration projects. However, the bulk of the programme resources were allocated to projects in the expectation that they would generate impacts beyond the achievement of project level objectives.

Second, few of the programme resources were allocated directly to dissemination, animation and promotional activities. Only towards the end of the programme period was basic information about the programme and projects supported, such as the MLIS brochure, made available. The leverage of MIDAS-NET resources was limited and there would have been benefit in having a comprehensive communications strategy and sufficient resources within the EC to implement it.

Third, the projects were co-funded by the EU after open calls for proposals. The procedures and contracts used were similar to those used to support Research and Technology Development (RTD) projects. These arrangements are complex and not well suited to some of the MLIS target groups. The appraisal process judges proposals on their merits and the subsequent negotiations over budgets and project participants led to some dilution of the initial project proposals. At the same time the procedures may constrain the scope for the EC to proactively define and engage activities that closely match the requirements of the programme objectives. The arrangements also put a strong onus on reporting and co-ordination, which limits the time available from both EC officials and the projects to draw lessons from the projects. The varied nature of projects needed varied levels of co-financing. The non-routine character of projects and the difficulty of foreseeing technical costs suggest a strong need for good benchmark data on for example the cost effectiveness of methods of making language resources available digitally. The programme management of MLIS did not benefit from support of a Technical Assistance Office (TAO).

Fourth, the intermediate evaluation took place soon after the first round of projects had been engaged.

Fifth, MLIS tended to assume low priority next to the larger INFO2000 and did not have a separate management committee.

In the light of these observations there would be benefit in a more flexible project funding model oriented towards take up of language services (demand), that would allow for more emphasis on outputs rather than inputs at project level.

Synergy, Complementarity and Sustainability

All of the projects contacted reported a high degree of support and interest from EC officials. However, there was a widely held view that not enough concertation took place. At the same time, those that were involved in concertation considered that the small number and varied nature of projects meant that the value of such activity was limited.

There were some similarities between MLIS and the content of other EU programmes. Several observations can be made:

- There is a close relationship between MLIS and INFO2000. The programmes share the same management committee and much of the MLIS dissemination and promotion work takes place via the MIDAS-NET nodes established within INFO2000. There are particular similarities between Action line 3 of INFO2000 and MLIS Action Line 2.1.However; INFO2000 is much larger than MLIS (65 million Euro compared with 15 million Euro).

- There similarities between MLIS and the RTD 4FP Programme on Language Engineering (78 million Euro between 1994-1998) and the 5FP on Human Language Technologies (1999-2002). However, the concept of MLIS was that it should be user oriented and more 'downstream' than the RTD programme in the same field.

- There are several EU programmes pertinent to the wider cultural rationales of MLIS and related to new and innovative approaches to language learning.

- The EU has supported a large number of initiatives to assist EU companies, particularly SMEs, overcome barriers to international trade. It is not evident that the potential synergies with MLIS were exploited.

- There is a large number of EU supported programmes that share some objectives with MLIS. Whilst these programmes are in the main complementary, some of the MLIS supported activities might have been able to receive support from other programmes and some activities in those programmes might have been eligible for support from MLIS. Whether or not this is the case the interlinkages between MLIS and these programmes are complex and it is difficult to ascertain a clear overall picture of EU activities concerning language and ICT from the publicly available information.

In the light of these observations there would be merit in the EC establishing an Inter Service Group on ICT and language issues and publishing via a crosscutting Web site a guide to EU supported activities in this area. This work could lead to the establishment of an EU language policy framework.

Pointers for Future Policy

Each of the aims of MLIS remains relevant. However, the appropriate weight of emphasis that should be placed on each has been affected by some of the changes that have taken place since 1996. There are three main potential policy rationales:

- The application of ICT has and is having a profound influence in the field of multilingualism. ICT can reduce the costs of translations and the costs of access to information. In both these respects the progress during the period of the MLIS programme has been larger than in the preceding decades and to some extent unforeseen. Translation costs have been reduced by the application of machine translation, translation memory and controlled language; the Internet in particular has enabled wider and faster access to information. If EU economic actors on the demand and supply sides of multilingual services do not stay at the forefront of these changes the EU will be disadvantaged.

- The optimal application of ICT for the benefit of multilingualism raises new technical, legal and co-ordination issues. Many of these need to be resolved through trans-national working and co-operation. Some issues are best addressed at the EU level.

- The application of ICT has the potential to increase and reduce the costs of access in one Member State to public sector information originating in another. This has in turn the potential to realise economic, cultural and political benefits and a strengthening of the EU.

Plans for a follow up to aspects of MLIS are already underway. For example, studies have been commissioned of development in the digital content and language sectors and the preparatory phase of e-content programme has been launched. There is virtue in linking the issues concerning language and ICT to the wider concerns over digital content development and ICT. Multilingual content is one aspect of digital content. However, multilingualism within the EU has a special significance because of the strong nexus of economic, cultural and political issues involved. In the light of the evaluation findings there is a strong case for incorporating the following priorities in future programmes:

- Priority 1 The need to stimulate demand for language services applying ICT. It is especially important to find ways to support SME in defining and implementing language strategies. Identifying good practice and stimulating interaction between resource holders, users and tool developers can assist this.

- Priority 2 The need for trans-European infrastructure to address legal and technical issues, for example, IPR, standards, terminology etc

- Priority 3 The need to encourage the localisation of the Internet.

- Priority 4 The need to improve multilingual access to public sector information.

Main recommendations

Policy Framework

* Recommendation: An EU language policy framework should be established.

* Recommendation: In considering future programmes emphasis should be given to stimulating demand for and take up of language services.

Policy Measures

* Recommendation: Greater emphasis should be given to measures concerning dissemination awareness raising and good practice, stimulating demand and catalytic role of the EU.

Relationship with other EU programmes

* Recommendation: The EC establishes an Inter Service Group on ICT and language issues and publishes via a crosscutting Web site a guide to EU supported activities in this area.

Design of the Programme

* Recommendation: Any follow up programme should include as part of the preparation work ex-ante assessments concerned with the problematique and the rationale for EU intervention, more closely defined and measurable objectives and appropriate indicators of progress.

* Recommendation: Given that many of the MLIS projects are not yet complete and are likely to generate interesting results, mechanisms should be put in place to ensure that lessons can be derived and disseminated

Procedures

* Recommendations: There would be benefit in a more flexible project funding model oriented towards take up of language services (demand), and that would allow for more emphasis on outputs rather than inputs at project level.

Funding and Resources

* Recommendation: Funding needs to be determined before details of any future Council Decision so that ambitions are commensurate with resources.

* Recommendation: Benchmarks should be established for assessing value for money at the project level.