Annexes to COM(2009)695 - Report on the Ex-post Evaluation of the Erasmus Mundus Programme 2004-2008 - Main contents
Please note
This page contains a limited version of this dossier in the EU Monitor.
| dossier | COM(2009)695 - Report on the Ex-post Evaluation of the Erasmus Mundus Programme 2004-2008. |
|---|---|
| document | COM(2009)695  |
| date | December 22, 2009 |
REPORT FROM THE COMMISSION TO THE COUNCIL, THE EUROPEAN PARLIAMENT, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS
Report on the Ex-post Evaluation of the Erasmus Mundus Programme 2004-2008
(Text with EEA relevance)
1. INTRODUCTION
This report is presented under Article 12 of the Decision 2317/2003/EC[1] of 5 December 2003 establishing the Erasmus Mundus programme, which requires an ex-post evaluation of the programme to be carried out. It puts forward the Commission’s position on the main conclusions and recommendations of the ex-post evaluation that can be obtained via the link: http://ec.europa.eu/dgs/education_culture/evalreports/index_en.htm . These conclusions and recommendations are based on extensive surveys of Erasmus Mundus participants and key stakeholders.
2. Background to the external evaluation
Erasmus Mundus 2004-2008 was a co-operation and mobility programme in the field of higher education intended to promote the European Union as a centre of excellence in learning around the world. It aimed to support the development of top-quality European Masters Courses and to enhance the visibility and attractiveness of European higher education in third countries. The programme had, as its strategic aims, to improve the quality of higher education in Europe and to promote intercultural understanding through co-operation with third countries.
The specific aims of the programme were to: promote quality and excellence in European higher education; encourage the incoming mobility of third-country graduate students and scholars; foster structured co-operation with third-country higher education institutions; and improve the profile, visibility and accessibility of European higher education in the world.
The 2004-2008 programme comprised four main Actions. These were: Action 1 - Erasmus Mundus Masters Courses, integrated courses at masters level offered by at least three universities in three different European countries; Action 2 - Erasmus Mundus scholarships for students and scholars from third countries; Action 3 - Partnerships with higher education institutions in third countries, including scholarships for students and scholars from EU countries for mobility towards third countries; Action 4 - Projects to enhance the worldwide attractiveness of European higher education.
3. The external evaluation
3.1. The terms of the evaluation
ECOTEC Research and Consulting were requested to carry out the evaluation under their framework contract with the Commission[2]. The scope of the ex-post evaluation was the period 2004-2008, during which a number of calls for proposals took place to implement the programme, with particular attention being paid to the period since the interim evaluation of the programme in 2006-2007. While Erasmus Mundus Masters Courses, scholarships and “attractiveness projects” (Actions 1, 2 and 4) commenced in the academic year 2004-2005, Partnerships (Action 3) began one year later in 2005-2006.
The primary objectives of the evaluation were to assess the impact of the first Erasmus Mundus programme and to reflect on lessons learned to support implementation of the successor programme. The focus of the evaluation was thus on the "effectiveness" of the Erasmus Mundus programme - the extent to which the programme has achieved the objectives established for it at the outset. Alongside this consideration of short- and longer-term effects, the evaluation also analysed the "efficiency" with which Erasmus Mundus achieved these effects[3] and the extent to which the courses it supported are sustainable in the longer term.
3.2. Methodology
The methodology included desk research of relevant literature; a survey of institutions participating in Erasmus Mundus Masters Courses, Partnerships and attractiveness projects (Actions 1, 3 and 4), and of participating third-country and EU students and scholars; and interviews with key stakeholders, including the Erasmus Mundus National Structures and participants in the four Actions of the programme. Case study visits covering twelve EMMCs were carried out. Triangulation and cross-verification of the collected evidence were present throughout the analysis.
3.3. The evaluator’s findings
The balance of evidence collected for the ex-post evaluation of Erasmus Mundus suggests that the programme has been effective in achieving its stated objectives and has created a significant Community added value. Erasmus Mundus has succeeded in bringing together some of the best higher education institutions in the EU to offer 103 new and innovative joint masters programmes, which were unlikely to have been created without the programme. These masters programmes are considered to be of high quality by both the academic staff and current and former Erasmus Mundus students consulted during the evaluation and have generally managed to attract large numbers of applications from third-country students.
3.3.1. Quality of Erasmus Mundus courses
The evidence from the evaluation suggests that the Masters Courses supported by the programme have indeed been of high quality. The excellence of the institutions involved, and that of their staff, remain a fundamental guarantor of academic quality. While the geographical distribution of participating institutions was broadly balanced, institutions from the new Member States remained proportionally under-represented.
The high academic level and content of courses was the characteristic most frequently perceived by course co-ordinators to add value over and above "mainstream" (domestic) masters courses in the same discipline. Course co-ordinators and graduates also perceived the mobility built into Erasmus Mundus courses and the "intercultural experience" they offer to be important components of their added value.
The general standard of support to students and academic facilities was rated highly by students. However, the evaluation suggests that, in some cases, more could be done to improve coherence between the curricula taught at different participating institutions and the different training paths offered to students. More structured formats, with common courses and a more limited number of study tracks, can facilitate course integration.
3.3.2. Quality of Erasmus Mundus students and scholars
With a limited number of exceptions, Erasmus Mundus courses have been successful in attracting high numbers of applicants from third countries. However, courses have frequently encountered difficulties in recruiting the expected numbers of EU students. This was often because of the level of fees charged by the courses (which could be higher than for "national" programmes) and the absence of scholarships for EU students in the 2004-2008 programme.
The evidence from the evaluation suggests that the overall academic standard of the students selected for Erasmus Mundus courses is high. Fifty-five per cent of co-ordinators and partners replying to the online survey stated that the average academic standard of Erasmus Mundus students was "significantly above" the average of masters-level students in their departments.
The use of additional funding for the targeted selection of students from specific geographical areas and countries as part of Action 2 (the "Windows") was not always consistent with the primary focus of the programme on academic excellence. In a limited number of cases, there is evidence that the primarily geographical criterion introduced by the "Windows" led to the selection of students of a below average standard.
The evidence from the evaluation suggests that the overall academic standard of scholars supported through Action 2 was high. However, over 75% of scholars supported were men. This level appears to be disproportionate, even taking into account possible variation in the gender profile of academic staff in different disciplines.
3.3.3. Impact on the "supply side" of higher education in Europe
Erasmus Mundus has added to the pre-existing supply of masters-level education in the EU in both quantitative and qualitative terms. The new courses supported have generally been additional to the existing higher education offer, as "domestic" courses have been maintained in parallel. The programme has also enhanced overall quality by creating new, high-quality courses of a type rarely seen before the programme was established.
In addition to the direct impact of creating new courses, the evidence from the evaluation illustrates that Erasmus Mundus has brought additional, indirect impacts in participating departments and institutions. In particular, the process of establishing and implementing joint courses appears to have strengthened a twin process of "Europeanisation" on the supply side (as higher education institutions from different countries worked together) and "internationalisation" on the demand side (as institutions sought to attract more students from abroad and respond to their needs).
The evaluation found that Erasmus Mundus has made a generally positive contribution to the EU's strategic objectives in the field of higher education. In particular, the integrated, trans-national nature of the courses has required participating institutions to engage directly with the detail of applying aspects of the Bologna Process, most notably the European Credit Transfer System. It has also increased mutual awareness among the academic community of the characteristics and functioning of higher education systems in other Member States.
The experience of a number of courses illustrates that the issue of joint degrees (diplomas) remains an area where further work is required to remove obstacles in national legislation, which prevent the accreditation of joint degrees. The enhanced level of joint quality assurance and mutual recognition involved in joint degrees is likely to contribute further to the development of the European Higher Education Area.
3.3.4. Impact on the international demand for European higher education
There is evidence that Erasmus Mundus has led to an increase in the number of third-country students attending participating departments and institutions, although this pattern is not universally reported and it is possible that some third-country students who would have come to Europe in any case have opted for Erasmus Mundus courses. Consolidated data on the numbers of third-country students attending individual institutions were not available to the evaluation in any of the cases examined, although such data would be a valuable means to explore the quantitative effects of programmes such as Erasmus Mundus.
Many of the course representatives consulted during the evaluation noted that Erasmus Mundus has allowed their departments to attract third-country students from a wider range of countries and thus to break with traditional patterns of recruitment.
3.3.5. Impact on students and scholars
Evidence from the Graduate Impact Survey shows that Erasmus Mundus graduates generally believe they have gained valuable and relevant skills and experience from their courses. However, evidence about the long-term academic or career progression of graduates after they leave the course is, although broadly positive, currently inconclusive.
The results of the online survey of Action 2 scholars suggest that Erasmus Mundus has allowed these individuals to strengthen their own (research-related) networks and improve their knowledge and understanding of higher education in the EU. It appears that the personal impact of participating in the programme and the impact on host departments and courses is enhanced when scholars are directly involved in the delivery of courses - something that has not always been the case.
3.3.6. Sustainability
The evaluation has demonstrated that a majority of co-ordinators and partners currently participating in Erasmus Mundus do not believe the courses in which they are involved could continue in their present form without continued EU funding.
Coming from beneficiaries of European funding, these findings are perhaps not surprising, particularly in countries where the higher education systems face significant funding constraints. However, as the longer-term sustainability of Erasmus Mundus is likely to require a reduction in the level of EU funding for scholarships attached to individual courses, it is clear that solutions other than the "status quo" need to be found if many Erasmus Mundus courses are to continue.
In principle, the internationally excellent quality of courses should allow the best among them to recruit high-quality, self-financing students from third countries, particularly as the tradition of paying for high-quality education is frequently more established outside the EU than inside it. While Erasmus Mundus is likely to have supported many students who would otherwise not have been able to afford to study in Europe, it is equally likely that the programme has supported students who would have been able to pay at least part of the cost of their studies.
Although the process of diversifying student funding sources is already built into the design of the new Erasmus Mundus programme, the evaluation suggests that courses need to be made more aware of the need to improve self-financing and of alternative funding sources.
3.3.7. Programme design and structure
While the evaluation found that Action 1 Masters Courses and Action 2 scholarships for students were clearly complementary and worked effectively together, the evidence concerning the effectiveness of Action 2 scholarships for scholars, Action 3 Partnerships and Action 4 projects is less compelling.
The impact of funding for Action 2 scholars on the quality of courses was not always evident. It seems likely that the contribution of funding under this Action is enhanced when scholars are employed specifically to contribute to the course in question (mainly through teaching).
Although Action 3 in several cases added a valuable extra dimension to Erasmus Mundus courses, partnership activities have often been insufficiently integrated with the courses to which they are linked and the Action as a whole has attracted fewer participants than expected. This has been addressed in the new Erasmus Mundus programme, by making third-country institutions eligible to be full consortium partners.
The evaluators consider that, while Action 4 has supported some interesting and potentially effective projects, the funding for the Action was too thinly dispersed among many projects with differing objectives. Action 4 could have been more effective if resources had been concentrated on a more limited number of larger projects.
3.3.8. Programme management
The procedures for the management of Erasmus Mundus were found to have been appropriate and largely effective. In particular, programme beneficiaries have not reported any major difficulties with the programme's selection and monitoring procedures. The evaluators consider, however, that the quality of consolidated monitoring data for the programme could be improved without considerable additional effort. In particular, more complete data on student numbers and graduation/completion rates would be useful, as they are simple, but valuable, "result" indicators.
3.3.9. Programme efficiency
The evaluators consider the programme has achieved a generally high degree of efficiency. In particular, the comparatively modest sums allocated to Erasmus Mundus consortia to administer the courses appear to have had a leverage effect, as considerable additional resources (notably administrative staff time) have been devoted to their implementation.
In view of the high proportion of Erasmus Mundus students on individual courses receiving full scholarship funding, it seems likely that overall efficiency could be improved by making efforts to ensure that more students attend courses with support from other funding sources.
4. MAIN RECOMMENDATIONS OF THE EXTERNAL EVALUATION AND COMMENTS FROM THE COMMISSION
The main recommendations of the evaluator are presented in bold, while the Commission’s answer is in italics.
4.1. Effectiveness
Recommendation 1
The factors explaining the continued "under-representation" of institutions from the new Member States in the Erasmus Mundus programme should be explored further and, within the scope of EU competence in the field, appropriate action taken to address the issues identified.
The Commission agrees with this recommendation. In order to increase the participation of less-represented EU countries in the programme, the Commission is supporting a project under Action 3 (co-ordinated by the Slovak National Structure) that aims to spread information on how to participate and provide support to potential participants from new Member States. It will be possible to assess the impact of this project in the summer of 2010.
Recommendation 2
In the ongoing monitoring of Erasmus Mundus courses, particular attention should be paid to the level of integration in the curricula and study tracks offered.
The Commission welcomes this recommendation as a way of further promoting the importance of integration. In future, applicants should be asked to explain and demonstrate more clearly the level of integration of their course and evaluators of proposals should be asked to take particular care in the analysis of integration and mobility mechanisms.
Recommendation 3
While scholarships for EU students have been introduced under the new Erasmus Mundus programme to increase the proportion of EU students on Erasmus Mundus courses, the actual application rates and numbers of EU students selected should be monitored closely during programme implementation.
The Commission agrees with this recommendation and will closely monitor the application and selection rates of EU students, as it has done up to now with non-EU students. This information is essential in order to make the future implementation of the programme as effective as possible.
Recommendation 4
In the new Erasmus Mundus programme, consortia should be asked to pay greater attention to the gender balance among the scholars supported in the framework of their Masters Course.
The Commission agrees in principle with this recommendation and will make every effort to raise awareness of this issue among participants in the programme.
Recommendation 5
In order to monitor the impact of Erasmus Mundus on the numbers of high-quality third-country students coming to Europe, improved data are required. Participating institutions should therefore be asked to provide data on the overall numbers of third-country students they recruit to allow more systematic analysis of the impact of Erasmus Mundus in this respect.
The Commission welcomes this recommendation, which will be used to spur participating institutions to provide more timely and accurate statistical information on the numbers and performance of their students. This information should be provided via the existing student database (managed by the Executive Agency), which should be as easy to use as possible.
Recommendation 6
The Graduate Impact Survey should be continued in future years, making use of a more systematic, but shorter, questionnaire, tailored to measuring the longer-term impact of the programme on participating students.
The Commission agrees with this recommendation. The Graduate Impact Survey is a key tool for understanding and interpreting the past and future trends of the programme. Discussions are presently ongoing with the contractor in charge of the GIS to agree on a more effective questionnaire for students and graduates.
4.2. Sustainability
Recommendation 7
In order to ensure the sustainability of the best Erasmus Mundus courses, while maximising the impact generated by EU funding, Erasmus Mundus should move away from the current situation in which the vast majority of third-country students on Erasmus Mundus courses receive a full scholarship. Existing courses should be strongly encouraged to move to a more differentiated system of student support, with a higher proportion of students financed from sources other than Erasmus Mundus scholarships.
The Commission agrees with the need to ensure the sustainability of Erasmus Mundus courses and has put a strong accent on this in the first call of the new programme. However, the Commission deems it necessary to reflect further on the future of the scholarships system. For example, this may in future become separated from the support given to excellent courses, with scholarships attributed as a function of the courses' capacity to attract students.
Recommendation 8
Action 3 of the new Erasmus Mundus programme should take a more strategic approach than was the case under Action 4 of the previous programme, funding a limited number of projects in clearly-defined priority areas and encouraging "clustering" activities. Strengthened, targeted marketing of courses, particularly at elite higher education institutions in high- and middle-income third countries and in Europe, should be undertaken to boost (self-financing) demand for Erasmus Mundus courses.
The Commission shares the concern of the evaluators about the lack of a precise strategic approach in the definition of specific objectives under Action 3. This year, the Commission has decided to avoid an open call for proposals and launch, for the first time, a call for "clusters" to draw the best out of past Action 4 projects. Future calls for proposals will, moreover, focus on a limited number of priority areas. The Commission will also continue funding some actions from the Erasmus Mundus Global Promotion Project to support the promotion of European higher education worldwide.
4.3. Efficiency - programme design and management
Recommendation 9
In the context of the new Erasmus Mundus programme, visiting scholars should be required to contribute directly to the course for which they receive their Erasmus Mundus grant, in order to enhance the quality of the course in question.
The Commission considers this recommendation interesting. Up to now, scholarships for scholars were mainly intended to allow teaching or research activities related to the course. Directly contributing to the course may imply some previous participation of the scholars in the definition of the curriculum. Certain courses may already require a direct contribution, in which case this good practice could be made known to other consortia.
Recommendation 10
The progress and final reporting formats for Erasmus Mundus courses should be reviewed to ensure that complete data on student numbers (EU and third-country) and graduation rates are collected.
As with Recommendation 5, the Commission agrees that courses should be spurred to provide more timely and accurate statistical information on the numbers and performance of their students. In particular, graduation rates should be clearly stated to allow a more transparent analysis of the success of the course.
5. THE COMMISSION’S CONCLUSIONS
The Commission shares the overall assessment of the evaluators that Erasmus Mundus has made an important contribution to the internationalisation of European higher education. The programme remains relevant to the challenges facing European higher education institutions, particularly in the current global economic climate, and retains a high level of enthusiasm among co-ordinators, partners, students and scholars. The results of the evaluation show that Erasmus Mundus continues to meet its political and operational objectives, as well as the objectives of Article 149 of the Treaty.
The fundamental quality of Erasmus Mundus Masters Courses, of the institutions behind them and of the students and academics they attract, are essential to the future development and health of the programme. The Commission welcomes the positive findings of the evaluator in this respect and has given reinforced status to the role of Quality Assurance in the new Erasmus Mundus programme.
The new programme has been designed to overcome certain structural shortcomings identified in the 2004-2008 programme, offering scholarships for EU students and allowing third-country institutions to participate as full partners in joint programmes, for example. The Commission notes the findings of this evaluation as to where further improvements could be made, and will pay due respect to these over the remainder of the 2009-2013 programme.
STATISTICAL ANNEXES ON APPLICATION AND SELECTION FIGURES
EU + EFTA/EEA Countries | HEI instances of participation in EM Masters Courses applying to EM (1) | HEI instances of participation in EM Masters Courses selected under EM (1) | HEIs participating in EM Masters Courses selected under EM |
Austria | 32 | 7 | 4 |
Belgium | 133 | 21 | 7 |
Bulgaria | 2 | 0 | 0 |
Cyprus | 1 | 0 | 0 |
Czech Rep. | 56 | 9 | 4 |
Denmark | 72 | 14 | 7 |
Estonia | 25 | 3 | 2 |
Finland | 59 | 11 | 5 |
France | 465 | 65 | 49 |
Germany | 320 | 54 | 33 |
Greece | 43 | 5 | 4 |
Hungary | 74 | 10 | 4 |
Ireland | 44 | 6 | 5 |
Italy | 361 | 44 | 26 |
Latvia | 21 | 0 | 0 |
Lithuania | 35 | 2 | 2 |
Luxembourg | 7 | 1 | 1 |
Malta | 17 | 1 | 1 |
Netherlands | 162 | 29 | 14 |
Poland | 123 | 16 | 8 |
Portugal | 170 | 26 | 11 |
Romania | 14 | 0 | 0 |
Slovakia | 14 | 1 | 1 |
Slovenia | 24 | 3 | 2 |
Spain | 464 | 62 | 27 |
Sweden | 140 | 22 | 11 |
UK | 257 | 42 | 28 |
Iceland | 5 | 0 | 0 |
Liechtenstein | 0 | 0 | 0 |
Norway | 63 | 14 | 9 |
Total | 3195 | 468 | 265 |
EU + EFTA/EEA Countries | HEI instances of participation in EM Action 4 projects applying to EM (1) | HEI instances of participation in EM Action 4 projects selected under EM (1) | HEIs participating in EM Action 4 projects selected under EM |
Austria | 41 | 8 | 6 |
Belgium | 84 | 18 | 16 |
Bulgaria | 18 | 2 | 2 |
Cyprus | 5 | 2 | 2 |
Czech Rep. | 27 | 4 | 3 |
Denmark | 22 | 6 | 4 |
Estonia | 20 | 5 | 5 |
Finland | 53 | 17 | 12 |
France | 135 | 26 | 20 |
Germany | 108 | 16 | 10 |
Greece | 27 | 5 | 3 |
Hungary | 41 | 4 | 2 |
Ireland | 11 | 3 | 3 |
Italy | 144 | 15 | 12 |
Latvia | 11 | 3 | 2 |
Lithuania | 23 | 4 | 3 |
Luxembourg | 2 | 0 | 0 |
Malta | 9 | 3 | 1 |
Netherlands | 73 | 14 | 9 |
Poland | 60 | 12 | 9 |
Portugal | 55 | 10 | 7 |
Romania | 31 | 3 | 3 |
Slovakia | 30 | 6 | 3 |
Slovenia | 17 | 1 | 1 |
Spain | 133 | 32 | 18 |
Sweden | 51 | 11 | 8 |
UK | 99 | 22 | 15 |
Iceland | 2 | 2 | 1 |
Liechtenstein | 1 | 1 | 1 |
Norway | 24 | 11 | 9 |
Total | 1357 | 266 | 190 |
Number of student applications received | Erasmus Mundus scholarships awarded | “Window” scholarships awarded | Total number of scholarships awarded |
2004-05 | n.a. | 140 | 0 | 140 |
2005-06 | 3030 | 455 | 353 | 808 |
2006-07 | 5500 | 741 | 636 | 1377 |
2007-08 | 12 766 | 1196 | 629 | 1825 |
2008-09 | 18 820 | 1957 | 74 | 2031 |
Total | 40 116 | 4489 | 1692 | 6181 |
[1] OJ L 345 of 31.12.2003.
[2] Framework Contract for Evaluation, Evaluation Related Services and Support for Impact Assessment (EAC/03/06 - 1st Renewal).
[3] The extent to which the effects of the programme were achieved at reasonable cost.