Explanatory Memorandum to COM(2010)190 - Report on the External ex-post Evaluation of the 3rd Phase of the Tempus Programme 2000-2006 - Main contents
Please note
This page contains a limited version of this dossier in the EU Monitor.
| dossier | COM(2010)190 - Report on the External ex-post Evaluation of the 3rd Phase of the Tempus Programme 2000-2006. |
|---|---|
| source | COM(2010)190  |
| date | 28-04-2010 |
Report from the Commission to the Council, the European Parliament, the European Economic and Social Committee and the Committee of the Regions - Report on the External ex-post Evaluation of the 3rd Phase of the Tempus Programme 2000-2006 /* COM/2010/0190 final */
[afbeelding - zie origineel document] EUROPEAN COMMISSION
Contents
- Brussels, 28.4.2010
- 1. Introduction 3
- 2. Background 3
- 3. Brief description of the programme 4
- 4. The evaluation work 4
- 5. The evaluation conclusions 5
- 5.1. Relevance 5
- 5.2. Efficiency 5
- 5.3. Effectiveness 6
- 5.4. Impacts 7
- 5.5. Sustainability 7
- 6. Recommendations of the evaluators and Commission Comments 8
- 7. Conclusion 11
- BACKGROUND
- BRIEF DESCRIPTION OF THE PROGRAMME
- The evaluation work
- THE EVALUATION CONCLUSIONS
- Efficiency
- Effectiveness
- Impacts
- Sustainability
- RECOMMENDATIONS OF THE EVALUATORS AND COMMISSION COMMENTS
- Recommendation 5 - Improve the effectiveness of field monitoring
- Recommendation 6 - Give greater support to the NCPs in New Member States
- Recommendation 7 - Better define the involvement of the EU Delegations to improve the (structural) dialogue with the national authorities on higher education
- Recommendation 12 - Support additional ambitions of projects such as in the case of joint degrees or double diplomas
- CONCLUSION
COM(2010)190 final
REPORT FROM THE COMMISSION TO THE COUNCIL, THE EUROPEAN PARLIAMENT, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS
Report on the External ex-post Evaluation of the 3 rd Phase of the Tempus Programme 2000-2006
TABLE OF CONTENTS
Report on the External ex-post Evaluation of the 3 rd Phase of the Tempus Programme 2000-2006
REPORT FROM THE COMMISSION TO THE COUNCIL, THE EUROPEAN PARLIAMENT, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS
Report on the External ex-post Evaluation of the 3 rd Phase of the Tempus Programme 2000-2006
Introduction
This report is presented pursuant to article 12 of Council decision 1999/311/EC of 29 April 1999. It puts forward the Commission position on the main conclusions and recommendations of the external ex-post evaluation of the 3rd phase of the Tempus programme (Tempus III) which can be found at: ec.europa.eu/dgs/education_culture/evalreports .
The first phase of Tempus, adopted in 1990 i and initially covering the Central and Eastern European Countries (CEEC), sought to contribute to socio-economic reform through co-operation between higher education institutions in the EU and in the Partner Countries. It was designed to promote the joint development of new curricula, the acquisition of new management skills by academic and administrative staff and the opening up of Partner Countries’ educational systems to civil society.
The 2nd phase of the programme extended the geographical coverage of the programme and also increased its ambition and expectation levels. Specific national priorities were introduced which complemented the original 'bottom-up' approach whereby initiative rested exclusively with Universities. This evolution allowed the programme to accompany the changes in the political context and contribute to the process of accession of the CEEC to the European Union.
Two innovative aspects were then introduced to the 3rd phase of Tempus (Tempus III) in 2000 i.In addition to the tried-and-tested country-specific approaches, particular emphasis was placed on the programme’s capacity to encourage regional co-operation and more explicit reference was made to the need to ensure consistency and complementarity with other Community programmes, and also to create synergies with other forms of assistance to the partner countries.
With the extension i of Tempus III to the Southern and Eastern Mediterranean countries, the programme also sought to promote inter-cultural dialogue and understanding as a means to secure sustainable growth, peace and stability and reinforce the intercultural and civil society dimension of the programme.
Tempus III came to an end in December 2006 i. The fourth phase of the programme began in 2007 and has been managed by the Education, Audiovisual and Culture Executive Agency since April 2009.
With a view to maximise the impact on reform processes, Tempus III combines a bottom-up approach, whereby initiative is left mainly to the universities, with a top-down approach, whereby national priorities are established for each partner country.
To achieve these objectives, Tempus III supported three types of projects:
- Joint European Projects (JEPs): multilateral projects designed to support efforts to develop and upgrade curriculum, retrain teachers, develop modern teaching and learning material, improve university management and contribute to building up the institutional tissue in the partner countries.
- Structural and Complementary Measures (SCMs): multilateral projects designed to support short term interventions aimed at system reform and linked closely to partner countries’ priorities.
- Individual Mobility Grants (IMGs): awarded to individuals to travel for work related to a particular reform process and academic cooperation.
Beneficiary partner countries under Tempus are:
- Albania, Bosnia and Herzegovina, Croatia, the former Yugoslav Republic of Macedonia, Montenegro, Kosovo under UNSC Resolution 1244, Serbia.
- Armenia, Azerbaijan, Belarus, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Mongolia, the Russian Federation, Tajikistan, Turkmenistan, Ukraine and Uzbekistan.
- Algeria, Egypt, Israel[5], Jordan, Lebanon, Morocco, Occupied Palestinian Territory, Syria and Tunisia.
The external ex-post evaluation aimed to gain more insight in the Tempus III programme, to provide conclusions and concrete recommendations on how the current phase could be managed to better address any identified weaknesses and to maximise its relevance and impact, as well as exploit examples of good practice.
Relevance
TEMPUS III remained relevant with respect to its global, specific as well as its operational objectives. This is due to the fact that socio-economic and democratic transformation processes are still ongoing, in different forms, and the quality of higher education remains an important issue across partner countries. The degree of relevance of each specific objective did however differ from country to country depending on their involvement in the Bologna process and/or whether they had significant national-level higher education reform initiatives underway.
The relevance of TEMPUS III was also high because of the bottom-up character of the programme, allowing the formulation of national priorities and taking into account the different stages and needs of the transformation process. In most if not all countries the TEMPUS programme fitted very well into national policies, although in some cases missing a clear strategic goal orientation.
The overall mixture of programme approaches was a relevant means of achieving the programme’s expected results, outcomes and impacts and with this variety in actions TEMPUS achieved its specific objectives, although this was more clear for objectives regarding curricula reform or management training for instance than for strengthening democracy, which is particularly difficult to assess.
Aside from its relation to the Bologna process, which further strengthened the relevance of the programme, and the European higher education modernisation agenda, TEMPUS also in other respects was not an isolated cooperation programme. It had a clear role in other EU policies, although the evaluation showed that this was open to improvement, in particular with regards to the complementary role between TEMPUS and Erasmus Mundus when the latter was launched in 2004 and which gained special relevance when the IMGs were removed from TEMPUS. It was also clear from the objectives of the Instrument for Pre-accession Assistance and the European Neighbourhood and Partnership Instrument that TEMPUS can play a distinct role, for instance regarding strengthening (higher education) institutions and promoting understanding between and rapprochement of cultures and human resources development, although its specific role appeared to be underexposed.
Aside from its role to the EU external policy programmes, TEMPUS can be claimed to have a complementary function to the Sixth Framework Programme for promoting European research and technological development. Complementarities can be found e.g. with the Marie Curie Actions, that are open to the nationals of third countries willing to work in Europe and to public and private/commercial organisations active in research willing to join Marie Curie international consortia. For the relation between these two programmes, the evaluation concluded that if academic education and academic research are to benefit to the full of each other, the complementarity or expected synergy between education reform programmes should get more attention than it received up till now.
In general the funding levels for TEMPUS III were considered too low for its ambitious objectives. Considering the large challenges in the eligible countries, the fact that the total budget was spent and that funding levels have been insufficient to fund all relevant and quality proposals was considered a factor for under-funding. The funding levels for the projects in general appeared to be sufficient.
TEMPUS was perceived as a cost-effective programme compared to (technical assistance) cooperation programmes and its' multilateral nature made the policy instrument in essence cost-effective, although the degree differed per instrument.
Respondents were generally satisfied with the rigour of the selection process. Their only main concern was the perceived strong technical focus where the weight given to the national needs and priorities on the ‘ground’ could have been increased.
The monitoring process improved considerably since the mid-term evaluation and the only remaining weakness identified concerned the limited feedback (to all consortium members) on field monitoring.
The National Tempus Offices (NTOs), National Contact Points (NCPs) and EU Delegations fulfilled supportive roles to the programme and its participants. The evaluation however showed that the way in which these actors fulfilled their respective roles varied considerably in the different countries and left room for improvement.
TEMPUS III made an important overall contribution to the development of higher education systems, including to the five specific action lines of the ‘Bologna Process’. This achievement included ‘core’ higher education goals such as curriculum development, development of comparable and compatible degrees organised in a three-cycle structure, quality assurance, recognition of foreign degrees, reform of higher education structures and establishments as well as development of skills profiles – although more so at the level of individual institutions than at the national policy level, where the duration and possibilities offered by SCMs and the inaccessibility of senior policy-level actors sometimes resulted in more indirect effects.
Depending on the actual content of projects and the characteristics of higher education systems TEMPUS also contributed to a broader societal relevance of higher education, for instance by establishing or improving the links between higher education, enterprises, research and innovation. TEMPUS III’s bottom-up nature and the high level of people-to-people contact definitely helped to promote understanding between and rapprochement of cultures but the contribution to the development of ‘free and flourishing civil societies’ and the role of NGOs would need to be clarified.
Teachers are those who mainly benefited from their involvement, notably through contacts with foreign colleagues, greater awareness of other cultures and EU institutions, improved training and the morale-boosting impact of career development and mobility opportunities. Students also benefited – through access to new learning materials/methods, improved qualifications and greater awareness of other cultures – though less so in terms of the enhancement of their employability, where effects were more indirect.
Participation was geographically balanced. EU higher education institutions viewed increased internationalization as the biggest benefit from involvement in TEMPUS III, with financial considerations rarely the sole reason for taking part and in some cases, the reason why they participated less and why participation remained relatively concentrated. The geographical coverage within Partner Countries was balanced, with most higher education institutions involved. Non-university bodies (e.g. Ministries or accreditation/training bodies) were generally involved although the level of involvement was not always as widespread or as prominent as expected.
The size, scope and unique nature of TEMPUS III meant that an identifiable, quality TEMPUS brand emerged in Partner Countries – particularly among university administrators, participating faculty staff and national higher education actors, but perhaps less so among students and other staff. In EU Member States, the ‘brand’ was less pre-eminent, with it often in competition with national cooperation or support programmes for third countries.
Notwithstanding significant variation at the regional, country and even individual higher education institution levels, TEMPUS III made an important contribution to modernise and make teaching more responsive to societal and labour market needs. Many new curricula were developed, often responding to highly specific and previously unmet labour market needs. However, some time will be required before a ‘critical mass’ of graduates trained through TEMPUS III can deliver a clear impact on the labour market at national level.
It had also a significant impact on individuals contributing to introduce a culture of change and motivate in particular young staff to support and perform the reforms needed. It contributed to change mentalities and to make them open to changes and innovations providing at the same time new career perspectives to those involved.
Impact in terms of higher education decision-makers utilising skills developed through TEMPUS III to influence their respective institutions was positive at the higher education institution level. Numerous decision-makers took part in projects and many of these will be able to play a future role in reforms of their own institutions. At the national level, TEMPUS III was not able to engage as actively with policy-makers, which, as a group, also often experience high staff turnover. This meant that although TEMPUS III was able to generate significant results at the level of individual institutions in domains such as quality assurance or general university governance, influencing national legislation, except in the candidate countries where there were a number of obvious examples, was more indirect in nature with the programme feeding into policy discussions rather than bringing about direct changes.
TEMPUS III helped create more internationally-oriented and flexible HEIs, as a platform on which individual HEIs can further professionalise in the future. These, together with the related impacts at the level of individual staff, were probably the single biggest impact of the programme. It also made an important contribution to the long-term convergence process between Partner Countries and EU higher education standards and practices – most notably encapsulated within the Bologna Process and the wider HE reform agenda. Finally, TEMPUS III had a significant impact on establishing or rekindling regional co-operation among Partner Countries across all regions and made an important, yet often overlooked, impact in terms of strengthening co-operation within individual Partner Countries.
Many benefits and outcomes of the programme were sustainable. The experience or the knowledge gained in the projects, increased knowledge of a study area, improved working practice, improved teaching and management skills, greater awareness of other cultures and EU institutions, new curricula, together with a more innovative culture in the institutions remained after the projects finished. The results of SCMs, due to their very nature of providing support to structural reform, were less tangible but the majority of policy outputs were still in use in the organisations involved.
In many cases the programme encouraged sustainable partnerships in higher education, involving EU and partner countries in various degrees depending on the character of the project, with academic cooperation continuing after the end of the project.
An important contribution for follow-up funding was made by the involved institutions: half of the partners continued their related activities by way of own funding sources such as national budgets, other multilateral or bilateral sources and external fund raising. In this respect, there appeared to be a high level of financial commitment by the various stakeholders involved to continue the activities and networks that were put in place. The stakeholders would need to feel more incentivised however to cooperate more actively in the dissemination of results and the promotion of achievements which tends to be limited due to the competition between HEIs.
The evaluation report makes a series of recommendations, which are listed below together with the Commission comments.
Recommendation 1 – Strengthen strategic programme orientation.
The Commission agrees that policy dialogue can certainly be reinforced. In Tempus IV, a lot of effort has been deployed to strengthen the dialogue with ministries. Ministries are setting the national priorities themselves, through the Delegations and following consultation with stakeholders at national level. Ministries are also consulted on the shortlist of projects to be funded. A group of Higher Education Reform Experts, endorsed by the Ministries, has been established in the partner countries and these experts are an important link between the academic world and policy-making and are in many cases involved in advising their Ministers. Last but not least, the performance indicators on the basis of which projects have to report have recently been reviewed and will facilitate linking the monitoring of the projects to the national strategic frameworks.
Recommendation 2 – Improve effective synergy with Erasmus Mundus and other Community actions and encourage links with research
The Commission has undertaken a considerable effort to conceive TEMPUS IV and Erasmus Mundus in a complementary perspective (institutional cooperation/reform vs. promotion of mobility). The NTOs now have an explicit mandate and related tasks to promote Erasmus Mundus, and other EU higher education initiatives. Updates on Erasmus Mundus are a standard item of regular NTO/NCP meetings and Erasmus Mundus is investing more and more to keep NTOs systematically informed about developments.
Encouraging links with research policies and programmes, and especially to the Marie Curie Actions, that aim at the international mobility of researchers and the development of their careers, is also one of the Commission's priorities. A specific theme of activities in Tempus IV is to support the development of the knowledge triangle in partner countries and a study has just been launched to take stock of doctoral studies in neighbouring countries, which includes also elements linked to research policies and programmes.
In fact, the recommendation regarding complementarity should not be limited to Erasmus Mundus and Research but should also cover the other bilateral support in higher education that the Commission is making available in many partner countries through the Delegations mainly (budget support, sector programme). Equally, it should also refer to relevant programmes covering other regions, namely EDULINK, which has been designed to promote regional and multilateral networking between HEIs in African, Caribbean and Pacific Group of States (ACP) and the EU, and Nyerere, which promotes academic mobility in the same region. Linking existing regional initiatives to country based policy dialogue in the area of education should be part of the recommendation.
Recommendation 3 –Reconsider level of funding in view of TEMPUS objectives
The Commission agrees that there are sufficient high quality projects which could justify an increase in budget, in particular given that there is a very high demand for the programme (1st Tempus IV call: 530 proposals received/76 projects selected – 2nd call: 610 received/69 selected). The selection rate is very low at around 12%. Many very good projects cannot be funded because of budget constraints. The absorption capacity of Tempus is very high.
Individual project budgets have increased in TEMPUS IV. A limited number of the first generation of TEMPUS IV projects will be completed at the end of 2011. It may be useful to investigate whether all the project funds have been spent and to eventually reconsider the individual project budgets in the light of future financial perspectives for the programme. It would also be important to reflect on the idea that increased funding could also be considered at the level of all involved regions in order to avoid large imbalances between countries (such as the Russian Federation) and regions (for instance the Mediterranean countries).
Recommendation 4 – Enrich quality of project proposals
Over recent years, the Commission has been investing heavily in the training of NTOs and NCPs in terms of project preparation skills. Information on past projects is made available more and more systematically (website, during conferences and other events – project compendia, studies with good practice from projects).
An updated field monitoring approach has been developed for Tempus IV. Three objectives of monitoring have been defined: preventative (early stages of project), advisory (mid-term) and control (ex-post). Quantitative targets have been fixed with a strong accent on the preventative function meaning that all newly selected projects should be visited during the first six months. Debriefings on the spot are organised and feedback is systematically given to contractors allowing them to improve their project.
In addition to these instruments, Result Oriented Monitoring visits are also undertaken to projects in many of the countries.
The most important actors in the new Member States, in terms of sharing their experience of recent beneficiaries of the programme, are the universities. Universities from Poland, Slovenia, the Czech Republic and the Baltic States have been particularly active in TEMPUS III and continue to be very engaged in TEMPUS IV. Considering funding opportunities for the NCPs, who are the Information Points in the Member States, is not possible.
Recommendation 7 - Better define the involvement of the EU Delegations to improve the (structural) dialogue with the national authorities on higher education
The involvement of the EU Delegations is probably the area where there has been the most improvement over the last years. The Delegations are associated during all steps of the programme’s implementation. They are involved in the consultation of Ministries for the programme priority setting, consulted on the short-listed projects, invited to the information days and associated in the field monitoring of projects. They are the prime interlocutor for the contacts with Ministries and are also involved in the group of Higher Education Reform Experts and are associated to their activities.
Recommendation 8 – Continuation of TEMPUS-like projects after accession to the EU
The instrument for cooperation in higher education for EU Member State institutions is the Lifelong Learning (Erasmus) Programme and higher education institutions in the candidate countries automatically become eligible for participating in Erasmus when they become Member States. The Commission is investing many efforts in preparing the candidate countries for their participation in the Lifelong Learning programme and proposes, in addition, to include a session on post-Tempus in the Tempus Information Days (as is already taking place in Croatia).
Recommendation 9 – More emphasis on best practices – information on linkages with the labour market and civil society
University-enterprise cooperation is one of the priority themes under Tempus IV. More and more projects are taking place in this area and the involvement of business is encouraged in the call for proposals (multi-actor partnerships). This is a concrete result flowing from the Amman conference on university/enterprise cooperation in 2007 (see also the report on "Linking the world of works and education through Tempus i").
Recommendation 10 – Give more priority to management reform projects
Governance reform is one of the explicit priority themes under Tempus IV and more and more projects are tackling this issue. A study on university governance was finalised in 2009, based on Tempus best practice in this field, and policy developments taking place both in the EU and in the partner countries. In 2010, three policy seminars will take place in Ukraine, Kazakhstan and Morocco, engaging experts and policy makers in discussions on how to make university governance more transparent and efficient.
Recommendation 11 – Give more attention to accessibility (equal opportunities) of the projects.
Equal opportunities are mentioned explicitly as a quality criterion in the Tempus IV call for proposals and can be mentioned more clearly during the Tempus Information Days.
Recommendation 12 - Support additional ambitions of projects such as in the case of joint degrees or double diplomas
The Commission will explore the kind of additional ambitions and investigate the problems that projects might encounter. The EC Delegation in Moscow has recently launched a study on double degrees in the Russian Federation and this study will no doubt raise issues which can be further investigated.
Recommendation 13 – Improve both dissemination and use of project results
Dissemination is a quality criterion for the assessment of proposals and is also a major point of consideration during project monitoring. A lot has been done in this area in recent years. Thematic studies on university-enterprise cooperation, quality assurance and governance have been carried out and followed up by thematic seminars with a wide range of stakeholders attending. Project compendia are established and are continuously updated and particularly successful projects are invited to share their experiences in different fora.
The TEMPUS Programme remains highly relevant with respect to its main objective of promoting reform and modernisation of higher education at institutional level. The activities have at various levels contributed to reforms that otherwise would probably not have been carried out, or gained momentum at a much slower pace. At the same time it is clear that in many countries further progress in reforming the higher education sector is still needed and that important challenges in reforming higher education structures are still ahead or have just started. Tempus represents, for quite a number of partner countries, the sole window for cooperation with other institutions in other countries and it is the only working cooperation programme that the EU (and Member States) can offer to some of the countries.
The intervention logic and the basic management mechanisms to support this process have been fully confirmed, and significant enhancements have been introduced to the latest phase of the programme, Tempus IV.
The originality and the success of the programme lies very much in the 'bottom-up', demand driven approach where the higher education institutions undertake their own initiatives within a call for proposals with a large framework of possibilities, as well as the strong focus on institutional cooperation. The high level of people-to-people contact has helped to promote understanding between and rapprochement of cultures. These characteristics should be kept. Tempus, through its multilateral nature, the project approach and its management mechanisms, has proven to be a very cost-effective policy instrument, in particular compared to other classical technical assistance interventions.
It is clear that Tempus projects are typically particularly strong in terms of their relevance to the reform and upgrading of partner countries’ higher education systems, their impact on the individuals, professions, contents, methods, improved qualifications, internationalization, and the organisation of studies and the institutions involved. The outcomes of the projects are sustainable, maintained, built upon and exploited within the institutions. Tempus is an instrument that the academic community in the partner countries can test, use and exploit to support their needs, and to feed into developments at national level and beyond that, within the wider Bologna higher education community. Tempus is an identifiable and quality brand within the partner countries. It is also a highly respected programme within the EU and the Member States institutions.
The programme continues to attract a high level of interest and many very good quality applications, which however cannot be funded due to budgetary constraints. The Commission could consider the possibility of increasing the funding for all the regions covered by the programme as far as the financial framework permits and without prejudice of other cooperation priorities, in order to fully support the very ambitious objectives and challenges in higher education and the significant developments underway in the international academic community.
[5] Israel’s participation is possible on a self-funding basis only.