Explanatory Memorandum to COM(2019)615 - Results of the interim evaluation of the ISA² programme - Main contents
Please note
This page contains a limited version of this dossier in the EU Monitor.
| dossier | COM(2019)615 - Results of the interim evaluation of the ISA² programme. |
|---|---|
| source | COM(2019)615  |
| date | 23-09-2019 |
Contents
- Brussels, 23.9.2019
- Table of contents
- 1. Introduction
- 2. Background
- 3. Common understanding of interoperability
- 4. Methodology
- 5. Evaluation issues and questions
- 6. Findings
- 6.1. Relevance
- 6.2. Effectiveness
- 6.3. Efficiency
- 6.4. Coherence
- 6.5. EU added value
- 6.6. Utility
- 6.7. Sustainability
- 7. Recommendations
- 7.1. Awareness-raising beyond national administrations
- 7.2. From user-centric to user-driven solutions
- 7.3. Paving the way to increased sustainability
- 8. Next steps
COM(2019) 615 final
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL
Results of the interim evaluation of the ISA² programme
{SWD(2019) 1615 final}
1. Introduction
As provided for in the ISA2 Decision 1 , on 1 January 2016 the five-year programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) was launched as a follow-up to the ISA programme 2 .
This report presents the findings and recommendations of the interim evaluation of the ISA2 programme 3 . Under Article 13(3) of the ISA2 Decision, the Commission is required to carry out this evaluation and communicate its results to the European Parliament and the Council by 30 September 2019.
The Commission carried out the evaluation using a team of independent experts from a consultancy company 4 . The evaluation was overseen by an interservice steering group 5 .
2. Background
The ISA2 programme’s ultimate objective is to promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens, via improved interoperability of European public administrations.
More specifically the programme aims to do the following:
–Facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens.
–Contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local administration levels.
–Promote a holistic approach to interoperability in the EU, by identifying, creating and operating interoperability solutions and facilitating their reuse by European public administrations. This will support the implementation of various EU policies and activities.
To meet these objectives 6 , the programme builds on the achievements of its predecessor, the ISA programme 7 . It also puts emphasis on fitting smoothly into the wider policy framework related to the digitalisation of public administrations in the EU. In this regard, the ISA2 programme is the main instrument to support the implementation of the European Interoperability Framework (EIF) 8 , which has been updated and extended as planned in the Communication on a Digital Single Market Strategy for Europe 9 . Other policy initiatives to which ISA2 contributes include the eGovernment Action Plan 2016-2020 10 , the 2017 Tallinn Declaration on eGovernment 11 and the Single Digital Gateway Regulation 12 .
In practical terms, ISA2 runs from 1 January 2016 until 31 December 2020 with a total budget of €130.9 million. It funds actions defined on a yearly basis in the annual rolling work programme. The programme’s management also fosters synergies with other EU programmes, like the Connecting Europe Facility (CEF) 13 or the Structural Reform Support Programme (SRSP) 14 .
The programme’s primary stakeholders are the European public administrations at all levels: EU, national, regional and local. Yet the programme affects a much wider group of stakeholders, including EU businesses and citizens. ISA2 is open to EU Member States, other countries of the European Economic Area and candidate countries. In addition to the 28 EU Member States, 3 other countries take part in the programme: Iceland, Norway (since 2016) and Montenegro (since 2018). The programme also encourages cooperation with other non-EU countries and with international organisations and bodies.
3. Common understanding of interoperability
In line with Article 1(2) of the ISA2 Decision, ‘the ISA2 programme shall ensure a common understanding of interoperability through the EIF and its implementation in Member States’ administrations. The Commission, through the ISA2 programme, shall monitor the implementation of the EIF.’
In this respect, the ISA2 programme is developing an integrated framework for monitoring, assessing and reporting the progress made in implementing the EIF 15 by both the Member States and the Commission. The design and implementation of this monitoring framework are conducted within the remit of the National Interoperability Framework Observatory (NIFO) action, funded by the ISA² programme.
The EIF implementation results across the EU will be presented yearly through an online scoreboard to better inform policy-making and to communicate to a wider public in a more visual and intuitive way 16 . In practice, the scoreboard aims to improve communication and capacity building to make Member States better aware of their current status in the EIF’s implementation, and to help them comply more easily with the EIF’s recommendations.
To achieve these goals, a toolbox will be made available to the Member States to provide practical guidance on implementing the EIF by means of good practices and solutions.
The monitoring framework for the EIF implementation is meant to provide evidence for the EIF’s future evaluation, which was initially planned to take place by the end of 2019 17 . However, in order to leave a realistic timeframe for Member States to adapt their national policies on eGovernment, digital services and interoperability to the EIF and given that the broader scope of the new EIF requires additional internal and external coordination to finalise a commonly accepted and integrated monitoring framework, it appears necessary to postpone the EIF evaluation until 2021. Indeed, these factors resulted in the data collection process only beginning in 2019, thus there is a lack of data to evaluate the EIF’s implementation in the same year.
In addition, by running the EIF’s evaluation back-to-back with the final evaluation of the ISA2 programme in 2021, several synergies can be created, since the programme is the main instrument supporting the EIF’s implementation.
4. Methodology
The interim evaluation of the ISA2 programme relied on quantitative and qualitative data collected through various methods, like desk research, public consultation, targeted online surveys, in-depth interviews and a kick-off workshop. The consultation activities allowed for a wide coverage of the different ISA2 stakeholders, ranging from representatives of Member States and Commission departments to citizens and standardisation organisations.
In order to better guide data collection, a sample of 20 actions was selected from the 53 actions — grouped into 9 packages — funded by the ISA2 programme up to 2018. Four pre-defined criteria steered this sampling to ensure that the selected actions are largely representative of the programme. Furthermore, the overall data collection was complemented by an expert assessment of the programme carried out by four technical experts in interoperability.
5. Evaluation issues and questions
The evaluation focused on the following seven main criteria 18 :
–Relevance — to what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels?
–Effectiveness — how far are the ISA² programme’s results in the process of achieving the programme’s objectives? Are there aspects that are more or less effective than others, and if so, what lessons can be drawn from this?
–Efficiency — to what extent has the programme been cost-effective? Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised? How is the programme performing relative to the planned work and budget?
–Coherence — to what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme (internal coherence)? To what extent is the ISA² programme coherent with other EU interventions, which have similar objectives, and with global initiatives in the same field (external coherence)?
–EU added value — what is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels?
–Utility — how do the ISA² programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address?
–Sustainability — to what extent is the financial, technical and operational sustainability of the developed solutions — maintained and operated through the ISA² programme — ensured?
6. Findings
Based on data collected from 129 consulted stakeholders 19 , extensive desk research, and expert assessments, the interim evaluation confirms that ISA2 performs well in all the evaluation criteria. However, the evaluation is confined to actions funded between 2016 and 2018; hence, the results of ongoing actions as well the longer-term outcomes of ISA2 can only be captured after the programme finishes.
6.1.Relevance
The objectives pursued by ISA2 are still pertinent in relation to the evolving needs and problems in the field of interoperability of digital public services, confirming the programme’s relevance. Most of the consulted stakeholders agree that the problem of administrative e-barriers and related needs originally identified by the programme are still valid. However, some of them (46 out of 129) currently have some additional needs (like a more prescriptive approach to design interoperable public services) and problems (including resources constraints), which ISA2 can only be partially address.
One additional need, which deserves special attention, is to ensure more collaboration and exchanges with regional and local administrations in order to increase awareness of interoperability and the take-up of ISA2 solutions at the sub-national level.
Raising public administrations’ awareness on interoperability was the first of the three recommendations in the final evaluation of the ISA programme 20 . The ISA2 programme acted upon this call by adopting its communication strategy and engagement plan in 2017 and by organising 10 major events between 2016 and 2018, with an average of 211 participants per event. Moreover, ISA2 representatives played an active role in 60 events held over the same period. The interim evaluation shows that these efforts paid off, as most of the consulted stakeholders indicated that they have expert knowledge of interoperability (91 out of 128) and of ISA2 (81 out of 128). In spite of this general awareness, the above-highlighted additional need signals that the programme should continue its awareness-raising activities targeting regional and local administrations and possibly indirect beneficiaries, like citizens and businesses too.
6.2.Effectiveness
The results achieved so far by ISA2 are aligned with the programme’s objectives. Nevertheless, they still do not fully match the expected results, as most of the actions are still ongoing and solutions are still being developed. The duration of the programme also influences the take-up rate of solutions. As such, actions that have been continued from previous editions of the programme have produced solutions that are now more widely used than solutions resulting from actions that have been established under ISA2.
There is one specific ISA2 objective where the evaluation found that the programme on its own is less effective: the development of a more effective, simplified and user-friendly e-administration at the national, regional and local level. Here Member States can in fact have a great impact, and complement the EU-level initiatives on interoperability and digitalisation.
External factors can improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The call for common standards and frameworks from public administrations represents an external factor contributing to the programme’s performance. By contrast, institutional complexity could hamper the achievement of interoperability across borders and across sectors.
Finally, the evaluation found that the ISA2 actions are largely compliant with the general principles listed in Article 4 of the ISA2 Decision, due to the rolling work programme process, which ensures that the principles are systematically taken into account when the work programme is being prepared.
6.3.Efficiency
The programme’s implementation is progressing as planned; all actions are either on track or close to achieving the planned level of work. Nevertheless, the heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness. For those ISA2 packages where it was possible to aggregate performance indicators of different actions, costs per end-user (e.g. businesses, citizens) have been estimated as very low.
The process for selecting actions funded by ISA2 is considered relatively efficient. The costs incurred to prepare a proposal are rather small, ranging from 0.07% to 0.4% of the potential funds that could be allocated to the proposal once accepted. Nonetheless, the selection process could be further improved by simplifying the rolling work programme (e.g. making the template more flexible) and launching thematic calls for actions, thus ensuring that the selection process is driven by objectives.
6.4.Coherence
According to the consulted stakeholders, substantial synergies and limited overlaps characterise the ISA2 actions. This strong internal coherence is in line with the second recommendation of the final ISA evaluation, which called for a holistic approach on interoperability within ISA2.
On external coherence, ISA2 followed the third — and last — recommendation of the final ISA evaluation, and built close cooperation with other EU policies and initiatives. It fostered synergies with CEF, Horizon 2020 and the Structural Reform Support Programme. ISA2 also interacts with initiatives such as the single digital gateway, Digital Single Market Strategy, eGovernment Action Plan, Tallinn Declaration on eGovernment, and the rolling plan for ICT standardisation 21 . Nonetheless, some overlaps were perceived between specific solutions under ISA2 and CEF, which could be addressed by better delineating the scope of each solution.
6.5.EU added value
The level of coordination ensured by ISA2 is crucial to improving the overall interoperability among European public administrations. In addition, 91 out of 109 respondents emphasised that ISA2 is able to achieve its objectives at a lower cost than comparable national or sub-national initiatives.
It is apparent that ISA2 has helped improve cross-border interoperability in the EU: i) it raises awareness about interoperability across the Member States; ii) helps put the topic on national agendas; and iii) creates networks and facilitates exchanges. ISA2 has also helped to further common EU policies: it plays a central role in implementing the EIF and supports the establishment of the digital single market.
6.6.Utility
User satisfaction with the ISA2 solutions tends to be positive with only 7 out of 110 stakeholders reporting ‘limited satisfaction’. As the programme is ongoing, it is expected that the take-up of ISA2 solutions will increase, thus improving the way that solutions meet user needs and, in turn, increasing overall user satisfaction.
6.7.Sustainability
The sustainability of the ISA2 solutions received mixed feedback from consulted stakeholders. While 66 out of 84 respondents believe that the programme’s results would last in the absence of future funding, over 55 out of 85 respondents consider that the need for operations and the maintenance costs required for the solutions could harm the ability of ISA2 solutions to deliver their results if the programme was terminated.
Stakeholders agree though, that ISA2 plays a central role in improving the interoperability landscape in the EU and its absence would jeopardise the efforts of European public administrations to improve interoperability and to foster the ICT-based modernisation of the public sector in Europe.
7. Recommendations
The interim evaluation paints a positive picture of the ISA2 programme’s performance so far. However, it also identifies areas for possible improvement. The recommendations below address these aspects of the programme, outlining both short- and longer-term measures. The short-term measures are those, which could be taken on board in the last rolling ISA2 work programme in 2020 and during the transition period leading to the set-up of the new Multiannual Financial Framework (MFF) 22 . The longer-term measures would need more years to yield results.
7.1.Awareness-raising beyond national administrations
The Commission, through the ISA2 programme, will continue its efforts to raise the European public sector’s awareness of interoperability, which is a key enabler of digitalisation, especially ensuring more collaboration with the regional and local administrations during this work.
To start with, targeted promotional activities should be designed, which emphasise the benefits of (re)using the ISA2 interoperability solutions. To substantiate this message, studies could be run in 2020 to quantify the impact of some ISA2 solutions on the efficiency and productivity of public administrations.
In the ISA2 programme’s final stage, even more emphasis should be put on ensuring that public administrations, academia and interested professionals share best practices among themselves. The programme’s recently-launched Interoperability Academy action should develop suitable training materials too.
To amplify the impact of the above-proposed measures, the ISA2 action owners should further leverage potential influencers (like researchers, committed public servants) and create a community of interoperability ambassadors.
In the longer run, advisory capabilities should be built around interoperability. These would help interested stakeholders to pick the appropriate interoperability solutions matching their needs, while at the same time providing support services and technical assistance for the effective implementation of the solutions.
7.2.From user-centric to user-driven solutions
To increase the programme’s utility, ISA2 could improve the quality of its existing solutions by better considering user needs. This approach could prevail during the implementation of the last ISA2 work programme, thus promoting the EIF principle on user-centricity both in the Commission and in the Member States.
However, in the coming years it is advisable to move from a user-centric to a user-driven paradigm, where users become involved in the design phase of an interoperability solution. An interoperability incubator could foster such a co-creation process, which would allow for new, user-oriented solutions to be experimented with and prototyped in a safe environment. The incubator would help with the take-up of emerging technologies and the exchange of innovative practices between pioneer public authorities.
Concerning the first two recommendations of this report, the Commission’s Digital Strategy 23 proves that the Commission makes good on its promises: both interoperability and user-centricity are among the strategy’s key principles and the EIF is one of its reference points. Moreover, ISA2 plays a consultative role in the ongoing implementation of the strategy, helping the Commission, in various policy areas, to modernise its existing IT systems and develop new digital solutions with interoperability in mind.
7.3.Paving the way to increased sustainability
When assessing the EU added value and sustainability of the ISA2 programme, the interim evaluation found that it plays an essential role in setting up a uniform interoperability landscape in the European public sector. ISA2 also helps to develop and deploy cross-border and cross-sector digital solutions among Member States’ administrations. ISA2 also helps with the advancement of common policies as they greatly rely on these interconnected and interoperable networks and systems.
After the ISA2 programme ends, it is vital to preserve and increase the ability of European public administrations to work together towards attaining mutually beneficial goals, involving the sharing of information and knowledge, which is at the core of interoperability at large. For that, strong political commitment and solid funding are needed. Regarding political commitment, the ministers in charge of eGovernment policy across Europe — in the Tallinn Declaration on eGovernment — confirmed that they were committed to building citizen-centric public services with digital-by-default design and achieving the vision laid out in the EIF. Concerning the future funding for interoperability, it is envisaged in the Digital Europe Programme (DEP) 24 , which is one of the Commission’s sectoral proposals under the MFF 2021-2027 legislative package.
In addition, the Commission should act upon the synergies created between ISA2 and other EU programmes to promote the EIF and interoperability in general and to facilitate the broad reuse of ISA2 solutions. Furthermore, this approach could result in efficient and streamlined programme management supporting the future implementation of the DEP proposal.
In parallel to the preparations for setting up the new MFF programmes, it is recommended to investigate the possibilities of increasing the sustainability of the results achieved by existing ISA2 solutions. The feasibility and costs of various sustainability measures should be assessed and the Commission should invest in them. For example, the Commission could consider transferring some ISA2 solutions to open source communities or encouraging companies to build services around free ISA2 solutions under the European Union Public Licence (EUPL) 25 .
Finally, the Commission may find it useful to assess in-depth the rationale and impacts of a possible binding interoperability instrument. Such assessment should rely on the findings of the ISA2 programme's final evaluation and on the evidence to be gathered when evaluating the EIF's implementation in 2021.
8. Next steps
In the ongoing implementation of the ISA² programme, the Commission will pay the utmost attention to the above findings and recommendations, analysing them to validate and address the issues raised, wherever appropriate in close cooperation with the Member States. The findings and recommendations of the ISA² programme’s interim evaluation will inform the transition to the new MFF programmes too.
(1)
Decision (EU) 2015/2240 of the European Parliament and of the Council of 25 November 2015 establishing a programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) as a means for modernising the public sector, OJ L 318, 4.12.2015, p. 1.
(2)
Decision No 922/2009/EC of the European Parliament and of the Council of 16 September 2009 on interoperability solutions for European public administrations (ISA), OJ L 260, 3.10.2009, p. 20.
(3)
The details of the evaluation process and results together with the supporting evidence is available in the Commission Staff Working Document: SWD(2019) 1615 final.
(4)
The independent experts’ evaluation study is available at: https://data.europa.eu/doi/10.2799/13397
(5)
Members of the group were DG CNECT, DIGIT, EMPL, FISMA, GROW, OP, SG, TAXUD and the JRC.
(6)
Article 1(1) of the ISA2 Decision (see footnote 1).
(7)
Article 1(3) of the ISA2 Decision (see footnote 1).
(8)
Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, European Interoperability Framework — Implementation Strategy, Brussels, 23.3.2017, COM(2017) 134 final.
(9)
Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, A Digital Single Market Strategy for Europe, Brussels, 6.5.2015, COM(2015) 192 final.
(10)
Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions EU, eGovernment Action Plan 2016-2020. Accelerating the digital transformation of government, COM/2016/0179 final.
(11)
Tallinn Declaration on eGovernment, at the ministerial meeting during the Estonian Presidency of the Council of the EU on 6 October 2017.
(12)
Regulation (EU) 2018/1724 of the European Parliament and of the Council of 2 October 2018 establishing a single digital gateway to provide access to information, to procedures and to assistance and problem-solving services and amending Regulation (EU) No 1024/2012, OJ L 295, 21.11.2018, p. 1-38.
(13)
Regulation (EU) No 1316/2013 of the European Parliament and of the Council of 11 December 2013 establishing the Connecting Europe Facility, amending Regulation (EU) No 913/2010 and repealing Regulations (EC) No 680/2007 and (EC) No 67/2010, OJ L 384, 20.12.2013, p. 129-171.
(14)
Regulation (EU) 2018/1671 of the European Parliament and of the Council of 23 October 2018 amending Regulation (EU) 2017/825 to increase the financial envelope of the Structural Reform Support Programme and adapt its general objective, OJ L 284, 12.11.2018, p. 3-5.
(15)
This monitoring mechanism also covers the implementation of the Interoperability Action Plan presented in Annex I of the EIF Communication mentioned in footnote 8.
(16)
The scoreboard will be available under the ‘NIFO’ collection on the Joinup platform in 2020.
(17)
See Section 6 of the EIF Communication mentioned in footnote 8.
(18)
See Tool #47 of the European Commission’s Better Regulation Toolbox.
(19)
Due to confidentiality and data protection reasons, the evaluation team asked the action owners to contact their solutions users as intermediaries for consultation activities. This two-step approach may have limited the number of responses received.
(20)
Report from the Commission to the European Parliament and the Council, Results of the final evaluation of the ISA programme, Brussels, 1.9.2016, COM(2016) 550 final.
(21)
See: https://ec.europa.eu/digital-single-market/en/rolling-plan-ict-standardisation
(22)
See: https://ec.europa.eu/commission/future-europe/eu-budget-future_en .
(23)
Communication to the Commission, European Commission Digital Strategy — A digitally transformed, user-focused and data-driven Commission, Brussels, 21.11.2018, C(2018) 7118 final.
(24)
Proposal for a Regulation of the European Parliament and of the Council establishing the Digital Europe programme for the period 2021-2027, Brussels, 6.6.2018, COM(2018) 434 final.
(25)
See: https://eupl.eu/