Annexes to COM(2008)866 - Towards a common understanding of the concept of tolerable risk of error - Main contents
Please note
This page contains a limited version of this dossier in the EU Monitor.
| dossier | COM(2008)866 - Towards a common understanding of the concept of tolerable risk of error. |
|---|---|
| document | COM(2008)866  |
| date | December 16, 2008 |
2.2.2. The assumptions made
The approach makes certain assumptions given that analysis at this stage has been limited to readily available and easily quantifiable data. The following assumptions and simplifications could potentially cause the tolerable risk point to be overestimated:
- controls have a dissuasive effect[11], but this is difficult to measure: in this model, cost and benefits of controls are set out in purely quantifiable financial terms.
- all (100%) projects need to be controlled annually on the spot to ensure 0% error: the model excludes the effect of additional training and guidance and the extrapolation of errors whereby corrections of systematic errors are applied to non-audited projects.
- the cost of controls axis assumes that each control costs the same amount and that there are no economies of scale from simultaneous checking of several projects for the same beneficiary or checking several years’ expenditure in a single control.
- the population of projects is assumed to be homogeneous in size and in terms of risk of error while in reality controls would be directed towards higher risk areas thereby reducing risk of undetected error at a lower cost.
- the multiannual corrective mechanisms in place (e.g. for cohesion policy) impact the error rate, since many of the projects audited in the Court's annual sample will not have been subject to the full set of controls required by legislation over the lifespan of programmes[12]. The (annual) tolerable risk point in this illustrative example is therefore totally distinct from the error level at the closure of each individual (multi-annual) programme and from the declarations of assurance in the Annual Activity Reports of Commission services which are based on an appreciation of systems' capacity to detect and correct errors over a number of years.
The following further assumptions in the model could potentially cause the tolerable risk point to be underestimated:
- improving existing ex ante controls to adjust the minimum error rate reported by the Court in 2006, for example through better guidance and training by the Commission to the Member States and by the Member States to their authorities and beneficiaries, is assumed to have zero (or no significant) cost.
- audit risk is not taken into account: controls carried out are assumed to identify and correct all errors in a project.
In addition, the examples provided were based on a single year and it is assumed that this year is representative. The combined (upward and downward) effect of these assumptions may balance each other out. Moreover, the introduction of more sophisticated mathematical approaches (using parabolic and logarithmic functions and the concept of opportunity cost), was found to produce comparable results, and the method is therefore believed to be reasonably reliable (see Annex).
Clearly, some of the above assumptions are quite far-reaching, and with the benefit of an extensive analysis of financial transactions and error rates[13] accompanied by an extensive collection and analysis of detailed data, at project level, the model could be further developed to take account of:
- the nature of projects, to identify the inherent levels of risk for different types of projects;
- the different risk levels of different activities and beneficiary types, to ensure that the highest risk ones are controlled first;
- the size of projects, so that at a given level of risk the largest ones could be prioritised for control, thus using control resources efficiently; and
- the duration of projects, so that controls can be timed to permit the maximum amount of funds to be controlled.
However, since developing this advanced version of the model would imply a time-consuming and costly data collection exercise in Member States, the option is not further pursued at this juncture.
3. ILLUSTRATIVE EXAMPLES OF COST OF CONTROL AND RISK OF ERROR
Using the simple model set out above, the following section looks at particular parts of two policy areas for which the Court concluded in its Annual Reports for both 2006 and 2007 that the error rate is high[14]. The underlying assumptions are simple and provide indicative results. In both cases, the basic model is adapted to accommodate the specificities of data available for each area. The results are not therefore comparable between the two areas. In both cases, the monetary amounts and percentages presented are rounded given their approximate nature.
3.1. European Regional Development Fund (ERDF)
In the 2000-2006 programme period, ERDF co-financed an estimated 700,000 individual projects. The Fund is implemented under shared management with implementation tasks, including the control of beneficiaries, delegated to Member States at national, regional and local level. Sectoral legislation sets out control structures and minimum levels of controls for managing authorities, paying/certifying authorities and audit bodies, and stipulates that errors and irregularities have to be prevented, detected and corrected in the first instance by the Member States[15]. The Commission supervises these management and control systems and DGs draw conclusions on their effectiveness in their Annual Activity Reports. In February 2008 the Commission adopted an Action Plan to strengthen its supervisory role under shared management of structural actions[16].
The Commission's own control strategies and practices are based on a multi-annual approach[17]. The control and correction mechanisms in the Member States are also multi-annual, comprising first level verifications (desk reviews and on-the-spot controls) before declaring expenditure to the Commission; on-the-spot ex post controls on operations after declaration of expenditure to the Commission; systems audits, and a closure declaration by an independent winding-up body.
Due to the annuality of the DAS enshrined in the Treaty, the following calculation of a possible tolerable risk point is annual, and is based on the Court's DAS 2006 and the error definitions and findings reported by the Court.
In 2006 and 2007, structural actions (including ERDF)[18] received a negative assessment from the Court of Auditors, which quoted an error rate in reimbursements for all Structural Funds of at least 12% and 11% respectively on the basis of its sample.
Using the current minimum error rate found by the Court of Auditors and reducing this to around 9%, to take account of the estimated benefits of improving the existing ex ante controls at limited or no significant additional cost, the Commission determined the annual cost of controls, together with the tolerable risk point as set out in section 2.2.1 above.
Based on data from the Member States, the Commission has made a preliminary estimate that the total cost of control for the ERDF in 2006 was some €215 million, around 0.7% of total public expenditure on activities co-financed by the ERDF in that year.
The following graph illustrates the theoretical relationship between error rates and the cost of control for ERDF (see annex for further details):
[pic]
Based on estimated figures for the amount of expenditure /number of projects controlled in 2006, the cost of achieving a theoretical zero error rate by controlling all expenditure/projects on the spot control annually would be €2.25 billion (equivalent to 8% of total public expenditure on activities co-financed by the Fund). On this basis, a curve was derived to illustrate the possible relationship between error rates and the cost of control[19].
The point on the curve where the amount at which any supplementary Euro spent on control yields exactly one Euro more in expected error detected was then calculated to determine the tolerable risk point (where the marginal cost of control equals the marginal benefit). This point could lie at an error rate of around 4% and a cost of control of €996 million, equivalent to 3.5% of total public expenditure. This would indicate that spending more on control of the ERDF would be cost effective up to the point of achieving an error rate of some 4%. A political decision on this would need to take account of capacity to increase expenditure on controls and could result in a target level of tolerable risk between the Court's current minimum error rate adjusted to 9% (to take account of the estimated benefit of improved ex ante controls at no significant cost) and the 4% theoretical target identified above.
As the information provided by the Member States was not always complete and consistent,[20] the Commission assessed how the calculation would be affected by a possible underestimate of the cost of controls. If these were 50% higher than estimated by the Member States, the tolerable risk point would increase by 1 percentage point (to 5%). This, together with the results of a sensitivity analysis using more sophisticated statistical techniques based on the same data, showed limited variation in the tolerable risk point (see Annex).
It should also be noted that the data used above relate to Structural Funds expenditure governed by 2000-2006 legislation. The rules governing the 2007-2013 programme period include reinforced and clearer control provisions for national authorities, which are intended to help reduce error rates. This effect should be accentuated by the preventive activities in relation to the 2007-2013 period being carried out under the Action Plan.
Additional simplifications have also been proposed recently to the legislative authority and these are also likely to have an impact on the error rate and therefore on the tolerable risk point. However, the majority of Structural Funds expenditure will not be governed by 2007-2013 legislation until 2010 onwards. Until then, the majority of expenditure declared by Member States to the Commission will be covered by the 2000-2006 legal framework[21]. As any further, more detailed, analysis of tolerable risk would need to incorporate the effect of the 2007-2013 legislation and of the Action Plan, this would need to be based on 2010 implementation data and therefore could be available as from 2011 (see section 4.1 below).
Key observations the tolerable risk level is above 2% and could lie, according to a simple theoretical cost/benefit approach, around 4%. This would imply imply a fivefold increase in the current control outlay from 0.7% to some 3.5% of total public expenditure. Accordingly, on the basis of the illustrative figures, €100 million extra spending on controls above the current levels could yield a net estimated benefit of some €285 million of detected errors. achieving a 2% error rate would, on the basis of the illustrative figures, require an eightfold increase in control expenditure to nearly 6% of total public expenditure on the Fund. because of the significant additional control costs to be borne by Member States to reach the theoretical tolerable risk point, a medium-term objective could be fixed to evolve from the Court's minimum error rate towards a target of around 5%. |
- 3.2. European Agricultural Fund for Rural Development (EAFRD)
EAFRD accounts for some 20% of total agricultural expenditure over the period 2007-2013. Its control framework is very close to that for European Agricultural Guarantee Fund (EAGF) expenditure. The Court concludes that rural development accounts for a disproportionately large part of the overall 2% to 5% error rate for agriculture, whereas for EAGF expenditure the value of the error rate is estimated to be slightly below 2%[22].The higher error rate in rural development is mainly due to the high incidence of errors in agro-environmental measures. Special report No 3/2005 of the Court of Auditors on the control of environmental expenditure in agriculture concludes that this expenditure is not only risky by its nature but that it is not possible to obtain assurance in this area at a reasonable cost.
The following illustrative exercise therefore looks specifically at these measures. In doing so, the Commission also responds to the discharge resolution of 22 April 2008[23].
It is not possible to estimate an error rate for agro-environmental measures on the basis of the Court's DAS exercise but the Commission requested Member States to provide statistics on the results of on-the-spot controls in financial year 2007 for several rural development measures, including agro-environmental measures. Member States provided the requested data which however was not verified and validated by the certification bodies – and in some cases was also incomplete. The data nevertheless confirmed that error rates for agro-environmental measures were higher than for other measures in rural development and indicated an error rate of around 4% for agro-environmental measures in 2007.
The Commission also requested Member States to provide information on their control costs for agro-environmental measures. On the basis of data received, these control costs presented for agro-environmental measures are roughly estimated at € 360 million, equivalent to 13% of total public expenditure on the measures in 2005. It is worth noting that this is three times more than for agricultural measures in general (around 4% of total public expenditure). Controls on agro-environmental measures also contribute to achieving the policy objective of protecting and improving the environment. These environmental benefits have not been quantified in this exercise.
[pic]
The cost of controlling all beneficiaries annually was estimated at some €1.4 billion. A curve was derived from these data to illustrate the possible relationship between error rates and the cost of control[24].
Key observations An increase in control costs above the present level (13%) would not be cost-effective, because a marginal increase of the number of on-the-spot controls by 1% of the beneficiaries would yield savings of irregular expenditure amounting to only around 10% of the costs of these additional controls. Reducing the level of error from the current 4% to the Court’s materiality level of 2%[25], would increase the cost of control from around 13% to almost 30% of total public expenditure on the measures, and would not be cost-effective. Agro-environmental measures contribute significantly to the overall error rate for rural development. As it would not be cost-effective to increase controls, the tolerable risk level for rural development measures is clearly above 2% and may lie above 5%. |
- 4. THE WAY FORWARD
4.1. Preliminary conclusions
It is important to stress that agreeing on a tolerable risk of error would not imply accepting inadequate management and control systems at the Commission or in the Member States. For all policy areas, the Commission and its implementation partners would need to remain vigilant and achieve the best possible control within given resources. Where weaknesses are found, these need to be addressed and the causes analysed to effectively prevent, detect and correct these.
The Commission also recognises that, in some sectors, control systems for Community programmes, including those operated by the Member States, could prevent, detect and correct errors more effectively within the current cost. It is taking action to achieve this[26].
Investment in controls beyond the current levels would detect and correct more errors. Any increase in controls would require additional resources to be made available by the Commission and/or by the Member States and decisions on the level of increase would need to be based on the expected benefits in terms of reduced error rates.
The Court of Auditors' assessments show that there is a real difference in error rates between policy areas. The above analysis confirms the view that the 2% materiality level for the annual Statement of Assurance (DAS) is not an appropriate measure of a cost effective control strategy for some policies. As a result the tolerable risk of error would need to be analysed and decided separately per policy area at Community level.
This tolerable risk level set by the political authorities would lie between the current error rate, as adjusted to take account of possible improvements in existing controls, and the theoretical tolerable risk point. The simple model set out in this Communication, based on available data and estimates, gives a first indication of these levels:
- For Cohesion policy, the Budgetary Authority could consider that for the purposes of the annual DAS exercise, as a first indication, the tolerable risk level may lie on an annual basis around 5%; and
- For rural development, on the basis of available data, the Budgetary Authority could consider that any further controls to reduce errors would not be cost effective. The Commission estimates in a first analysis that the tolerable risk level for this area is around 5%.
While this Communication has concentrated on two areas where the Court of Auditors gave a negative assessment in 2007, the approach could also be applied to other policy areas including internal policies, in particular research where intensified auditing has provided accurate data on errors and where costs of control are readily measurable, and external actions which, while enjoying strong political and public support, operate in often risky and difficult environments. Elements of simplification in both areas would be taken into account where these have an effect on risk of error and costs of control. The tolerable risk analysis is also relevant for areas already covered by a positive assessment by the Court of Auditors. For Administration expenditure, for example, the Commission will examine whether the low inherent risk and the effective controls already put in place justify a tolerable risk of error below the current 2% materiality threshold.
For cohesion policy the model could be made more robust to allow for differences in risk between members of the population. This would require a more extensive and costly analysis which could be completed at the earliest at the end of 2010, while still being based on 2000-2006 legislation. In the Commission's view, this option is neither necessary, nor cost-effective. On the other hand, revised data for the area of cohesion policy, based on actual implementation of the new legal framework for 2007-2013, including newly proposed simplifications, could prove useful. This would need to be based on 2010 data, by which time the new programmes will be well under way, and the results (which would be available in 2011) could be used to review and refine the tolerable risk point in this area.
4.2. Possible steps towards an agreement on implementing a tolerable risk approach
The implementation of a tolerable risk approach would in the Commission’s view be a sound investment and would provide a firm basis for the Discharge Authority to judge the quality of the Commission’s management of risk. Clearly, as the auditee, the Commission cannot itself fix the levels of risk to be tolerated.
The major steps for moving to a tolerable risk approach could be as follows:
- re-launch the Inter-institutional debate with Council, Parliament and the Court of Auditors on the basis of the present communication and position taken by the Budgetary Authority on the levels of risk to be tolerated in Cohesion Policy and Rural Development based on the analysis above;
- should the conclusions of this debate allow, the Commission will present further analyses of tolerable risk, in particular for research, energy and transport and for external aid, development and enlargement, before mid-2010.
[1] Opinion 2/2004 of the Court of Auditors of the European Communities on the single audit model (and a proposal for a Community internal control framework) (OJ C107, 30.4.2004, p. 1).
[2] The DAS methodology, European Court of Auditors, available onhttp://eca.europa.eu/portal/page/portal/audit/StatementofAssurance, 7.10.2008.
[3] The threshold of 2% is derived from auditing standards relating to financial audits in the private sector and was originally set as a benchmark by the Court itself in the absence of a comparable international standard for the audit of underlying transactions or any indication to the contrary at political level. Also some supreme audit institutions in the Member States apply a 2% materiality threshold.
[4] Commission Action Plan towards an Integrated Internal Control Framework - COM(2006) 9, 17.1.2006.
[5] Press release 2688th Council Meeting Economic and Financial Affairs, 8.11.2005, paragraph 17.
[6] Council of the European Union, note from the Presidency to the delegations– An improved sound financial management of EU funds 10284/08 FIN 217, 3.6.2008.
[7] Working document No 2 on the budget review, “Tolerable risk of error”, MEP Herbert Bösch, 8.4.2008.
[8] Annual Report of the Court of Auditors on the implementation of the budget concerning the financial year 2007 (OJ C 286, 10.11.2008, p. 1).
[9] Example: a first level check on an operation in a Structural Funds programme carried out on the spot by the managing authority in a Member State could have been improved if the control tools (eg relevant guidance and adequate checklists) had been available and had been used effectively during the check.
[10] For ERDF (see section 3.1) the error rate at point A is the lower error limit in the Court's 2006 DAS sample reduced by the estimated impact of improved quality of ex ante controls. For agro-environmental measures (see section 3.2), point A represents the average error rate reported by Member State Paying Agencies.
[11] This may include the possibility of being controlled and the preventive effect resulting from continuous system improvements. For example the effectiveness of IACS under CAP can partly be explained by the dissuasive effect of sanctions coupled with a reasonable likelihood of being checked by the national authorities.
[12] Programme implementation may span 9 years and thus final closure declarations on each programme - which confirm the legality and regularity of underlying transactions, or any qualifications to it - may be submitted up to 10 years after the programme launch.
[13] The Court's assistance in error definition and sampling methodology could add value to this exercise.
[14] Structural actions (ERDF) and Agriculture (rural development-agro-environmental measures).
[15] Article 8BPVWXvwxyÞæÿ' 1 2 7 8 ¥ ¾ ¿ æ é ô ¦:@?ž¹wz{·¸é꾿ghÂìí&B?ùõêãßõßãêãêãêõãÛ39.1 of Council Regulation (EC) No 1260/1999.
[16] COM(2008) 97 final.
[17] The multi-annual approach is reflected in the declarations of assurance in Commission services’ annual activity reports.
[18] Structural actions in 2006 included Directorates General REGIO, EMPL, AGRI (EAGGF Guidance) and MARE (FIFG).
[19] It should be noted that both x and y axes were prepared originally with the same scale (€ million) to facilitate analysis, and percentage error rates are shown in the graph for ease of reference.
[20] Some Member States provided data on only some types of control. Data collection was often decentralised within the Member States leading to inconsistent interpretation of the methodology defined by the Commission for evaluating costs of control.
[21] The impact of the Action Plan on the error level in 2000-2006 programmes is only likely to be seen at closure.
[22] The Commission estimates that the Court's findings in 2007 indicate an error rate in rural development above 5%. The higher incidence of errors for EAFRD is also highlighted by some Member States.
[23] “[…] invites the Commission to thoroughly investigate and assess the possible costs and benefits in the area of agro-environmental measures, as well as the connection to other areas of spending, and to present this analysis to the Council, Parliament and the ECA as a minimum basis for discussing the need for reform”. The attached Commission Staff Working document provides a first rough estimation of their costs of control of agro-environmental measures together with an overview of their widely-recognised benefits.
[24] It should be noted that both x and y axes were prepared originally with the same scale (€ million) to facilitate analysis, and percentage error rates are shown in the graph for ease of reference.
[25] This is the result of a calculation based on the hypothesis that the error rate decreases in a linear manner when the rate of controls increases and excluding any extra dissuasive effect of more controls.
[26] For example, the 'Commission Action plan towards an integrated internal control framework' - COM(2006) 9 - and 'An action plan to strengthen the Commission's supervisory role under shared management of structural actions' - COM(2008) 97.