Legal provisions of COM(2017)113 - Ex post evaluation report on the Drug Prevention and Information Programme (2007-2013)

Please note

This page contains a limited version of this dossier in the EU Monitor.

dossier COM(2017)113 - Ex post evaluation report on the Drug Prevention and Information Programme (2007-2013).
document COM(2017)113 EN
date March  6, 2017
EUROPEAN COMMISSION

Brussels, 6.3.2017

COM(2017) 113 final

REPORT FROM THE COMMISSION TO THE COUNCIL AND THE EUROPEAN PARLIAMENT

Ex post evaluation report on the Drug Prevention and Information Programme (2007-2013)


TABLE OF CONTENTS


1    INTRODUCTION    

1.1    OBJECTIVE OF THE REPORT    

1.2    METHODOLOGY AND SOURCE OF INFORMATION    

1.3    OVERVIEW OF THE PROGRAMME    

2    RESULTS OF THE EVALUATION    

2.1    RELEVANCE OF THE PROGRAMME    

2.2    COHERENCE AND COMPLEMENTARITY    

2.3    EFFECTIVENESS    

2.4    SUSTAINABILITY AND TRANSFERABILITY    

2.5    EFFICIENCY AND SCOPE FOR SIMPLIFICATION    

2.6    EUROPEAN ADDED VALUE    

3    CONCLUSIONS    


1INTRODUCTION

1.1OBJECTIVE OF THE REPORT

1 2 The Decision establishing the Drug Prevention and Information Programme (DPIP) requires the Commission to present an ex post evaluation covering 2007 to 2013. This evaluation was performed by an independent external evaluator assisted by Commission staff.


This report is based on that evaluation. The report is structured according to the main evaluation criteria and corresponding questions. These include relevance, coherence and complementarity, effectiveness, impact and sustainability, efficiency and scope for simplification, and European added value.

1.2METHODOLOGY AND SOURCE OF INFORMATION

This final evaluation of the DPIP is based on the following:

• an extensive review of the available documentation of all 51 action grants and operating grants funded by the programme between 2007 and 2013;

• a review of programme documentation, such as the decision setting up the programme, the annual work programmes and calls for proposals for both grants and public procurement contracts;

• a review of other information available online — for example EU policy documents, websites, founding decisions of related EU programmes;

• a quantitative analysis of the 51 DPIP-funded projects and activities;

• an analysis of the 23 responses to the online survey received from DPIP grant beneficiaries;

• the write-ups of the five follow-up interviews with coordinators of projects/organisations receiving DPIP 2007-2013 grants.

1.3OVERVIEW OF THE PROGRAMME

The DPIP was established by Decision No 1150/2007/EC of the European Parliament and of the Council as part of the General Programme ‘Fundamental Rights and Justice’. The programme was implemented for seven years from 2007-2013. It is referred to in the EU Drugs Strategy (2005-2012) and the EU Action Plan on Drugs 2005-2008 and 2009-2012.


The three general objectives of the programme, as listed in Article 2 of the Decision, are:

• to prevent and reduce drug use, dependence and drug-related harm;

• to contribute to the improvement of information on drug use;

• to support the implementation of the EU Drugs Strategy.

The programme’s specific objectives (set out in Article 3 of Decision No 1150/2007/EC) are:

• To promote transnational actions to:

• Set up multidisciplinary networks;

• Ensure the expansion of the knowledge base, exchange of information and identification and dissemination of best practices;

• Raise awareness of the health and social problems caused by drug use;

• Support measures aimed at preventing drug use, including reduction of drug-related harm and treatment methods taking into account the latest state of scientific knowledge;

• To involve civil society in the implementation and development of the EU Drugs Strategy and Action Plans; and

• To monitor, implement and evaluate the implementation of specific actions under the Drugs Action Plans (2005-2008 and 2009-2012).


In order to achieve these objectives, the programme funded activities through three different types of financial instruments 3 : ‘action grants’, ‘operating grants’ and procurement. Action grants were awarded for specific transnational projects of interest to the whole of the EU, or for national projects preparing for or complementing transnational projects or Union measures, or which helped develop innovative methods and/or technologies with a potential for transferability. Operating grants were awarded either to non-governmental organisations or other entities pursuing an aim of general European interest in the area of drugs policy. Funds were also available for undertaking specific projects outlined by the Commission (Commission’s initiatives), such as feasibility studies, topic-specific research, organising conferences or building IT systems. 4


The total planned budget for the DPIP for the period January 2007 to December 2013 amounted to EUR 22 332 million 5 (see Table 1 1 Planned budgetary breakdown for DPIP (2007-2013) ). Total committed amount for the same period was EUR 20 738 million.


Table 11 Planned budgetary breakdown for DPIP (2007-2013)

YearAvailable budget for grants (action grants and operating grants) and contracts
Projects (Action grants)Operating grantsCommission initiativesTotal Annual Budget
Value (€)%Value (€)%Value (€)%Value (€)%
20072 150 00072 %N/A-750 00025 %2 900 000100 %
20082 150 00072 %500 00017 %350 00012 %3 000 000100 %
20092 400 00080 %N/A-600 00020 %3 000 000100 %
20101 717 60056 %400 00013 %958 00031 %3 075 600100 %
20113 045 20074 %250 0006 %800 00020 %4 095 200100 %
20122 058 00067 %500 00016 %520 00017 %3 078 000100 %
20133 040 00099 %N/A-43 3341 %3 084 000100 %
Total16 560 80075%1 650 0007 %4 021 33418 %22 232 800100 %

Source: DPIP — Annual work programmes (2007-2013)


The intervention logic of the DPIP is illustrated in Figure 1 1 below.


Figure 11 Intervention logic of the Drug Prevention and Information Programme (2007-2013)


Table 1 2 summarises the number of different initiatives funded each year in which the programme was implemented. Calls for proposals for action grants were split between 2009-2010 and 2011-2012.


Table 12 Number of initiatives funded per year

200720082009-20102009201020112011-201220122013TOTAL
Action grants96
10
N/A0011
0
Not considered 636
Operating grants05
0
N/A420
4
Not considered15
Public procurement11
N/A
5117N/A
1
228


DPIP projects and activities were mainly led by NGOs/national networks (33 % of all lead organisations), followed by universities (23 %), research institutes (22 %) and European networks/platforms/forums (14 %). 7

Figure 1 2 illustrates the main activities funded by action grants (on the left) and operating grants (on the right) under the DPIP. Action grant projects most often focused on awareness-raising, information and dissemination (31 %) and analytical activities (31 %), followed by mutual learning, exchange of good practices and cooperation (18 %). The fourth most frequent activities of action grant projects were training activities (12 %). Operating grants also focused on awareness-raising, information and dissemination activities (27 %), followed by mutual learning, exchanges of good practice and cooperation (20 %). The third and fourth most common initiatives implemented by operating grants were analytical activities (16%) and support to key actors (16 %). Public procurement contracts focused on three main activities: studies and the organisation of events and meetings. 8






Figure 12 DPIP action grants (left chart) and operating grants (right chart) by main activity            

2RESULTS OF THE EVALUATION

2.1RELEVANCE OF THE PROGRAMME

The relevance of the DPIP is assessed in terms of the extent to which its actions logically address its objectives, the wider EU policy needs and the needs of the target audiences.


2.1.1Priorities set in the calls for proposals and selected actions and their relevance to policy

Between 2007 and 2013 there were three major policy and legislative developments at EU level in the areas relevant to the DPIP programme: Council Framework Decision 2004/757/JHA, the EU Drugs Strategy and related action plans, and developments in relation to new drugs such as new psychoactive substances.

As of the 2011-2012 call for proposals, new priorities 9 were introduced not only to reflect these policy and legislative developments, but also to take into account the latest scientific knowledge. This flexibility made the programme more relevant to the context in which it was implemented.

At project level, selected initiatives were found to have adequately supported policy developments both because this was their main project objective, but also because many initiatives disseminated their results to policy-makers and/or fed into policy-making at EU/national level. 10

In addition to projects, public procurement contracts funded by the programme were specifically aimed at developing and implementing policy and legislation in the area of drug prevention. Evidence shows that some contracts were particularly relevant for developing policy/legislation in this area (for example the impact assessment leading to a new legislative proposal on new psychoactive substances). 11


2.1.2Relevance of the programme to target group needs

Overall, the DPIP targeted the needs of grant beneficiaries. Firstly, the programme was tailored to this policy area. Secondly, it filled a funding gap at national level for drugs-related initiatives. Moreover, both the conceptual framework of the programme and its priorities matched the needs of actors working to reduce drug demand. 12  

Most DPIP projects undertook a needs assessment 13 in order to ensure their activities were relevant to the target group. In total, 16 of the 23 respondents to the online survey (67 %) indicated that the project or activities implemented had been designed on the basis of needs assessments. Nine of these respondents stated that these needs assessments had been conducted in the year of, or the year preceding, the start of the project, demonstrating that the assessment considered the most up-to-date data. 14

2.2COHERENCE AND COMPLEMENTARITY

2.2.1Complementarity with other EU programmes

Council Decision No 1150/2007/EC 15 establishing the DPIP for the period 2007-2013 indicates that the EU can add value to the Member States’ actions ‘in the field of drug prevention and information, including treatment and reduction of drug-related harm, by complementing those actions and by promoting synergies’. Article 11 of the Decision highlights that synergies and complementarity must be sought with other Community instruments, in particular with:

• the General Programme ‘Security and Safeguarding Liberties’;

• the 7th Research and Development Framework Programme; and

• the Health Programme 2008-2013.

The Decision also indicates that complementarity should be ensured between the DPIP and the work of the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA), by using the Centre’s expertise, methodology and best practices, with a special focus on the statistical element of information on drugs.

In addition to the instruments indicated in the Council Decision, there was scope for complementarity and synergies between the DPIP and the other DG Justice programmes, such as Criminal Justice Programme (JPEN), Civil Justice Programme (JCIV), Fundamental Rights Programme (FRC) and Daphne III. Also, with regard to the risks of overlaps there was a formal safeguard in the annual work programmes of the DPIP to avoid duplication between projects funded under other programmes.

The DPIP shared similarities with the Prevention of and Fight against Crime (ISEC) programme, under the General Programme on Security and Safeguarding Liberties, to the extent that both programmes had a focus on drug-related initiatives. However, the DPIP aimed to reduce drug demand while the ISEC focused on reducing drug supply, in particular through activities aimed at curbing drug trafficking.

The Health Programme (2008-2013) managed by DG SANTE through its integrated approach to health, covered also drug policies and prevention of drug consumption from a public health perspective promoting healthy lifestyles. The actions financed under this Programme focused on the issue of drug use exclusively, including in the instances when this related to poly-drug use with a more direct approach than DPIP. Health information initiatives cover highly technical/scientific information or measures aimed at improving access to information for patients and doctors alike.

The 7th Research and Development Framework Programme (FP7) (2007-2013) funded projects that aimed to strengthen scientific evidence on the consequences of drug consumption in the fields of public health, socioeconomic sciences and the humanities. As the programme focused mainly on large-scale research projects, they complemented the limited-scale projects funded by the DPIP. 16

At programme level, the evaluation showed that the DPIP complemented other EU programmes, as explained above, with regard to thematic areas and objectives, eligible grant applicants, end beneficiaries and the types of measures and interventions funded. At the level of calls for proposals, findings confirm that there was no scope for overlap with the other DG Justice calls and other EU instruments in terms of eligible actions, given that they differed in terms of priorities and objectives. 17

The findings of the evaluation confirm that the DPIP is coherent. Firstly, its intervention logic (see Figure 1 1 Intervention logic of the Drug Prevention and Information Programme (2007-2013) ) is internally consistent, meaning the programme’s objectives, inputs, activities and expected results are non-contradictory. 18

Secondly, the various funding instruments (action grants, operating grants and public procurement contracts) are complementary and do not overlap.

Finally, there are limited data suggesting the existence of synergies between the DPIP and other EU and national interventions. However, some synergies were reported during the evaluation. Some of the activities which helped create synergies were the dissemination of project findings; the development of networks; joint actions; and cooperation. 19

2.2.2Complementarity at project level

Project summaries with information on all the projects funded by the DPIP were publicly available on the DG Justice website dedicated to the DPIP calls for proposals. Selected information on projects can also be found on the EMCDDA’s website. 20 Furthermore, the publication ‘Projects, studies and research on illicit drugs funded by the European Commission 2007-2010’ 21 provided a comprehensive overview of projects funded under the DPIP, the Health Programme, FP7 and ISEC between 2007 and 2010. Another publication ‘New Psychoactive Substances Projects, Studies and Research’ 22 also provided information on projects focusing on new psychoactive substances funded from the DPIP, the Health Programme, FP6, FP7 and ISEC (from 2007 to 2013).

The risk for overlap between projects was further reduced by the requirements and priority setting of the DPIP calls for proposals. Data collected through mapping projects, follow-up interviews and the online survey indicated that some DPIP projects complemented projects funded by other EU programmes. Examples of project complementarity can also be observed among DG Justice programmes (JPEN, JCIV, FRC and Daphne III). 23

2.3EFFECTIVENESS

The effectiveness of a programme refers to the extent to which the programme has been successful in achieving its objectives, and the extent to which the DPIP projects were successful in achieving their own project objectives.

2.3.1Programme achievements

24 The evidence collected to date suggests that overall the DPIP was effective in achieving its general programme objectives. However, it should be noted that only 51 initiatives were co-financed during the 2007-2013 funding period. Therefore, the contribution to achieving the objective of preventing and reducing of drug use in the EU could only be relative. Nevertheless, the DPIP contributed to fostering intra-European awareness-raising and information on drugs and associated harm, in particular among young people and drug users.

25 Further, DPIP-funded initiatives led to an improved dialogue on drugs, and encouraged the exchange of best practice among stakeholders, particularly NGOs, social workers, policy-makers and experts. The initiatives achieved a wider impact in the Member States where most of the activities were implemented namely: Italy, the United Kingdom, Germany, the Netherlands, France, Spain, Portugal, Belgium, the Czech Republic, Bulgaria, Estonia and Belgium.

However, DPIP-funded initiatives contributed to some extent in implementing the EU Drugs Strategy and related action plans. This is due to the fact that the implementation and development of the Drug Strategy and related Action Plan was covered to a lesser extent by the actions compared to other objectives/ priorities.

26 The programme achieved its general objective also through public procurement contracts. The programme was focused less on supporting the development of legislation and policies at national and EU level and more on developing new practices related to prevention/drug reduction. However, it was not equally successful in incorporating those practices into policies at EU level. Also, only few projects contributed to the development of national policy. Although policy-makers were actively engaged in the majority of DPIP projects, there is little evidence that contacts with these stakeholders led to a real impact on policy development.


General objective 1: to prevent and reduce drug use, dependence and drug-related harm


DPIP-funded interventions implemented a wide range of activities which improved awareness-raising and the dialogue on drugs. They also improved prevention and helped reduce drug use, dependence and drug-related harm. This objective was mainly addressed by awareness-raising and mutual learning activities.

Mutual learning activities in particular led to an improved dialogue on drugs, and encouraged the exchange of best practice among stakeholders from different environments. Moreover, they helped create sustainable transnational multidisciplinary networks specialised in drug prevention (12 networks were established or expanded during the funding period). This impact was more prominent in the Member States where such activities were directly implemented: Italy, United Kingdom, the Netherlands and Austria, followed by — to a limited extent — Denmark, Bulgaria, Portugal and the Czech Republic. Nearly half of the actions organised workshops or focus groups (20 held in total). 27


Awareness-raising activities improved awareness and information on drugs and associated harm. This impact was again more pronounced in the Member States where such activities were directly implemented, namely: Italy, United Kingdom, Germany, the Netherlands and Spain, followed by France, Lithuania, Portugal, Denmark, Estonia and Slovenia. 28

Activities mainly targeted people at risk and vulnerable groups (such as young people, women, disadvantaged groups and prisoners) to raise their awareness about the consequences of drug use. Awareness-raising activities also targeted the general public. 29


General objective 2: to help improve information on drug use


DPIP-funded interventions implemented a wide range of activities, which improved the provision of information about drugs by developing innovative intervention programmes and harm-reduction strategies. These interventions mainly consisted of analytical activities 30 focusing on the latest developments in drugs policy (for example new trends in relation to new psychoactive substances, poly-drug users and drug use by vulnerable groups including inmates, former convicts and sex workers).


The analytical activities implemented under the programme helped to promote innovation in science, and by designing innovative treatment methods they helped to reduce drug-related harm. This impact was more prominent in the Member States were such activities were directly implemented namely: Germany, United Kingdom, Italy, Spain, the Netherlands, Czech Republic, Denmark, Belgium, Poland, Portugal, Latvia, Estonia, Austria, Romania and Bulgaria. The impact of projects developing these activities appears to have been wider due to the high number of Member States involved in this type of activities, but also because most project outcomes were easily disseminated within the Union.


The inventory of documentation carried out for this evaluation showed that 41 projects implemented analytical activities. Out of these, 15 engaged in scientific research and/or produced reports, 13 developed surveys and questionnaires to collect data from drug users and/or organisations dealing with drug issues and five focused on data collection/analysis or statistics. 31

These analytical activities resulted in a number of reports: 21 guides and manuals (e.g. manual of the intervention produced), 20 reports outlining statistics (in relation to the drug market and length of various drug using periods), and 18 surveys were conducted (e.g. surveys to collect direct information from target groups).

The analytical activities also produced 43 other types of reports. These included reports collecting conclusions and recommendations from workshops or seminars, analytical reports on the effectiveness of tools used for screening, interventions and referral to treatment programmes and summary reports on survey results and other types of stakeholder consultation. 32

Scientific research was used mainly to identify components of new psychoactive substances and to develop new methods for detecting these substances in the human body.

The main contributions of analytical activities/support measures can be summarised as follows:

• filling gaps in knowledge and in scientific research in relation to drugs (i.e. new synthetic drugs/ NPS) and related health risks;

• fostering debate on changes and reforms to drugs policy; and

• developing new harm-reduction strategies and treatments approached to address rapid changes in drugs usage (new psychoactive substances, poly-drug users, etc.).


General objective 3: to support the implementation of the EU Drugs Strategy


The implementation and development of the EU Drugs Strategy and Action Plans was primarily meant to be achieved by concluding public procurement contracts. A study on ‘Minimum quality standards in drug demand reduction (EQUS)’ contributed to the policy process in this area. 33 This in turn helped implement the related action in the EU Drugs Strategy and Action Plan. Further, an impact assessment on new psychoactive substances was used as basis to draft a new legislative proposal on new psychoactive substances. 34 In addition to this contribution to the development of the EU drugs policy, the impact assessment became a document of reference for the scientific community within the Union.

Some projects funded under the DPIP supported policy-makers at national/EU level in shaping new policies and legislation. Others developed tools for screening, brief interventions and for referring young people with drugs problems for treatment. These tools were then incorporated in national drugs plans and in other national drug prevention strategy documents. While the majority of the DPIP projects reached out to relevant policy-makers at national and EU level, their results varied. 35

The initiatives funded under the DPIP used a range of methods to reach policy-makers at national and EU level, for example: 91 % of online survey respondents who were beneficiaries of DPIP funding had invited policy-makers to workshops and/or other dissemination events; 78 % of online survey respondents who were beneficiaries of DPIP funding had shared communication materials (brochures, leaflets, flyers) with policy-makers; 65 % of grant beneficiaries organised project meetings involving policy-makers; 65 % of online survey respondents who were beneficiaries of DPIP funding had invited policy-makers to events such as briefings and conferences. 36

Policy-makers responded by participating in project-related events such as seminars, conferences and workshops, or by using some elements of the project, such as the approach/method used or activities implemented.

2.3.2Project achievements

The majority of the DPIP projects achieved their objectives. This is confirmed both by the online survey results and the follow-up interviews with all the grant beneficiaries consulted as part of this evaluation, who stated that they were or would be able to achieve all or most planned objectives/results on time. Nearly all survey respondents stated that their projects were or would be able to reach the expected target groups, adding to the evidence that the projects had achieved their goals. 37

2.4SUSTAINABILITY AND TRANSFERABILITY

It was analysed whether the results, outcomes and impact of the projects were sustainable beyond the project funding period. Three levels of sustainability were identified:

• short-term sustainability, achieved mainly by disseminating projects’ results;

• medium-term sustainability, which includes continuing project results and/or partnerships; and

• longer-term sustainability, achieved mainly by successfully transferring projects’ results to other contexts, organisations and Member States without additional funding (or with only limited funding).

2.4.1Continuation of project activities and outputs and dissemination

Both the European Commission and grant beneficiaries made considerable efforts to disseminate DPIP project results. This helped increase the impact of the projects on the ground. Using the different dissemination methods put in place helped to reach different types of stakeholders, for example EU and national policy-makers, NGOs, social workers, young people and policy experts. At EU level, project results were disseminated through the Commission’s and the EMCDDA’s websites which further improved the visibility of the programme itself.

38 Public procurement contracts were also used to help disseminate project results by producing information material and organising dissemination events. At project level, project managers planned and effectively used a number of dissemination tools, including organising events and publishing printed and audio-visual material to publicise projects’ results. Moreover, most projects used their own websites to disseminate project outputs. However, dedicated project websites were generally only set up for a limited time period, thus affecting the sustainability and transferability of project outputs. The majority of grant beneficiaries surveyed indicated that, even when arrangements were made for the continued use and/or availability of the outputs (e.g. websites), additional funding was necessary.

39 The analysis of project documentation confirms that most projects developed a sustainability and/or dissemination plan (see Figure 2 1 ). However, the extent to which project beneficiaries were successful in ensuring the sustainability and transferability of project outputs/activities varied substantially. In most cases, beneficiaries only produced dissemination materials which were circulated among partners and target groups within the six months following the end of the funding period.

40 Most DPIP project leaders shared project results with the EMCDDA throughout the lifetime of the project, but some also shared the results of their projects through the Reitox network. Project results were also presented during meetings and conferences organised by the EMCDDA, the Civil Society Forum on Drugs and the Council’s Horizontal Working Party on Drugs (HDG) on Drugs.


Figure 21 Type of dissemination of project results


Source: Survey for the ex post evaluation of the five DG Justice programmes 2007-2013.

Question 24. Please comment on the following statements with regard to the dissemination of the results of your project/ activities

The quantitative analysis shows that at least one event was organised per project (54 events across 51 projects), covering an audience of just over 41 000 people, while a large number of projects appear to have produced published materials and/or information or advice websites. However, only a limited number of projects appear to have engaged in more proactive forms of dissemination, such as press releases/media involvement or campaigns. Nevertheless, since young people were one of the main target groups of the DPIP, some projects sought to be innovative and/or creative in their dissemination activities to be able to attract a generally ‘difficult to engage’ target group and to be more effective in delivering information about drugs to them. 41

2.4.2Continuation of partnerships after the project’s completion

42 Most partnerships established during the implementation of DPIP projects/activities continued after the finalisation of the grant, as demonstrated by the results from the online survey and the follow-up interviews. Most of the respondents to the online survey (82 %) reported that their partnership did or would continue after the project was complete. In most cases, continuing the partnership meant further collaboration on similar projects or a sustained commitment from all partners to continue disseminating the projects’ results. However, only few DPIP-funded actions could demonstrate that they already ensured medium-term sustainability, i.e. a continuation of project results.

2.4.3Potential sustainability and transferability of outputs

Though there is some evidence of the medium-term impact and sustainability of DPIP-funded initiatives, it is difficult to assess their sustainability in the longer term. Only a few DPIP projects could demonstrate that they put a mechanism in place in order to achieve longer-term sustainability by transferring their project results or by continuing the partnership.

Transferability in most cases related to the outputs of the DPIP projects/activities (not results) and their potential to be implemented in other Member States, with minor changes/inputs (as indicated by 78 % of online survey respondents and two out of five interviewees). Half of both groups reported that outputs of DPIP projects have already been successfully implemented in other Member States. The majority of stakeholders consulted considered the results of the projects/activities as suitable for feeding into policy activities in other Member States, but this only happened in a few cases. 43

The evaluation also identified possible challenges to the transferability of DPIP project outputs and results. Firstly, the transferability of outputs or approaches developed within the framework of DPIP projects might be affected by policy differences between Member States. Secondly, the transferability of results could be affected by the level of capacity building of the NGOs and networks involved in the delivery of prevention and risk reduction strategies. 44

2.5EFFICIENCY AND SCOPE FOR SIMPLIFICATION

This section assesses the efficiency of the programme in terms of the sufficiency and efficiency of financial resources available, the appropriateness of the money spent in comparison to the positive impact achieved and the extent to which the allocation of funds among the different funding tools was appropriate and sufficient to implement the project’s objectives. It also reviews the management efficiency of the Commission and considers scope for simplifying the programme management.

2.5.1Sufficiency and efficiency of available financial resources

The assessment of efficiency showed that the funding made available for implementing the DPIP may not have been entirely sufficient, considering the level of ambition of some of the objectives, the very high demand for funding and the high absorption rates of grants. When looking at the extent to which the initial programme allocation was effectively committed, in total EUR 12.9 million was committed to action grants (i.e. in terms of grant agreements signed). EUR 0.9 million was committed to operating grants, with approximately EUR 2.7 million committed to procurement contracts. Compared to the initial allocations, both action grants and operating grants received less than initially envisaged (respectively EUR 0.6 million and EUR 0.8 million less than planned). Despite these differences, overall the grants were in very high demand. 45 The lower commitments to operating grants which became sharper in the second half of the funding period are shown in Figure 2 2 .

More funds could have been committed to the programme, in particular with a view to achieving the objective of reducing drug demand. The very high number of applications for funding under the DPIP (as opposed to the small number of funded projects), and the fact that the most commonly funded organisations were NGOs who largely rely on external financing, confirms the general need for adequate financial resources in the area of drugs policy. Overall, the financial resources available were used in an efficient way, judging by the comparison of inputs/outputs in projects’ budgets. 46

Though some of the expected impacts of DPIP were very ambitious as they derive from the programme objectives, the implemented projects achieved good number of outputs and positive results which suggest that the amount of money spent was reasonable in comparison to the achievements. The impact of the projects is demonstrated in the development of new prevention tools; new treatment methods corresponding to the latest state of scientific knowledge in the drugs field; and awareness-raising approaches targeting specific (vulnerable) groups. Furthermore, the high demand for DPIP grants has allowed the Commission to select those projects which showed the most potential and represented the best value for money. 47


Figure 22 Total allocated, committed and paid DPIP funds and total over/under commitment and underspending per call (action grant above, operating grant below)


Note: No information was available on paid funds and underspending for 2011-2012 action grants


DPIP — Calls for Proposals (2007-2012) 2013 could not be considered within the scope of this evaluation, because the final project reports were not submitted during this evaluation process.


2.5.2Extent to which the money spent was reasonable in view of the impact achieved

The DPIP was expected to:

• improve the prevention of drug use in the EU;

• create better understanding/improved information on drugs and associated harm in the EU;

• improve the implementation of the EU Drugs Strategy and specific actions under the action plans.

It is reasonable to expect the programme to achieve its objectives with the resources available, especially with regard to improving the provision of information on drugs and associated harm in the EU. Nevertheless, the first objective, i.e. improving prevention, was the most challenging to achieve given not only the scale of the problem of drug use versus the size of the programme, but also the fact that reducing drug use is inherently associated with behavioural changes and lifestyle choices which are generally difficult to influence and may require a series of long-term interventions. The second objective was worked on through the dissemination of outputs and results by the Commission and individual projects and networks (more on this see the section 2.4.1 Continuation of project activities and outputs and dissemination). The third objective — better implementation of the EU Drugs Strategy and action plans — was achieved mainly through a series of high-level policy reports covering a host of issues relating to drugs policy in the EU. These policy reports were outsourced to external contractors. 48

The evidence collected as part of this evaluation is still insufficient to firmly conclude that the resources spent on the programme were reasonable in view of the long-term outcomes and impact achieved.

On the other hand, the outputs and results of DPIP action grants and operating grants which form the basis for the long-term outcomes and impact, were positive, in particular in relation to 49 :

• developing new prevention tools, harm-reduction measures and treatment approaches to address new developments and rapid changes as regards drugs, or developing tools (e.g. quality standards for online helplines) supporting the work of organisations dealing with target groups;

• developing new approaches to raise awareness and provide information among specific target groups/vulnerable groups;

• establishing multidisciplinary networks and/or platforms to increase the exchange of information and cooperation between organisations, further disseminate project results/good practices and gain visibility with policy-makers;

• capacity building and training to strengthen the implementation of the EU Drugs Strategy and action plans and relevant actors in prevention of drug use/dependence and associated harm.

DPIP funded 51 mostly transnational projects, representing approximately 200 leading and partner organisations. The analysis of the projects finalised so far shows that at least 63 % of the finalised action grants and operating grants show evidence of having achieved their objectives, although action grants showed slightly less evidence of achieving their objectives (60 %) than operating grants (67 %). 50

2.5.3Appropriateness of the allocation of funds among the different funding tools

According to the annual work programmes, the total budget allocated for implementing the DPIP for 2007-2013 amounted to EUR 22 232 million. The planned budgetary breakdown is presented in Table 1 1 Planned budgetary breakdown for DPIP (2007-2013) . The average value of action grants increased between 2007 and 2012, with most of the funding allocated during the second half of the implementing period. The budget absorption of action grants (payments as a share of commitments) was acceptable, i.e. 88 %. This suggests that funding for action grants has been allocated in an efficient manner. 51 Note that this analysis does not consider most of the grants awarded under action grants for 2011-2012.

Operating grants received more than half of the funds initially allocated. The average value of OGs decreased considerably between 2008 and 2012 (reaching its lowest value in 2011), ranging from EUR 20 499 (2012 operating grant) to EUR 200 000 (2008 operating grant). The lower average value of operating grants can be explained by their shorter duration (maximum 12 months) and the fact that one operating grant funds only one organisation, whereas a partnership of organisations is financed under action grants. Budget absorption of operating grants was acceptable, i.e. 85 %. In the same way as for action grants, operating grant spending appears to have been efficient. 52

During the implementation period, the Commission committed approximately EUR 2.7 million to a total of 28 procurement contracts (or about 17 % of the total committed budget of DPIP). Approximately half of the procurement budget was committed to the European Action on Drugs (EAD) — a multiannual awareness-raising campaign – and to regular meetings of the Civil Society Forum on Drugs. With an initial allocation of over EUR 4 million, procurement has been under-used and this might have had a negative effect on, for example, the dissemination of programme results at EU level. 53

Finally, the funding tools were all implemented through a competitive process, using calls for proposals (for the grants) and calls for tenders (for procurement) which attracted high numbers of applications. Action grants and procurement contracts are more effective and efficient funding tools. 54

2.5.4Commission management and scope for simplification

Based on the online survey results, the requirements for applicants to access DPIP funding were deemed appropriate overall, as presented in Figure 2 3 below.


Figure 23 Application requirements


Source: Survey for the ex post evaluation of the five DG Justice programmes 2007-2013.

Question 36: With regard to the Commission’s management of the five programmes, including the Commission’s monitoring and evaluation of your project/activities, please comment on the following statements (see Figure above):


The information provided in calls for proposals throughout the funding period was straightforward and explained the various elements required to apply for funding. However, the document became lengthy, partly reflecting the increased level of detail required in the application form, the number of priorities presented and the introduction of the PRIAMOS IT system. The information included in the calls for proposals was clear and easy to understand according to the majority of the respondents to the online survey. The 2013 call introduced the Single Guide for Applicants, covering the five DG Justice programmes, ISEC and PROGRESS. The Single Guide made it easier and more efficient for applicants to submit proposals for different projects.

Throughout the programme, the application form asked applicants to describe the project in terms of general project information, implementation, financial management, results, evaluation and dissemination. In addition, applicants were required to complete budget estimation forms, a staff-cost analysis and a partnership declaration. From 2010, applicants were also asked to provide indicators to assess results, evidence of previous programme experience, add more detail on the partners and work streams. This increased the potential quality of the projects, motivating the applicants to develop a more rigorous plan and estimate the cost of each of their activities.

Overall, the Commission’s management of the DPIP improved over time, thanks to:

• the introduction of the single programme management unit,

• the publication of a handbook for Commission officials,

• indirectly, through the dissemination of a project management guide.

Overall, grant beneficiaries’ experience of cooperation with the Commission was positive. The grant application requirements followed a similar process to other Commission centrally-managed programmes and required more detailed information, which improved the quality of the applications and the projects. The changes to reporting also created a more balanced approach between financial justification on the one hand and the evaluation/assessment of a project’s results and potential impact on the other.


In 2007, the application form required the following information: the project’s objectives and relevance; project implementation (impact, methodology, duration, concrete outputs and foreseeable risks and difficulties); follow-up, sustainability and visibility (including dissemination and European added value); information on the applicant and partners; a declaration by the applicant; and an application package and a checklist.

In addition to the application form, applicants were asked to submit annexes comprising, among others, partner and associate declarations, co-financing declaration forms, budget forms, a staff-cost analysis form, timetable, financial identification forms, legal entity forms, a declaration on exclusion criteria and CVs.

According to the information collected in the survey, in order to respond to the call for proposals just under half of the respondents (48 %) had to request at least some assistance. Moreover, more than half of the respondents (61 %) confirmed that they knew of potential applicants who did not respond to the calls for proposals due to the complex requirements set by the Commission. These findings are illustrated in Figure 2 4 .


Figure 24 Commission’s management of the five programmes during the application phase


Source: Survey for the ex post evaluation of the five DG Justice programmes 2007-2013.

Question 36: With regard to the Commission’s management of the five programmes, including the Commission’s monitoring and evaluation of your project/activities, please comment on the following statement:


In spite of the difficulties faced by applicants, the total number of applications received for action grants increased steadily from 39 in 2007 to 117 in 2013. However, the number of applications for operating grants decreased from 16 applicants in 2008 and 19 in 2010, to 7 in 2011 and 12 in 2012.


Information on the calls for proposals, both for action grants and operating grants, was available on the DG Justice website. 55 It included documents for applicants, practical information on how to apply, on reporting fraud and irregularities and the projects selected under each call. The vast majority of survey respondents (78 %) regarded the call for proposals clear and easy to understand. Similarly, the procedure for submitting an application for an action grant or operating grant was perceived as straightforward by a majority of respondents (74 %).

Overall, the reporting arrangements concerning the progress and achievements of the projects/activities were considered to be appropriate by 56 % of respondents to the online survey. Moreover, just under half of the survey respondents considered the Commission’s monitoring arrangements at least partially good and helpful during the implementation of the projects and activities. 56

Nearly half (43 %) of the grant beneficiaries who responded to the online survey considered the Commission’s monitoring arrangements as partially good and helpful during the implementation of the project/activities. Moreover, progress reports for grants lasting 24 months or more were regarded as useful by grant beneficiaries since these would provide both project managers and the Commission with an overview of the results achieved so far and would allow them to introduce adjustments to work streams, when relevant. 57

2.6EUROPEAN ADDED VALUE

EU added value refers to the extent to which the EU nature of the programme brings value to its stakeholders and the extent to which the EU has a comparative advantage over national and international actors working in the area. First, the EU nature of the programme and its geographical coverage is discussed. The EU added value is then analysed in terms of value brought to the EU and to beneficiaries. However, it has not been possible to identify and measure EU added value already in terms of impacts.

The DPIP had a strong transnational dimension. This is borne out by the objectives of the programme and by the kinds of action deemed eligible in the 2007 Decision. Article 4 of the Decision states that the DPIP ‘shall support specific transnational projects of Union interest presented by at least two Member States or by at least one Member State and one other country which may either be an acceding or a candidate country.’ Eligible actions also included national projects.

2.6.1Geographical coverage and involvement of Member States

The DPIP covered lead and partner organisations from a total of 25 EU Member States and Norway. Overall, there was a relatively good geographical coverage of activities across the EU. The nature of the programme brought added value to the EU and to most grant beneficiaries. However, some Member States received a larger number of grants and participated in a larger number of partnerships than others (see Figure 2 5 ). Lead organisations were clustered within three Member States: Germany, the United Kingdom and Italy. Together these Member States led 61 % of all projects.

From the total of 457 applications received through the DPIP, the highest number was submitted by Italian organisations (24%, 109), followed by those based in the United Kingdom (60) and Spain (55). These top three applicant Member States submitted 49 % of all applications for funding from the programme. Against this background it is worth noting that the highest success rates of the submitted applications were found to be those from Austria, Germany and the Netherlands.

The partnership structure of the Member States with the highest number of lead organisations (Germany, the United Kingdom and Italy), is shown in Figure 2 6 . Data show that lead organisations are more likely to partner with organisations from their own Member State than with organisations from other EU Member States. This is especially pronounced for Italy. However, this was not the case for the projects led by the United Kingdom, where the lead organisations were most likely to form partnerships with German organisations. For example, Italian lead organisations formed partnerships with six Italian and three UK organisations, while German lead organisations formed partnerships with four German and three Belgian, three Austrian and three Lithuanian organisations. Lead organisations from the United Kingdom formed partnerships with six German, five French and four Italian organisations.

Figure 25 Total number of organisations involved in DPIP action grant projects, including lead (left) and DPIP action grants partner (right) organisations


Figure 26 Partnership structure for the top three Member States of lead organisations



2.6.2Added value for the EU and for grant beneficiaries

The findings of this evaluation show that the main features that brought EU added value to the programme for grant beneficiaries were:

• mutual learning and exchanges, and increased knowledge through cooperation across the borders;

• the ‘EU brand’ and reputation provided by the DPIP;

• EU funding which made the implementation of the projects possible.

The findings of the survey and stakeholder consultations showed that the transnational dimension of the DPIP has allowed organisations established in different Member States to cooperate together and to develop and implement transnational activities aimed at reducing drug demand. While lead organisations are geographically concentrated in just a few Member States, the partnerships they formed allowed a high number of organisations from other Member States to participate, collaborate and learn with/from the project leaders and to implement similar activities. Overall, transnational partnerships contributed to mutual learning, collaboration and to sustainable support both at national and EU level. This is also shown in the results of the stakeholder consultation which are further explored in this section.

The evaluation concluded that the key specific benefits of transnational partnerships while implementing DPIP projects were: the creation of a network of international partners (90 % of survey respondents), an increased knowledge/expertise in the area (85 % of survey respondents) and an increased knowledge of policy and practice in other Member States (70 % of survey respondents). Hence, transnational partnerships also contributed to achieving the specific objectives of DPIP which refer to supporting transnational actions such as creating multidisciplinary networks, expanding the knowledge base, exchanging information and identifying and disseminating good practices. Furthermore, when implementing transnational activities in the area of drug demand reduction, DPIP projects also promoted transnational learning through partnerships. Follow-up interviews with project beneficiaries showed that transnational partnerships helped disseminate best practices, encourage transnational learning and identify information gaps and common issues in drug demand reduction.

The specific benefits of transnational partnerships (see Figure 2 7 ) also resulted from good working experiences between project partners. The majority of survey respondents said they had good working relationships with their transnational partners (74 %) and were satisfied with the partnerships established by the projects (62 %). These positive relationships enabled and strengthened transnational learning — the majority of respondents (68 %) exchanged experiences and lessons learned with their partners during the project implementation. In this respect, 48 % of the survey respondents agreed that it would have been beneficial to involve more partners from different Member States in the project implementation to multiply this effect.


Figure 27 The extent to which project beneficiaries had a good experience with the transnational partnership


Source: Survey for the ex post evaluation of the five DG Justice programmes 2007-2013.


Other form of value that the programme brought was the “EU brand” and reputation of the DPIP. This meant a higher visibility, interest in the projects and their higher impact on EU and national policy-makers, practitioners and the wider public.


Finally, the evaluation showed that EU funding was essential to implement the DPIP projects and achieve the EU objectives. Therefore, the likelihood of achieving the EU’s objectives — and thus meeting the objectives of the EU Drugs Strategy — without EU funding is intrinsically limited. 58


3CONCLUSIONS

Relevance of the programme 59

• Overall, the priorities of the calls for proposals and the selected initiatives were relevant to the objectives of the programme as defined in the legal base. The general and specific objectives of the DPIP are broadly formulated and overly ambitious in the legal act. However, the annual priorities set by the Commission were not only clearly defined, but were also realistic and attainable and addressed the key policy developments in the policy area. They became an increasingly important tool for the Commission to influence the scope of the funded projects throughout the life of the programme.

• The selected actions are relevant to the programme objectives. The project/work programme objectives of all 51 projects were in line with the specific objectives of the programme. Operating grants were only partially aligned with the specific objectives of the DPIP.

• At project level, the selected initiatives appear to have adequately supported policy developments. Procurement contracts were also particularly relevant for developing policy and legislation in this area.

• Overall, the programme was relevant to the needs of grant beneficiaries. The programme is unique in this area and filled a funding gap at national level. Moreover, the conceptual framework of the programme and its priorities matched the needs of actors working in the area of drug prevention.

Coherence and complementarity

• Complementarity of the DPIP with other EU programmes and interventions was almost fully achieved. Complementarity was reached through mechanisms that the Commission put in place at programme design stage and at the stage of designing calls for proposals.

Effectiveness

• Overall, the DPIP was effective in achieving its general programme objectives, although the impact was somewhat limited by the relatively low budget and number of projects funded. The DPIP contributed to fostering intra-European awareness-raising and information on drugs and associated harm, in particular among young people and drug users. Furthermore, DPIP-funded action led to an improved dialogue on drugs, and facilitated an exchange of best practice among stakeholders.

• Some DPIP projects contributed to EU policy-making/legislative development. Procurement contracts, being specifically aimed at developing and implementing policy and legislation in the area of drug prevention, were more effective in this respect because they triggered policy debates and steered the policy-making process at EU and Member State levels.

• At project level, most initiatives achieved their own objectives, in particular thanks to good working relationships with partners and to a clear intervention logic with regard to the target group, objectives, method and activities to implement. No major obstacles were experienced in the implementation of project activities.

• DPIP-funded actions also developed tools recognised as innovative in a number of areas. These tools included prevention and harm-reduction measures and treatment approaches, research methods or contributions to new research in order to fill knowledge gaps and innovative approaches to provide information and raise awareness among vulnerable groups.

Sustainability and dissemination

• Overall, the dissemination of DPIP results helped increase the impact of the projects on the ground, in particular as a result of the dissemination methods put in place, which helped to reach a wide range of stakeholders (EU and national policy-makers, non-governmental organisations/civil society organisations, social workers, young people and drug experts).

• Project results were disseminated through beneficiaries’ websites and networks as well as through Commission and EMCDDA websites. This further improved the visibility of the programme itself (demonstrated by the increasing number of applications received). In addition, public procurement contracts were used to produce information material and organise dissemination events to publicise the results of the projects. However, mechanisms and strategies for dissemination were not exploited to their fullest potential.

• Most initiatives developed a sustainability and/or dissemination plan. However, the extent to which project beneficiaries were effective in securing the sustainability and transferability of project outputs and activities varied substantially.

Efficiency

• The funding made available for implementing the DPIP may not have been entirely sufficient considering the level of ambition of some of the objectives, the high demand for funding, the high absorption rates of grants and the high numbers of outputs and results achieved. Nevertheless, the funding made available under the DPIP was sufficient for beneficiaries to achieve their own objectives.

• The Commission’s management has improved over time and grant beneficiaries experienced cooperation with the Commission as positive.

• As regards scope of simplification, the level of detail required in the application form has increased from the 2010 call onwards and included the introduction of work streams. This has benefited both the Commission and the applicants. Reporting requirements were considered appropriate by the vast majority of the respondents. The Commission’s monitoring arrangements were at least partially considered as good and helpful during the implementation of the project/activities. The reporting requirements reflect a more balanced approach between financial justification on the one hand and evaluation of actual results and potential impacts of the projects on the other.

EU added value

• Most EU Member States participated in the DPIP either as a lead organisation or a partner.

• The EU added value of the programme lay in the fact that the DPIP enabled organisations based in various Member States to make a difference in the area of drug demand reduction. The partnerships formed within the projects helped promote transnational learning and improve the visibility of the initiatives carried out, and also helped identify information in the area of prevention and fight against drugs. The partnerships also improved cross-border cooperation, contributed to the exchange and dissemination of best practices and information, developed mutual trust among Member States and supported the creation of practical tools and solutions to address global challenges. However, the extent of the improvement and contribution, i.e. the EU added value in terms of impacts could not yet be measured.

Key recommendations

• Better define the priorities: the Commission should invest more time and human resources in the process of setting priorities in order to ensure that the priorities can be adequately achieved within an earmarked budget.

• Realistic assessments of project risks and better risk mitigation strategies: the Commission should better monitor risks throughout the project duration, for example by asking for brief progress reports that identify any potential risks as they arise during the implementation of the project.

• Increase focus on assessment of impacts at all levels and not merely on outputs, as regards monitoring and evaluation. This goes hand in hand with the need to collect, analyse and use objective and independent evidence in order to perform project and programme evaluations.

• Explore ways of enhancing the uptake of project outputs, results and best practices by other organisations, including in other Member States, including more resources for translations, communication and dissemination. 

• Sharpen the programme's intervention logic; further to the scope of the programme and its general and specific objectives and priorities, types of action and types of intervention and implementing measures, the Commission will seek to sharpen the intervention logic 60 , and make the relations between the rationale, objectives, inputs, outputs, beneficiaries, expected outcomes and impacts more articulate, precise and concrete in any future continuation of the programme.

(1)

Article 15(3)d, Decision No 1150/2007/EC of 25 September 2007 establishing for the period 2007-2013 the Specific Programme ‘Drug prevention and information’ as part of the General Programme on ‘Fundamental Rights and Justice’.

(2)

The ex-post evaluation report by external evaluator is published here: Main report: http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf , Annexes: http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_annex_1,_2_and_3.pdf , http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_annex_4_quantitative_analysis.pdf

Mid-term evaluation report by external evaluator is published here: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52011DC0246&qid=1467122450426&from=EN  

(3)

Article 8, Decision No 1150/2007/EC of 25 September 2007 establishing for the period 2007-2013 the Specific Programme ‘Drug prevention and information’ as part of the General Programme ‘Fundamental Rights and Justice’

(4)

 Ex post evaluation of five programmes implemented under the 2007-2013 financial perspective. Specific programme evaluation: Drug Prevention and Information Programme (DPIP), ICF, 28 July 2015, p. 2

http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf  

(5)

Article 12, Decision No 1150/2007/EC of 25 September 2007 establishing for the period 2007-2013 the Specific Programme ‘Drug prevention and information’ as part of the General Programme ‘Fundamental Rights and Justice’


(6)

Not considered within the scope of this evaluation, because the final reports were not submitted during this evaluation process.

(7)

 Ex post evaluation of five programmes implemented under the 2007-2013 financial perspective. Specific programme evaluation: Drug Prevention and Information Programme (DPIP), ICF, 28 July 2015, p. 3

http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf  

(8)

Ibid., p. 2

(9)

  http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_annex_1,_2_and_3.pdf  

(10)

 Ex post evaluation of five programmes implemented under the 2007-2013 financial perspective. Specific programme evaluation: Drug Prevention and Information Programme (DPIP), ICF, 28 July 2015, p. 9

http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf

(11)

Ibid.

(12)

Ibid., p. 11

(13)

 Needs assessment should include relevant and reliable data and should contain a robust analysis clearly demonstrating the need for the action. The applicant can refer to existing research, studies, previous projects which had already identified the need. The needs assessment must make it clear to what extent the action will meet the need and this shall be quantified. The applicant should be specific and focus on the actual needs that the project will aim to address and not limit the analysis to general statements and information about the problems and needs of the target group in general.

(14)

Ibid., p. 11. Based on an analysis of the 23 responses to the online survey received from DPIP grant beneficiaries; and the write-ups of the five follow-up interviews with coordinators of projects/organisations receiving DPIP 2007-2013 grants

(15)

DECISION No 1150/2007/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 25 September 2007 establishing for the period 2007-2013 the Specific Programme ‘Drug prevention and information’ as part of the General Programme ‘Fundamental Rights and Justice’.

(16)

 Ex post evaluation of five programmes implemented under the 2007-2013 financial perspective. Specific programme evaluation: Drug Prevention and Information Programme (DPIP), ICF, 28 July 2015, p. 15-16

http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf

(17)

Ibid., p. 14

(18)

Ibid.

(19)

Ibid.

(20)

  http://www.emcdda.europa.eu/  

(21)

European Commission (2011) Projects, studies and research on illicit drugs funded by the European Commission, 2007-2013: http://ec.europa.eu/justice/anti-drugs/files/2007-2010_drug_related_projects_en.pdf .

(22)

European Commission (2014) New Psychoactive Substances. Projects, Studies and Research funded by the European Commission: http://ec.europa.eu/justice/anti-drugs/files/nps_report_2014_en.pdf.

(23)

 Ex post evaluation of five programmes implemented under the 2007-2013 financial perspective. Specific programme evaluation: Drug Prevention and Information Programme (DPIP), ICF, 28 July 2015, p. 19

http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf

(24)

Ibid., p. 22

(25)

Ibid.

(26)

Ibid.

(27)

Ibid., p. 23

(28)

Ibid.

(29)

Ibid.

(30)

Ibid., p. 24

(31)

Ibid., p. 25

(32)

Ibid.

(33)

  http://ec.europa.eu/justice/anti-drugs/files/equs_main_report_en.pdf  

(34)

  http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52013SC0319&from=EN  

(35)

 Ex post evaluation of five programmes implemented under the 2007-2013 financial perspective. Specific programme evaluation: Drug Prevention and Information Programme (DPIP), ICF, 28 July 2015, p. 27

http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf

(36)

Survey question 23b: Have the policy makers responded to the information provided by the project/activities? (Multiple choice question). In: Ibid., p. 27. Based on an analysis of the 23 responses to the online survey received from DPIP grant beneficiaries; and the write-ups of the five follow-up interviews with coordinators of projects/organisations receiving DPIP 2007-2013 grants.

(37)

Ibid., p. 28

(38)

Ibid., p. 31

(39)

Ibid., p. 32

(40)

Ibid., p. 33

(41)

Ibid., p. 34

(42)

Ibid., p. 35

(43)

Ibid., p. 36

(44)

Ibid.

(45)

Ibid., p. 39

(46)

Ibid., p. 37

(47)

Ibid.

(48)

Ibid., p. 44

(49)

Ibid., p. 45

(50)

Ibid.

(51)

Ibid.

(52)

Ibid., pp. 45-46

(53)

Ibid., p. 46

(54)

Ibid.

(55)

  http://ec.europa.eu/justice/grants1/programmes-2007-2013/drug/index_en.htm .

(56)

Based on an analysis of the 23 responses to the online survey received from DPIP grant beneficiaries; and the write-ups of the five follow-up interviews with coordinators of projects/organisations receiving DPIP 2007-2013 grants. Ex post evaluation of five programmes implemented under the 2007-2013 financial perspective. Specific programme evaluation: Drug Prevention and Information Programme (DPIP), ICF, 28 July 2015, pp. 48-49 http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/dpip_programme_evaluation_final_report.pdf

(57)

Ibid., p. 50

(58)

Ibid., p. 59

(59)

Ibid., pp. 60-62

(60)

See for instance Figure 1 1 Intervention logic of the Drug Prevention and Information Programme (2007-2013)