Legal provisions of COM(2024)568 - - Main contents
Please note
This page contains a limited version of this dossier in the EU Monitor.
| dossier | COM(2024)568 - . |
|---|---|
| document | COM(2024)568 |
| date | December 12, 2024 |
1. Introduction
In line with Article 225(dc) of Regulation (EU) No 1308/2013 (the CMO Regulation)1, this report to the European Parliament and the Council assesses the use of new information and communication technologies (ICTs) and reflects on whether they ensure better market transparency.
Market transparency helps the market to operate more efficiently by facilitating the exchange of market signals between the supply and the demand side. More information on prices, production volumes, stocks etc., along the different stages of the supply chain would generate numerous advantages, such as:
(1) extending the information that is publicly available, particularly in the intermediate stages of supply chains, which would increase trust in the operation of markets;
(2) helping to reduce asymmetries in bargaining power, in particular those affecting farmers, which would level the playing field between participants in terms of their access to supply chains and, more generally, to markets;
(3) enabling a better design of relevant economic policies, going beyond anti-competitive practices2.
This report is largely based on the findings of a Commission study on the role of ICTs in improving market transparency (the ICT study)3. The study describes and analyses the potential of different ICTs to improve the existing EU systems for data on agri-food markets, as well as to assess the governance aspects that would make potential improvements possible.
1. The EU agri-food market information system’s contribution to market transparency
The EU agri-food market information system based on the CMO Regulation includes ISAMM (the Information System for Agricultural Market Management and Monitoring, established by Commission Implementing Regulation (EU) 2017/1185 (the ISAMM Regulation))4, market observatories and the market section of the agri-food data portal. While ISAMM is used for data collection, both market observatories and the agri-food data portal are used for data dissemination.
This system also includes data from several other information sources, such as Eurostat (on production, prices and trade) and DG TAXUD’s Customs Surveillance System (on daily imports and exports).
2.1. Data collection - ISAMM
ISAMM is one of the data collection IT support systems for the Common Agricultural Policy (CAP). The market data is provided by Member States and not directly by individual undertakings. Around 4 650 communications per month are sent via ISAMM. Information is submitted via standard forms, whose content depends on the data required and the frequency of reporting, which ranges from weekly to yearly.
The data sent by Member States to the Commission is often entered manually into ISAMM by Member States officials. Occasionally, data files are uploaded to complete individual forms, or xml files are submitted to complete multiple forms at once. Regarding the data that Member States collect from the original sources, documents are mainly submitted by operators in electronic format, via online questionnaires or forms.
2.2 Data dissemination - Agri-food data portal and market observatories
The markets section of the agri-food data portal5 covers a wide range of products. Most of the sector-specific subsections include: (i) datasets covering prices, production and trade; (ii) dashboards compiling key recent data for each sector; and (iii) a short-term outlook, featuring estimates for production, consumption and trade for the months ahead.
The portal supports the work of market observatories for seven agricultural products (milk, meat, sugar, arable crops, fruit and vegetables, wine and olive oil), as well as of the Fertiliser Market Observatory and the recently created EU agri-food chain Observatory. These market observatories collect information from their members, and publish information, analyses and minutes of board meetings on their respective web pages6. The role and working practices of agricultural market observatories have been described in a recent report7.
2. Assessment of the current agri-food market information system under the CMO
3.1 The CMO agri-food information system ensures overall market transparency despite some weaknesses
Regarding data quality, the current system generally provides a comprehensive compilation of data, that is deemed reliable overall by actors in the EU food supply chain8. It puts in place a common knowledge base to underpin economic and market analyses, which is useful when making decisions on production and investment. It also provides the necessary data for policymaking and further research. Agri-food information systems in the main non-EU countries do not significantly differ from the EU system.
However, several margins for improvement have been identified:
Timing and frequency of updates
While the data in the CMO agri-food market information system is updated frequently (weekly, monthly, etc.), the data entered by the sources are sometimes delayed by quality validation requirements. In addition, delays in publication can be caused by administrative steps that are not automated, such as data collection by Member States from the relevant economic actors, transmission to the Commission and final dissemination. As EU-level data collection and dissemination relies on submissions from each Member State, some of which can be held up by the factors mentioned above, EU data is sometimes published later than that of certain non-EU countries.
Data reliability and occurrence of material errors
Despite the overall reliability of the CMO agri-food market information system and its internal quality control procedures, material errors are likely to occur when human handling of data is required at different stages in the process.
The comparability of data between Member States can be uncertain due to the different methodologies used9 (for example, one Member State may only collect data from the most representative operators in the market, while another may collect them from a wider sample of operators), and to specific national characteristics (market structure, different product characteristics, etc.).
Data accessibility and user-friendliness
The vast majority of data users positively assess the overall usefulness of dissemination tools and platforms for agri-food information. Nevertheless, they point to a lack of clarity about the availability of data and about the meta-data (context, methodology of collection, processing). Users also expressed a need for more flexible features for the extraction/download of data.
Non-compliance with data submission obligations
Although most Member State authorities notify on time, some do not submit information to the Commission within the legal deadlines, which leads to incomplete data and delays. Certain Member States have not fulfilled all of their data collection obligations. The extent of non-compliance varies between Member States and notification types, and is more significant for the new data series requested as of 2021.
Data gaps that need to be addressed
The system is less exhaustive for certain type of data. For stocks, despite the recent notification obligation for cereals, oilseed and oilseed products, data seem to be difficult to collect as operators consider such information subject to business confidentiality. This hampers the usefulness of such data to inform on the EU’s food security status. For trade, data on intra-EU trade and re-direction of international flows from key trade hubs within the EU is not sufficiently incorporated in the dissemination part of the system.
3.2. New challenges for the CMO agri-food information system
Data on upstream and downstream sectors
Although some agricultural inputs are already covered by the agri-food market information system (such as fertilisers, which have their own market observatory, and seeds with data collected on area, production and stocks), data on the upstream stages of the supply chain (input supply) as well as on the downstream stages (retail distribution, consumption) are less developed than those concerning production, and to a lesser extent processing, in this system.
Since there are some other public and several private systems that collect data on the upstream and downstream stages, any extension of the CMO agri-market information system to up- and downstream sectors should be assessed as regards its added value. Any possible duplication of data collection and dissemination should be avoided.
Data in times of crisis
It became clear during the COVID-19 pandemic that in times of crisis, normal reporting delays are too long, and frequent dissemination of timely transparent and evidence-based information and data is essential to maintain the trust and confidence of actors in the food supply chain.
There is currently no data stream specific to a crisis situation in the agri-food market information system. The European Food Security Crisis preparedness and response Mechanism (EFSCM) provides a dashboard and a platform to exchange views but does not cater for market data collection. Its Food Alert System10 section compiles data from other early warning systems on weather-related events, animal diseases, energy costs, freight costs, etc. This mechanism bases itself on qualitative information provided voluntarily by economic operators rather than on data.
Incorporating rapid alert systems, based on automated, real-time monitoring of critical availability, accessibility and affordability of agri-food products, and possibly inputs, at different levels of the supply chain, would represent a significant change to the CMO agri-food market information system.
3. Improving agri-food market transparency through a better use of ICTs
Developing ICTs would provide opportunities to address some of the shortcomings of the CMO agri-food information system and therefore further improve its capacity to deliver market transparency.
4.1. Optimising the existing agri-food information system
The use of specific technological solutions can make data collection and processing more efficient. Notably by using already available data to the maximum extent possible, avoiding duplication of efforts by all actors involved and saving resources.
The CMO agri-food market information system can be optimised in several ways. For example, by importing data already available in other information systems. This could be done by interconnecting the Commission information systems with other systems to enable interoperability between all systems concerned. This requires technological, organisational, and possibly also legislative measures alongside the Interoperable Europe Act, which entered into force on 11 April 2024 with the aim of strengthening the cross-border interoperability of IT systems and public sector cooperation across the EU.
According to the stakeholders consulted, six main innovative solutions could improve the quality of the information provided by the agri-food market information system:
1. The use of ICTs could improve the current processes or enable new automated systems to be used for data collection and management (processing, storage and validation of data). Examples include:
- Putting in place a machine-to-machine (M2M) interface so that information can flow directly from Member States’ systems to the Commission (via ISAMM). This would contribute to both the timeliness and quality of the data collected and would also help reduce the administrative burden and costs related to data collection and validation. To enable full M2M communication, both semantic and technical interoperability challenges need to be tackled11.
- Automating the data collection processes from operators to the Member States. This could include integrating the Member States’ data collection and validation systems within the Commission’s system (ISAMM).
- Carrying out automated checks on data, according to pre-defined parameters that would help ensure the reliability of data validation more systematically.
2. The use of advanced ICT tools, including artificial intelligence (AI) tools, could improve publication and sharing of the collected data, by:
- Ensuring better data accessibility and visualisation, for example by including direct links to original sources, providing information on collection/elaboration methods, using alternative extraction systems, providing more formats that allow editing, and creating a mobile app version of the website. All of these measures would help improve data user experience.
- Providing dissemination materials tailored to the needs of key user typologies, thus raising awareness of the system and the available resources.
- Helping reduce risks of market manipulation due to ‘time sensitive’ data, that actors have access to before others and that can be used to take business beneficial decisions (e.g. information on low stocks could incite some actors to advance their purchases)12.
The role of artificial intelligence in agri-food market data collection and dissemination
Artificial intelligence (AI)13 is a broad concept that encompasses applications of machine learning (ML), deep learning (DL), and generative AI (Gen AI)14. Deep learning is a subsection of machine learning and forms the models for Gen AI.
The current use of AI in agri-food market information systems, operated by the Commission, and in general, is rather limited, although there is awareness of its potential. Elsewhere, the Agricultural Marketing Service (AMS) of the USDA has held preliminary discussions on whether to incorporate AI into the system for data validation.
One of the key AI applications being developed is the use of machine learning to assess web scraped data (e.g. by training the tool to recognise items and classify them according to product definitions). However, quality controls by humans may nonetheless be necessary to fine-tune the dataset extracted and refined with these tools.
Examples of cutting-edge AI technology being used in other projects related to agri-market data include:
- ongoing FAO projects exploring opportunities and challenges in the use of AI by different divisions to improve the provision and integration of remote sensing data;
- data validation and analysis in the World Food Programme’s ‘Data Bridges’ application which aims to quickly identify emerging food security issues, including early warning indicators such as the Alert for price spikes (ALPS).
Beyond market data provision, some private providers are experimenting with AI applications to provide forecasts based on available data. For example, the Swiss Federal Office for Agriculture (FOAG) is carrying out AI-based forecasts for milk prices for internal use.
The advancement of generative AI and the availability of user-friendly AI tools will allow more organisations to develop or use AI-based tools. More specifically, applying AI to an agri-food market information system can be a viable option for data collection, cleaning and analysis automation, for creating summaries and visualisations and for developing predictive models for crop yields, prices and demand forecasts.
3. They could enable to improve the collection and processing of data from the sectors indirectly related to agricultural supply chains, for instance logistics and transportation (such as real-time monitoring of seaborne shipments). The technologies behind this type of solution are already being developed and used by some private data providers15 and international organisations16.
4. They could enable to improve the collection and processing of big data directly from farms thanks to the use of agri-tech solutions by professional farmers, including for precision agriculture. To improve on-site collection of farm-level data, more work is needed on data intermediaries and the potential of digital farm passports/wallets. Increasing market data collection and further data sharing between farmers could also be beneficial. Developments in the field should be closely monitored as they could open up new possibilities for data collection. For example, as announced in the European Strategy for Data, a Common European Agricultural Data Space is under development contributing to the aim of creating a single market for data while a European Digital Infrastructure Consortium is being formed that will foster the set-up and use of so-called digital farm passports/wallets in agriculture.
5. The ability of ICTs to collect data on up- and downstream sectors could be boosted by expanding the practice of web-scraping17. Automatic scraping of market information (often by using AI) from various online sources, is an increasingly used cost-effective method of collecting data. Especially in downstream stages, such as consumption and retail prices, which have larger online exposure, web-scraping enables the automation of data collection processes, allowing for real-time or near-real-time monitoring of market information from diverse sources, covering a wide range of online sources, including websites, social media platforms and online marketplaces. However, challenges associated with web-scraping include difficulties in validating data (only a very structured and advanced system of checks can ensure the reliability and representativeness of data collected) and legal issues (data privacy, for example). For the upstream sectors, web-scraping is less promising, given that reliable and representative input prices are less available online.
6. On data at times of crisis, the use of ICTs could enhance real-time data capabilities, establish dedicated communication channels and likely improve predictive analytics. Such data could be critical during crises, in particular if supported by ad hoc new data collection streams and integrated into a dedicated rapid alert system that combines data from a range of other IT systems, both public and private, and has trigger functions integrated. Such triggers could cover data indicators relevant to measure food availability/affordability metrics, for example critical level of stocks, transport time or delays at the border, etc. Having such a rapid alert system could allow business operators, and also consumers, to prepare for possible supply-chain disruptions.
4.2. Conditions for the successful development of technological solutions
The ICT solutions identified above may only be useful if their implementation is feasible, which depends on the following conditions:
1. A large-scale redesign of the agri-food data collection and validation systems would require a close coordination between the Commission and all Member States and considerable investment in technology. This is particularly relevant in the case of ISAMM as the data collection, aggregation and validation is entirely carried out by the Member States, and therefore relies on their commitment. This is an area where the harmonisation of IT systems brings the most value added by automating processes and making them more effective. This redesign of data collection and validation systems should not be done in isolation and should not create parallel or duplicate systems. It should tie-in with the development of data governance and data sharing mechanisms in the agri-food sector, considering the benefits for greater data sharing in the sector and with other sectors.
2. The cost of setting up various IT systems with different scopes and operability must be assessed. The current report does not include any assessment of the costs, as there are too many variables for such a comprehensive analysis. Most stakeholders and experts have not been in the position to provide cost estimates for setting up the IT solutions mentioned and only limited data on past setting up costs is available18.
3. Data systems must be designed with the user in mind. The reasons a user has for searching and using agri-food market data will influence their perception of its timeliness and coverage. Although the level of detail and frequency of the data provided by the current systems is generally enough for long-term market analysis, academic research and policymaking, it tends to be insufficient for day-to-day business decision-making or for complex systems for making financial market transactions.
4. Regarding data submission, business confidentiality data de-anonymisation is a concern for private operators. Data on stocks, marketed volumes and prices are considered as particularly sensitive by business operators as data breaches can compromise their competitive position. Reaching a balance between improving market transparency and protecting operators’ confidentiality could entail: (i) legal and IT protections for confidentiality; (ii) strengthened dialogue and cooperation between supply-chain stakeholders and the Commission; (iii) involving non-governmental institutions like public research institutes; and (iv) using a ‘privacy-enhancing technology’ model (a depersonalised system that uses inputs and outputs to generate useful information, without any knowledge of its internal workings) in the data crunching process.
5. Finally, EU legislation on agri-food market information systems and on the use of ICTs needs to be effective, up to date (for example, on confidentiality, personal data protection, use of AI, etc.), and applied and enforced consistently across the EU.
5. Conclusions and further debate
This report presents, for the purpose of a policy debate, the main features of the CMO agri-food information system, some of the challenges it faces and some potential ICT-based solutions that could contribute to addressing those challenges.
Existing IT technology could help improve several aspects of the EU agri-food market information. For example, ICTs could help expand the scope of the current system by automating data collection and dissemination and extending it to sectors and stages not currently covered. Moreover, further integration of the EU and national agri-food information systems through machine-to-machine interaction would also contribute to the development of a single agricultural data space in Europe. Such integration would, however, require considerable financial and administrative investment. Finally, ICTs could help with the collection and processing of market data during times of crisis, when data must be frequently updated and provided with minimum delay.
As cutting-edge ICTs or elements of them are already being used in the private sector, fostering public-private partnerships to develop agri-food market data solutions warrants further exploration. This would help improve the quality and accessibility of data in a cost-efficient way.
Thanks to technological advances, tomorrow’s ICTs have a huge potential to improve the efficiency of the agri-food market information system. AI can help to automate the data processing workflow, making it faster, more accurate and more efficient. AI algorithms and machine learning can manage large volumes of data, making sense of complex patterns that humans might not identify. While still under-used in the field of agri-food market data, the role of AI will certainly grow in the future.
1 Regulation (EU) 1308/2013 of the European Parliament and of the Council of 17 December 2013 establishing a common organisation of the markets in agriculture and repealing Council Regulations (EEC) No 922/72, (EEC) No 234/79, (EC) No 1037/2001 and (EC) No 1234/2007, OJ L 347, 20.12.2013, p. 671.
2 European Commission: Joint Research Centre and Ménard, C., Market transparency in food supply chain – Goals, means, limits, Publications Office, 2018. https://data.europa.eu/doi/10.2760/285157.
3 ‘Study on the role of information and communication technologies to improve market transparency’, upcoming publication.
4 Commission Implementing Regulation (EU) 2017/1185 of 20 April 2017 laying down rules for the application of Regulations (EU) No 1307/2013 and (EU) No 1308/2013 of the European Parliament and of the Council as regards notifications to the Commission of information and documents and amending and repealing several Commission Regulations.
5 https://agridata.ec.europa.eu/extensions/DataPortal/agricultural_markets.html.
6 https://agriculture.ec.europa.eu/data-and-analysis/markets/overviews/market-observatories_en.
7 Report from the Commission to the European Parliament and the Council on Union Market Observatories, COM(2023) 679 final of 31.10.2023.
8 According to stakeholder interviews and surveys carried out for the ICT study.
9 Article 9 of the ISAMM Regulation provides for such methodological flexibility.
10 https://agridata.ec.europa.eu/extensions/DataPortal/food-alert-system.html.
11 Such challenges are being addressed for example in the interoperability study carried out by the Digital Europe programme and the plant variety databases (Crop code list).
12 See also Regulation (EU) No 596/2014 (Market Abuse Regulation).
13 AI is regulated in the EU by ‘The AI Act’ (Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence) which provides AI developers and deployers with clear requirements and obligations on specific uses of AI.
14 See explanations of the differences between AI, ML, DL and Gen AI at: https://synoptek.com/insights/it-blogs/data-insights/ai-ml-dl-and-generative-ai-face-off-a-comparative-analysis/ or https://www.sumologic.com/blog/machine-learning-deep-learning/.
15 For example, supply chain data solutions for suppliers, buyers and service providers in the fresh produce industry.
16 For example, for real-time tracking of vessels to monitor grain and wheat maritime trade and food security: https://globaltradedata.wto.org/real-time-data.
17 Web-scraping is already used by the Commission in the European Market Observatory for Fisheries and Aquaculture Products (EUMOFA), which has a custom-made price scraper instrument for retail prices: https://eumofa.eu/.
18 For example, the setting-up costs of the US system of high-frequency mandatory reporting of market data in the livestock sector, based on the 1999 Livestock Mandatory Reporting Act (LMRA), were estimated at around USD 2 million (1999). According to the US Agricultural Marketing Service, the operational costs of LMRA cover 15-20 reporters and auditors, plus USD 1 million a year in IT support.
EN EN