The Strengthened Code of Practice on Disinformation, presented in June 2022, included also a specific commitment (41) to develop and adopt Structural Indicators. This is an important progress compared to the earlier version of the Code that did not contain any performance indicators. Structural Indicators, if adequately designed and implemented, should unveil relevant dimensions of the disinformation phenomenon and how they evolve over time across platforms and countries. Consequently and to the extent possible they should also assess the effectiveness of the Code as a policy instrument to tackle disinformation online. The complexity of the disinformation phenomenon is reflected in the challenge of establishing comprehensive as well as applicable indicators. Since it is a pioneering attempt, it requires rounds of piloting and testing different solutions and approaches.
The first pilot measurement of Structural Indicators for the Code was commissioned by the platform-signatories and delivered by TrustLab, global technology start-up founded by former executives in some of the leading online platforms. The company was selected from the pool of entities that expressed interest in carrying out the pilot measurement. TrustLab proposal was evaluated as most complete, considering many constraints (time, access to platform data, funding, etc.), by relevant working groups of the Code’s Task-force. The Task-force is a permanent component of the updated version of the Code, aimed at evolving and adapting the Code, including the ability to update its Commitments and Measures in view of technological, societal, market and legislative developments. The Task-force should also serve as an important forum of exchange among Signatories, the European Commission, the European External Action Service (EEAS), the European Regulators Group for Audiovisual Media (ERGA), the European Digital Media Observatory (EDMO) and other invited third-parties.
The Centre for Media Pluralism and Media Freedom (CMPF) has been coordinating the Policy Research and Analysis work under EDMO, which is a research project composed of EDMO.eu and a network of local and regional EDMO hubs that are all established as collaborative projects. Therefore, CMPF has also been leading the EDMO work on Structural Indicators by participating in all the relevant working groups of the Code of Practice Task-force. Having a long experience and expertise in developing comprehensive methodologies for cross-country and interdisciplinary assessments of complex phenomena, CMPF developed an initial proposal for Structural Indicators to support the process developing under the Code’s framework. CMPF-EDMO’s initial proposal included six areas of measurement: prevalence, sources, audiences, demonetization of disinformation, as well as the impact of and investment in fact-checking, and investments in the overall implementation of the Code. Each indicator foresees an assessment for each relevant signatory, the member state level, and the entire online ecosystem in the EU.
The EDMO proposal marked the initial step in what should be a broader and more systematic effort to monitor disinformation and assess the policy effectiveness of the Code. Given the complexity of the phenomenon, EDMO’s contribution also acknowledges the need to adjust Structural Indicators over time and in collaboration with wider group of experts and stakeholders. To this end, EDMO has established a dedicated Expert Group, which comprises over 20 experts volunteering to contribute to improving the methodology and advising on the implementation framework of the indicators.
The TrustLab pilot is the first testing of a methodology to measure Structural Indicators. It is limited to two indicators: prevalence and sources of disinformation, and three countries (Poland, Slovakia, Spain), and covers six platforms (Facebook, Instagram, LinkedIn, TikTok, Twitter (X), and YouTube). The resulting pilot is the outcome of a self-standing implementation that utilises the methodology and metrics put forward by TrustLab. However, it was framed and inspired by the initial EDMO proposal. Structural Indicators are planned to be measured and reported bi-annually, with an expectation that their implementation framework will include academic projects dully funded and supported in a way that protects integrity and independence.