I. Plain language summary
In this research paper we lay the foundations for a data-driven perspective on how to appropriately use open data to build business intelligence, with proven usefulness in applications to public health and water resource management.
II. Introduction
The world’s globalization phenomena unveiled awareness of worldwide problems, such as the climate crisis, but also to common efforts to find solutions to those problems, as was the case of the several COVID-19 collaborative actions. There are many obstacles still today on such global strategies to which innovative technology and data-driven solutions can help. In this paper we propose the concept of a global observatory based on text mining algorithms that is able to answer the wide range questions that are core to global solutions, using Big Data analytic methods over the layered information it is ingesting in real-time. The main perspectives of this global observatory fall on: (i) the monitoring and exploration of news articles and social media feeds; (ii) the analysis of combinations of indicators through time and what stories can they tell; and (iii) the exploration of published scientific knowledge. All of these perspectives can be combined to provide complementary answers to main topics from health to engineering.
III. Methods
Taking into account the schema in Figure 1, we consider the construction of the Global Water Observatory into phases going from lower to higher complexity. We start by putting together data sources that are meaningful to a range of stakeholders that are targeted, from engaged citizens to decision makers that can leverage the information provided to established evidence-based policies.
Figure 1. The approach used leading from data sensing to the digital twin and its approximation to local priorities.
At the data collection phase, we are concerned with addressing properly the challenges in the heterogeneous nature of the data, their different frequency and size, as well as the levels of access to it established by data providers. These parameters to take into consideration ensure the appropriate data ingestion into the system. The selection of data sources is done manually, but their ingestion is automated, and their frequency of update depends solely of the data provider. At this stage we are collecting data from many different data sources (such as, e.g., the Worldwide news, the Microsoft Academic Graph, the Word Bank, the United Nations Sustainable Development Goals), according to their relation to the focus topics and priorities.
A forthcoming stage is in the data cleaning, data processing and data integration prior ingestion. This step is highly important to allow for the data quality that is needed in order to obtain useful insights from it. In this step we include the data curation, where the most meaningful datasets are selected and parsed. We also include the exploratory data analysis and some data visualisation for the purpose of prototyping what is then available at the Water Observatory.
The Observatory phase is then possible when the curated data streams of a selection of dynamic data sources are live in the system and can be used to obtain insight on particular topics of interest, monitor Key Performance Indicators associated with business priorities, and allow for a global and local perspective on related topics. These include interactive data visualisations of indicators and statistical data, the dynamic view of the news sources over priorities, or the user query over a scientific research topic. This allows for insight on topics in analysis (such as water topics like, e.g., water scarcity and water quality, and public health topics like, e.g., ebola or the new coronavirus) that will be put into the context of local data when sourced from the shared interest of users.
The path ahead is a novel concept of a meaningful Digital Twin (i.e., a dynamical model which, given a current state of an observed system, is capable of a digital partial reconstruction of such a system) that builds over the Global Water Observatory to rise above data complexity towards data interoperability. This is usually difficult to achieve in full due to the heterogeneity of the data, the different characteristics of the data sourced (frequency, data types, etc.) and the domain knowledge needed to identify new challenges covering a wide range of business intelligence priorities. Nevertheless, useful aspects of it can be achieved, some of which are already evident from the implementations discussed in Section 6 and Section 7. An example of this is to track a topic in the news, its impact in the social media, and explore the range of the problem in the published scientific research, as well as extract good practices to deal with this problem.
We add a final stage to this diagram that is usually forgotten in a theoretical framework, which is the adaptation of the system to the needs and priorities in the user side. Here we consider the ingestion of local data, the customization of news streams, the availability of exploratory dashboards, the shareable instances for policy makers, and the APIs for 3rd party integration. A typical example of the aimed outcome of such an intelligent system is the following sequence of events: (i) a new technology is identified in scientific research; (ii) the patents around it multiply, alerting for its importance; (iii) new mentions of its usage arise in the market landscape; (iv) companies relating to it are now able to guarantee new investment; (v) news media is more and more aware of the importance of the technology (unknown in step (i)); (vi) GitHub mentions show the growing technology communities in contact with the trend; (vii) the job market also reacts to the trend.
The system that is able to access the data sources that relate to the items above, is also able to track the term throughout the several phases of popularization. It is also able to show the current status of a particular topic of interest, and optimally alert for potentially trendy topics in the future.
The analysis of indicators in pair with other data-driven perspectives can put the problem into a global context. In the visualisation of Figure 2 we consider five dimensions, x-axis, y-axis, bubble size, bubble color, and time, to represent the many aspects of the representation of the information. The explorations based on this analysis lead us to aspects of the causality inherent to the problem itself, in the sense that some of the answers to the problem in analysis lie in the analysis of the causes of that problem.
Figure 2. The 5D animated visualisation of indicators and their relations.
IV. On The Coronaviruswatch Observatory
When the World Health Organization (WHO) announced the global COVID-19 pandemic on March 11th 20201, following the rising incidence of the SARS-CoV-2 in Europe, the world started reading and talking about the new Coronavirus. The arrival of the epidemic to Europe scaled out the news published about the topic, while public health institutions and governmental agencies had to look for existing reliable solutions that could help them plan their actions and the consequences of these.
Technological companies and scientific communities invested efforts in making available tools (e.g. the GIS2 later adopted by WHO), challenges (e.g. the Kaggle COVID-19 competition3), and scientific reports and data (e.g. the repositories medRxiv4 and Zenodo5).
In March 2020 we have released the first implementation of this global observatory as the Coronavirus Watch portal6, aiming to contribute to a multinational response to the global crisis. It was made available by the UNESCO AI Research Institute (IRCAI), comprehending several data exploration dashboards related to the SARS-CoV-2 worldwide pandemic (see the COVID19-configured news portal in Figure 4). This platform aimed to expose the different perspectives on the data generated and trigger actions that can contribute to a better understanding of the behavior of the disease (see Figure 3).
Figure 3. The IRCAI observatory for the COVID-19 pandemic.
Figure 4. The SDG6 observatory focused on water-related priorities.
The portal includes a real-time news monitoring system that can be focused at European and national level, side-by-side with the data on the progress of the pandemics made available by the Worldometer7 and the Center for Disease Control8. The visual representation of the details of those indicators were made available over animations showcasing the live comparisons in 5D (as in Figure 2), the trajectories of the most affected countries, and the details of the progression of the disease. It also included perspectives on the mobility, sourced on the Google Community mobility data9, a social distancing simulator, and exploration tools based on the published biomedical research (see 10).
To improve the resolution of these results and to optimise their relevance for European public health agencies, we have developed a set of COVID-19 focused tools on the MIDAS platform11. This system was designed for evidence-based decision-making in public health. This approach allows us to validate the usability of the global observatory on a cross-EU level within the COVID-19 context, integrating both health news and biomedical research exploration (see Figure 5).
Figure 5. Exploring scientific research through complex data visualisation.
V. On The Water Observatory
Climate change is a global problem that in the recent years has been in the focus of European and Worldwide strategies. The priorities in European Union are rapidly changing towards sustainability and environmental efficiency, transversely to most domains of action. The European Commission’s Green Deal aiming for a climate neutral Europe in 2050, and boosting economy through green technology12 provides a new framework to understand and position water resource management in the context of the challenges of tomorrow13. In this context, the NAIADES project14 aims to improve the water resource management in a global context, including European regions where water scarcity is predicted, also dealing with concerns as, e.g., saline intrusion and groundwater contamination. To contribute to this cause, we deployed a Global Water Observatory15 that is focusing on water-related aspects allowing the user to explore the several layers of information it is providing, from news and social media to published science, weather models and indicators. The NAIADES Global Water Observatory will not only contribute to the improvement of European sustainability in water-related matters, but will also provide the local actors on the water resource management an active role in that.
Water is fundamental to all human activity and ecosystem health, and is a topic of rising awareness in the context of the recent discussions on climate change. Water resource management is central to those concerns, with the industry accounting for over 19% of global water withdrawal, and agricultural supply chains are responsible for 70% of water stress16. In 2015 the UN established ”clean water and sanitation for all” as one of the 17 Sustainable Development Goals, aiming for eight targets to be achieved by 203017. The UN secretary-general points out in April 2020 that SDG 6 is ”badly off track” compromising the progress on the 2030 Agenda18. As noted by the Organisation for Economic Cooperation and Development (OECD), the ‘water crisis’ has often proven to be a crisis of governance19, where water scarcity is largely caused by mismanagement of resources, leading to a global prioritisation20.
The intention to globally monitor water resources is not new, and already in the late 1960s21 the first spatially-distributed water resources model appeared, with first operational uses of satellite observations in water resources developed in the early 1980s22. The reliable management of water resources is only possible under condition of availability of adequate qualitative and quantitative information about state of the water body at any moment of time. Taking advantage of the recent technological progress enabling much innovation that was unthinkable a few years ago, the concept of the Digital Twin is increasingly entering the water sector as an innovation driver. Due to the rapidly growing awareness of the sustainability challenges that we are facing in Europe and worldwide in the context of the water resource management, there has been much work done to develop systems that are able to collect information about the available water and even simulate and forecast that in the near future. These are usually geolocation-based systems ingesting water-related data to enable real-time monitoring of resources and usage23. The other typical approach is the systems focused on workflows in the water sector, including the management of water distribution networks, hydraulic efficiency or leak/fraud detection, better suited to those companies that already have their infrastructure in place and know well what do they want to monitor24.
The approach we proposed in this paper is novel in many ways. The news monitoring perspective is monitoring water scarcity and water quality in worldwide and, in particular, in the surrounding regions of the water resource management agencies is is mainly addressing, together with their audiences. This is also including a Twitter observatory that adds to the already implemented measure of impact of the monitored news in Facebook, a social media component to the observatory. The global indicators that are already available for visualisation, sourced over the UN Open Data Portal, the water-focused Sustainable Development Goal 6, and the Wold Bank Data Portal, can help us understand water-related aspects of the climate crisis.
The important role of scientific research in this context, and the best practices that can be extracted from this data, is explored with a complex data visusalisation technology that allows the user to powerful Lucene-based queries over the article’s metadata aiming to refine search by moving a pointer over clusters of related topics (see Figure 5). We will also be including other data analytics technologies to analyse simultaneously multiple time-series providing interactive exploration tools to understand trends in the weather and water-related impacts to it.
The localisation of this global system entails the customisation of its functionality in news monitoring, ingestion of local indicators and exploration of scientific research on observed problems in, e.g., groundwater contamination. In that, the observatory is synchronising with the priorities of regional water providers. These agencies (e.g. Aguas de Alicante) are collecting data on their water resource management services to improve the customer satisfaction and optimise their system.
VI. Conclusions nnd future work
The results discussed in this paper show the potential impact of the proposed data-driven global observatory in contexts like public health and climate change preparedness. This integrated system is capable to monitor in real-time the worldwide news and social media, statistics, published science, weather and many other data streams that are identified as useful and can be provide complementary value to those considered already. We will be deploying this system in the context of other global problems where there is enough data to provide useful and meaningful contribution, either in other aspects of the climate crisis to better plan response, in addressing other epidemiological concerns to serve as early warning, or in addressing a new focus in the context of data science for social good.
We are now working on extending this system to integrate the information retrieved by the topics searched over the internet provided by Google Trends, regarding issues related to the context in focus. The user will be able to explore a wide range of indicators and compare trends in a global and local level throughout a meaningful timeline. We will also be reusing EC-funded open datasets and initiatives in order to ingest this information as European-level indicators to complement the analysis. Furthermore, we will be further investigating the validity of the localization of this Global Water Observatory, integrating some of the local data that can be provided by the user, and customizing news sources to their own priorities, as well as making available data exploration dashboards that allow for further insight and evidence-based policy.
VII. Data availability
For this paper, we used only open data. In particular, we have used the MEDLINE dataset25 and the worldwide news are being collected by 26, freely available online, but which the dataset we do not have permission to share.
VIII. Ethics and consent
Ethical approval and consent were not required
Faculty Opinions recommendedReferences
- 1.
World Health Organization: WHO Director-General’s opening remarks at the media briefing on COVID-19 -11 March 2020. Reference Source
- 2.
ArcGIS: WHO COVID-19 Dashboard. 2020. Reference Source
- 3.
Kaggle: COVID-19 Open Research Dataset Challenge. Reference Source
- 4.
medRxiv: COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv. 2020. Reference Source
- 5.
Zenodo Coronavirus Disease Research Community. 2020. Reference Source
- 6.
IRCAI: Coronavirus Watch portal. 2020. Reference Source
- 7.
WorldoMeters. 2020. Reference Source
- 8.
Center for Disease Control and Prevention. 2020. Reference Source
- 9.
Google: COVID-19 Community Mobility Report. 2020. Reference Source
- 10.
Massri B, Costa JP, et al.:
A global COVID-19 observatory, monitoring the pandemics throughtext mining and visualization. Informatica (manuscript in revision), Slovenian Society Informatika, 2021.
- 11.
Pita Costa J, Grobelnik M, Fuart F, et al.:
Meaningful Big Data Integration For a Global COVID-19 Strategy.
Computer Intelligence Magazine.
IEEE, 2020; 15(4): 51–61. Publisher Full Text
- 12.
European Commission: European Green Deal. 2019. Reference Source
- 13.
European Commission:
Water Scarcity and Droughts in the European Union. 2019. Reference Source
- 14.
NAIADES Project Portal. 2021. Reference Source
- 15.
NAIADES Water Observatory. 2020. Reference Source
- 16.
Our World in Data: Water Use Stress. 2020. Reference Source
- 17.
United Nations Development Programme: Goal6 -clean water and sanitation. 2020. Reference Source
- 18.
Blazhevska V:
United Nations launches framework to speed up progress on water and sanitation goal. United Nations Sustainable Development, 2020. Reference Source
- 19.
OECD: Water governance in OECD countries: A multi-level approach. OECD, 2011. Reference Source
- 20.
Akhmouch A, Clavreul D, Glas P:
Introducing the OECD principles on water governance.
Water International.
2018; 43(1): 5–12. Publisher Full Text
- 21.
Freeze RA, Harlan RL:
Blueprint for a physically-based, digitally-simulated hydrologic response model.
J Hydrol.
1969; 9(3): 237–258. Publisher Full Text
- 22.
Ramamoorthi A:
Snow-melt run-offstudies using remote sensing data.
Sadhana.
1983; 6(3): 279–286. Reference Source
- 23.
Idrica: Blue Dot Observatory. 2018. Reference Source
- 24.
Idrica: GoAigua - Smart Water for a Better World. 2019. Reference Source
- 25.
North American National Library of Medicine:
MEDLINE: Description of the Database. 2022. Reference Source
- 26.
Institute Jozef Stefan:
IJS Newsfeed: a clean, continuous, real-time aggregated stream of semantically enriched news articles from RSS- enabled sites across the world. 2022. Reference Source
