Introduction
Eutrophication of coastal marine waters and freshwater ecosystems is of global concern as these aquatic systems continue to face significant pressure from human activities and climate change (European Commission, 2021; Riemann et al., 2016; Schmidt et al., 2023). Therefore, it is crucial to monitor the quality of aquatic ecosystems. Recent advancements in sensor technology have enabled rapid collection of high-resolution sensor data from freshwater systems, offering new opportunities for high-frequency (HF) monitoring (Bieroza et al., 2023; Blaen et al., 2024; Rode et al., 2016). Although some sensor technologies may claim to require minimal maintenance and quality assurance (QA)/quality control (QC), integrity and accuracy of HF data are essential and important to avoid producing unreliable and useless data. Therefore, data QA and QC are important, and there is a pressing need to implement standardized QA-QC applications (Bieroza et al., 2023; Wollheim et al., 2018). In a study by Rozemeijer et al. (2025) they provide a guideline for the best practice when working with HF water quality monitoring, including a description of field practices together with considerations about the purpose of deploying a sensor. Their guidelines are based on international experiences and can be used for optimizing QA workflows, contributing to a complete and standardized QA-QC process when working with HF water quality monitoring.
HF sensor monitoring can be useful both regarding scientific understanding of transport pathways of solutes in catchments, hydrochemistry, and aquatic ecology (Bieroza et al., 2023) and concerning more management aspects through national monitoring programs. In Europe, the chemical status of water bodies (lakes, streams, etc.) is observed with the goal of achieving good status for Europe’s water bodies according to rules in the Water Framework Directive (WFD) (European Commission, 2000). National monitoring programs typically use traditional infrequent discrete grab sampling techniques, but these may miss crucial events and information (Cassidy & Jordan, 2011; Levi et al., 2024). To address this issue, HF sensor technology has the potential for future integration into national monitoring programs (Rode et al., 2016; van Geer et al., 2016) to improve both groundwater and surface water quality management. However, when using HF sensors, there is a need for robust QA and QC, data storage infrastructure, and open data access to enable the investigation of long-term trends. Bieroza et al. (2023) advocate for the integration of HF water quality measurements into existing monitoring programs as an important step toward improving water quality, together with increased stakeholder engagement, particularly due to changing environmental conditions and stricter regulatory requirements.
Examples of larger international sensor networks for national monitoring are the U.S. Clean Water Act (CWA), the National Ecological Observatory Network (NEON) in the USA, and the Australian Water Act (AWA) (Bieroza et al., 2023; Chabbi & Loescher, 2017; Dekker & Hestir, 2012; Edmonds et al., 2022; van Geer et al., 2016). In Europe, several countries use sensors to monitor the quality of their freshwater systems. Ireland has pioneered this approach by incorporating sensors into its Agricultural Catchments Program (ACP) for over a decade (Jordan et al., 2012). In 2005, the Global Lake Ecological Observatory Network (GLEON) was launched. It is a voluntary international network connecting researchers working with high-resolution sensor data from lakes and reservoirs with the aim to share and communicate their findings (Read et al., 2016; Weathers et al., 2013).
In most other European countries, apart from Ireland, sensors are not fully implemented in national monitoring programs but are often used in special cases, for example in the regional monitoring in Sweden since 2017 (Fölster et al., 2019) and in a limited number of national river monitoring programs in Norway (Kaste et al., 2022; Skarbøvik et al., 2023b). Another example is Lake Maggiore, which has been part of the Long-Term Ecosystem Research (LTER) network, with continuous monitoring of its physical, chemical, and biological characteristics since the 1980s. In 2020, a HF monitoring system was introduced, featuring a buoy equipped with sensors to complement traditional discrete sampling methods. This HF monitoring program, developed through a cross-border collaboration between Italy and Switzerland, enhances lake quality monitoring and supports effective freshwater ecosystem management (Rogora et al., 2023). QA-QC procedures and other main features (hardware and software) for such an HF monitoring system of lakes are under development based on the data from Lake Maggiore (Tiberti et al., 2021). The QA-QC of the HF sensor data collected in the different countries varies considerably as shown in a study of turbidity sensor results from 11 different monitoring programs/research projects from six different north-western European countries (Skarbøvik et al., 2023a). Some countries use manual methods for data QC, but this can be time-consuming and makes it difficult to compare not only the data between the countries but also between the different monitoring programs (Lannergård et al., 2019; Skarbøvik et al., 2023b; Skarbøvik et al., 2023a; Yang et al., 2023). In addition, large-scale monitoring programs like NEON and GLEON are continuously expanding their geographic reach and sensor equipment (Schmidt et al., 2023). Furthermore, this enormous amount of HF sensor data requires new smart methods of data quality assurance to ensure that the quality of the collected sensor data is good and acceptable and also follows the FAIR principles (Wilkinson et al., 2016). In some countries, for example Poland, HF sensor data is QA and QC by private companies applying confidential methods. In these cases, it is difficult to understand how the QC has been performed.
Several methods have been developed for the QA and QC of aquatic sensor data from fresh waters (Barnett et al., 2012; Belay et al., 2023; Jones et al., 2022; Read et al., 2016; Schmidt et al., 2023; Schmidt & Kerkez, 2023; Talagala et al., 2021). Some methods such as B3 software, ODM Tools Python, the Great Expectations Framework, and the GCE Data Toolbox are mainly based on manual QC and statistics (Barnett et al., 2012; Gong & Campbell, 2022; Horsburgh et al., 2015; Read et al., 2016; Sheldon, 2008), while others, such as pyhydroqc and SaQC, use more advanced machine learning methods (Belay et al., 2023; Jones et al., 2022; Leigh et al., 2019; Schmidt et al., 2023; Schmidt & Kerkez, 2023).
Belay et al. (2023) performed a numerical assessment of 13 deep-learning algorithms on two time-series of sensor datasets and found that there are still challenges in anomaly detection techniques. This is primarily due to the lack of a universal approach as each method is specifically designed for a particular application (Belay et al., 2023). Moreover, these machine learning (ML) algorithms can be difficult to follow for the user and are time-consuming to set up and train to fit a specific dataset, and the computational costs involved in training/testing are large (Belay et al., 2023). Schmidt & Kerkez (2023) also emphasized the need for further research into their QC machine learning method to evaluate the performance of the ML model over longer periods and to assess the generalizability and transferability of ML models trained on various sensor data monitoring identical environmental processes.
Schmidt et al. (2023) developed the SaQC method, which includes 32 different QC functions to flag anomalies. They divided the methods into three overall parts: processing, basic QC tests, and advanced QC tests, where the third part includes machine learning algorithms. This means that the user can choose which QC test they want to use for a specific sensor. For example, they included the Search and TRace AnomalY (STRAY) algorithm for multivariate spike detection. The STRAY algorithm is unsupervised and uses k-nearest neighbor distances for score calculations for anomaly detection (Talagala et al., 2021). Leigh et al. (2019) introduced a ten-step approach for automated detection of anomalies in HF water quality data from in situ sensors. They reviewed and compared different anomaly detection methods such as automated classification rules and several regression and feature-space methods and advocated for a QC approach, that highlighted the importance of understanding end-user requirements and data characteristics to determine the specific type of anomalies and their importance for each sensor type (Leigh et al., 2019). This framework also allows users to select the most appropriate anomaly detection methods for their specific sensor data.
Anomaly detection concerning HF data has different definitions depending on the focus of the study and input data (Talagala et al., 2021). Goldstein and Uchida (2016) identify three different types of anomalies: i) Global anomalies, which are easy to detect since they are very different than the rest of the dataset, ii) local anomalies, which can be difficult to detect since they can only be identified as an anomaly when compared to its neighborhood data points, and iii) clusters of anomalies, which can be very difficult to detect as these types of anomalies can look like the normal dataset when comparing each datapoint to its neighborhood data points. The third anomaly type has received less attention compared to the other two categories (Goldstein & Uchida, 2016; Talagala et al., 2021). In this study, we define an anomaly closely related to the general definition in (Talagala et al., 2021) that states that an anomaly significantly deviates from the majority of the data, with a much larger distance between typical observations and anomalies compared to the distances among typical observations.
Quality assurance (QA) procedures for environmental sensor data ensure that sensors measure reliable concentrations and meet established requirements. This is achieved by following procedures when using in situ sensors. Quality control (QC) involves testing the validity of the sensor data, thereby validating and controlling the data after collection. In this study, we developed a five-step workflow outlining the overall steps for working with environmental sensor data in freshwater environments. This includes essential steps such as sensor deployment, maintaining an electronic logbook, and managing a database for the large volume of data. As described above, Rozemeijer et al. (2025) provides QA workflows with practical guidelines for sensor deployment, contributing to a complete QA-QC process.
This study aimed to explore the feasibility of implementing a simplified robust QC approach, called SentemQC, to HF sensor data from multiple freshwater systems. The focus was on developing a method that was computationally efficient, transparent, and had a user-friendly setup/interface. The method was designed to identify and mark anomalies and detect errors or patterns in the raw sensor data. SentemQC identifies a data point as an anomaly if it significantly deviates from the other data points within a predefined “moving window”. This deviation is characterized by a much larger distance to the other data points in the moving window, based on statistical calculations such as average, standard deviation, and/or mean. Using SentemQC, all three defined types of anomalies by (Goldstein & Uchida, 2016) can be detected.
The objectives were i) to develop a simple, open access, and computationally efficient method for QC procedure of HF sensor data called SentemQC; ii) to investigate the robustness of SentemQC by testing it on different sensor measurement types from different aquatic systems and locations; iii) to explore the advantages and disadvantages of the SentemQC program compared to other available programs.
Study sites
In this study, we examined water quality sensor data from three different freshwater systems (Table 1). The higher measuring frequency in the case of streams is chosen because headwater streams are known to be extremely dynamic systems, which necessitates a higher monitoring frequency to capture the changes in water quality parameters. In contrast, lakes generally respond more slowly to environmental changes compared to small headwater streams, allowing for a lower measuring frequency.
Table 1. Overview of the examples of high-frequency sensor data included in this study.
This Table illustrates the data examples in the study used for quality assurance and quality control (QA-QC) using SentemQC of high-frequency sensor data.
| Type of waterbody | Field site | Data parameters | Measuring period | Measuring
frequency |
|---|
| Stream | Lyby-Grønning | Nitrate | 2021-05-05 to
2023-01-03 | 1 min |
| Stream | Horndrup | Turbidity, nitrate | 2021-03-10 to
2023-01-02 | 1 min |
| lake | Ormstrup | Dissolved oxygen | 2023-07-01 to
2023-10-01 | 15 min |
Lake warming mesocosm
experiment (LWME) | Lemming | Dissolved oxygen, pH | 2014, 2015, and
2016 | 30 min |
The two stream sensor stations were established in Horndrup Stream and Lyby-Grønning Stream in Jutland, Denmark (Figure 1). The Horndrup stream station covers a catchment area of 5.48 km2, of which 70% is agricultural land and 19% forests (Levin et al., 2017). The soils are primarily dominated by moraine deposits: fine sand mixed with clay (59%) and fine clay mixed with sand (27%). The stream station is positioned in a part of the stream covered by forest. The Lyby Grønning Stream sensor station covers a catchment area of 11.3 km2 where 84% is agricultural land (Levin et al., 2017). The soils are dominated by fine sand mixed with clay (47%) and fine clay mixed with sand (52%). Both sensor stations are part of the Danish national monitoring program of the aquatic environment, NOVANA (Petersen et al., 2021; Windolf et al., 2012).
Figure 1. Study sites.
The location of the different sensor stations in Denmark. Red dots represent the stream sensor stations Lyby-Grønning Stream and Horndrup Stream, the blue dot represents the lake sensor station Lake Ormstrup, and the yellow dot represents the Lake Warning Mesocosms experiment station in Lemming, Denmark.
The lake sensor station is placed in Lake Ormstrup in Jutland, Denmark. The lake is 11 ha and has an average depth of 3.4 m and a hydraulic retention time of >1 year. The lake catchment is small (ca 1 km2), and the lake only has outflow during heavy rainfall events, often in the winter period. The lake is eutrophic with high concentrations of chlorophyll and total phosphorus (TP) (Davidson et al., 2024; Søndergaard et al., 2023).
The lake warming mesocosm experiment (LWME) sensor stations are located in Jutland, Lemming, Denmark. Here, 24 flow-through outdoor freshwater mesocosms were established in 2003 and are still running in 2024. The overall aim of the mesocosms is to experimentally study how increased temperatures affect shallow lake systems at contrasting nutrient concentrations. A description of the mesocosm setup can be found in Liboriussen et al. (2005).
Data and methods
Sensor data
This study applied various sensors and data parameters (see Table 1). At the stream stations, nitrate sensors from HACH. In-Situ Nitratax™ plus UV (HACH, Düsseldorf, Germany) was employed, and at Horndrup an additional turbidity sensor from HACH was installed. These sensors gathered data continuously for nearly two years (Table 1). The sensors were manually cleaned in the field with acid every two weeks and installed with a measuring frequency of 1 minute. The nitrate sensors were in-situ Nitratax plus UV purely photometrical sensors that are based on the principle that nitrate and nitrite ions absorb ultraviolet (UV) light at a wavelength of 210 nm. The Nitratax plus UV sensors had the manufacturing series number 1640674 for the nitrate sensor installed in Horndrup Stream, and 2060341 for the nitrate sensor installed in Lyby-Grønning Stream. The sensor sends a light signal every minute. However, there is a built-in automatic function that averages the last three measurements to stabilise the measurements. The sensors also have a built-in wiper technology (double Wiper) for lens cleaning. The double wiper is designed to keep the optical system free of blockages and growth of e.g. algae and deposits (HACH, 2012; HACH, 2014). The nitrate sensors have a measurement range between 0.1 and 100.0 mg N/L. The precision/uncertainty is 3% or ±0.5 mg N/L (HACH, 2012; HACH, 2014). Additionally, the nitrate sensor measurements were before using the SentemQC method corrected for zero offsets, which occur linearly for the nitrate sensors. This correction was achieved by using the sensor measurements taken before and after manual cleaning, along with linear interpolation.
The turbidity sensor installed at the Horndrup stream station were in-situ optical sensor called Solitax (HACH, Düsseldorf, Germany) with the manufacturing series number 1942974. These sensors determine turbidity by an infrared light measuring technique, which quantifies the intensity of light back-scattered by the water at 90° where the light source is infrared light (860nm). The turbidity was measured in the unit FNU (Formazin Nephelometric Unit), which complies with the European ISO 7027. The turbidity sensor has a measurement range between 0.001 to 4000 FNU and a measuring precision/uncertainty of <5% or ±0.01 FNU (Table 2) as well as a built-in function stabilizing the measurements. The turbidity sensor has an inferred light source that measures constantly and provides a 1-minute measurement that is an average of the values recorded during the last minute. Additionally, it has a built-in wiper technology (double Wiper) for lens cleaning (HACH, 2017; HACH, 2018).
Table 2. Overview of sensor uncertainty/accuracy for the high-frequency sensor data included in this study.
The uncertainty/accuracy of the sensor equipment used to collect the high-frequency sensor data in this study are provided by the manufactories and can be found following the sources in this Table for the different high-frequency sensor data.
| Field site | Data
parameters | Uncertainty/accuracy | Source |
|---|
Lyby-Grønning
and Horndrup | Nitrate-N | 3% of the measured value or ±0.5 mg N/L | (HACH, 2012; HACH, 2014) |
| Horndrup | Turbidity | Turbidity to 1000 FNU:
without calibration <5% of the measured value or ±0.01 FNU | (HACH, 2006) |
| Ormstrup | Dissolved
oxygen | ±0.1 mg/L (from 0 to 20 mg/L)
±2% of reading (from 20 to 60 mg/L) | (In-Situ, 2021) |
| Lemming | Dissolved
oxygen | < ±2% of actual value | (OxyGuard International A/S, 1994) |
| Lemming | pH | 0.1 of the actual value is used as a constant. This was chosen
because no accuracy is provided in the sensor manual. However,
we must be aware that the calibration itself is somewhat
uncertain; pH is a log scale and is rounded to 2 decimal places. | (OxyGuard International A/S, 2020) |
Dissolved oxygen (DO) and pH sensors were installed in LWME mesocosm named G2 in Lemming. The dissolved oxygen sensor used was an OxyGuard® two-wire sensor, model 420, and the pH sensor was an OxyGuard light-duty submersion sensor connected to a Manta pH measurement system. Both sensors had a measuring interval of 30 minutes, and all sensors were manually calibrated every week. Further description of the sensors and the mesocosm setup can be found in Liboriussen et al. (2005), and the uncertainty is provided in Table 2.
The dissolved oxygen sensors (DO) at Lake Ormstrup station 100011 were Aqua TROLL 500 (In-Situ, Fort Collins, CO, USA) multisondes installed near the surface (ca. 1.0 m) and at a deeper-water depth (ca. 3.8 m). The measuring frequency was 15 minutes, and the sensor had a wiper function that automatically cleaned the sensor every 15 minutes before measuring. The uncertainty is ±0.1 mg/L (from 0 to 20 mg/L) and ±2% of reading (from 20 to 60 mg/L) (Table 2) (In-Situ, 2021). In addition, the sensors were manually cleaned and calibrated every week. Further description of the sensor setup in Lake Ormstrup can be found in Davidson et al. (2024).
QA-QC method
When working with HF sensor data, we recommend following a specific workflow as outlined in Figure 2, ensuring QA-QC (quality assurance and quality control) of the HF sensor data. However, QC using the method described in this study (Step 4) is just one step of the process and stands out for its robustness and simplicity. The SentemQC method can be applied to various HF sensor data by fine-tuning the parameters as described in the following sections.
Figure 2. Conceptual model of a hypothetical workflow for data preprocessing high-frequency (HF) sensor data.
Step 1 includes an illustration of a sensor installed in a waterbody. Step 2 represents establishing a database containing all HF sensor data. Step 3 represents the calibration of the sensor data with an example showing correction for zero offset of a nitrate sensor. The y-axis is the nitrate-N, and the x-axis is the datetime. The blue line represents the raw data, and the orange line is the corrected sensor data. Step 4 represents and illustrates the quality control (QC) workflow. An example from SentemQC is shown in a Figure where the y-axis is the pH and the x-axis is the datetime. The gray dots are the approved data points and the red dots are the datapoints marked as anomalies. Step 5 represents the use of the electronic logbook to mask out anomalies (red box). Step 6 represents the gap filling of missing data, and step 7 represents the use of the sensor data to for example load calculations.
Implementation
The SentemQC method was developed as a freely available Python code to make the method available and easy to apply for future users. The method is free to download from Zenodo at: https://zenodo.org/doi/10.5281/zenodo.13830624 (Andersen et al., 2024) as an open-source package that aims to develop a smart, simple, open-access, and cost-efficient method for the QC procedure of HF sensor data.
The method is based on simple mathematical algorithms using averages, median, and standard deviation calculations. The algorithm uses moving windows of different widths to flag different data patterns as anomalies. To flag large continuous periods of anomalous data, very wide moving windows are applied, yielding a more stable moving average, while individual anomalies and anomalies are flagged with narrow windows. The narrow moving windows can only be used when the large continuous periods of anomalous data are flagged. The mean, median, and standard deviation were computed for the moving windows using the Pandas library in Python (The pandas development team, 2024). Specifically for each mowing window, rolling statistics were calculated using the Rolling object from pandas.core.window.rolling, with the standard deviation calculated via the Rolling.std() method. Similarly, the moving mean and median were calculated using the Rolling.mean() and Rolling.median() methods, respectively. The moving windows are all centered, and the calculations for a given data point within the window will therefore have the same number of data points before and after. When a moving window passes through data, calculations are only made on the data points that are not flagged. If a given data point lies outside the window, it will be flagged and excluded from the subsequent window calculations. The width of the individual windows is defined by the user as a given number of data points or a time interval (Table 3), while the height of the window measurements changes dynamically depending on the data included in the moving window. The upper limit of the moving window for a given data point is the sum of the moving average or median, the product of the moving standard deviation, a predefined factor, and a constant (Equation 1). The lower limit is the difference between the moving average or median, the product of the moving standard deviation, a predefined factor, and a constant (Equation 2).
Table 3. Information about the five different moving windows in the SentemQC method.
Moving windows
in SentemQC | Number of
datapoints | Calculations | Purpose of window |
|---|
| 1 | 960 | Average | To capture clusters of anomalies |
| 2 | 960 | Average | To capture clusters of anomalies |
| 3 | 48 | Median | To include some robustness in the method |
| 4 | 5 | Average | To capture single anomalies, both local and
global anomalies |
| 5 | 5 | Average | To capture single anomalies, both local and
global anomalies |
The upper boundary for the moving windows for a given sensor measurement is defined by Equation 1:
The lower boundary of the moving windows for a given sensor measurement is defined by Equation 2:
Where topwin is the top of window/ upper window boundary, botwin: bottom window/ lower window boundary, avg: average of the nearest measurements, std: standard deviation of the nearest measurements, std_factor: a constant representation of the factor by which the standard deviation is multiplied (sensor dependent), topadd: a constant that expands the top height of the first moving window (sensor dependent), bottomsub: a constant that expands the bottom height of the first rolling window (sensor dependent).
In total, the method uses five windows, and for the fifth moving window, these calculations are weighted with a triangular window, which means that the data points in the middle of the window are given greater weight than those at the beginning and end. Window three does not calculate moving averages but moving medians to lend some robustness to the method. The specific purpose and characteristics of the five different windows can be found in Table 3. The five windows are chosen to ensure that all types of anomalies are flagged, both global and local individual anomalies, and clusters of anomalies as defined in the Introduction section. In addition, warm-up and cool-down periods are defined in SentemQC to include 480 records for both warm-up and cool-down, however, this number of records can be adjusted by the user. This is implemented in the method to ensure enough data points for average and mean calculations in each moving window. SentemQC will warn if there are insufficient data points for the warm-up and cool-down periods.
Information about the five different moving windows in the SentemQC method. The moving windows are calculating either moving averages or medians based on the input data. Each window serves a unique purpose in identifying clusters of anomalies or single anomalies, both on a global and local scale.
The uncertainty of the sensor measurements related to the sensor technology can be defined in the configuration when conducting the parameterization of the QC method as ± in the first two moving windows. The uncertainty can be included as a percentage called “uncertainty_pct” in the parameterization or as a constant called “uncertainty_con”, depending on how the uncertainty is described by the sensor company, see extended data for the final parameterizations (vant Veen et al., 2024). For example, for the NITRATAX sensor from HACH, the uncertainty is 3% of the measured value or ±0.5 mg N/L (HACH, 2012; HACH, 2014). The different uncertainties used in this study are given in extended data 1(vant Veen et al., 2024) and Table 2.
The method identifies anomalies in the data by comparing the average concentration within a moving window with the established uncertainty range. If a data point falls outside the upper or lower limits of this uncertainty window (represented by ±), it gets flagged as an anomaly. We assume that the uncertainty is found by comparing discrete water grab samples with sensor measurements under controlled conditions in the laboratory and is equally distributed around the measurement point. More specifically, when there is an uncertainty factor associated with the sensor parameterization, SentemQC evaluates whether the sensor measurement plus or minus the uncertainty falls outside the calculated moving window. If the measurement remains within the bounds defined by the uncertainty range, it is not flagged as an anomaly. This is because the SentemQC model is not permitted to outperform the defined uncertainty threshold. A conceptual figure illustrating the QC method is seen in Figure 3.
Figure 3. Conceptual model showing the principle behind the SentemQC method.
The y-axis is the oxygen while the x-axis is the datatime in both figures. The scatted black boxes represent the different moving windows in the SentemQC program. The gray dots are the approved datapoints, while the red dots are the anomalies flagged by SentemQC. The calculation of the moving average in each window is simplified by the red, blue, and purple boxes.
Operation
SentemQC requires a recent version of Python installed. An environment file that includes the Python versions and packages to run SentemQC is available for download at: https://zenodo.org/doi/10.5281/zenodo.13830624
All files including SentemQC software and the test data for this paper are free to download from Zenodo at: https://zenodo.org/doi/10.5281/zenodo.13830624.
SentemQC operation flow consists in i) Setting up the SentemQC model, and ii) Running the SentemQC model. In total, SentemQC includes six steps illustrated in Figure 4. Steps 4 and 5 are described in more detail in the following section. SentemQC is set up and controlled by a Python file called Main.py, in which the data period, sensor parameter, and input data are defined. This file loads the QC module and parameterization file. It is important and necessary to have electronic logbooks from the sensor station for the calibration and validation/evaluation of the QC method. In the logbook, each field visit, manual cleaning, calibration, and physical distribution should be noted. Therefore, collaboration between researchers and technicians is important. The QC method produces three outputs: 1) Data Frame. This table contains detailed information about data points identified as anomalies. It includes columns for: The raw data points and whether the data point was flagged as an anomaly or not. If flagged, the specific window that identified it as an anomaly is recorded. Additionally, it shows all valid data points. 2) Time Series Plot. This visual representation displays the entire dataset as a time series. Anomalies are marked out to provide a quick overview of their occurrence. 3) An output file. This file illustrates the total number of flagged anomalies and valid data points in each window. Please download and read the file: SentemQC – Description.pdf for a more detailed description of the operation and folder structure of SentemQC available at https://zenodo.org/doi/10.5281/zenodo.13830624.
Figure 4. The different steps in the SentemQC quality control (QC) method.
The boxes represent the six steps and workflow for data preprocessing in SentemQC.
Parameterization of the SentemQC model
The SentemQC method passes through the sensor data five times in total, as different moving windows cover different amounts of data. The windows are different, however, two are repetitions of the previous window in relation to the number of datapoints included in the window (Table 3). For example, window 1 has the same number of datapoints (width) as window 2. The windows are moving and begin directly after each other, which means that the windows do not interact with each other and are not connected. An example of the parameterization of the oxygen sensor at the Lemming field site from the library file (*RollWindowStdConfig.json*) and explanations of the components in the parameterization of different windows can be seen in the extended data (vant Veen et al., 2024).
Parameterization of the SentemQC model’s windows (step 4 in Figure 4) was done individually for the different sensor types installed at the field sites by fine-tuning the model to the specific sensor and field site. The optimization of the parameterization was performed by changing the constants in Equation 1 and Equation 2, which changes the heights of the moving windows (the parameters: topadd, bottomsub, and std_factor). Equation 1 and Equation 2 are the equations used for windows 1, 2, 4, and 5 for the sensor parameterization (window 3 calculates the median instead of the average). For each sensor measurement, the standard deviation and average were calculated using the temporally nearest neighboring measurements, depending on the windows number of measurements (for example, 960 for windows 1 and 2), centered on the measurement. As described above, the average, median, and standard deviation were calculated for the moving windows using the Pandas library in Python (The pandas development team, 2024).
For example, in Lyby-Grønning Stream, the nitrate concentration was relatively high compared to the concentration in Horndrup Stream, and some nitrate peaks occurred with a short duration on a daily basis, probably due to point sources. Therefore, it was important that the QC model did not mark out these high peaks as anomalies but still marked out real anomalies. For the parameterization of SentemQC, a shorter calibration period was used for each individual dataset compared to the entire dataset. This approach helps identify the most optimal parameterization for the different windows across individual sensors and field sites. Following the parameterization of SentemQC for the specific datasets, the model's performance was validated (step 5 in Figure 4). This validation process involves reviewing datapoints excluded from the parameterization of SentemQC to ensure the model accurately flags anomalies during periods with known issues using the electronic logbook. Consequently, the validation heavily relies on the electronic logbook.
The size of the parameterization/calibration and validation datasets depends on the available data points from the specific dataset. For Lemming, the year 2014 was chosen as the calibration period because the mesocosms were exposed to some extreme temperature experiments in 2014. Consequently, the parameterization was calibrated to handle extreme conditions.
See the final parameterization and the calibration periods for the individual sensors in extended data 1 (vant Veen et al., 2024). For more information about the parameterization read the file: SentemQC – Description.pdf available here: https://zenodo.org/doi/10.5281/zenodo.13830624.
Results
The results of using the SentemQC method show the robustness and adaptability of various sensor data types. Adaptability is achieved through model parameterization fitted to the specific characteristics of each sensor data set. Several snapshot examples illustrating the sensor data cleaning process enabled by the QC method are presented in the following sections.
Lake Warming Mesocosm Experiment (LWME) in Lemming
The QC model demonstrates its robustness by its effectiveness in cleaning the oxygen sensor measurements from LWME (Figure 5). The program identifies and flags various anomalies using different "moving-window rolls” (Figure 5). It was able to flag different anomaly types like: i) individual isolated anomalies, which were the most frequent and occurred in most months, ii) sensor error, which was observed in October 2015, indicating a specific issue with the sensor itself, iii) clumps of anomalies, which occurred in March, April, September, and December 2015, suggesting that external factors potentially affected the measurements, iv) calibration uncertainty, which was found in the April and July measurements, possibly due to inaccurate calibration of the sensor, and v) unstable sensor anomalies, which appeared around November 2015, indicating sensor malfunction.
Figure 5. Example of quality control (QC) of oxygen sensor measurements in the lake warming mesocosm experiment using SentemQC.
The example illustrates quality control (QC) of Oxygen sensor measurements in the lake warming mesocosm experiment (LWME) in Lemming 2015 using SentemQC and shows that the program flagged different types of anomalies such as: i) individual anomalies (marked with blue circles), ii) sensor error (red), iii) clumps of anomalies (black), iv) calibration uncertainty anomalies (orange), and v) unstable sensor anomalies (pink). The gray dots (Approved data) are the valid data points. The red dots (Flagged anomalies) are the flagged anomalies by the SentemQC program, and the numbers close to the red dots indicate the moving window flagging the anomaly.
Remarkably, the oxygen sensor data revealed more anomalies near the lower limit compared to the upper limit, which is also flagged by the program. This highlights its ability to handle anomalies and distortions, which contributes to its overall robustness. By using the automatic QC program instead of manual cleaning, 10 to 15% more valid measurements were obtained, demonstrating the efficiency and value of the program. Additionally, the calculation time for the QC algorithm for 1.3 million data points was 40 seconds (LWME, Lemming 2015 all mesocosms and parameters).
The QC model also demonstrates its robustness by effectively cleaning the pH sensor measurements from Lemming (Figure 6). Thus, the program identified and flagged various anomalies using different "moving windows” dominated by windows 1 and 3 in the period from March 18 to April 12, 2020 (Figure 6). As for the oxygen sensor data, the pH sensor data also contained more anomalies near the lower limit compared to the upper limit, which were flagged by the program. This also highlights its ability to handle anomaly distortions, which contributes to the QC programs’ overall robustness (Figure 6).
Figure 6. Example of quality control (QC) of pH sensor measurements in the lake warming mesocosm experiment using SentemQC.
The example illustrates quality control (QC) of pH sensor measurements in the lake warming mesocosm experiment (LWME) in Lemming 2020 using SentemQC. The gray dots (Approved data) are the valid data points. The red dots (Flagged anomalies) are the flagged anomalies, and the number indicates the moving window flagged the anomaly.
However, when zooming in, some individual data points in the data series appeared as potential anomalies although they were not flagged as anomalies by the QC program. This intentional design avoids overly restricting the QC algorithm, which could lead to mistakenly labeling valid data points as anomalies. Therefore, we accepted that anomalies falling inside the upper or lower limits of the uncertainty with values close to the overall trend might not be identified.
Lyby-Grønning stream and Horndrup stream
The QC model cleaning the nitrate sensor measurement from Lyby Stream is parameterized to include nitrate concentration peaks as valid even if the peak is brief (Figure 7). We know from discrete samples in the stream that the peaks around September 12, 2021, are valid. This evidences the model’s robustness because it is possible to make adjustments to include such peaks and not mark them as anomalies. At the same time, the QC model could mark the real anomalies that were not real changes in the nitrate concentration measurements in the stream. The high and low concentrations on 7 September 2021 reflect that the sensor was visited and manually cleaned with acid and checked for zero offsets that affect this nitrate sensor before and after manual sensor cleaning (Figure 7). The SentemQC model perfectly marked out these measurements as anomalies (Figure 7).
Figure 7. Example of quality control (QC) of nitrate sensor measurements in Lyby-Grønning Stream using SentemQC.
The example illustrates quality control (QC) of nitrate sensor measurements in Lyby-Grønning Stream in September 2021 using SentemQC. The gray dots (Approved data) are the valid data points. The red dots (Flagged anomalies) are the flagged anomalies, and the number indicates the moving window flagging the anomaly. The blue line represents the water level in the stream.
The QC model marked out single anomalies during the manual cleaning of the nitrate sensor measurement from Horndrup Stream on February 15, March 3, 15, and 30, and April 20 (Figure 8). From February 21 to 23, 2022, the sensor was blocked by branches, leaves, and sand from a heavy rainfall event that transported a lot of sediment to the stream. The QC program marked out these not valid measurements as anomalies. Also, the four single high anomalies around the March 8, 2024, were perfectly marked out as anomalies. In the same period there was no change in the water level (Figure 8).
Figure 8. Example of quality control (QC) of nitrate sensor measurements in Horndrup Stream using SentemQC.
The example illustrates quality control (QC) of nitrate sensor measurements in Horndrup Stream from February to May 2022 using SentemQC. The gray dots (Approved data) are the valid data points. The red dots (Flagged anomalies) are the flagged anomalies, and the number indicates the moving window that flagged the anomaly. The blue line represents the water level in the stream.
The use of the QC model cleaning turbidity sensor measurement also showed that the model marked out and captured anomalies and not-valid measuring periods (Figure 9). The model marked out anomalies such as single noise but also not-valid measurements during a field visit and during manual cleaning of the sensor for the nitrate sensor measurements in Horndrup and Lyby.
The model parameterization was set to include peaks in the turbidity measurement during, for example, heavy rainfall events where the turbidity is increasing. An example is December 2, 2021, where the model did not mark out the peak but only the single very high concentrations as anomalies (Figure 9). Window 3 marked out anomalies during this heavy rainfall event, showing that the model can easily be adjusted for cleaning turbidity sensor measurements (Figure 9).
Figure 9. Example of quality control (QC) of turbidity sensor measurements in Horndrup Stream using SentemQC.
The example illustrates quality control (QC) of turbidity sensor measurements in Horndrup Stream during a heavy rainfall event at the beginning of December 2021 using SentemQC. The red dots (Flagged anomalies) are the flagged anomalies, and the number indicates the moving window that flagged the anomaly. The gray dots (Approved data) are the valid data points. The blue line represents the water level in the stream.
Figure 10. Example of quality control (QC) of oxygen sensor measurements in Lake Ormstrup using SentemQC.
The example illustrates SentemQC quality control (QC) of oxygen sensor measurements from Lake Ormstrup from August 21 to September 1, 2023. The red dots (Flagged anomalies) are the flagged anomalies, and the number indicates the moving window that flagged the anomaly. The gray dots (Approved data) are the valid data points.
Lake Ormstrup
The SentemQC model effectively cleaned the oxygen sensor data from Lake Ormstrup (Figure 10). It identified anomalies between August 21 and September 1, 2023 (Figure 10), showing the ability of the model to handle data distortions and improve the data quality. For this oxygen sensor, the SentemQC model also flagged more anomalies near the lower limit than the upper limit (Figure 10). Some data points might look like anomalies, for example around September 25. However, these are likely caused by wind-induced partial mixing with lower oxygen water, which means that the data points are real.
Total amount of anomalies in the different sensor data
Table 4 summarizes the number of anomalies flagged by the different moving window settings during QC in SentemQC. The table also shows QC results excluding uncertainty to show the difference and importance of including this parameter (Table 4). Sensor data from the stream stations yielded a low anomaly rate of 0.1%, 0.1%, and 0.2% including uncertainty, and 0.2%, 0.3%, and 0.5% without uncertainty (Table 4). This was lower than the anomaly rates observed for sensors in Lake Ormstrup and LWME, which were 0.5%, 0.6%, and 0.8% with uncertainty and 1.6%, 1.4%, and 1.2% without uncertainty, respectively (Table 4). The differences in anomaly percentages between the different sensor types may be because the sensors installed in the lake and ponds measure oxygen and pH that are considerably more variable parameters than nitrate and turbidity (Table 4). As expected, SentemQC flagged more anomalies when uncertainty was not included in the program (Table 4). Additionally, SentemQC also flagged data points as anomalies due to sensor errors that may occur, for example, when sensors are undergoing calibration or maintenance in the laboratory (vant Veen et al., 2024). Although SentemQC captured and flagged these points as anomalies, this does not reflect the software's performance in identifying anomalies, and sensor errors were therefore excluded in Table 4.
Table 4. Summary of the number of sensor data points flagged as anomalies using SentemQC.
This table illustrates a summary of the number of sensor data points flagged as anomalies in the different windows using the quality control (QC) method SentemQC with and without uncertainty included in the method. The number of datapoints flagged and the percentages of anomalies flagged are illustrated.
Sensor/
parameter | Station | Number of data points
flagged as anomalies | Percent flagged as
anomalies |
|---|
With
uncertainty | Without
uncertainty | With
uncertainty | Without
uncertainty |
|---|
| Nitrate-N (mgN/L) | Horndrup Stream | 1399 | 4818 | 0.2 | 0.5 |
| Turbidity (FNU) | Horndrup Stream | 1298 | 1944 | 0.1 | 0.2 |
| Nitrate-N (mgN/L) | Lyby-Grønning Stream | 1025 | 2663 | 0.1 | 0.3 |
| Oxygen (mg/L) | Lake Ormstrup | 56 | 123 | 0.6 | 1.4 |
| Oxygen (mg/L) | Lemming LWME | 449 | 643 | 0.8 | 1.2 |
| pH | Lemming LWME | 84 | 264 | 0.5 | 1.6 |
There was no general pattern in the number or percentage of flagged anomalies across the different months for any of the sensors investigated, see extended data (vant Veen et al., 2024). The nitrate and turbidity sensors almost consistently showed an anomaly rate below 0.4% throughout the year. The sensor measurements therefore seem to be robust and not highly affected by seasonal disruptions during the year. However, the nitrate sensor in Horndrup Stream showed a higher anomaly rate during the winter months (December – February), perhaps due to a heavy bed sediment load, which might have buried the sensor and thus affected its measurements. Throughout the year, sensor height was therefore adjusted in Horndrup Stream based on the amount of rain in the catchment area to ensure that it remained submerged. Unfortunately, in February 2022, the sensor was buried by sediment, and in December 2022 it was positioned too high. In December, the water levels dropped, and the sensor ended up measuring air instead of water, which led to a significant anomaly spike for that specific month (vant Veen et al., 2024). Additionally, both the number of anomalies flagged for the pH sensor in Lemming in February and March 2020 at 1.1% and 2.3% and for the oxygen sensor in Lemming in February and March 2015 at 2.9% and 3.3% were relatively high compared to the other months (vant Veen et al., 2024).
Discussion
The main strengths of SentemQC are its simplicity, robustness, and cost-efficiency concerning time resources, low computer power, and the fact that it is freely available for download and use (Andersen et al., 2024). In SentemQC, all data points are connected because the algorithm compares the data points with nearby data points ("neighboring points") using narrow windows and then compares them with points further away using wider moving windows. By examining both nearby and distant data points, this approach ensures that both individual data points (global and local anomalies) and potential clusters of anomalies are effectively flagged (Figure 5). SentemQC is working even if the dataset contains periods with gaps, however, it does not account for missing data. Consequently, the program calculates the moving average or median in the defined moving windows, including the subsequent datapoints following a gap. This can result in the first datapoint after a period of missing data being flagged as an anomaly if the concentration, for example, has changed significantly. However, since the program only flags the data, it is possible to remove such flagging afterward. Additionally, SentemQC can provide a QC of 1.3 million sensor data points in 40 seconds. In general, methods such as the rate of change or recursive methods are often used for anomaly detection in dataset (Barnett et al., 2012; Géron, 2019). However, the rate of change method does not work well for our HF sensor data because the anomalies in the sensor data are sometimes grouped in clusters as described in the Results section. Therefore, this method only identifies and marks out the first and last data points of each cluster as anomalies, thereby missing the remaining anomalies within the group. Using recursive methods was also ineffective for the QC of our sensor data, they were simply too slow and very computer RAM demanding. Although we have not tested SentemQC on datasets where it fails to perform, exploring its limitations in highly complex environments, such as those with frequent biological disturbances or abrupt events, for example, gas bubble events for methane (Davidson et al., 2024), could be an interesting direction for future work. Importantly, SentemQC does not discard data but flags it, making it a valuable tool for identifying periods of change. This is especially interesting if supporting parameters are included in the method.
The B3 QA-QC software from GLEON (Barnett et al., 2012; Read et al., 2016) allows the user to investigate the sensor data semi-automatedly and correct the data periods in the QA-QC tools provided by the program (Barnett et al., 2012; Read et al., 2016). This can be time-consuming as the user has to manually tap and go through the different QA-QC tools and choose which test to use. Additionally, the user must choose between different QC tools and may select a moving average. However, the moving average is only calculated once compared to our five windows, which means that the data points in B3 are only compared to the neighboring data points a single time. This can be inefficient for identifying clusters of anomalies.
Leigh et al.'s (2019) ten step-framework also allows users to select the most appropriate anomaly detection methods for their specific sensor data. The first steps involve a collaborative visual examination of the water quality sensor data with end users and identification of the most important anomalies for the specific sensor data. However, this approach can be very resource intensive. In the framework, the user can choose between regression-based and feature-based methods. It requires some level of expertise to choose the right methods, which highlights the importance of clear communication between end-users and anomaly detection developers (Leigh et al., 2019). Additionally, regression-based methods are semi-supervised and require training, just like ML methods.
The SaQC method allows the user to choose between 32 different QC functions. As described in the Introduction, some of these functions include ML algorithms (Schmidt et al., 2023). These methods also require high-level expertise in the specific sensors and the QC methods. Moreover, there may be differences in how the same type of sensor data, such as turbidity, is cleaned because it is based on the user’s individual choice, and this can result in non-harmonized QC approaches. This can render a comparison between the different turbidity data difficult (Skarbøvik et al., 2023a). Furthermore, the SaQC method’s first part involves preprocessing of the data input as the data needs to be preprocessed before performing the actual QC. This process includes aligning timestamps as well as smoothing and transforming the data (Schmidt et al., 2023). This is not needed in SentemQC as it can perform QC of all, also shifting time stamps. The input data only needs to contain the correct date-time format. This is an advantage as information can get lost when shifting from 1-minute measuring frequency to 15 minutes.
Additionally, as described in the Introduction, the use of ML approaches for QC often requires substantial computer power and RAM, and it can be difficult to understand the underlying mathematical principles even when spending considerable time on understanding the programming (Belay et al., 2023). Furthermore, it also takes time to train the ML model compared to the simple SentemQC. Often the responsibility for conducting QC of sensor data quality falls on specialists with in-depth knowledge of the field (domain experts) but less experience with IT technology (Schmidt et al., 2023). Therefore, the transparency and simplicity of our method are a great strength, making it ideal for specialists, even with limited IT knowledge. Furthermore, the user is not required to choose between different methods and can use SentemQC for QC of all types of aquatic sensor data.
Moreover, as pointed out by Belay et al. (2023), ML approaches are all very specific for the type of sensor for which they are trained and require input data without noise. This means that it can be difficult to use ML approaches for different types of sensors. Belay et al. (2023) consequently concluded that ML methods are promising but that challenges remain as to the current state-of-the-art multivariate time series anomaly detection techniques. They concluded that the challenges are primarily due to the lack of a universal approach, as each method is specifically designed for a particular application (Belay et al., 2023).
The studies of Leigh et al. (2019) and Schmidt et al. (2023) both provide a general workflow for conducting QC of HF sensor data (Figure 4). Our simple automatic SentemQC could potentially be integrated as an additional QC test within the SaQC program or in Leigh et al.'s (2019) framework. As described in the Methods section, the SentemQC program only requires filling in one step to obtain good data quality (step 4, Figure 2). After the QC step (step 4, Figure 2), steps such as implementation of the electronic logbook are needed to mark out periods where the sensor measurement should not be included in the final data time series.
Another step in processing HF sensor data prior to use is gap-filling (step 6, Figure 2), e.g. calibration of the sensor data based on gap samples for correction of zero offsets (step 3, Figure 2), which SentemQC cannot presently perform. For the nitrate sensors in the streams used in this study, a zero-offset of the sensor data needs to be performed based on measurements from before and after manual cleaning of the sensor in the field. These zero offsets are most probably caused by algae, biofilm, and dirt that settle on the sensor lenses when installed in the streams (van’t Veen et al., 2020), and for these sensors, this offset is linear but depends on both stream-specific and season-specific conditions. Schmidt et al. (2023) corrected for zero offset of a dissolved organic carbon (DOC) sensor due to biofilm growth and dirt accumulation at the sensor lenses using the SaQC program. The offset exhibited an exponential development, and they used regression to correct for the drift. Therefore, we assume that offsets are site-specific for each sensor and stream.
In future work, both the electronic logbook and methods for gap-filling and calibration against grab samples can be implemented in SentemQC. Incorporating these important steps is a promising direction for the future development of SentemQC. From a technical perspective, integrating electronic logbooks could be included as a next step after the five moving windows have masked out anomalies. It would require users to adopt a standardized format for the logbook, particularly for datetime conventions, to ensure compatibility with SentemQC. This step would enable the tool to automatically identify and mask periods as anomalies marked as errors in the logbooks. For gap-filling, several methods could be considered as a final step in SentemQC. Rozemeijer et al. (2025) describe different options for gap-filling in water quality sensor time series. A simple approach, such as the linear interpolation method could be implemented as an option. Alternatively, more advanced techniques could be offered, including machine learning-based methods such as Random Forest, which seems like a promising method (Barcala et al., 2023). Further investigation is needed to determine the most suitable approach for integration into SentemQC. The different methods for gap-filling could be implemented as optional modules in SentemQC, allowing users to select the most appropriate strategy for their specific dataset. However, this would affect the complexity and universality of the SentemQC approach between different sensor data types.
The use of supporting sensor data parameters could be included in our QC method, e.g. water level, discharge, temperature, or conductivity. For example, Schmidt et al. (2023) used water level and water temperature as supporting parameters when performing QC of the spectral absorption coefficient at 254 nm (SAC254), which served as a proxy for measuring the dissolved organic carbon (DOC) concentration using the SaQC program. However, this required more transformation to normalize the input data. Leigh et al. (2019) also tested the use of supporting parameters for QC of environmental data, applying feature-based models. They also tested dynamic regression models (RegARIMA models) to forecast the concentration of one parameter based on multiple water quality variables. Yet, both methods can be difficult to follow and require transformation of the input data, and both regression methods and feature-based methods require great computer power for classification (Leigh et al., 2019). Using water level to QC of turbidity datasets could be useful. During high rainfall events, this should not lead to excessive flagging in the event of a rapid increase in turbidity. Similarly, for nitrate, programming the system to be less sensitive during rainfall events is possible, but comes with a delay, as seen for Horndrup and Lyby-Grønning streams. However, determining the appropriate delay is challenging due to site-specific and seasonal variations, rendering, as described earlier, the method more complicated. Another obvious co-comparison is oxygen and pH, which often follow each other, in lakes and mesocosms. Using supporting parameters would make SentemQC stronger but requires more computational resources. Furthermore, it would make the program less simple and transparent to follow when performing the QC of the sensor data, depending on the type of sensor data and supporting parameters. In addition, Lannergård, Fölster and Futter (2021) found that turbidity sensor measurements were not only correlated with an increase in discharge (Q). Thus, for low-flow events (mean Q < 2 m3/s), the turbidity of the water showed short, repeating increases that were largely independent of changes in the discharge and high turbidity concentrations occurred during periods with decreasing discharge after a high-discharge event. This implies that users should be careful when choosing supporting parameters, for example when performing QC of turbidity sensor data.
As mentioned in the Introduction, there is a need for harmonized and automated QA-QC methods for use on aquatic HF sensor data to allow comparison of data between countries, and because the use of manual methods for QC of large amounts of data is time-consuming (Bieroza et al., 2023; Skarbøvik et al., 2023a). SentemQC has been tested and used for QC of different sensor data (nitrate, pH, oxygen, turbidity) collected from three different freshwater systems such as streams, lakes, and LWMEs. The method is thus robust, and our study shows that the method can be a sufficient solution for standardizing QA and QC of aquatic sensor data as it is simple, cost-efficient, and open source.
However, SentemQC could also be used on other types of HF sensor data technologies such as lab-on-a-chip, nanosensors, DNS-based biosensors, or molecular biosensors that all measure with different measuring frequencies and offer different advantages and limitations (Bieroza et al., 2023), and on sensors applied in wastewater treatment plants where sensor drift is a known problem that may generate high economic costs and energy usage (Hansen et al., 2022). Former studies have shown that QC of sensor data from wastewater treatment plants can be difficult and sometimes even not possible due to insufficient data and electronic logbooks with missing logs and low data resolution (Hansen et al., 2022). As pointed out by Scarisbrick-Hauser and Rouse (2007) and Hansen et al. (2022), insufficient data quality is a problem in multiple industrial cases. Companies invest in data collection to gain valuable insights, but insufficient data quality often creates a disconnection between intention and reality. We believe that SentemQC, with its simplicity and robustness, can be of great relevance for gaining more reliable data.
In our study, the use of SentemQC on the sensor data collected at the different sensor stations identified anomalies, flagging from 0.1% to 1.6% of the data as described in the Results section (Table 4). While the sensors used here contained relatively few anomalies (<2%), they may represent a best-case scenario in terms of use and maintenance. Sensors used for monitoring and/industrial applications may not be maintained as intensively as under research applications, and this may increase the potential for anomalies and the need for QC. By applying SentemQC, we ensured that only reliable and accurate data was considered. For comparison, Lannergård, Fölster and Futter (2021) identified anomalies and removed 0.4% of their sensor turbidity measurements in streams, recorded with a frequency of 10 or 15 minutes during six years (2012–2017). They used a manual approach to identify the anomalies by calculating different statistical elements such as standard deviation, mean, maximum, and minimum. The sensor observations were log-transformed, and all turbidity measurements falling outside three standard deviations of the daily mean were identified. However, they also followed several steps during the QC of their sensor data, including calibration using discrete manual samples (Lannergård et al., 2021).
In SentemQC, we include the uncertainty related to the sensor itself. This uncertainty is given by the manufacturer or company, for example, HACH. As described in the Methods section, uncertainty is supposed to be equally distributed around the measurement point. The uncertainty given by the companies is usually stated as plus/minus a percentage to be understood as the relative standard deviation of a large number of analyses where the correct concentration of the chemical substance is known. Normally, analysis results are assumed to have a normal distribution around the true concentration; therefore, it can be discussed whether the uncertainty interval should be extended to include plus/minus three standard deviations. If we assume that the sensor measurements exhibit a normal distribution around the true concentration, it covers 96.6% of all outcomes compared to only 68.2% with plus/minus one standard deviation.
The difference between including and excluding uncertainty when performing SentemQC is given in Table 4, and, as expected, SentemQC flagged more anomalies when excluding the uncertainty (Table 4). For November 2021, the nitrate measurement in Lyby-Grønning Stream is displayed in Figure 11 as an example. More specifically, the SentemQC flagged 58 anomalies including uncertainty compared to 106 without uncertainty out of 43,200 data points in November 2021 (0.1% vs 0.2%). While the program can identify anomalies, flagging them within the uncertainty range presents challenges, as this requires absolute certainty within the uncertain area. However, this is not the case, and the SentemQC program cannot be better than the uncertainty. Therefore, it is important to include the uncertainty. Additionally, none of the currently explored QC methods consider the uncertainty of sensor measurements provided by the sensor companies when performing QC on sensor data.
Figure 11. Example of quality control (QC) of data from Lyby-Grønning Stream using SentemQC with and without uncertainty included.
These two examples illustrate SentemQC quality control (QC) of nitrate sensor measurements from Lyby-Grønning Stream in November 2021 with and without uncertainty included in the method. I) shows the QC of the data without including uncertainty, and Ii) shows the QC of the data including uncertainty. The gray dots (Approved data) are the valid data points. The red dots (Flagged anomalies) are the flagged anomalies, and the number indicates the moving window that flagged the anomaly. The blue line represents the water level in the stream.
Conclusion and perspectives
SentemQC is a cost-efficient and simple program designed to identify and mark out anomalies and detect errors or error patterns in HF sensor data, and this is needed as sensors are neither free of errors nor maintenance-free. SentemQC has, by testing and application, proven effective for QC of sensor data (such as nitrate, pH, oxygen, and turbidity) from various freshwater ecosystems, including streams, lakes, and lake warming mesocosm experiments (LWME). This broad application shows the robustness of SentemQC.
Given its simplicity, cost efficiency, robustness, and open-source availability, SentemQC is a promising solution for standardizing QC of different aquatic sensor data. The SentemQC concept that connects all the data points in the input data by examining both nearby and distant data points ensures that both individual anomalies and potential clusters or groups of anomalies are effectively flagged.
There is a high demand for standardized and automated QA-QC procedures for HF aquatic sensor data to ensure cross-national comparability, and here, SentemQC can be useful. Moreover, SentemQC saves workforce compared with the time-intensive manual QC methods, which are impractical for handling enormous amounts of sensor data.
The future development of SentemQC includes integrating the electronic logbook and implementing a calibration procedure for sensor data. Additionally, establishing a mathematical correlation between the dimensions of the moving windows (height and width) for various sensor types, together with supporting parameters would improve SentemQC. Furthermore, future developments of SentemQC might include an ML approach, a method to include classification of anomaly types within the method, and a gap-filling method also using ML. This would enable SentemQC to provide the QC in real time and to do forecasting, which is needed in connection with future warning systems and sensor maintenance. However, this requires considerable effort, complicates the method, and increases the computational demand, as earlier discussed. Currently, we do not have sufficient data to optimize SentemQC by including an ML approach and to retain the universality of the method between different sensor data types. Additionally, a more detailed analysis of how sensor measurement frequency influences data characteristics and the performance of SentemQC across different sensor types is a promising direction for future work.
