Introduction
Nowadays, the Internet of Things (IoT) term is often used due to its significant role in enabling smart interactions between machines, sensors, and the environment1. The IoT approach aims to orchestrate a set of "things" or technologies such as sensors, actuators, radio frequency identification (RFID) tags, near field communications (NFC), and machine-to-machine (M2M) communications, by means of network protocols, in order to achieve the required goal by the developed IoT system1,2. IoT applications and use cases can vary from manufacturing and agriculture to healthcare and transportation, with a wide spectrum in between1. One specific example of an IoT application could be an indoor localization system using an IoT testbed as described by Elkenawy et al.3.
To build a complete IoT system as a Wireless Sensor Network (WSN) infrastructure, an efficient prototyping procedure must be carried out as a first step. Testbed facilities are a great tool for prototyping purposes compared to other simulation or emulation tools, as they represent the real-world conditions more precisely, which in turn speeds up the development process and could make the process of developing WSNs, making debugging and testing less time consuming4,5. A plain definition for the WSN testbed facility would be a realistic/physical environment consisting of a large number of permanently deployed sensor nodes (25+ nodes according to Ruskuls et al.6) with a software backend that provides a basic set of functionalities such as node reprogramming and remote interaction. On top of that, a lot of testbed facilities provide additional features such as data logging, experiment scheduling7, energy metering8, etc.
A good dataset demonstrating testbed facilities, in terms of facilities’ capabilities versus market needs, would provide guidelines with regard to design choices for testbed facilities and provide useful information during the creation and execution of scientific experiments, in addition to fueling innovative solutions and remarkable competition9 within the market. To the best of our knowledge, a comprehensive and up-to-date dataset for testbed facilities is a scarce resource that is not yet available to the research community. An illustrative example of this is that querying "IoT testbed" in Google Dataset Search would result in:
Data generated during a testbed experiment;
Specifications of a testbed facility;
Description of a cluster of testbed facilities (e.g. cybersecurity testbeds category).
A broader survey has been done by Judvaitis et al.10 by extracting information about 3059 sensor network deployments according to different categories, which is one of the few existing attempts for synthesizing a complete dataset for actual sensor network deployments. This data set was gathered with the aim of providing a definite overview of existing Wireless Sensor Networks and Internet of Things testbed facilities available for scientific and industrial use and identifying possible gaps to be filled by future testbed facility developments.
The research question addressed by this article is how many and what testbed facilities are available and how do they compare against one another.
Methods
This dataset was gathered following the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA)11 checklist. The initial search was done in two databases Scopus and Web of Science (WoS) using the following queries:
Scopus: TITLE ( testbed ) AND TITLE-ABS-KEY ( wsn OR iot OR "sensor network*" OR "internet of thing*" ) AND SUBJAREA ( comp ) AND PUBYEAR < 2021 AND PUBYEAR > 2010
WoS: TI = ("testbed") AND (AB = (wsn OR iot OR "sensor network*" OR "internet of thing*") or AK= (wsn OR iot OR "sensor network*" OR "internet of thing*") ) and SU="Computer Science" and py =(2011–2020)
The raw results returned 346 articles from Scopus and 176 articles from WoS. After 163 duplicates were removed, 359 unique articles were left for further analysis. In the phases described below each article was mainly processed by one reviewer independently. For each reviewer in each phase articles were assigned randomly. This minimized the risk of biases while assigning the articles to a particular reviewer. The confusing and difficult-to-evaluate articles were discussed in weekly meetings or re-evaluated by another reviewer.
First phase
In this phase, only the abstracts of all 359 were analyzed. The aim was to filter articles that did not contain a description of the testbed facility. With a testbed facility, we understand the following: the facility provides remote access to the embedded hardware which can be used freely without any restrictions regarding the usability or functionality, and it should meet the following minimal requirements:
Designed to run a variety of different experiments, where the devices under test are completely controlled by the user;
Users do not need physical access to the hardware- reprogramming or any other interaction with software can be performed remotely;
Provide a user interface specifically designed for testbed facility purposes.
Access to the testbed facility can be restricted and is not necessary for a testbed facility to qualify, so it does not necessarily need to be public. Articles containing descriptions of improvements to testbed facilities were also to be included, even if they did not contain a description of the testbed facility itself. The guidelines for researchers who did the screening were the following:
(a) Open the article you are going to evaluate.
(b) Read the abstract.
(c) Make a decision about the article, does it contain a testbed facility description, and, if so, mark it as included for further analysis.
(d) Mark the article as screened.
(e) Go to the next article.
We used Mendeley (https://www.mendeley.com/) to quickly share the progress between the team members and automatically obtain the abstracts for the articles. 9 team members (3 of which are the authors) screened the abstracts. As a result of the first phase, 170 articles were dropped, as they did not contain a testbed facility description. A total of 189 articles were left for further analysis.
Second phase
The aim of this phase was to filter out non-testbed facility articles and group the articles by distinct testbed facilities, assuming that there can be multiple articles per testbed facility. In this phase, the full article was screened, and, using the same criteria as in the previous phase, the article was marked as “containing testbed facility” or not. We used a Microsoft Excel worksheet to track the progress and split the work between researchers. 4 team members (3 of which are the authors) screened full texts of articles. After the second phase, 125 articles were marked as not containing a testbed facility description.
Third phase
In this phase, to minimize the risk of missed articles in the second phase, all 189 articles (the same articles as in the previous phase) were processed and predefined values (with their descriptions) were extracted: device under test (DUT), Sensors (the type), location (generic description of the place where the DUTs are located), mobility (can the DUTs physically move while the experiment is ongoing), architecture (the internal design of the testbed facility), workstations (workstation is a DUT-to-server relay, typically a Linux-capable device that forwards the data and commands between server and DUTs), DUT location accuracy (what is the precision to which the user knows the DUT location), cost of implementation (if provided), deployment options (is the testbed suited and intended to deploy outside of the laboratory), facility count (assuming there might be more than one facility), functionality (the features this testbed facility provides to users), access level (what is the level of user control), user interface (the type of used UI), assistive tools (any tools designed to facilitate or improve the user experience), DUT connection and interaction interfaces (physical connections available), and whether the testbed facility is available as Open Source.
All articles were processed again by the whole team to minimize the risk of error in the previous phase. We used JSONForms (https://jsonforms.io/) to make the extraction process smoother and obtain machine-readable data in JSON format. After the third phase, 134 articles (125 from the previous phase + an extra 9 articles) were excluded and 55 articles were included in the dataset. There were multiple testbeds with more than one article describing them. In total, 41 unique testbed facilities were described in 55 articles. An overview of all the systematic review phases described above is summarized in a flow diagram in Figure 1, depicting the whole process.
Figure 1. Flow diagram overview of the systematic review methodology and results at each phase.
Data processing
As the data describing testbed facilities contains data entries that are not machine-readable, the nature of the resulting data set does not allow processing using programming tools such as Python/Jupyter Notebooks, it can only be viewed as a textual compilation of different specifics about the evaluated testbed facilities. As a remedy to this situation, we compiled a subset of all the extracted data unifying and simplifying the extracted values to be more easy to use. The unifying process and actions taken were discussed with the team members to gain a unified vision of the process. Still, the process itself was done by the leading researcher to minimize the interpretation difference risks. After this process, the team members reviewed the unified dataset. The newly obtained machine-readable data set is in unified JSON format and thus can be read and analyzed by any modern programming language script or data analytics application, the data set includes some initial analytic scripts written in Python. The machine-readable data set contains less information overall, as similar values were merged for increased readability. In the published dataset we have included both versions, together with the tools used to obtain the datasets.
Ethical approval and consent
Ethical approval and consent were not required.
Data availability
Underlying data
Zenodo: Available Wireless Sensor Network and Internet of Things testbed facilities: dataset, https://doi.org/10.5281/zenodo.715722112.
This project contains the following underlying data:
• raw_dataset contains the files for each article containing a testbed facility description with the extracted information in json format;
• json contains json files for each testbed facility processed with the aim of improving the machine readability of the dataset;
• output.json contains a json array with the names of extracted features and the list of testbed facility IDs corresponding to the json file names with such features;
• feature_extraction.ipynb contains the script used for the initial analysis of the JSON files about testbed facilities.
Extended data
Zenodo: Available Wireless Sensor Network and Internet of Things testbed facilities: dataset, https://doi.org/10.5281/zenodo.715722112.
This project contains the following extended data:
• Supplementing_info/raw_exports/contains the raw list of articles exported from SCOPUS and WoS databases;
• Supplementing_info/processed/1-combined.xlsx contains the combined list of articles;
• Supplementing_info/processed/2-deduplicated.xlsx contains the deduplicated list of articles;
• Supplementing_info/jsonforms/contains the web-based tool used for testbed facility feature extraction in the Third phase;
For further details on the structure and contents of each folder please view the README.md file.
The dataset is licensed under CC-BY 4.0 International (CC-BY 4.0) Public Domain Dedication license.
Acknowledgements
In addition to the authors of this article, the following individuals provided their work to the acquisition and processing of this data: Janis Arents, Vladislavs Medvedevs, Artis Mednis, Andris Ivars Mackus, Krisjanis Nesenbergs, and Rihards Balass. All six collaborators participated in the weekly meetings and discussed articles that in all three phases were selected as difficult or confusing. All acknowledged contributors have agreed to be named in this manuscript.
Faculty Opinions recommendedReferences
- 1.
Shah SH, Yaqoob I:
A survey: Internet of things (iot) technologies, applications and challenges.
2016 IEEE Smart Energy Grid Engineering (SEGE).
2016; 381–385. Publisher Full Text
- 2.
Atzori L, Iera A, Morabito G:
The internet of things: A survey.
Computer Networks.
2010; 54(15): 2787–2805. Publisher Full Text
- 3.
Elkenawy A, Judvaitis J:
Transmission power influence on wsn-based indoor localization efficiency.
Sensors (Basel).
2022; 22(11): 4154. PubMed Abstract
| Publisher Full Text
| Free Full Text
- 4.
Salmins A, Ozols K, Ruskuls R:
Data management in testbed for large scale wireless sensor networks. In:
2015 Advances in Wireless and Optical Communications (RTUWO).
IEEE, 2015; 54–57. Publisher Full Text
- 5.
Salmins A, Judvaitis J, Balass R, et al.:
Mobile wireless sensor network testbed. In:
2017 25th Telecommunication Forum (TELFOR).
IEEE, 2017; 1–4. Publisher Full Text
- 6.
Ruskuls R, Lapsa D, Selavo L:
Edi wsn testbed: Multifunctional, 3d wireless sensor network testbed. In:
2015 Advances in Wireless and Optical Communications (RTUWO).
IEEE, 2015; 50–53. Publisher Full Text
- 7.
Lapsa D, Balass R, Judvaitis J, et al.:
Measurement of current consumption in a wireless sensor network testbed. In:
2017 25th Telecommunication Forum (TELFOR).
IEEE, 2017; 1–4. Publisher Full Text
- 8.
Werner-Allen G, Swieskowski P, Welsh M:
Motelab: A wireless sensor network testbed. In:
IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005.
IEEE, 2005; 483–488. Publisher Full Text
- 9.
Chapman A, Simperl E, Koesten L, et al.:
Dataset search: a survey.
VLDB J.
2020; 29(1): 251–272. Publisher Full Text
- 10.
Judvaitis J, Mednis A, Abolins V, et al.:
Classification of actual sensor network deployments in research studies from 2013 to 2017.
Data.
2020; 5(4): 93. Publisher Full Text
- 11.
Moher D, Liberati A, Tetzlaff J, et al.:
Preferred reporting items for systematic reviews and meta-analyses: the prisma statement.
Ann Intern Med.
2009; 151(4): 264–269, W64. PubMed Abstract
| Publisher Full Text
- 12.
Judvaitis J, Abolins V, Elkenawy A, et al.:
Available Wireless Sensor Network and Internet of Things testbed facilities: dataset. 2022. http://www.doi.org/10.5281/zenodo.7157221
