In this section, we present the datasets used for the experiments, the parameter settings, the setup for the experiments, and finally, the obtained results. The source code of the experiments is available in the GitHub repository9.
5.1 Experiments for tabular data classification
In this subsection, we describe the two experiments that we have developed to analyze the utility of data reduction for classification tasks with tabular datasets. In the first place, we detail the methodology to apply the different data reduction methods to a dataset and measure its efficiency, representativeness and performance. Then, we give some details on the two datasets we used. Finally, we show the results obtained for both experiments and discuss the main conclusions.
5.1.1 Datasets for classification
The two datasets that we have used in our experiments are:
Collision dataset This tabular dataset was provided by our colleagues Maurizio Mongelli and Sara Narteni, based on 57. It can be downloaded from the repository9, that also contains the code for the experiments and our results. The classification task consists of predicting whether a platoon of vehicles will collide based on features such as the number of cars and their speed. The dataset consists of 107,210 examples with 25 numerical features and 2 classes:
collision = 1, with 69,348 examples.
collision = 0, with 37,862 examples.
We decided to use this dataset in experiments to test the usefulness of data reduction methods to reduce resource consumption in a task related to safe mobility. Before the experiments, we discarded the two features “N” and “m” since they are constant and do not help us in the classification task.
Dry Bean dataset This dataset (see 58 and 59) was created by taking pictures of dry beans from 7 different types and calculating some geometric features from the images, such as the area, the perimeter and the eccentricity. The classification task consists of predicting the type of dry bean based on these geometric features. The dataset contains 13,611 examples with 16 features and 7 classes:
Barbunya, with 1,322 examples.
Bombay, with 522 examples.
Cali, with 1,630 examples.
Dermason, with 3,546 examples.
Horoz, with 1,928 examples.
Seker, with 2,027 examples.
Sira, with 2,636 examples.
The classes were encoded from 0 to 6 for the experimentation following the listed ordering. We decided to use this dataset to test the usefulness of data reduction methods for classification tasks with several unbalanced classes.
5.1.2 Methodology
The methodology of the experiments for both datasets consists of the following three steps:
1. Dataset preprocessing:
It is a common practice to scale or standardize a dataset before building the DL model because this increases the likelihood that the training process will be fast and will not be conditioned by some features simply due to their greater magnitude60,61. In our case, we decided to apply the scikit-learn function MinMaxScaler62.
Each feature pj has a maximum value pj,max = maxxi∈X pj(xi) and a minimum value pj,min = minxi∈X pj(xi). We say that the range of pj is the interval [pj,min, pj,max] because all its possible values lie in it. Rescaling with MinMaxScaler is just changing the example xi ∈ X by:
After scaling, the range of all the features is [0, 1]. That means that all of them have similar values and can be compared between them.
2. Fixing the architecture and hyperparameters:
In both experiments, we used a neural architecture with 10 layers with the following dimensions:
All layers except the last one, use the Rectified Linear Unit (ReLU) activation function ReLU(x) = max(0, x). Additionally, these layers use dropout as a regularization technique. The probability of zeroing a neuron during dropout each time is a hyperparameter called the dropout probability, which we have set equal to 0.50 for the experiment with the Collision dataset and equal to 0.25 for the experiment with the Dry Bean dataset. The differences in neural architecture for both experiments are in the last layer and in the output space O.
For the Collision dataset, the last layer has only one neuron (that is, O = ℝ), with sigmoid activation function σ(x) = 1/(1 + e−x). The output of 𝒩 (also called the logit) for an input xi is a number zi ∈ [0, 1]. The predicted class for xi is:
To train the network for the Collision dataset, we used the Binary Cross Entropy as a loss function. Given a neural network 𝒩θ and a dataset 𝒟, it has the following formula:
For the Dry Bean dataset, the last layer has 7 neurons, one for each class (that is, O = ℝ7). The logits zi,0, ···, zi,6 for an input xi are transformed into a probability distribution with the softmax activation function, given by and the predicted class for xi is:
To train the network for the Dry Bean dataset, we used the Categorical Cross Entropy loss function. If we denote yi,k = 1 if f(xi) = k and yi,k = 0 otherwise, the formula of Categorical Cross Entropy is:
Here, wk = N/Nk is a weight assigned to class k to give more importance to the least populated classes and prevent a bias towards the most populated ones.
In both cases, we used the Adam optimizer63 to minimize the loss function, specifying a learning rate of γ = 0.001 and letting the default values for the rest of the required hyperparameters.
Regarding the other learning hyperparameters, the network for the Collision dataset was trained for ne = 600 epochs with a batch size of β = 1, 024. When the FES reduction was applied, the model was trained for ni = 200 epochs with the full training dataset and the remaining 400 epochs with the reduced dataset. For the Dry Bean dataset, the number of training epochs was ne = 150 (ni = 50 for the first part of training with FES reduction) and the batch size was of β = 32.
3. Data reduction and model training: Now that the dataset is scaled and the neural architecture and the learning hyperparameters have been set, in this step we analyze how the data reduction methods affect the efficiency and performance of the training of a neural network. This step is divided into the following 4 sub-steps:
• Train-Test dataset split: The dataset 𝒟 is randomly split into a training dataset 𝒟train and a test dataset 𝒟test. The DL model will be trained using 𝒟train and its performance will be evaluated using 𝒟test. The test dataset contains a proportion ptest ∈ (0, 1) of the total number of examples in 𝒟. For both experiments, we set ptest = 0.25.
• Training with no reduction: In this step the DL model is trained using the whole training dataset 𝒟train with no reduction, and then, the computation time and carbon emission of the training are calculated. After that, the model is used to classify the test dataset 𝒟train and the accuracy, macro average precision, macro average recall and macro average F1-score are computed.
• Training + reduction for non-wrapper methods: In this step, 𝒟train is reduced getting 𝒟train,R as a result, and the ε-representativeness of 𝒟train,R respect to 𝒟train is computed. The model is then trained for ne epochs using 𝒟train,R, and the total computation time and carbon emission of the reduction and the training are calculated. The model is used to classify 𝒟test as in the previous step. This is repeated for each non-wrapper data reduction method (all but FES) and for each reduction percentage p ∈ {0.1, 0.2, ···, 0.9}.
• Training + reduction for FES: In this step, the DL model is trained using 𝒟train for the first ni epochs. After applying the FES reduction, the model is trained for the remaining epochs using 𝒟train,R, and the ε-representativeness of 𝒟train,R with respect to 𝒟train is also computed. The total computation time and carbon emission of the training and the reduction are computed. The model is used to classify 𝒟test as in the previous steps. This step is repeated for each reduction percentage p ∈ {0.1, 0.2, ···, 0.9}.
This last step is repeated 10 times to test how the data reduction works for different train-test splits and mitigate possible overfitting or bias caused by a specific split of the dataset.
Algorithm 11 shows the experiment pipeline for tabular data classification.
Algorithm 11. Pipeline of the Experiments for Tabular Data Classification
Data: 𝒟 = (X, f)
1 Dataset Preprocessing;
2 Scale 𝒟 using MinMaxScaler;
3 Fixing the architecture and hyperparameters;
4 Set a test size proportion ptest ∈ (0, 1);
5 Set a neural architecture and create the DL model 𝒩;
6 Set a loss function ℒ : Θ → ℝ+;
7 Set an optimization algorithm to minimize ℒ and its associated hyperparameters;
8 Set a regularization technique and its associated hyperparameters;
9 Set a number of training epochs ne ∈ ℕ;
10 Set a number of initial training epochs for FES reduction ni ∈ ℕ, with ni < ne;
11 Set a batch size β ∈ ℕ;
12 Set a number of iterations niter ∈ ℕ;
13 Data reduction and model training;
14 for i = 1 to n_iter do
15 Train-Test dataset split;
16 Set Ntest = ⌊ptest · N⌋;
17 Split 𝒟 into 𝒟train and 𝒟test, being the size of 𝒟test equal to Ntest;
18 Training with no reduction;
19 Train 𝒩 for ne epochs using 𝒟train;
20 Calculate the computing time and carbon emission of the training;
21 Validate the model with 𝒟test and calculate the accuracy, macro average precision, macro average recall and macro average F1-score;
22 Training + reduction for non-wrapper methods:;
23 foreach non-wrapper method do
24 for p ∈ {0.1, 0.2, · · · , 0.9} do
25 Get 𝒟train,R as the reduced dataset of 𝒟train with the corresponding data reduction method and the reduction ratio p;
26 Calculate the ε-representativeness of 𝒟train,R respect to 𝒟train;
27 Train 𝒩 for ne epochs using 𝒟train,R;
28 Calculate the computing time and carbon emission of the reduction and the training;
29 Validate 𝒩 using 𝒟test and calculate the accuracy, macro average precision, macro average recall and macro average F1-score;
30 Training + reduction for FES;
31 for p ∈ {0.1, 0.2, · · · , 0.9} do
32 Train 𝒩 for ni epochs using 𝒟train;
33 Get 𝒟train,R as the reduced dataset of 𝒟train with FES reduction and the reduction ratio p;
34 Calculate the ε-representativeness of 𝒟train,R respect to 𝒟train;
35 Train 𝒩 for ne − ni epochs using 𝒟train,R;
36 Calculate the computing time and carbon emission of the reduction and the training;
37 Validate 𝒩 using 𝒟test and calculate the accuracy, macro average precision, macro average recall and macro average F1-score;
5.1.3 Results and discussion
All results in this Section are the median values after 10 repetitions. We chose to use the median for this experiment because it provides a robust measure of central tendency that is less affected by outliers, ensuring that our analysis is not influenced by extreme values.
Collision Dataset The median results we obtained for the efficiency metrics (computing time and carbon emission) can be seen in Figure 2 and Figure 3. The first thing that we can note is that both metrics express the same information since they are almost proportional. Approximately, each minute of computation during our experiment emitted 0.22 g of CO2 into the atmosphere. It is important to clarify that this occurs because we are primarily measuring CO2 emissions during the training time, which present the same characteristics regardless of the method used. However, as will be seen in the experiments for object detection, the CO2 emitted during the data reduction phase is not proportional to the computing time, because the reduction methods are not similar in terms of code. In general, the use of data reduction methods before network training helped to reduce the computing time and the carbon emission of model building with respect to the reference case (when the model is trained over the whole training dataset), but we find three particular exceptions. When the CLC reduction method is applied with a reduction ratio of 80% or superior, the efficiency metrics are worse than those obtained for the reference case. We have the same situation for MMS and DES when the reduction ratio is equal to 90%. That suggests that, if we extract a reduced dataset with too many examples, it is possible that the time needed to compute the reduction does not compensate the time saved during the network training. Because of that, if the size of the dataset is equal to or larger than the size of the Collision dataset, we recommend applying the three reduction methods only to perform reductions with small reduction ratios. In all the other data reduction methods, we can observe that the efficiency metrics always improve those of the reference case. In the sense of efficiency, the two top data reduction methods are SRS and NRMD, with similar results in both the computation time and carbon emission. Additionally, we can see that the total time and carbon emission from the reduction and training are proportional to the number of examples in the reduced dataset. That indicates that the reduction takes a small time in the process, and almost all the measured time and carbon correspond to the network training. We observed that the efficiency of NRMD reduction depends on the type of matrix decomposition selected. We used SVD decomposition but the results may be different if we select another decomposition type.
Figure 2. Collision: Reduction + training time.
Figure 3. Collision: Reduction + training carbon.
About the ε-representativeness of the reduced datasets with respect to the whole training dataset, the median results can be seen in Figure 4. The first thing we can observe is that MMS reduction is always the best at preserving the ε-representativeness for all the possible reduction ratios, which seems natural if we recall the definition of ε-representativeness and the way the MMS method selects each new example in the reduced dataset. CLC reduction also gives us datasets with good ε values. In contrast, the data reduction method that has the highest ε values for all the possible reduction ratios is NRMD.
Figure 4. Collision: Reduction + ε-representativeness.
The results on accuracy, macro average precision, macro average recall and macro average F1-score can be seen in Figure 5, Figure 6, Figure 7 and Figure 8 respectively. With respect to the accuracy, we can see that the model trained with the whole training dataset has an median success probability of 91%. In general, all the data reduction methods work very well for this dataset. In fact, there are many specific cases where the model obtained with a reduced dataset performs better on the test dataset than the one trained with the complete training dataset. We observe that when the reduction ratio is above 50% the best performing method is FES, while in other cases it is DES. Most of the compared methods manage to maintain accuracy almost intact despite the significant reduction in training size. If we look at the results when we reduce the training dataset to 10% of its size, the model trained after applying DES loses 1.8% of accuracy, while the loss less than 3% when CLC and SRS are applied. In all cases, this loss in accuracy is more or less linear for all methods except for FES. In this case, the accuracy remains stable while the reduction ratio is high, but it undergoes a drastic drop when a high percentage of examples is removed.
Figure 5. Collision: Reduction + Accuracy.
Figure 6. Collision: Reduction + macro average Precision.
Figure 7. Collision: Reduction + macro average Recall.
Figure 8. Collision: Reduction + macro average F1-score.
We observe a similar situation when analyzing the macro average precision. FES is the best method to preserve this metric (even improving the reference case) when the reduction ratio is greater than 50%, while for other ratios the best one is DES. macro average precision dropping as training data set size decreases also appears to be approximately linear except for all the methods but FES, which suffers a significant drop when the reduction ratio is under 30%.
The results that we get when we analyze the macro average recall are quite different. As we saw with accuracy and macro average precision, this metric is generally well preserved even if the reduction ratio is very low, although it suffers a very significant drop when FES is applied with a reduction ratio under 30%. But contrary to what we have seen for the previous metrics, no method clearly outperforms the rest in terms of macro average recall. Depending on the reduction ratio, the method that best preserves the macro average recall is one or another. All methods except NRMD have given the best median result for some of the chosen percentages.
All the general observations we have made when analyzing accuracy and macro average precision can also be seen for the macro average F1-score. In general, all reduction methods preserve this metric well, being FES the best performing method when the reduction ratio is higher than 50% and DES otherwise. The drop in macro average F1-score is also noticeable when many examples are removed with the FES method, while this tendency is not as pronounced for the other data reduction methods.
Finally, we have found an interesting relationship between the ε-representativeness of the reduced datasets and the macro average F1-score of the models trained with them. Given any reduction ratio p = 0.1, ⋯ , 0.9, we have got the ε-representativeness and the macro average F1-score for each reduction method and each iteration in the experiment (in total there are 6 reduction methods × 10 iterations = 60 pairs (ε,F1) for each p). We computed for each p the Spearman’s rank correlation coefficient64 of its respective cloud of 60 points to test if there exists a dependence between ε-representativeness and the macro average F1-score that can be described with a monotonic (always increasing or always decreasing) function. This coefficient is a real number ρ ∈ [−1, 1], where ρ close to 1 indicates a strong positive monotonic correlation, implying that as one variable increases, the other variable also increases, ρ close to −1 indicates a strong negative monotonic correlation, and ρ similar to 0 indicates no monotonic correlation. We also compute the associated p-value to test if the correlation ρ is significantly different from 0. A p-value under a certain threshold (in our case 0.05, which is a standard choice) indicates that ρ is likely not null, while a p-value above it suggests that the observed correlation might be coincidental and not due to a true dependence between both variables. We performed this statistical analysis independently for each p to eliminate the possible effect that the reduction ratio could have if we used all the possible pairs (ε, F1) altogether.
The results that we got can be seen in Table 3. All the computed ρ values are negative, although they are only significantly different from zero when the reduction ratio is below 40%. That indicates that, when data reduction methods remove a large number of examples, the best-performing models are those trained with the reduced datasets that best preserve the ε-representativeness of the entire training set. In few words, when we reduce the Collision dataset with a small reduction percentage, the smaller the ε value, the better the model will perform.
Table 3. Collision: Correlation between ε-representativeness and macro average F1-score.
This table displays the non-parametric Spearman correlation coefficient and its p-value. We have marked in green the columns with a significative correlation setting a significance level of 5%, that is with a p-value less or equal than 0.05.
| 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% |
|---|
| Spearman’s ρ | -0.38 | -0.43 | -0.42 | -0.39 | -0.22 | -0.15 | -0.19 | -0.07 | -0.14 |
| p-value | 0.0 | 0.0 | 0.0 | 0.0 | 0.1 | 0.24 | 0.14 | 0.58 | 0.3 |
Dry Bean Dataset The median results for the computing time and carbon emission can be seen in Figure 9 and Figure 10. There is also a proportional relation between the computation time and the carbon emission in this experiment since each minute of computations emitted approximately 0.21 g of CO2 into the atmosphere. As happened with the Collision dataset, the use of data reduction methods prior to network training helped to reduce the computation time and the carbon emission of the model building with respect to the reference case. The only exception to this rule is when we apply PRD reduction with a reduction ratio greater than 70% (see in Figure 9 and Figure 10). In that situation, both the computation time and the carbon emission of reduction and training exceed those of the reference case. There is no reduction method that runs faster than all the others for this dataset. SRS, MMS, DES, NRMD and even CLC reduction(all of these are overlappingby NRMD in these figures) , which was the slowest method for the Collision dataset, run equally fast for the Dry Bean dataset. They hardly need any time to reduce the training data set, so almost all the measured time and, therefore, the carbon emission correspond to the network training.
Figure 9. Dry Bean: Reduction + training time.
Figure 10. Dry Bean: Reduction + training carbon.
The median values on the ε-representativeness, which can be seen in Figure 11, show us similar results to those observed with the Collision dataset. MMS is still the best data reduction method to preserve the ε-representativeness with respect to the full training dataset, being CLC the second best option. On the contrary, NRMD and PRD are the two methods that generally give the less ε-representative reduced datasets.
Figure 11. Dry Bean: Reduction + Epsilon.
The median results on accuracy, macro average precision, macro average recall and macro average F1-score for the Dry Bean dataset can be seen in Figure 12, Figure 13, Figure 14 and Figure 15 respectively. In this experiment, the reference case has a median accuracy in the test dataset of 89.9% and, contrary to the Collision dataset, no model trained on a reduced dataset improves this value. No reduction method outperforms the others for every reduction ratio in terms of accuracy. What we observe is that, while, in general, the accuracy is well preserved when the reduction ratio is high, it suffers a drastic drop when many examples are removed from the training dataset. If we wanted to lose at most 5% of accuracy (i.e., have at least 84.9%) we would have to select at least 40% of the data, and not all reduction methods would guarantee that maximum accuracy loss.
Figure 12. Dry Bean: Reduction + Accuracy.
Figure 13. Dry Bean: Reduction + Precision.
Figure 14. Dry Bean: Reduction + Recall.
Figure 15. Dry Bean: Reduction + F1-Score.
With respect to macro average precision, we also find that no method is clearly better than the rest for all reduction ratios, although FES seems to dominate the statistics for the central ratios (30% ≤ p ≤ 60%). Here, the drop in the metric as the reduction ratio decreases is not as pronounced as it is for accuracy. This could be because the macro average precision is a more robust measure as it is less influenced by the larger classes.
We can see similar results when analyzing the details on macro average recall. No method outperforms all the rest, and SRS, PRD, CLC, PHL and FES give the best result for at least one reduction ratio.
Finally, when analyzing the macro average F1-score, we observe that four methods (PRD, CLC, PHL and FES) give the best score for some reduction ratio, but FES seems to give the best performing reduced datasets for 30% ≤ p ≤ 70%. As we said when we analyzed the accuracy, it is not possible to extract a reduced dataset with 30% of its size or less without losing more than 5% of macro average F1-score.
Description of data structure for accessing the final results The results of both experiments were saved in a Python dictionary with a nested structure, which we will call the results dictionary. On the first level, the result dictionary is composed of 10 dictionaries containing a dictionary for each iteration. The dictionary of each iteration has a dictionary for each data reduction method used. The dictionary of each data reduction method contains an item for each reduction ratio p = 0.1, ⋯ , 1.0, and each one of them is a dictionary containing all the results obtained for that data reduction method and the p value in that iteration. The results obtained when the training dataset is not reduced are saved in the item with key p = 1.0. Finally, the dictionary associated with a specific iteration, data reduction method, and reduction ratio, contains a key for the following metrics:
For each class k:
– pre_k: To store the model precision for class k over 𝒟test
– rec_k: To store the model recall for class k over 𝒟test
– f1_k: To store the model F1-score for class k over 𝒟test
pre_avg: To store the model macro average precision for class k over 𝒟test
rec_avg: To store the model macro average recall for class k over 𝒟test
f1_avg: To store the model macro average F1-score for class k over 𝒟test
Once we have this results dictionary, the next step is to summarize the information of all the iterations in a more simple dictionary. This object, which we will call the median results dictionary, has the same structure as the entry that we got for each iteration in the results dictionary. For each data reduction method and each reduction ratio, the entry of a specific metric is the median value of the 10 metrics obtained during the 10 different iterations of the experiment. This way we can obtain a more stable representation of the performance of each method across iterations, mitigating the potential influence of outliers or variability in each individual run. All the Figures that we have seen in this subsection present the metrics from the median results dictionary.
5.2 Experiments for object detection
In this subsection, we describe the methodology we have used in our experiments to extend data reduction techniques to images. Observe that we need to adapt the methodology depending on the type of data reduction method. Later, we present the datasets used for the experiments done on object detection, including the parameter settings and the setup, and finally, we discuss the results that we have obtained.
5.2.1 Methodology
The proposed methodology for the data-dependent methods, illustrated in Figure 16, consists of the following five steps:
Figure 16. Diagram of the workflow for the proposed methodology to apply data reduction techniques on images dataset.
1. Feature extraction: The objective of this stage is to convert the raw pixel values of images into a set of meaningful and concise features that capture pertinent information. These features should empower the model to distinguish between different patterns, objects, or structures within the images. This process entails utilizing a computer vision model, to extract features from all the images in the training set. In our case, we used a pre-trained YOLOv5 model on the COCO dataset (Common Objects in Context)65, a widely used collection in computer vision. The COCO dataset, consisting of approximately 330,000 annotated images with object location and category information, is one of the largest and most diverse datasets available for object detection and segmentation tasks. Utilizing pre-trained models on COCO proves advantageous because of its scale and diversity. This pre-training allows models to learn generic features and representations from a vast array of real-world images, enhancing their ability to generalize across various downstream tasks. This approach can lead to improved performance and efficiency when fine-tuning or adapting these models to specific applications.
2. Categorizing images: In this step, we categorize each image based on the objects present in them. This categorization is essential for applying data reduction techniques, as explained in Section 5.2.2, where we detail how each dataset is categorized.
3. Global Average Pooling: Regarding the output of Step 1, we apply Global Average Pooling10,66 to the output of the last layer of the backbone, in order to transform the features maps into an n-dimensional vector representing their extracted features. Subsequently, these feature vectors can be used to calculate distances or similarities between images, and reduction methods can be applied to them.
4. Applying data reduction technique: Reduction techniques are applied to decrease the amount of samples in the dataset with a specified reduction rate on the matrix produced in Step 3, comprising x images and n dimensions, along with the labels from Step 2.
5. Fine tuning with the reduced dataset: This step allows us to assess whether satisfactory performance is achieved, potentially maintaining the same level as with the complete training set. Performance evaluation is conducted on the test set using YOLOv5 pre-trained on the COCO dataset, with the backbone frozen. In this context, "fine-tuning with a specific part frozen" implies that some of the model’s parameters are kept fixed during the training process on the new task. This approach leverages prior knowledge gained during initial training, enabling more efficient adaptation to the new task without completely discarding previously learned information.
When employing wrapper methods, we must adopt a slightly different methodology than the one previously described (see Figure 17). Initially, we must categorize the images, similar to the preceding methodology, as these images are intended for object detection and lack specific labels, instead featuring multiple elements within them. Subsequently, we create a straightforward classification model, incorporating the reduction technique during training to yield the reduced dataset. Finally, we train the YOLOv5 detection model using the reduced dataset, aligning with the objective of Step 5 in the aforementioned methodology.
Figure 17. Diagram of the workflow for the proposed methodology to apply wrapper methods.
Additionally, to apply the CLC method to images, we introduce a slight modification by employing KMeans67 on X with c (the number of classes) clusters. Then, we determine the number of samples closest (based on the Euclidean distance) to each centroid, resulting in our reduced dataset 𝒟R guided by the specified reduction rate. This adjustment is necessary because the centroids generated by the KMeans method, derived from the representations obtained by our image methodology, do not correspond to specific images from our dataset. Consequently, they do not convey information about what we genuinely aim to detect and localize. This modification is called Representative KMeans (RKM).
5.2.2 Datasets for object detection
Roboflow The dataset Roboflow1 68 comprises 514 RGB images, each 416 pixels in both height and width. These images feature pedestrians and people in wheelchairs. The training dataset comprises 463 RGB images, in which a total of 499 pedestrians (annotated as P) and 616 wheelchair users (annotated as W) appear. The test dataset is composed of 51 RGB images, in which a total of 55 pedestrians and 65 wheelchair users appear. To label this dataset, we stick to the following criteria:
| Number of P | Number of W | label |
|---|
| 0 | 1 | 0 |
| ≥ 1 | 0 | 1 |
| ≥ 1 | ≥ 1 | 2 |
| 0 | ≥ 2 | 3 |
Mobility Aid The dataset Mobility Aid2 69 is composed of 17079 RGB images, 10961 are part of the training dataset and the 6118 remaining are part of the test dataset. In this dataset we can find five types of objects: pedestrians (8371, it will be annotated as P), wheelchair users (6458, it will be annotated as W), person pushing a wheelchair (3323, it will be annotated as PW), person with crutches (5374, it will be annotated as C) and person with a walking-frame (7649, it will be annotated as WF). The test dataset is composed of 6208 P, 1993 W, 782 PW, 1883 C, and 2174 WF. To label this dataset, we stick to the following criteria:
| Number of P/PW | Number of W/C/WF | label |
|---|
| 1 | 0 | 1 |
| 0 | 1 | 2 |
| ≥ 1 | 0 | 3 |
| 0 | ≥ 1 | 4 |
| ≥ 1 | ≥ 1 | 5 |
5.2.3 Parameter setting
For the fine-tuning of YOLOv5, we maintain the backbone frozen (pretrained on the COCO dataset) while training the remaining parts of the model. We configure the training with 100 epochs for the Roboflow dataset and 50 epochs for the Mobility Aid dataset. The batch size is set at 16, and the image size is fixed at 640. We employ the SGD optimizer, and the learning rate is set to 0.01 for both datasets. The number of classes (nc) is adjusted based on the specific dataset, such as 2 for the Roboflow dataset and 5 for the Mobility Aid dataset.
5.2.4 Experiments setup
We utilized Python 3.9 and PyTorch27 on Ubuntu 20.04 to conduct our experiments. The training phase was executed on an NVIDIA QUADRO RTX 4000 with 8 GB of RAM and an Intel Xeon Silver 4210 preprocessor. Data partitioning followed the default specifications for each dataset.
To assess the performance difference between the full training set and the reduced set, we conducted training five times for the reduced training dataset. We calculated the arithmetic mean and standard deviation of the results for the test set to gauge the performance of each reduction method. For the Roboflow dataset, given its limited sample size, we applied rate reductions of 50% and 75%. In contrast, for the Mobility Aid dataset with a larger sample size, we implemented rate reductions of 75% and 90%.
5.2.5 Comparison metrics
We use five metrics to assess the performance of training YOLOv5 with the complete dataset compared to training it with different reduction methods and reduction rates. Initially, we employ three performance metrics: precision, recall, and mean average precision setting a confidence threshold of 0.5 (mAP@0.5), which means that only detections with a confidence of 50% or more are considered. These metrics are computed individually for each class and globally by averaging the values across all classes. Additionally, we consider the time required for data reduction, measured in seconds for each reduction method, and the model fine-tuning time. Our primary goal with these metrics is to determine whether we can maintain similar performance while considering the time saved.
Furthermore, we calculate ε-representativeness to gauge how well the reduced dataset 𝒟R represents the original dataset 𝒟. We also compute carbon emissions during both the data reduction process and during fine-tuning.
5.2.6 Results and discussion
The source code to implement data reduction techniques in these datasets is available at 9 in the folder ObjectDetection.
Roboflow Dataset In our initial experiment, we compared the performance of fine-tuning with the complete training dataset against training with a reduced dataset using various reduction methods. Table 4 displays the reduction time, ε-representativeness of the 𝒟r, training time, model performance, and CO2 emissions during the training and reduction phases at a 50% reduction rate. We can observe that, despite training with only 50% of the data, we maintained performance comparable to using the entire dataset. Additionally, we reduced the training time by approximately 40%, decreasing from 9 minutes and 20 seconds to around 5 minutes and 45 seconds with 50% of the samples. This time reduction was accompanied by a similar decrease (about 40%) in CO2 emissions throughout the process. The emission during the application of the reduction methods was small compared to the training time. Notably, effective methods in this scenario include SRS, DES, MMS, RKM, and FES. On the contrary, NRMD and PHL perform worse, with a slight loss of performance. Data reduction times were generally light, except for FES, which exhibited excessive duration compared to other methods. Conversely, a lower ε-representativeness did not seem decisive for performance improvement or degradation.
Table 4. Table results for Roboflow dataset and 50% reduction rate.
The ’Precision’, ’Recall’ and ’mAP@.5’ columns display mean and standard deviation values for the specified variables. The ’CO2(g)’ column indicates the grams of CO2 emitted during the application of the reduction method and during the fine-tuning. The ’R Time(s)’ column shows the time in seconds for data reduction, while the ’FT Time’ column displays the time spent on fine-tuning the model. We have highlighted in red the values obtained during fine-tuning with the complete dataset,which serve as the reference. Additionally, in green, we highlight the best reduction method for each metric.
| Method | R Time(s) | ε | FT Time | Precision | Recall | mAP@.5 | CO2(g) |
|---|
| - | - | - | 9m 19s | A: 0.951±0.001
P: 0.926±0.022
W: 0.976±0.015 | A: 0.897±0.019
P: 0.832±0.031
W: 0.96±0.018 | A: 0.944±0.009
P: 0.906±0.014
W:0.984±0.005 | 5.5 |
| SRS | 0.002 | 2.58 | 5m 44s | A: 0.945±0.021
P: 0.921 ±0.035
W: 0.97±0.011 | A:0.897 ±0.019
P: 0.836±0.03
W: 0.958±0.015 | A:0.897 ±0.019
P: 0.836±0.03
W: 0.958±0.015 | 0+3.3 |
| DES | 0.29 | 3.12 | 5m 44s | A: 0.94±0.001
P: 0.925±0.015
W: 0.95±0.015 | A: 0.885±0.015
P: 0.795±0.03
W: 0.975±0.015 | A: 0.945±0.005
P:0.905±0.005
W: 0.985±0.005 | 0.002+3.3 |
| NRMD | 0.09 | 2.27 | 5m 47s | A: 0.925±0.015
P: 0.905±0.02
W: 0.945±0.02 | A: 0.89±0.016
P: 0.82±0.014
W: 0.956±0.019 | A: 0.932±0.009
P: 0.892±0.016
W: 0.972±0.007 | 0.001+3.3 |
| MMS | 0.09 | 1.99 | 5m 46s | A: 0.951±0.01
P: 0.921±0.017
W: 0.981 ±0.005 | A: 0.894±0.015
P: 0.821±0.029
W: 0.967±0.011) | A: 0.939±0.006
P:0.9±0.011
W: 0.981±0.006 | 0.0004+3.26 |
| RKM | 1.26 | 1.25 | 5m 45s | A: 0.948±0.005
P:0.907±0.015
W: 0.99 ±0.009 | A: 0.895±0.017
P: 0.819±0.032
W: 0.971±0.008 | A: 0.94±0
P: 0.894±0.003
W: 0.985±0.002 | 0.005+3.3 |
| PRD | 0.92 | 1.67 | 5m 44s | A: 0.944±0.02
P: 0.916±0.04
W: 0.97±0.008 | A: 0.89±0.015
P: 0.814±0.025
W: 0.965±0.008 | A: 0.937±0.007
P: 0.894±0.015
W: 0.974±0.008 | 0.002+3.29 |
| PHL | 0.64 | 2.85 | 5m 38s | A: 0.942±0.028
P: 0.9±0.045
W: 0.982±0.014 | A: 0.863±0.21
P: 0.773±0.041
W: 0.954±0.01 | A: 0.927±0.006
P: 0.875±0.017
W: 0.978±0.011 | 0.004+3.24 |
| FES | 8.14 | 2.28 | 5m 39s | A:0.921±0.013
P: 0.871±0.026
W: 0.972±0.017 | A: 0.903±0.011
P: 0.844±0.018
W: 0.962±0.006 | A: 0.948±0.004
P: 0.913±0.009
W: 0.986±0.003 | 0.08+3.24 |
Table 5 presents the same analysis with a 75% reduction rate, where the overall training time was reduced from 9.5 minutes to approximately 4 minutes, a 60% increase in speed. Data reduction time remained insignificant, with some methods displaying longer computation times, such as FES. Despite a slight performance reduction across all metrics with all methods, SRS, MMS, RKMEANS, and PHL emerged as more resilient options. In particular, SRS is the one that best maintains precision and mAP. On the contrary, NRMD showed the most significant performance loss. A lower ε-representativeness did not appear to be a determining factor for better or worse performance. CO2 emissions were also reduced by 60%. On average, the overall mAP for reduction methods saw only a 3% reduction compared to the substantial computational time and CO2 emission savings of 60%. In Figure 18, the mean mAP values for each category and method, along with the full dataset, illustrate the best-performing methods. At a 50% reduction rate, performance is nearly maintained, while at a 75% reduction, some performance loss is evident. SRS stands out as the most effective method. Additionally, a general improvement in accuracy is observed for Wheelchairs compared to People, potentially attributed to a slight imbalance in the dataset between the two categories. We only present the mAP figure (Figure 18) because it is the most comprehensive performance metric for object detection evaluation.
Table 5. Table results for Roboflow dataset and 75% reduction rate.
The Precision’, ’Recall’ and ’mAP@.5’ columns display mean and standard deviation values for the specified variables. The ’CO2(g)’ column indicates the grams of CO2 emitted during the application of the reduction method and during the fine-tuning. The ’R Time(s)’ column shows the time in seconds for data reduction, while the ’FT Time’ column displays the time spent on fine-tuning the model. We have highlighted in red the values obtained during fine-tuning with the complete dataset,which serve as the reference. Additionally, in green, we highlight the best reduction method for each metric.
| Method | R Time(s) | ε | FT Time | Precision | Recall | mAP@.5 | CO2(g) |
|---|
| - | - | - | 9m 19s | A: 0.951±0.001
P: 0.926±0.022
W: 0.976±0.015 | A: 0.897±0.019
P: 0.832±0.031
W: 0.96±0.018 | A: 0.944±0.009
P: 0.906±0.014
W :0.984±0.005 | 5.5 |
| SRS | 0.001 | 2.7 | 4m 2s | A: 0.931±0.013
P:0.9±0.017
W: 0.963±0.021 | A: 0.886±0.013
P: 0.815±0.016
W: 0.957±0.01 | A: 0.937±0.005
P: 0.906±0.008
W: 0.968±0.004 | 0+2.25 |
| DES | 0.24 | 3.21 | 3m 59s | A: 0.894±0.022
P: 0.824±0.04
W: 0.964±0.03 | A: 0.86±0.031
P: 0.761±0.045
W: 0.96±0.019 | A: 0.919±0.006
P: 0.855±0.009
W: 0.984±0.005 | 0.002+2.24 |
| NRMD | 0.09 | 2.3 | 3m 58s | A: 0.901±0.016
P: 0.887±0.021
W: 0.914±0.023 | A: 0.846±0.018
P: 0.743±0.043
W: 0.949±0.011 | A: 0.908±0.009
P: 0.857±0.016
W: 0.958±0.003 | 0.001+2.23 |
| MMS | 0.05 | 2.3 | 4m | A: 0.935±0.012
P: 0.927±0.017
W: 0.943±0.015 | A: 0.858±0.009
P: 0.776±0.002
W: 0.94±0.01 | A: 0.922±0.01
P: 0.885±0.019
W: 0.958±0.006 | 0.0004+2.23 |
| RKM | 1.22 | 1.18 | 3m 52s | A: 0.908±0.015
P: 0.821±0.029
W: 0.995±0.006 | A: 0.881±0.009
P: 0.829±0.019
W: 0.934±0.009 | A: 0.927±0.007
P: 0.881±0.015
W: 0.972±0.004 | 0.005+2.18 |
| PRD | 0.38 | 2.68 | 4m | A: 0.895±0.024
P: 0.849±0.038
W: 0.941±0.02 | A: 0.872±0.033
P: 0.793±0.066
W: 0.951±0.011 | A: 0.916±0.013
P: 0.856±0.022
W: 0.967±0.014 | 0.001+2.22 |
| PHL | 0.67 | 3.74 | 3m 59s | A: 0.897±0.02
P: 0.816±0.034
W: 0.977±0.017 | A: 0.887±0.016
P: 0.834±0.023
W: 0.941±0.016 | A: 0.923±0.004
P: 0.877±0.007
W: 0.97±0.004 | 0.004+2.23 |
| FES | 8.63 | 3.12 | 3m 55s | A: 0.884±0.002
P: 0.824±0.033
W: 0.945±0.016 | A: 0.874±0.014
P: 0.796±0.024
W: 0.951±0.009 | A: 0.911±0.008
P: 0.854±0.013
W: 0.968±0.004 | 0.08+2.17 |
Figure 18. mAP values on Roboflow dataset when using a 50% reduction rate (first column) and when using a 75 percent reduction rate (second column).
Based on the outcomes obtained from this dataset, we can affirm that employing reduction methods within the proposed methodology, followed by fine-tuning YOLOv5 for object detection, led to a substantial reduction in CO2 emissions and computation time. Importantly, this reduction did not adversely affect the model’s performance in localization and object detection tasks.
Mobility Aid Dataset Initially, we observe the training benchmark results with the complete dataset comprising 10,961 instances. This yields commendable outcomes, such as a mean average precision of 0.93, with a narrow standard deviation of 0.003 across all classes. Notably, the performance stands out in the category of people in wheelchairs (W), surpassing the overall average, while performance in other categories is noteworthy. Other categories were not included in Table 6 and Table 7 In general, superior performance is evident for the wheelchair, push-wheelchair, and walking-frame categories compared to the pedestrian and crutches categories, which exhibit below-average performance.
Table 6. Table results for Mobility Aid dataset and 75% reduction rate.
The ’Precision’, ’Recall’ and ’mAP@.5’ columns display mean and standard deviation values for the specified variables. The ’CO2(g)’ column indicates the grams of CO2 emitted during the application of the reduction method and during the fine-tuning. The ’R Time(s)’ column shows the time in seconds for data reduction, while the ’FT Time’ column displays the time spent on fine-tuning the model. We have highlighted in red the values obtained during fine-tuning with the complete dataset, which serve as the reference. Additionally, in green, we highlight the best reduction method for each metric.
| Method | R Time(s) | ε | FT Time | Precision | Recall | mAP@.5 | CO2(g) |
|---|
| - | - | - | 2h 8m | A: 0.91±0.008
W: 0.994±0.001 | A: 0.875±0.007
W: 0.896±0.01 | A: 0.93±0.003
W: 0.941±0.003 | 73.85 |
| SRS | 0.006 | 0.86 | 1h 2m | A: 0.912±0.007
W: 0.979±0.029 | A: 0.876±0.007
W: 0.887±0.011 | A: 0.932±0.004
W: 0.94±0.003 | 0+34.12 |
| DES | 7.98 | 0.87 | 1h 2m | A: 0.91±0.009
W: 0.989±0.005 | A: 0.876±0.006
W: 0.893±0.004 | A: 0.93±0.006
W: 0.941±0.001 | 0.04+34.1 |
| NRMD | 3.83 | 1.04 | 1h 2m | A: 0.857±0.004
W: 0.986±0.007 | A: 0.83±0.006
W: 0.865±0.031 | A: 0.903±0.004
W: 0.932±0.002 | 0.015+33.94 |
| MMS | 5.88 | 0.66 | 1h 2m | A: 0.911±0.006
W: 0.994±0.003 | A: 0.875±0.004
W: 0.891±0.007 | A: 0.928±0.005
W: 0.939±0.005 | 0.03+34.13 |
| RKM | 31 | 0.74 | 1h 2m | A: 0.845±0.013
W: 0.961(0.008 | A: 0.795±0.006
W: 0.871±0.008 | A: 0.89±0.006
W: 0.92±0.005 | 0.023+34.1 |
| PRD | 457 | 0.59 | 1h 1m | A: 0.912±0.011
W: 0.989±0.005 | A: 0.876±0.007
W: 0.889±0.008 | A: 0.931±0.004
W: 0.939±0.002 | 1.82+33.91 |
| PHL | 314 | 0.9 | 1h 3m | A: 0.912±0.007
W: 0.994±0.003 | A: 0.875±0.009
W: 0.896±0.003 | A: 0.931±0.003
W: 0.38±0.001 | 1.79+35.09 |
| FES | 289 | 1 | 56m 33s | A: 0.912±0.004
W: 0.993±0.001 | A: 0.875±0.005
W: 0.887±0.009 | A: 0.928±0.004
W: 0.939±0.003 | 2.9+31.83 |
Table 7. Table results for Mobility Aid dataset and 90% reduction rate.
The ’Precision’, ’Recall’ and ’mAP@.5’ columns display mean and standard deviation values for the specified variables. The ’CO2(g)’ column indicates the grams of CO2 emitted during the application of the reduction method and during the fine-tuning. The ’R Time(s)’ column shows the time in seconds for data reduction, while the ’FT Time’ column displays the time spent on fine-tuning the model. We have highlighted in red the values obtained during fine-tuning with the complete dataset, which serves as the reference. Additionally, in green, we highlight the best reduction method for each metric.
| Method | R Time(s) | ε | FT Time | Precision | Recall | mAP@.5 | CO2(g) |
|---|
| - | - | - | 2h 8m | A: 0.91±0.008
W: 0.994±0.001 | A: 0.875±0.007
W: 0.896±0.01 | A: 0.93±0.003
W: 0.941±0.003 | 73.85 |
| SRS | 0.018 | 1.15 | 48m 31s | A: 0.894±0.01
W: 0.993±0.003 | A: 0.867±0.007
W: 0.886±0.015 | A: 0.926±0.002
W: 0.942±0.002 | 0+26.13 |
| DES | 8.13 | 1 | 43m 28s | A: 0.875±0.014
W: 0.971±0.02 | A: 0.862±0.007
W: 0.893±0.011 | A: 0.917±0.006
W: 0.94±0.003 | 0.04+24.06 |
| NRMD | 2.69 | 1.18 | 43m 17s | A: 0.826±0.005
W: 0.976±0.008 | A: 0.773±0.02
W: 0.846±0.02 | A: 0.873±0.009
W: 0.923±0.005 | 0.017+23.93 |
| MMS | 2.9 | 0.84 | 43m 26s | A: 0.904±0.006
W: 0.996±0.001 | A: 0.875±0.004
W: 0.87±0.01 | A: 0.927±0.004
W: 0.935±0.002 | 0.017+24.06 |
| RKM | 22.92 | 0.95 | 43m 26s | A: 0.812±0.008
W: 0.968±0.007 | A: 0.767±0.012
W: 0.855±0.024 | A: 0.858±0.004
W: 0.918±0.003 | 0.024+24.04 |
| PRD | 66.4 | 1 | 43m 28s | A: 0.908±0.01
W: 0.99±0.006 | A: 0.868±0.007
W: 0.883±0.01 | A: 0.926±0.007
W: 0.938±0.004 | 0.29+24.05 |
| PHL | 317 | 1.04 | 43m 36s | A: 0.898±0.006
W: 0.989±0.005 | A: 0.869±0.005
W: 0.889±0.007 | A: 0.926±0.003
W: 0.937±0.002 | 1.35+24.11 |
| FES | 269 | 1.46 | 43m 29s | A: 0.906±0.007
W: 0.992±0.005 | A: 0.87±0.005
W: 0.875±0.033 | A: 0.927±0.003
W: 0.936±0.002 | 2.69+24.02 |
Moving to Table 6, which illustrates the results for a reduction rate of 75%, we observe a significant reduction in training time, approximately 50%. Reduction times are just seconds for most methods, extending to minutes for PRD, PHL and FES. CO2 emissions also witness a substantial decrease, around 55% for all methods, except for PRD, PHL and FES, which emit more CO2 due to an extended computation time during data reduction. Finally, across all methods, we managed to maintain the performance achieved with the complete training set. This underscores the practical significance of these methods in reducing computation time and consequently lowering CO2 emissions during model fine-tuning. The exceptions are the RKM and NRMD methods, which exhibit a performance drop.
In Table 7, we present similar findings, but this time with a reduction rate of 90%. The primary observation is a decrease in performance across various metrics, including precision, recall, and mean average precision. Notably, while there is an overall performance loss due to reduced metrics in other categories, the decline in the wheelchair category is comparatively less pronounced. Despite this reduction in performance, we managed to cut down the training time to 43 minutes, representing about 67% of the training time without dataset reduction. A corresponding decrease in CO2 consumption is observed. Despite the general decline, certain methods, such as SRS, MMS, PRD, PHL and FES, demonstrate a relatively robust maintenance of performance.
A visual representation of the mean mAP values for wheelchairs, overall, and for each method alongside the full dataset is provided in Figure 19. This visualization offers a clearer insight into the methods that yield optimal results. With a reduction rate of 75%, we almost maintain performance in all methods, except NRMD and RKM. However, at a rate of reduction 90%, some performance loss is evident, highlighting the efficacy of methods such as SRS, MMS, PRD, PHL and FES. In particular, the drop in performance for the wheelchair category is less pronounced compared to other categories. We only present the mAP figure (Figure 19), as it serves as the most comprehensive performance metric for object detection evaluation.
Figure 19. mAP values on Mobility Aid dataset when using a 75% reduction rate(first column) and when using a 90% of reduction rate (second column).
With these results obtained for this dataset, we can confirm that the use of reduction methods within the proposed methodology, followed by fine-tuning YOLOv5 for object detection, led to a substantial reduction in both CO2 emissions and computation time. Importantly, this reduction did not affect the model performance in object detection and localization.
