Introduction
Much of our understanding of the past derives from the study of sediments and soils. The sedimentary record preserves a rich archive of information and, as such, constitutes a primary focus of research across numerous disciplines, including archaeology, soil sciences, and geology. Soil micromorphology, the microscopic study of undisturbed soil and sediment in thin sections, has become a cornerstone analytical method in these disciplines. By examining sediment composition, fabrics, pedogenic features and their subsequent micro-scale spatial relationships, micromorphologists reconstruct depositional environments, and site formation processes, revealing both natural microenvironments and anthropogenic activities that shaped the geological and archaeological record (Courty, 1989; Goldberg et al., 2022; Goldberg and Aldeias, 2018; Karkanas and Goldberg, 2018; Nicosia and Stoops, 2017).
Despite its central analytical power, micromorphology faces three persistent methodological constraints. First, analysis is labor-intensive and time-consuming, often requiring months of specialist effort for comprehensive site studies. Second, interpretations depend heavily on expert experience and knowledge for recognizing diagnostic features through years of training; yet these features are often subtle, variably expressed, or context dependent. Third, this interpretive complexity creates persistent inter-observer variability, reducing reproducibility, and limiting our ability to build large, comparable datasets across sites and regions (Shahack-Gross, 2016). These challenges hinder cumulative comparisons, as quantitative syntheses are essential for reconstructing past environments and behaviors at regional scales.
One prominent example of these challenges is frost action, where repeated freeze-thaw cycles create diagnostic microscopic signatures, including granular, blocky, and platy-lenticular microstructures, capped grains, and characteristic porosity patterns (Van Vliet-Lanoë, 1998, 1985; Vliet-Lanoë, 2010). From an archaeological perspective, recognizing these frost signatures is critical for multiple reasons. First, they enable reconstruction of past climatic conditions and understanding of hominin adaptations to glacial and periglacial settings (Courty and Vallverdu, 2001; Davies et al., 2003; Dibble et al., 2017; Holcomb et al., 2023; Kehl et al., 2025; Murphree et al., 2025; Straus, 1991; Vachula et al., 2019; Van Vliet-Lanoë, 1998). Second, frost action affects the preservation of archaeological materials (Deeben et al., 2010; Texier et al., 1998; Todisco and Monchot, 2008) and disturbs the spatial integrity of previously deposited archaeological assemblages (Bertran et al., 2015, 2010; Masson, 2010).
Yet despite extensive characterization in archaeological contexts, frost features vary along a continuum from weakly developed to well-defined forms, their expression is dependent on textural features (e.g., grain-size), and they can be obliterated by or overlap with other pedogenic processes (Bertran et al., 2017, 2015; Phillips et al., 2013; Van Vliet-Lanoë, 1998, 1985; Van Vliet-Lanoë et al., 2004). These characteristics make frost feature identification a suitable test case for evaluating whether computational approaches can improve consistency and reproducibility in micromorphological classification.
Computational approaches have been adopted in related fields to address similar interpretive challenges arising from subtle features, contextual variability, and observer subjectivity. In geological petrography, where similar interpretive challenges exist, Convolutional Neural Networks (CNNs) approaches have proven effective for diverse classification tasks, such as rock and mineral classification (Basso et al., 2025; Dalhat and Osman, 2025; Faria et al., 2025; Iyas et al., 2020; Maitre et al., 2019; Polat et al., 2021; Su et al., 2020), microfacies classification (Pires de Lima et al., 2020), fossil classification (Liu et al., 2023; I. C. Romero et al., 2020a), porosity and grain size analysis (Rubo et al., 2019; Tang et al., 2020).
However, CNNs are often treated as “black boxes”, limiting their adoption in domains where diagnostic criteria are well established and where high accuracy does not necessarily indicate that models rely on meaningful features (Ebert-Uphoff and Hilburn, 2020). To address this, Class Activation Maps (CAMs) are particularly relevant as they visualize the spatially discriminative regions that influence model predictions (Barredo Arrieta et al., 2020; McGovern et al., 2019; Şahin et al., 2025), improving the justifiability and trustworthiness of computational approaches (Bergen et al., 2019; Jiang et al., 2024; Reichstein et al., 2019). Several studies in geological petrography have incorporated interpretability analysis on their classification scheme, demonstrating both the utility and challenges of interpretability analysis (Bai et al., 2025; Seo et al., 2022; Zheng et al., 2024).
Despite these advances in geological petrography and the shared interpretative challenges, the application of machine learning to archaeological micromorphology remains remarkably limited. To date, Arnay et al. (2026, 2021) have demonstrated the viability of deep learning for archaeological thin section analysis, first for pore type classification and more recently for segmentation of bones, charcoals, and lithic materials and estimate their subsequent abundance in thin sections. To our knowledge, no studies have yet addressed frost feature identification in archaeological sediments, compared models and expert performance on micromorphological classification tasks or used interpretability analysis to evaluate whether models learn micromorphologically meaningful features.
Here, we address this gap by evaluating machine learning and expert performance in parallel to the classification of frost-related features in archaeological thin sections. First, we develop and evaluate CNN-based models that identify and classify frost features in photomicrographs. Second, we apply interpretability analysis to visualize model attention patterns and assess whether models learn micromorphologically meaningful diagnostic criteria. Third, we quantify inter-expert variability in frost classification, establishing a baseline for practicing micromorphologists’ performance. Fourth, we directly compare model and expert classifications to evaluate whether computational approaches provide complementary capabilities. Together, these analyses provide a framework for assessing how automated approaches can meaningfully support identification and classification of frost features in archaeological micromorphology.
Materials and methods
Site selection and data preparation
We selected thin section samples from Plio-Pleistocene archaeological sites spanning a broad range of geographical and environmental contexts, specifically targeting deposits containing frost-affected and non-frost-affected sediments. European sites included Quinçay and La Ferrassie (France), Ortvale Klde (Republic of Georgia), El Cierro, La Garma (Spain), Lapa do Picareiro (Portugal), and Bacho Kiro (Bulgaria). Sites beyond Europe included Boker Tachtit (Israel, Levant), Ali Toyta (Ethiopia), and Contrebandiers Cave (Morocco). We chose these sites because they represent different periglacial and non-periglacial depositional environments with the hope to build a model that can (later) be generalized across diverse archaeological settings.
We analyzed 154 thin sections, yielding 733 photomicrographs captured under plane-polarized light (PPL) using a Nikon SZM25 stereomicroscope equipped with a DS-Fi3 digital camera at 5x magnification and 1x zoom. At lower magnifications (e.g., scanned thin section images) the broader depositional context is visible, but individual diagnostic features, such as aggregate boundaries and shape, porosity distribution, lack sufficient resolution for the classification. At higher magnifications, fine details are resolved, but the spatial relationships between features that define the microstructure (granular, blocky, platy-lenticular) types fall outside the field of view. In our case, a 5x magnification provides a balance between the feature resolution and spatial context, which is necessary for the classification. We also used PPL entirely to create a controlled, single-modality imaging procedure, which is particularly useful since diagnostic criteria are directly observed under PPL. Also, merging PPL and XPL computationally would necessitate either multi-model fusion structures lacking proven methods in micromorphological imaging or the possibility of adding methodological artifacts through correlated duplicate observations in the same region.
All the images we acquired are in RGB format, with a resolution of 2880 x 2048-pixel resolution and saved as uncompressed.tif files. These raw images formed the input dataset for subsequent data preparation and model training described in Section 2.2.
We cropped each photomicrograph into six equal parts to capture local textural variability, and we excluded areas with no relevant sedimentary information (e.g., slide edges, large rock fragments). This allowed us to increase dataset size and ensuring models learn from localized diagnostic features suitable for classification (Lakshmanan et al., 2021). After cropping, we resized the images to 512 x 512 (width x height) pixels, ensuring that all images are the same size before model implementation.
We established reference labels based on two annotation levels: (1) binary classification (Frost vs. non-frost), and (2) multiclass classification of Frost images into Blocky, Granular, or Platy-lenticular microstructure types following the standardized micromorphological nomenclature (Stoops, 2021), though simplified here to enhance the practical applicability.
Our dataset is comprised of 921 images in total and randomly split into training (70%), validation (15%) and test (15%) sets (Supplementary Information 4, Table S4.2.1). We augmented only the training set with random rotations, flips, and height/width shifts, and to address class imbalance, we computed class weights and applied label smoothing (Müller et al., 2019; Spelmen and Porkodi, 2018).
Machine Learning workflow
We applied five CNN architectures for the two-stage classification framework: MobileNetV2, EfficientNetV2B0, ResNet50, a custom CNN, and a custom Bayesian Neural Network (BNN). On one hand, we selected three widely adopted CNN initialized with ImageNet weights models (MobileNetV2, EfficientNetV2B0, ResNet50) because they represent different trade-offs between computational efficiency and pattern recognition capability while being effective for small-to-medium dataset (Lakshmanan et al., 2021). Specifically, MobileNetV2 emphasizes speed and has a simple feature extraction design (Sandler et al., 2019). ResNet50 uses a more complex layered structure to capture detailed patterns (He et al., 2015), while EfficientNetV2B0 provides a balanced approach by combining both computational efficiency and complex detection capability (Tan and Le, 2021). On the other hand, Custom CNN and BNN models were designed from scratch specifically for this task. We selected BNN specifically because it allows for uncertainty estimation (confidence intervals) in predictions and to do so we used Monte Carlo (MC) dropout (Jospin et al., 2022; Wang and Yeung, 2021).
To evaluate all models’ generalization, we applied 5-fold stratified cross-validation (CV) with each fold presenting class proportions and ensuring that every image is used for validation once in one of the five subsets (Ghojogh and Crowley, 2023; Gorriz et al., 2024). After completing CV, a final model for each one of the CNN-based models was trained on the entire augmented training set and evaluated on an independent held-out test set, providing an unbiased estimate of model performance on unseen data (Ghojogh and Crowley, 2023). For all models, we used ReLU as the activation function followed by global average pooling and fully connected dense layers. To prevent overfitting, we applied high dropout rates (0.3–0.5), l2 regularization (Gavrilov et al., 2018), and adaptive learning rate scheduling (Santos and Papa, 2022; Ying, 2019).
We evaluated all models’ performance using accuracy, precision, recall, F1-scores, and confusion matrices, as well as visualizing the misclassified images per model (Raschka, 2020; Tohka and van Gils, 2021). To clearly assess which model performed better and how large the performance differences were across both classification tasks, we compared metrics across all folds and the test set using the Friedman test with Nemenyi post-hoc tests to evaluate significance between models (Demšar, 2006a; Rainio et al., 2024).
To understand which parts of an image influences the model’s decision-making process, we applied Score Class Activation Map (Score-CAM), a visualization technique that generates maps which highlight the image regions that mostly affect a model decision that an image belongs to one class or another (Minh, 2023; Wang et al., 2020). To see if these models can be used for archaeological deployment and if the models learn meaningful patterns, we also qualitatively compared if these activation maps align with the established micromorphological criteria, such as frost-related porosity, aggregate shape and arrangement, or microstructure geometry (Stoops, 2021; Van Vliet-Lanoë, 1985; Vliet-Lanoë, 2010). In addition to qualitative inspection, we quantified Score-CAM attention concentration by summarizing what proportion of pixels is needed to reach fixed activation mass thresholds per model and per image. Full implementation details are provided in Supplementary Information 4 (Section S4.2.2).
We determined optimal hyperparameter values for learning rates, dropout rates, regularization coefficients, and models’ architecture-specific parameters (e.g., number of dense layers) through systematic iterative experimentations. We tested variations of these hyperparameter values, evaluated performance on the validation set, and refined settings based on results. This process balanced multiple objectives: maximized accuracy while preventing overfitting and ensuring stable convergence.
Since the random splitting of the operates at crop level, some crops form the same photomicrograph may appear in different sets. However, in our case, the photomicrograph-level split was not feasible because it would have drastically reduced sample size given the already limited dataset. To assess whether the crop-level splitting substantially inflates performance estimates, we performed a small robustness test on full photomicrographs (n = 21; 11 frost, 10 non-frost), applying all five models and computing the same metrics used for the cropped images, together with Score-CAM maps to assess the whole-image behavior (See Supplementary Information 4, Section S4.3 for details).
Expert survey design
Model performance metrics alone cannot assess whether experts would agree with model classifications or whether the reference labels themselves are reliable. Therefore, we conducted an anonymous online survey among practicing micromorphologists (Microsoft Forms v25.7), distributed via direct invitations to micromorphologists and remained open from May 10, 2025, to October 31, 2025.
The survey serves three purposes: (1) quantify inter-expert agreement to identify potential label noise in the training data, (2) establish whether certain images are inherently ambiguous even to experts, and (3) provide a baseline for further comparison between CNN performance to human expert performance.
We selected a subset of 23 images that are representative of our dataset in a two-stage process. First, we identified potentially ambiguous images based on frequent misclassification by preliminary CNN models, (misclassified by ≥3 of the models), and cases where reference labels require extended expert discussion without reaching initial consensus. Second, we refined this subset through expert review to ensure representation of specific ambiguity types, such as poorly defined frost features, frost features not dominating the field of view, and non-frost images displaying potentially confounding characteristics.
In the first task (first screening assessment, n = 9), participants evaluated whether the sediment features within the resized image are sufficient to attempt classification. We selected these 9 images specifically because preliminary CNN models frequently misclassified them and because reference label assignment required extended discussion, suggesting that diagnostic features were difficult to identify at the cropped and resized scale.
In the second task (binary and multiclass classification, n = 14), participants classified images in a two-stage process mirroring the CNN framework: first assessing whether images contain any frost features, and if they do, then the specific type of microstructure. This conditional design prevents forced classification and allows direct comparison with model outputs.
We assessed inter-rater agreement using Fleiss’ kappa, individual expert-reference labels alignment using Cohen’s kappa (Landis and Koch, 1977), raw percentage agreement, and relationships between response variability and agreement using Spearman’s ρ (Mukaka, 2012). We also quantified the proportion of “Unsure” responses per participant and per image. Full statistical procedures are detailed in Supplementary Information 4, Section S4.4.
Experts and CNN models alignment
To evaluate whether CNN-based models align with expert micromorphological practice, we compared model predictions and expert classifications across the 14 images from the survey’s second tasks, the images where both experts and models completed full two-stage classification. All comparisons used the reference labels as reference.
Evaluating experts and CNN models’ alignment addresses three questions: (1) do CNN models agree with expert classifications, (2) do they make similar errors on the same images, and (3) does model uncertainty correspond with expert-perceived classification difficulty.
To evaluate how models behave on images where experts judged diagnostic features as insufficient for classification, we used the nine images from the survey’s first-screening task. Since these nine images presented challenges in feature visibility at the cropped and resized scale, they were exclusively for assessing whether models can distinguish between images with sufficient and insufficient diagnostic information. The remaining 14 images, where features were considered sufficiently visible for classification, were used for the two-stage classification tasks (binary and multiclass) and form the basis of the expert-model comparison described below. For the nine first-screening images, we categorized each by majority expert vote as “Visible”, “Not visible”, or “Unsure”. For each model, we generated frost/non-frost prediction on all nine images and summarized (i) class proportions on “Not visible” images, (ii) mean confidence scores (Equation 1, Supplementary Information 4, Section 4.5) for “Visible” vs. “Not visible’ subsets, and (iii) For the BNN, Monte Carlo dropout uncertainty (standard deviation over 20 passes).
For the classification task, we assessed the overall degree of agreement using Fleiss’ kappa first for the CNN models only (n = 5) and then all participants combined (n = 24; 5 models and 19 experts). We also computed Cohen’s kappa for all 95 expert-model pairs and classification accuracy against reference labels for each participant. To test whether experts agreed more with one another than with models, we compared Cohen’s kappa distributions using Mann-Whitney U test (Conover, 1999), with the same comparison applied to accuracy values. We quantified the error overlap between experts and models using Spearman’s ρ between misclassification patterns. We also used Spearman’s ρ to examine relationships between agreement proportions, model confidence scores, and BNN uncertainty metrics.
We categorized images into difficulty categories based on classification performance: (1) “Hard for CNN” correctly classified by ≥70% of experts but misclassified by ≥3 models; (2) “Hard for Experts” correctly classified by ≥3 models but correctly classified by <70% of experts; (3) “Hard for Both” misclassified by ≥3 models and correctly classified by <70% of experts. These thresholds were selected to identify images with clear difficulty patterns while maintaining sufficient sample sizes per category. Complete statistical pipelines and details are provided in Supplementary Information 4.
Workflow and interactive classification tool
The entire workflow, including image pre-processing, classification, evaluation, and interpretability were implemented in Python 3.12 with Tensorflow (Abadi et al., 2016) and Keras (Chollet, 2015) libraries within Google Colaboratory (Bisong, 2019) using an A100 GPU. Statistical analyses of the survey data and model comparisons were performed using Scikit-learn (Pedregosa et al., 2011) and Statsmodels (Seabold and Perktold, 2010). A workflow image and a detailed glossary of the machine learning and statistical terms used in this study is included in Supplementary Information 4, Figure S4.2.1; Table S4.1.1.
To support practical adoption and reproducibility, we also developed an interactive web application using Gradio (Abid et al., 2019) and Hugging Faces that enables users to upload their thin section photomicrographs and obtain frost classification predictions with confidence scores, uncertainty and interpretability, implementing the same two-stage classification framework (binary followed by multiclass). The tool is publicly available at online (https://huggingface.co/spaces/skkou/frost_classifier) and detailed guidelines on how to use it are provided in Supplementary material 4, Section S4.8.
Results
CNN models achieve high performance but rely on spurious correlations
Overall, all five models (MobileNetV2, EfficientNetV2B0, ResNet50, Custom CNN, and BNN) perform well on both binary (frost vs. non-frost) and multiclass (blocky, granular, and platy-lenticular) tasks in both 5-fold cross-validation and the test set. Models achieve better performance on binary tasks, while distinguishing among the three frost-related microstructures is more challenging (see Supplementary Information 1 and 4; Section S4.6–7).
Learning dynamics shows stable convergence for all models in the binary task, while there are more variable validation patterns with oscillations during the training procedure in the multiclass task (Supplementary Information 4, Figure S4.6.1-S4.6.3). Cross-validation results (precision, recall, and F1-score) are summarized in
Figure 1. In binary classification, MobileNetV2 and EfficientNetV2B0 achieve the highest mean accuracy (98–99%), with all other models above 94%. In multiclass classification, performance is slightly lower. MobileNetV2 performs best (~ 96%), followed by EfficientNetV2B0 and ResNet50 (~ 94–95%). Images with granular features are classified consistently across all models (>93% precision and recall), while images with blocky and platy-lenticular features show greater variability. Evaluation of the test set confirms these cross-validation patterns, where MobileNetV2 and EfficientNetV2B0 achieve the highest performance on binary classification and show slightly lower performance in multiclass classification (Supplementary Information, Tables S4.6.1–2). Statistical comparisons using Friedman’s test confirm significant differences in binary classification, with MobileNetV2 and EfficientNetV2B0 achieving the highest ranks. No significant differences are detected in the multiclass classification (
Table 1A).
Figure 1. Comparative performance of five CNN architectures (BNN, Custom CNN, EfficientNetV2b0, MobileNetV2, and ResNet50) based on mean Accuracy, and per class Precision, Recall, and F1-Score and the standard deviation over 5-Fold cross-validation (CV).
(a) Binary classification (Frost vs. non-frost) performance comparison. (b) Multiclass classification performance across Blocky, Granular, and Platy-lenticular microstructure classes.
Table 1. Statistical comparison of CNN model performance and expert-model agreement across 14 classification images.
(A) Model ranking based on Friedman’s test with significant pairwise differences identified by Nemenyi post-hoc test. (B) Pairwise agreement comparison between expert-model pari and expert-expert pairs, assessed using Mann-Whitney U test. (C) Group-level agreement using Fleiss’ kappa in three configurations (inter-expert, inter-model, and models-experts).
| A. Model ranking and significant pairwise differences |
|---|
| Model | Rank | Significant differences |
|---|
| MobileNetV2 | 1 | >BNN (p = 0.039); >ResNet50 (p = 0.001) |
| EfficientNetV2B0 | 2 | >ResNet50 (p = 0.005) |
| Custom CNN | 3 | - |
| BNN | 4 | - |
| ResNet50 | 5 | - |
| B. Expert-model vs expert-expert pairwise agreement |
|---|
| Comparison | Cohen’s kappa (SD) | Accuracy (SD) | N pairs | Statistical test |
|---|
| Expert-Model
| 0.387 (0.343) | 70.4 (21.1) | 95 | U = 7397, p = 0.224 |
| Expert-Expert
| 0.447 (0.361) | 74.1 (21.2) | 171 | Reference |
| Δ (difference) | −0.060 | −3.7 | - | Not significant |
| C. Group agreement configurations |
|---|
| Configuration | Fleiss’ kappa | Mean agreement (%) | N raters | Interpretation |
|---|
| Inter-Expert
| 0.202 | 60.9 | 19 | Slight |
| Inter-Model
| 0.676 | 91.4 | 5 | Substantial |
| Models-Experts
| 0.212 | 64.6 | 24 | Fair |
Misclassification patterns are consistent in both binary and multiclass classification (
Figure 2). In the binary classification, non-frost images contain high textural variability, while in the frost class and the subsequent microstructure classification, misclassifications are concentrated in images where the diagnostic features are either poorly developed, densely packed or occupy less than 30% of the image area.
Figure 2. Systematic misclassification patterns across CNN models and representative examples of challenging images.
(a) Number of models (out of 5 total: Custom CNN, BNN, MobileNetV2, EfficientNetV2B0, and ResNet50) that misclassify images in each class. (b) Representative Non-Frost images exhibiting high textural variability that mimics frost patterns. (c) Frost images displaying characteristics that challenge models’ performance: densely packed granular aggregates obscuring pore space (two top row images), poorly developed microstructures (bottom left), and limited spatial coverage of diagnostic features (bottom right).
To assess whether models rely on micromorphologically diagnostic features (such as aggregate geometry, frost-related porosity, and sediment arrangement), we applied Score-CAM interpretability analysis. For both classification tasks, MobileNetV2, EfficientNetV2B0, and ResNet50 models focus on scattered or fixed activation patterns that do not align with diagnostic features used by micromorphologists (
Figure 3). ResNet50 consistently highlighted a central rectangular region regardless of the image content. In contrast, Custom CNN and BNN show more interpretable activation patterns (>0.6). These models show higher attention to aggregate geometry and characteristic porosity, especially in granular and platy-lenticular classes. However, both models occasionally attended to irrelevant elements within the image, such as bone fragments. The cumulative distribution of Score-CAM activation values across pixels shows that off-the-self models have highly concentrated attention with the top ~10–25% of pixels accounting for roughly half of the activation mass. Custom CNN and BNN require ~40–45% of pixels to reach the same threshold (Supplementary Information 4, Figure S4.6.6).
Figure 3. Score-CAM visualizations across five CNN architectures illustrating model attention patterns during (a) binary (Frost vs. non-frost) and (b) multiclass (Blocky vs. Granular vs. Platy-lenticular) classification.
Original images are in PPL, where the porous space is white. Each heatmap highlights the image regions contributing most to model predictions. Differences in activation distribution reveal model-dependent focus and spurious correlations (for EfficientNetV2B0, MobileNetV2, and ResNet50), whereas other models (BNN and Custom CNN) emphasize more on diagnostically relevant areas (e.g., porosity) consistent with micromorphological criteria.
Quantitative Score-CAM analysis shows differences in attention concentration across architectures. Off-the-self models (MobileNetV2, EfficientNetV2B0, ResNet50) require only ~10–20% of pixels to reach 50% of activation mass, while Custom CNN and BNN needed approximately 40–45% of pixels showing more diffuse patterns (Supplementary Information 4, Figures S4.6.7–8 and Table S.4.6.1).
The complementary robustness test on the entire photomicrographs shows that off-the-shelf models, and particularly MobileNetV2, retained relatively balanced frost/non-frost performance. Whereas the custom CNN and BNN achieve lower performance metrics, but focused more consistently on micromorphological relevant features, with all models more prone to misclassify frost than non-frost images at whole scale (Supplementary Information 4, Figures S4.6.11–12 and Table S4.6.3).
Expert agreement on frost identification and classification is low.
Of the 20 practicing micromorphologists who responded to the survey call, 19 completed it, with 4 self-reporting little to no experience, 10 having some experience, and 5 having significant experience with frost action. Due to small and uneven sample sizes, the statistical analysis performed has a more exploratory and descriptive intention rather than a definitive and inferential one. The survey’s raw data and extended results are provided in Supplementary Information 2 and 4.
Overall, agreement among experts is low in both tasks. Fleiss’s κ indicated only slight agreement (κ ≤ 0.20; McHugh (2012)) (
Table 1C).
Figure 4a shows the broad spread of responses in both visibility and classification tasks relative to reference labels. The first-screening tasks produced roughly twice as many “Unsure” responses as the classification tasks, showing that experts find it harder to determine a feature that is present in the cropped and resized photomicrographs than to identify its class.
Figure 4. Summary of survey results evaluating expert reliability in frost identification and classification.
(a) Comparison of response patterns between first screening (in the legend is indicated as visibility task) and classification tasks across five reliability metrics (0–80% scale): agreement with reference labels, inter-participant consensus, disagreement with reference labels, disagreement among participants, and unsure response frequency. (b) Distribution of inter-rater agreement (Cohen’s Kappa) against the ground truth across frost experience levels during classification tasks, showing an inverted U pattern where participants with moderate frost experience have higher consistency among participants than participants with little to no experience and significant experience with frost-action. (c) Relationship between uncertainty (% of unsure responses) and performance (% of agreement with ground truth) for the first-screening task, colored by frost-experience level. The red dashed line shows downward trend, indicating strong negative correlation (Spearman r = −0.800, p < 0.001). (d) Relationship between uncertainty and performance for the classification task, colored by frost-experience level. Flat red dashed line indicates no correlation (Spearman r = −0.012, p = 0.961).
Individual performance varies widely. For the classification task, Cohen’s κ relative to reference labels range from near chance to moderate agreement (Landis and Koch, 1977; McHugh, 2012), and percentage agreement per image ranges from 7 to 71%. In the first-screening tasks, percentage agreement ranges from 11 to 89%, but Cohen’s κ could not be calculated because all reference labels are identical (assuming that all features are visible; Supplementary Information 4, Section S4.7). Frost-specific experience does not follow linear patterns. Experts with “some experience” show the highest median Cohen’s κ (~ 0.45), while those with little to no experience (~ 0.20) and “significant experience” (~ 0.17) score lower with greater variability, producing an inverted U-shape pattern (
Figure 4b).
Experts select “Unsure” for approximately 25–33% of images on average, with large individual variation (Supplementary Information 4, Table S4.7.4). In the first-screening task, uncertainty is negatively correlated with performance (Spearman r = −0.800, p < 0.001;
Figure 4c), while no correlation is observed for the classification task (Spearman r = −0.012, p = 0.961;
Figure 4d).
Models and experts show complementary strengths in frost classification.
To compare models with expert performance, we evaluated how models’ predictions aligned with reference labels and with the 19 experts who classified the same 14 images in the survey’s first screening and classification task. Extended results are provided in Supplementary Information 3 and 4.
On the nine first-screening images, models classified images confidently even when experts judged the images as insufficient for classification. Across the four images categorized as “Not visible” by majority vote, most models predicted an even split between frost and non-frost (2 each), with the exception of EfficientNetV2B0, which predicted frost for three of the four images. Mean confidence scores were consistently higher for “Visible” images (ranging from 0.85 to 0.98 across models) that for “Not visible” images (0.52 to 0.87), but confidence remained moderate to high rather than dropping to chance level (Supplementary Information 4; Table S4.7.2). The BNN shows the clearest differentiation, with higher mean uncertainty on “Not visible” images (σ = 0.089) than “Visible” (σ = 0.052), though even this difference was modest (Supplementary Information 4; Figure S4.7.4 and Tables S4.7.3). Overall, models produced confident predictions regardless of whether experts considered diagnostic information sufficient or insufficient.
In the classification task, model agreement with reference labels varies significantly (Friedman x2 = 33.4, p < 0.001;
Table 1A). MobileNetV2 shows the highest alignment (Cohen’s κ = 0.536), significantly outperforming BNN (p = 0.039) and ResNet50 (p < 0.001). EfficientNetV2B0 also performs well (κ = 0.490), significantly exceeding ResNet50 (p = 0.005).
Agreement between experts and models approaches the level of agreement observed between experts. Across 95 expert-model pairs, the mean Cohen’s κ is 0.39 (70% accuracy), compared to 0.45 (74% accuracy) of 171 expert-expert pairs (
Figure 5a-b;
Table 1B). Differences are not statistically significant for either κ (p = 0.224) or accuracy (p = 0.125), showing that CNN models achieve expert-level inter-rater reliability.
Figure 5. Expert-model agreement and image-level complementarity analysis.
(a-b) Distribution of Cohen’s kappa (a) and accuracy (b) for each CNN model against 19 experts. Red dashed lines indicate the expert-expert baseline (κ = 0.447, accuracy = 0.741). MobileNetV2 achieves the highest alignment with experts, approaching the expert-expert baseline. (c) Comparison of expert error rates (blue dot) and model error rates (red dot) for each of the 14 classification images, with thumbnail previews. Lines connect expert and model error rates per image, colored by complementarity patterns: green lines indicate images where models outperform experts (n = 8), cyan lines indicate similar performance (n = 2), and orange lines indicate images where experts outperform models (n = 4). Background shading corresponds to these categories.
Agreement patterns differ within groups (
Table 1C). The five models show substantial consensus agreement (Fleiss’ κ = 0.676, > 90% mean agreement), exceeding inter-rater agreement (Fleiss’ κ = 0.202, ~ 61% mean agreement; McHugh (2012)). When models are combined with experts (n = 24), Fleiss’ κ (0.212, fair agreement; (Landis and Koch, 1977) approaches expert-only levels.
Image-level analysis reveals independent error patterns, where models and experts fail on different images, demonstrating complementary capabilities (
Figure 5c). Eight images are difficult for experts, and models help in classification; four images need expert verification in the models’ prediction, and in two models, experts’ and models’ classifications align. Finally, BNN uncertainty shows an inverse relationship with expert-perceived difficulty (“Unsure” answers; Spearman ρ = −0.497, p = 0.071). For images where experts express the highest uncertainty (~32–63% “Unsure” rate), BNN uncertainty remains consistently low (~0.03–0.07; Supplementary Information 4, Figure S4.7.3).
Discussion
This study evaluates whether CNN-based classifications can identify frost features in archaeological thin sections with accuracy and reasoning comparable to practicing micromorphologists. We assessed model performance and interpretability, quantified expert variability, and directly compared models and expert classifications to determine whether computational and human approaches provide complementary capabilities. This framework enables us to evaluate not only classification accuracy but also decision mechanisms and the conditions under which CNNs can meaningfully support micromorphological practice.
We report four main findings: (1) CNN models achieve high classification performance, but best-performing off-the-shelf models rely on spurious correlations rather than diagnostic micromorphological features; (2) expert agreement on the same classification task is low, reflecting inherent interpretive challenges rather than methodological failure; (3) models and experts make independent errors on different images, indicating complementary strengths rather than redundant capabilities; and (4) model uncertainty does not align with expert-perceived difficulty, highlighting the need for expert oversight in hybrid workflows.
All models performed better on binary (frost vs. non-frost) than multiclass (blocky, granular, platy-lenticular) classification, and off-the-shelf models (MobileNetV2, EfficientNetV2B0, and ResNet50) performed better than those trained from scratch (Custom CNN, and BNN), a result that is consistent with the advantages of transfer learning for small-to-medium datasets reported in geological and petrographic imaging (Cai et al., 2024; Dawson et al., 2023; Pires de Lima et al., 2020). Among three widely adopted CNN architectures, lightweight models (MobileNetV2 and EfficientNetV2B0) generalized more effectively that heavier architectures (ResNet50), likely because their simpler designs are less prone to overfitting on limited training data (M. Romero et al., 2020b; Zhao et al., 2024). A similar pattern was also observed in carbonate rock classification where more complex CNN architectures performed better with bigger datasets (Dawson et al., 2023). The performance drop from binary to multiclass classification reflects a fundamental challenge, and several factors contribute to this performance decline. First, the classification threshold in the binary is easier to learn, and multiclass classification threshold boundaries might be vague or overlap with another class (Moral et al., 2022). Second, frost microstructures form a visual and functional continuum rather than discrete categories. Blocky, granular and platy-lenticular forms can be transitional, poorly developed or overlapping with other pedogenic processes through equifinality, for example granular microstructures from freeze-thaw cycles can resemble those produced from bioturbation (Van Vliet-Lanoë, 1985). This overlap between categories contrasts with machine learning techniques that demand discrete categorical boundaries, creating an epistemological mismatch between natural variability and the fixed category labels required for classification (Petrelli, 2024).
Furthermore, interpretability analysis showed that classification accuracy did not necessarily reflect the use of meaningful micromorphological features. Score-CAM maps showed that the highest-accuracy off-the-shelf models (MobileNetV2, EfficientNetV2B0, ResNet50) focused on scattered or central image regions unrelated to diagnostic criteria. This pattern is consistent with shortcut learning where deep networks exploit spurious regularities rather than focusing on the actual sediment features (Adebayo et al., 2020; Geirhos et al., 2020; Rosenfeld et al., 2018). In our case, the spatial attention patterns observed in off-the-shelf models likely reflect biases inherited from ImageNet pretraining, where images are predominantly object-centric and models focus on the part of the image with the most variability (Zhang et al., 2022). In ResNet50, the repeated activation of a central rectangular region across all photomicrographs exemplifies this center-of-the-photo bias. The additional quantitative Score-CAM analysis we conducted confirms that decisions in these off-the-shelf models are driven by compact regions of the image that only partially (or do not) overlap with micromorphological diagnostic sediment features, such as porosity and matrix arrangement, highlighting that high performance can coexist with non-diagnostic attention patterns.
In contrast, Custom CNN and BNN showed lower performance (varying from 86–95% in accuracy) but had clearer interpretability. Quantification of the pixel activation mass showed that the attention is more diffused, and Score-CAM overlays emphasized mostly on sediment matrix textures, frost-related porosity, and aggregate boundaries and shape, indicating that simpler architectures may force attention onto visual features that align with diagnostic criteria (Lipton, 2017; Rudin, 2019). This interpretability-performance trade-off suggests that simpler architectures focus on diagnostic features but lack representational capacity to integrate them across the frost pattern for robust generalizations (Raghu et al., 2019). At the same time, difference in feature extraction design across architectures (inverted residual blocks in MobileNetV2 (Sandler et al., 2019), compound-scaled blocks in EfficientNetV2B0 (Tan and Le, 2021), residual and squeeze and excitation blocks in ResNet50 (He et al., 2015), and the more linear custom CNN and BNN) and the specific last convolutional layer used for Score-CAM introduce distinct inductive biases in which regions are treated as informative.
In addition, the crop-level splitting may contribute to the optimistic performance estimates through spatial autocorrelation between crops from the same photomicrograph. However, the robustness test on the entire photomicrographs, which is independent splitting strategy, reproduces the same architecture-dependent pattern: off-the-shelf models retain better classification performance while custom models show better interpretability alignment. This suggests that the core findings are not primarily by crop level autocorrelation.
Therefore, high classification performance alone is insufficient for evaluating models in micromorphological contexts, where feature identification must be grounded in diagnostic criteria. Interpretability analyses are then essential not merely to explain model behavior, but to verify that classifications reflect the same process-based reasoning that experts apply (Ebert-Uphoff and Hilburn, 2020; Rudin, 2019b). Similar interpretability paradoxes have been reported in geological petrography, where well-performing CNN’s focus on visually irrelevant or non-diagnostic regions (Cai et al., 2024; Zheng et al., 2024). In other words, without explicit interpretability validation, deep learning models in micromorphology may be “right for the wrong reasons” and thus unreliable for process-based archaeological interpretations.
Interestingly, our survey results showed only a slight expert agreement both on ambiguous images and with reference labels. This reflects how micromorphological analysis typically unfolds: dynamically scanning thin sections while switching between cross-polarized and plane-polarized light, changing magnifications, rotating stages to observe crystal orientations, checking the overall geological context. In the survey, we effectively removed these workflows by providing static images at fixed lighting and resolution without a more general geological provenience context. Furthermore, many of the photomicrographs provided to the experts had weakly developed frost features or features that mimic those related to frost action (e.g., developed pedogenic pedality). Experts therefore lacked crucial information for confident classification, explaining low agreement, high uncertainty rates, and disagreement with the reference labels.
These constraints intersect with deeper challenges. Frost features can overlap with other pedogenic (planar voids and peds in vertisols) or biogenic (coalescent granular structures derived from bioturbation) processes, producing microstructures that can mimic those typically derived from frost action, particularly at the photomicrograph scale (Stoops, 2021). This overlap is compounded by the continuum-to-category mismatch discussed above, which amplified the subjectivity in expert judgement (where to draw boundaries) and rigidity in model classification framework, which must assign a single category. Shahack-Gross (2016) showed similar low agreement among micromorphologists when looking at basic component and mineralogical identification, suggesting an interpretive variability in micromorphology analysis. Such subjectivity arises because micromorphological features vary in degree of development (from incipient to well-developed structures), diagnostic criteria involve development of quality judgments, and expert experience shapes recognition patterns.
This inherent subjectivity, however, also explains why model-experts complementarity is advantageous. The near-zero error correlation between models and experts indicates that machine versus human expert errors arise from distinct heuristics: where CNN models ask, “what does this look like?”; experts ask, “how did this form?”. These are different epistemic questions, meaning that CNNs are not simply faster versions of experts but different kinds of observers. Complementarity arises not from matching abilities but from non-overlapping strengths. Because models and experts succeed under different conditions, hybrid workflows are justified (Dvijotham et al., 2023). Once validated, CNNs can rapidly screen large image sets, provide consistency unaffected by fatigue, and flag uncertain cases. Experts can then adjudicate ambiguous cases, provide contextual interpretation, and integrate into sedimentary and paleoenvironmental narratives; neither replaces the other (Hemmer et al., 2025). This epistemic divergence is further evidenced by BNN uncertainty estimates, which were well calibrated for model predictions but inversely correlated with expert-perceived difficulty; where experts expressed uncertainty due to equifinality, missing context, or weak feature expression, the BNN remained confident because visual patterns matched training distributions. This confirms that Monte Carlo captures parameter uncertainty, not interpretive ambiguity (Gal and Ghahramani, 2016). Expert oversight therefor remains essential for low-confidence cases, non-diagnostic attention patterns as revealed through the expert-models comparison and interpretability analysis, especially when specific images challenge all classifiers (Bianchi et al., 2024; Steyvers et al., 2022).
However, several limitations constrain the generalizability of our findings. Our simplified classification scheme does not capture the full spectrum of frost action variation. The five CNN architectures evaluated here represent a subset of available deep learning approaches, and other architectures designed for spatial relationship capture may better accommodate the relational reasoning central to microfeatures. Reliance on PPL at single magnification eliminates information from XPL examination and multi-scale observation central to standard micromorphological practice. While our dataset encompasses eleven Plio-Pleistocene sites, broader testing across diverse periglacial and non-periglacial contexts is needed to establish robust generalization. Future work should expand datasets across more sediment types and depositional environments, incorporate multiple imaging modalities, and explore more model architectures suited for capturing spatial relationships between microfeatures. Additionally, the low inter-expert agreement observed in our survey suggests that single label annotations may not fully capture the interpretive variability inherent to micromorphological classification. Future work should incorporate multi-annotator protocols that preserve disagreement as information rather than resolving it into single labels, thus enabling uncertainty-aware training strategies that reflect genuine diagnostic ambiguity (Sheng and Zhang, 2019).
To facilitate the adoption and reproducibility of this framework, we provide a publicly accessible classification tool that implements the two-stage classification pipeline with interpretability maps, enabling researchers to obtain frost feature predictions for their own thin-section images with full transparency into model attention patterns (available at: https://huggingface.co/spaces/skkou/frost_classifier). However, the outputs should be treated as a first-pass suggestion and not as a definitive micromorphological diagnoses and must be checked with established criteria and site context.
Conclusions
This study evaluated whether deep learning models can identify and classify frost-related micromorphological features at an expert level while maintaining interpretability aligned with established diagnostic criteria. We trained five CNN models on photomicrographs from multiple archaeological sites and validated them against both interpretability analysis and an expert survey.
Three principal findings emerge. First, off-the-shelf models achieve better classification performance, but interpretability analysis revealed reliance on spurious correlations rather than diagnostic features, demonstrating that good performance alone is not sufficient for evaluating models in micromorphology. Second, expert agreement on the same task was low, confirming that inter-observer variability is intrinsic to frost feature identification rather than a limitation specific to computational approaches. Third, mode and expert errors were largely independent, indicating that CNNs and expert micromorphologists capture different and complementary aspects of frost feature recognition.
Together, these findings establish a proof-of-concept for model-expert collaboration that addresses persistent challenges in archaeological micromorphology: scaling comparative analysis beyond what the manual examination permits, improving reproducibility through consistent pattern recognition, and broadening analytical access for researchers without extensive frost-specific training. The proposed framework, combining interpretability validation, uncertainty quantification, multi-expert consensus labelling, and model-expert workflow, provides a transferable template for computational integration across disciplines facing similar challenges with subtle, overlapping features subject to inter-observer variability.
Expanding this framework to incorporate broader sedimentary contexts and multiple imaging modalities will be essential for moving towards computational micromorphological tools. An effective implementation requires integrated workflows, where models handle the first pass screening while experts provide targeted validation and contextual synthesis into behavioral narratives. This complementarity preserves interpretive integrity while enabling analyses at scales previously constrained by time and expertise availability.
Ethics and consent
No ethical approval was needed for this study. The data were completely anonymous and we used no identifiable information on the survey participants.
