1.
Introduction
Cetacean bioacoustics and passive acoustic monitoring
1.1
Over the past few decades, Passive Acoustic Monitoring (PAM) has become a widely-used, non-invasive tool for monitoring acoustically active species over time,1 especially when environmental conditions make visual sightings logistically impossible. As researchers increasingly deploy PAM across multiple sites and extended timeframes, they generate massive datasets that require highly advanced analytical frameworks. Monitoring cetaceans2 is one of the most challenging applications of this technology. Marine mammals produce a wide range of vocalizations across wide frequency bands, sometimes in habitats heavily disrupted by intense human noise. Because of these environmental complexities, maintaining high accuracy in population monitoring supports reliable long-term assessment.
Growing human impacts on marine ecosystems have intensified the need for comprehensive, large-scale conservation efforts. Since marine mammals rely almost entirely on sound to communicate, navigate, and forage, PAM serves as the foundation for studying these vulnerable species. Visual surveys are constrained by weather, daylight, and observer availability, while PAM can enable continuous, unattended recording enabling researchers to scale their monitoring efforts efficiently across both coastal waters and remote offshore environments.3 On a technical level, underwater microphones called hydrophones capture these vocalizations by converting pressure changes into electrical signals. When paired with low-noise preamplifiers, these systems generally operate at sampling rates up to 200 kHz. PAM networks are expanding to support a range of management objectives, including estimating population density, assessing behavioral response to anthropogenic noise, and – as in the present work – reducing harmful interaction between cetaceans and commercial fisheries.4
Once the audio is digitized at high frequencies, operators can either monitor the signals in real time for immediate detection or archive the recordings for later study. Both approaches rely on a mix of manual audio-visual inspection and advanced automated workflows.5 For tonal vocalizations such as whistles, the dominant analytical representation is the spectrogram. Generated via the Short-Time Fourier Transform (STFT), the spectrogram converts raw time-domain signals into a two-dimensional time-frequency image. This visual format is ideal for identifying tonal vocalizations such as whistles, which appear as smooth, continuous, frequency-modulated lines that stand out clearly against sharp, impulsive sounds like echolocation clicks or background static.5 The field has steadily shifted away from manual spectrogram reviews toward automated detection. Early digital signal processing (DSP) techniques—such as template matching,6 active contour methods,7 and entropy-based detectors8 —contributed to automated whistle identification. Over the past decade, deep learning approaches – particularly with convolutional neural networks (CNN) applied to spectrogram images – have achieved detection and classification performance that consistently exceed earlier DSP methods on published benchmarks, though conventional approaches remain widely deployed in operational systems.
Deep learning for cetacean PAM: state of the art
1.2
Convolutional neural networks are now a widely used architecture for detecting cetacean vocalizations via spectrograms, and have outperformed traditional methods in several tasks. Jiang et al.9 presented a two-stage pipeline in which an initial CNN detects if a brief spectrogram frame contains a whistle, and a second network classifies the sound by species, distinguishing between killer whales and long-finned pilot whales. This has been obtained utilizing small grayscale spectrogram tiles; they achieved 97% detection and 95% classification accuracy. This success showed that compact, single-channel inputs can effectively capture the geometric shape of whistle contours, provided that the frequency limits and frame durations are carefully selected. While Jiang et al.9 focused on pre-rendered spectrogram images, Shiu et al.10 showed that these models could adapt to varying recording conditions by using a different input format. Instead of treating the spectrogram as a standard visual image, two-second audio clips were extracted, and the Short-Time Fourier Transform (STFT) is calculated using a 128 ms Hann window with a 50 ms step size. This process generated a matrix of time-frequency magnitude values, which they normalized using the squared Frobenius norm and passed directly into the CNN as a numerical tensor. When tested on North Atlantic right whale upcalls between 39 Hz and 358 Hz, their model delivered significantly better precision-recall metrics and cut false positives per hour by orders of magnitude compared to existing baselines. The model maintained its high performance on entirely new recordings from different times and locations, successfully passing the real-world stress test that operational passive acoustic monitoring systems regularly encounter.
Li L et al.11 utilized a depthwise-separable CNN to classify multiple types of whistles from Indo-Pacific bottlenose dolphins (Tursiops aduncus). They used three-channel spectrogram inputs combined with a comprehensive set of time-domain data augmentations, including pitch-shifting, time-stretching, noise mixing, temporal shifting, and gain adjustments. Classification accuracy increased from 93.7% on the original samples to 98.53%, proving that data augmentation can effectively offset the challenges of small training sets and unbalanced classes. In this study, the authors admitted that their chosen STFT parameters might not represent the ideal spectrogram configuration for whistle classification, treating the spectrogram generation as an arbitrary choice rather than an optimized variable.11
Li P et al.12 addressed the annotation bottleneck by training a deep network for whistle contour extraction on samples synthesized from background recordings, requiring minimal manual annotation, and evaluated it on common (Delphinus spp.) and bottlenose dolphin whistles. In a follow-up study, the same group13 removed the need for human-annotated training data by training on pseudo-labels generated by existing whistle extractors, including a sequential Monte Carlo tracker, and evaluated the method on five odontocete species. White et al.14 extended CNN detection beyond whistles to four sound-source classes (delphinid tonal calls, biological clicks, vessel noise, and ambient noise), stacking spectrograms computed at three FFT sizes (1024, 2048, and 4096 points) as the RGB channels of a single EfficientNet-B0 input.
Passing from simple binary detection to tracing the exact paths of vocalizations, researchers like Jin et al.15 have reframed the challenge as a semantic segmentation task. Their model processed high-resolution grayscale spectrograms to generate pixel-level masks of whistles, which were then refined using morphological post-processing to map out centerlines and untangle overlapping calls. Despite evaluating their model on noisy coastal recordings filled with snapping shrimp interference, they achieved a precision of 0.96, an accuracy of 0.89, and an F-score of 0.86. These results show that dense, high-resolution inputs are essential when a study needs to preserve precise contour curves and thickness for subsequent bioacoustic analyses. Taking a different approach, Nur Korkmaz et al.16 investigated the potential of transfer learning for dolphin whistle detection. Single-channel grayscale spectrograms have been duplicated into a three-channel format, and a pre-trained VGG16 network using automated hyperparameter optimization has been used. This method achieved a 92.3% mean accuracy on a substantial, annotated dataset, easily outperforming traditional detectors while keeping the initial preprocessing straightforward. Their work underscores the practical benefit of adapting established computer-vision models when annotated underwater audio datasets are scarce. Based on these studies, the way input data is represented is just as critical to performance as the neural network design itself. Small, single-channel tiles work well for basic detection and generalize efficiently with minimal computing power, three-channel inputs allow researchers to leverage powerful pre-trained models, and high-resolution imagery preserves fine pixel-level details of sound contours. Preprocessing choices such as frequency filtering, STFT configuration, and image scaling are therefore not incidental: they shape what the network learns and how well it transfers. While this reality is widely accepted in theory, the field still lacks a rigorous, quantitative evaluation of how these preprocessing choices interact within a unified, reproducible pipeline for cetacean monitoring.17
Table 1 presents a summary of these cited studies.
Table 1. Summary of CNN-based cetacean vocalization studies reviewed in Section 1.2.
| Study | Species/call type | Input repr. | Size | Ch. | Task | Preprocessing treatment |
|---|
| Jiang 9 | Orcinus orca/Globicephala melas, whistle | Grayscale image | ~180 × 120 | 1 | Det. + Class. | Fixed band-limiting & frame duration. |
| Shiu 10 | Eubalaena glacialis, upcalls | Num. matrix | 40 × 40 | 1 | Binary Det. | Fixed STFT (128 ms Hann, 50 ms adv.) |
| Li L 11 | Tursiops aduncus, whistles | RGB image | 200 × 200 | 3 | Multiclass | Fixed STFT; acknowledged as non-optimal. |
| Li P 12 | Delphinus spp./Tursiops truncatus, whistles | Spectrogram patches | 50 × 50 | 1 | Contour extr. | Fixed STFT (8 ms Hamming, 2 ms adv.); training samples synthesized from background noise |
| Li P 13 | Five odontocete species, whistles | Grayscale patches | 64 × 64 | 1 | Contour tracking | Same STFT as12; trained on pseudo-labels, no human annotation |
| White 14 | Multiple marine sources | Stacked multi-resolution spectrograms | 224 × 224 | 3 | Multiclass (4 cl.) | Three FFT sizes stacked as channels; Hann, 75% overlap |
| Jin 15 | Tursiops truncatus, whistles
| Grayscale image | 1000 × 1000 | 1 | Sem. Segm. | Fixed high-res for contour fidelity. |
| Korkmaz 16 | Tursiops truncatus, whistles | RGB image | 224 × 224 | 3 | Binary Det. | Minimal and fixed for VGG16. |
A second, related challenge concerns the annotation of training data. The studies reviewed above employ diverse and largely undocumented labeling conventions: event boundaries, minimum duration thresholds, and the criteria for distinguishing whistle from click train or burst pulse are rarely made explicit. This fragmentation has been identified as a systemic limitation of bioacoustic machine learning17 and has prompted community efforts to standardize acoustic metadata — notably the Tethys database framework,18 the emerging ANSI/ASA S3/SC1.7 standard for bioacoustic metadata,19 and broader PAM data management guidelines.20 On the training-framework side, several open-source toolkits have appeared: Koogu,21 Ketos,22 and ANIMAL-SPOT23 for bioacoustic deep learning; DeepAcoustics for object-detection-based whistle identification24; and the PAMGuard platform for real-time detection, classification, and localization.25 Inter-annotator variability in marine PAM contexts has been shown to depend on both vocalization type and signal-to-noise ratio,26 introducing a source of performance variance that is rarely quantified in comparative studies. The proposed pipeline addresses this gap by formalizing an annotation protocol with explicit decision criteria for each label class (Section 2.2) with a unified, formally defined framework that makes every preprocessing and evaluation decision explicit, reproducible, and systematically configurable.
Objectives and contributions
1.3
Objective 1 — A formally described, six-stage methodological framework. The proposed pipeline formalizes each stage of the CNN-based detection workflow — from annotation protocols and segmentation strategies through spectrogram parameterization and preprocessing to cross-validation design and performance reporting — making every decision point explicit and reproducible. Each stage generalizes to other taxa and vocalization types through YAML-based configuration.
Objective 2 — A full open-source implementation: ai-pam-pipeline. The toolkit provides a command-line interface whose entire parameter space is defined by a single YAML experiment file, ensuring that any reported result can be exactly regenerated. A plug-in architecture allows the CNN model to be substituted without modifying the training loop. Full technical documentation is provided as Extended Data.
Objective 3 — End-to-end validation on two complementary tasks. The framework is demonstrated on (A) a binary whistle detector for T. truncatus with cross-domain evaluation on an independent open-ocean dataset, and (B) a five-class vocalization classifier on the same training set, exercising the multiclass configuration and the scroll augmentation mechanism. Together, the two experiments show all six pipeline stages operating as an integrated, reproducible workflow.
2. Materials and methods
2.1 Overview and design principles
Implementing deep learning in Passive Acoustic Monitoring (PAM) requires a coordinated sequence of interdependent steps. The process begins with raw data analysis, in which PAM operators screen recordings to extract, identify, and classify individual signals. These signals are then converted into spectrogram images, a step that involves choosing among a wide range of parameters — color scales, image resolution, and STFT settings — each of which can meaningfully affect the outcome. The resulting images may further undergo preprocessing or augmentation to compensate for data scarcity, followed by visual inspection to verify their quality. This stage alone can be time-consuming and prone to undetected inconsistencies in training data composition. Once the image dataset is assembled, model training can proceed.
Although individual processing stages are described to varying degrees in the literature, the complete sequence is less often documented as a single executable and reproducible workflow, even though existing toolkits cover substantial parts of it. Yet undocumented workflows directly undermine reproducibility: CNNs are inherently stochastic and sensitive to parameter choices, so even minor untracked variations can produce divergent outcomes. The framework proposed here addresses this by formalizing the entire workflow into six integrated stages (
Figure 1), all governed by a single configuration file. The sequence spans from raw data annotation (Stage 1) to final performance evaluation (Stage 6), with intermediate steps covering audio segmentation into discrete windows (Stage 2), spectrogram generation (Stage 3), optional image filtering for feature enhancement (Stage 4), and CNN model training (Stage 5).
Figure 1. Six-stage pipeline block diagram of ai-pam-pipeline.
Module names are annotated for each stage. Architecture-level labels use general PAM terminology; inline annotations report the cetacean validation parameters adopted in Section 3 as an example instantiation.
2.2 Stage 1 — Audio acquisition and PAM annotation (
Figure 1)
In PAM applications, data acquisition at monitoring sites is typically carried out using autonomous devices equipped with a hydrophone, which converts acoustic waves into an electrical signal, followed by a preamplifier, an analog-to-digital converter (ADC), and a recording system. Depending on the target species and the type of vocalization of interest (e.g., tonal or pulsed signals), the relevant signal bandwidth may range from a few hertz up to approximately 25 kHz, as is typical for bottlenose dolphin whistles, and extend beyond 160 kHz for high-frequency echolocation clicks. Consequently, the sampling rate must be selected accordingly, in compliance with the Nyquist criterion, typically adopting values of 96, 192, or 384 kS/s. The acquired data are usually stored on integrated memory media, in WAV format or in lossless compressed formats such as FLAC. In long-term monitoring deployments, recordings are typically partitioned into fixed-duration segments (e.g., 1, 5, 30, or 60 minutes). It is considered best practice to adopt a standardized naming convention that includes a timestamp in a convenient format, such as YYYYMMDD_hhmmss. When multiple recording units are deployed, prepending a unique station ID (e.g., HYD1_) and appending the sampling rate (e.g., _192) is highly recommended to ensure traceability. Such structured filenames are fundamental for enabling downstream automated processing and metadata extraction. The acquired data subsequently undergo extensive processing aimed at detecting vocalizations, which may be performed through visual and acoustic inspection by trained personnel and/or with the support of detection software such as PAMGuard.25 This annotation process, in which segments of recordings are labeled by associating specific tags with particular vocalizations, remains — as previously highlighted — non-standardized.26 In the case of Tursiops truncatus, there is not even a universally accepted classification of the different vocalization types. The proposed approach consists of leveraging the tagging functionalities available in the open-source software Audacity (or similar tools). The user can graphically select portions of the audio signal containing specific sounds and assign them a label. Labels may overlap in the case of superimposed sounds. The final output is a tab-separated values (TSV) file containing three main fields:
,
, and
. The first two represent the temporal offsets, expressed in seconds as floating-point values, relative to the beginning of the recording. Audacity supports a variety of audio formats (e.g., WAV, FLAC), while labels are exported as plain text (TXT) files. To facilitate automated pairing within the pipeline, it is strongly recommended that label files maintain the same stem as the source audio file (e.g., HYD1_20200101_100025_192.txt). In this work, a simplified tagging scheme was adopted for the general detection of whistles produced by Tursiops truncatus:
W (whistle): tonal, continuous, frequency-modulated signal with a smooth contour in the spectrogram.
MW (multiple whistle): segments containing overlapping whistles.
ECT (click train): regular series of broadband pulses, typically appearing as vertical striations.27
BPS (burst pulse sound): rapid bursts of irregular pulses.27
FB (feeding buzzes): a rapid sequence of broadband pulses with progressively decreasing inter-pulse intervals, typically reaching <5 ms at peak rate. On the spectrogram they appear as a dense band of vertical striations, with total event duration generally between 0.5 and 3 seconds.27 In the Oltremare recordings, feeding buzzes are associated with foraging behavior, though the annotation criterion is the acoustic structure rather than the behavioral context.
NOISE: vessel interference, flow noise, and non-target biological sounds.
The same approach can be extended by adopting a more articulated labeling scheme or by applying the same principles to other species. Although a standardized classification framework would be desirable, in its absence, it is essential to maintain a consistent labeling criterion across all stages of the processing pipeline. The establishment of a universally shared annotation scheme would significantly increase methodological consistency and facilitate inter-calibration across diverse research efforts. For example, a classification approach similar to the FAO alpha-3 codes, employed for fish species classification, could be adopted.
The temporal boundaries of each annotation follow class-specific criteria. For whistles (W), a new annotation begins when the fundamental frequency contour shows a gap exceeding 50 ms or a discontinuity in frequency trajectory. For echolocation click trains (ECT), the annotation spans from the first to the last click in a continuous train, with a maximum permitted inter-click gap of 200 ms. For burst-pulse sounds (BPS), boundaries correspond to the onset and offset of the visible pulse train. These criteria follow the species-specific guidelines established for the Oltremare corpus (Screpanti et al.28) and are documented in the Reference Manual (Extended Data, S4.2).
An advantage of maintaining paired audio and label files is the ability to extract specific tagged and classified fragments automatically, ensuring consistency and replicability across large datasets. It could also facilitate the systematic creation of specialized databases, where vocalizations are indexed by type and optionally integrated with metadata (date, time, and environmental parameters).
2.3 Stage 2 — Audio handling, segmentation and windowing
Inference models based on neural networks require input data with strictly defined dimensions. Whether the architecture operates directly on the raw acoustic signal or on spectrogram representations, a fixed temporal observation window must be established. A wider window can capture finer acoustic details and potentially improve detection resolution, though this comes at the cost of reduced system responsiveness in real-time applications and greater computational overhead at inference time. Narrower windows, conversely, allow faster response and lower resource consumption, but may fail to capture the contour features that define a vocalization, with a corresponding effect on model metrics. In the work of Scaradozzi et al., which used the pipeline described here for the detection of T. truncatus whistles, a 0.8-second frame proved effective for capturing the bioacoustic modulation of interest.29 For continuous temporal analysis, overlapping consecutive windows is standard practice, preventing signals from being split across boundaries and subsequently misclassified. That study employed a 50% overlap, generating a new observation window every 0.4 seconds. These values are adopted throughout the present work as the cetacean validation parameters; other taxa or vocalization types may require different window lengths, all of which are configurable in the YAML experiment file (e.g., pipeline.stage_2.as_len, pipeline.stage_2.as_overlap). Species with longer stereotyped vocalizations — such as the song units of mysticetes or the multi-second call sequences of some odontocetes — are accommodated by increasing as_len accordingly, with no further modification to the pipeline. The choice of window size and overlap also has direct implications for real-time deployments, particularly on edge devices with constrained computational resources.30 When a tagged signal is shorter than the defined window, ai-pam-pipeline automatically centers the acoustic event within the frame to preserve spatial consistency for the CNN.
Isolating signals by class and rigorously excluding target-class events from the background class are essential to prevent inconsistency during training. Without this, the CNN risks learning to distinguish not between pure background and the target bioacoustic feature, but between categories that are acoustically contaminated by one another. The problem is especially acute in multi-class settings, where training samples labeled for one vocalization type must not contain fragments of another: such contamination introduces ambiguity directly into the feature space. To address this, the library implements an automated padding mechanism that expands exclusion zones around each annotation boundary as a function of the sliding window step; the technical details of this logic are provided in the Extended Data.
In PAM recordings where multiple vocalization types co-occur simultaneously, this exclusion and erosion logic becomes the primary means of obtaining acoustically unambiguous training samples. The mechanism is fully general: any annotation class whose temporal extent overlaps with the target class generates an exclusion zone, and the morphologically eroded residual of the target span defines the sampling domain for that class. The design applies identically regardless of species or vocalization type, provided the annotations follow the label-file convention described in Section 2.2.
Figure 2 illustrates this process for an exemplary extraction from a multi-label recording. The target species is Tursiops truncatus and the class of interest is the echolocation click train (ECT), but the same logic applies to any acoustic class in any annotated PAM dataset. Panel A shows the raw Audacity label file, in which NOISE, BPS, W, and ECT annotations are present and temporally overlapping. Panel B traces the construction of the final usable ECT signal: the raw ECT span (C3_raw) is first reduced by the exclusion zones generated from the overlapping NOISE, BPS, and W annotations (C3_post_excl), after which a one-dimensional binary erosion (C3_eroded) is applied using a structuring element of length equal to the minimum duration expressed in frames. The operation acts on the binary timeline representation, where active frames denote the presence of the target class after exclusion filtering. The effect is inherently contractive: by shrinking each contiguous active segment at both boundaries, it eliminates residual fragments that contain insufficient signal. Panel C shows representative spectrogram tiles extracted from the eroded span, each containing an unambiguous click train free of contamination from adjacent vocalizations. The exclusion zone width is parametrically controlled by the analysis window length and step size, as detailed in the Extended Data.
Figure 2. Multi-step segmentation and exclusion logic for class-specific sample extraction from a dense multi-label acoustic recording.
(A) Raw Audacity label file: four annotation classes (NOISE, BPS, W, ECT) are present and temporally overlapping in a
s segment. (B) Construction of the usable ECT (C3) signal: the raw ECT span (C3_raw) is reduced by exclusion zones derived from all overlapping non-target annotations (yellow), yielding the net usable signal (C3_post_excl, blue) and its morphologically eroded version (C3_eroded, green), which constitutes the final sampling domain for spectrogram window extraction. (C) Representative
spectrogram tiles extracted from the eroded span, showing uncontaminated echolocation click trains. Although the example is drawn from a T. truncatus recording, the exclusion and erosion procedure is identical for any class in any annotated PAM dataset.
An additional mechanism for expanding the training set is scroll augmentation: for a given annotated event, the analysis window is slid from the position where only the minimum required vocalization duration appears at the right edge, generating multiple time-shifted copies of the same event. This is the primary data augmentation strategy implemented in ai-pam-pipeline, activated via the YAML parameter pipeline.stage_2.augmentation_scroll. Image processing filters (Section 2.5) can also serve as a basis for augmentation: applying one or more transformations to a clean spectrogram — for instance, adding synthetic pink noise at varying intensity levels to a whistle image — produces training samples spanning a range of apparent signal-to-noise ratios. This approach is fully supported within the framework: running Stages 2–4 with different filter configurations and merging the resulting spectrogram directories before training is sufficient to compose a multi-condition dataset without any modification to the pipeline. Practical guidance for windowing parameter selection is provided in the Reference Manual (Extended Data, S4.3–S4.5).
2.4 Stage 3 — Spectrogram generation
As summarized in
Table 1, the most widely adopted methodological approach involves the use of Convolutional Neural Networks (CNNs) that process spectrograms as input. Spectrogram images are generated via the discrete STFT, defined as:
where
represents the window function,
is the discrete time index, and
is the hop size. In this framework, a Hanning window is applied to minimize spectral leakage. The hop size
is dynamically calculated based on the chosen FFT length (
) and the overlap percentage (
), following the relation
To align the representation with the logarithmic perception of sound intensity and to compress the dynamic range for the CNN, the magnitude of the STFT is converted to a decibel (dB) scale:
The spectrogram image generation process must allow for the systematic and uniform adjustment of core acoustic parameters, such as the FFT size (
) and the Hanning window overlap. Additionally, it must support customizable visualization settings, including output image dimensions (width and height) and the selection between grayscale representations or specific predefined colormaps. In the experiments reported in Section 3, the grayscale mode is used throughout with an image size of
pixels. This differs from the
format adopted in the reference architecture,29 reflecting an updated preprocessing choice evaluated as part of the present framework.
While band-pass filtering in the time domain is a common practice, the proposed library performs frequency selection directly on the generated spectrogram. This spectrogram cropping restricts the analysis to a specific frequency range
, effectively discarding components outside the bandwidth of interest. In the case of T. truncatus whistle detection, a frequency crop in the range of 3–25 kHz is generally employed, as determined by the species’ whistle bandwidth. This range encompasses the most discriminative frequency-modulated contours of the species while optimizing the input dimensionality for the CNN. The image dimensions (width_px, height_px) are likewise decoupled from the window length: a longer analysis window produces a wider raw spectrogram that is resampled to the same fixed pixel grid, preserving CNN input consistency regardless of species. By restricting the vertical axis of the spectrogram image to these limits, the pipeline ensures that the model focuses on the biologically relevant signal features, improving both classification accuracy and computational efficiency during inference. The heterogeneity of spectrogram formats across the literature (
Table 1) underscores the value of recording every STFT setting explicitly in the configuration file, so that the representation used in any given study can be reproduced without ambiguity. The adoption of a single-channel grayscale representation provides distinct computational advantages, particularly for deployment in resource-constrained environments. By reducing the input dimensionality from the three channels required by standard RGB images to a single data stream, the model minimizes memory footprint and processing overhead. Criteria for selecting FFT parameters for new datasets and species are documented in the Reference Manual (Extended Data, S5.1).
2.5 Stage 4 — Spectrogram image preprocessing
While raw spectrograms provide a fundamental time-frequency representation, they often suffer from non-uniform background noise, low contrast, and interference from anthropogenic sources. Preprocessing the spectrogram can substantially improve the signal-to-noise ratio of target vocalizations — particularly the frequency-modulated contours of dolphin whistles — making it easier for the model to extract the relevant features and learn to recognize them reliably.31 This step is optional in the proposed pipeline, but if it is used, it must be applied with strict consistency: the same transformations that are applied to the training data must be applied identically to every input processed by the inferential model during detection and classification.32 In real-time deployments, the computational cost of these filters also deserves careful consideration, as it directly affects system responsiveness. In PAM applications, filtering is commonly applied to the raw acoustic signal14 to attenuate background noise or bring out vocalization features that would otherwise be difficult to detect. Conversely, applying filters to the spectrogram image does not involve the original signal, but rather operates directly on the time-frequency representation generated via STFT before it is fed into the neural network. While this image-based approach is widely established in computer vision — for instance, in the enhancement of underwater photography33 — has only recentely begun to be explored in cetacean vocalization detection.
The toolkit supports a set of standard image-domain filters from OpenCV — including morphological operators (Top-Hat), edge detectors (Laplacian, Canny, Sobel, Scharr), and adaptive contrast enhancement (CLAHE) — that can be applied individually or in combination to the raw spectrogram before CNN input. A global polarity inversion is also available. The complete filter specifications are documented in the Extended Data.
Beyond image filtering, Stage 4 includes a normalization step applicable to each spectrogram image prior to CNN input. Per-image normalization rescales the pixel intensity values to the range
independently for each sample, compensating for global amplitude variability across recording sessions, hydrophone sensitivities, and environmental conditions. This operation ensures that the CNN receives input distributions of consistent dynamic range, regardless of the absolute signal level in the original recording, and is a prerequisite for stable gradient-based optimization. In all experiments reported in Section 3, per-image normalization is enabled via the YAML parameter pipeline.stage_4.image_normalization: true and constitutes the only active Stage 4 operation; image-domain filters are not applied in this work.
The toolkit introduced in this study provides a dedicated script, ai_pam_preprocessing.py, designed to execute the operations detailed in Stages 2 through 4 as a unified, streamlined workflow. This script directly interfaces with raw audio recordings and their associated label files to generate spectrograms, optionally filtered, according to the specific experimental parameters encoded in the configuration file described in the following section.
2.6 Stage 5 — CNN training with k-fold cross-validation
As established in Section 1.2, CNNs operating on spectrogram images are a widely used approach for automated vocalization detection in PAM. The proposed framework is implemented for CNNs but is not restricted to them: the plug-in architecture convention — any Python file exposing a build_model(input_shape, n_classes, kernel_size) function — allows recurrent (RNN, LSTM), hybrid (CRNN), or transformer-based (ViT) architectures to be substituted without modifying the training loop. For models requiring sequential inputs, the toolkit produces time-series spectrogram extracts by modulating the frame overlap, as described in Section 2.3.
The present study utilizes a baseline CNN architecture,29 specifically engineered for efficiency and deployment on resource-constrained hardware,30 implemented using the TensorFlow framework.32 The network consists of three convolutional blocks with an increasing number of filters (32, 64, and 128, respectively); each block incorporates a Rectified Linear Unit (ReLU) activation and a
max-pooling layer. The architecture concludes with a fully connected layer of 128 neurons and a sigmoid activation function for the binary output layer. For multiclass tasks, the sigmoid output is replaced by a softmax activation over N classes, and the loss function switches from binary to categorical cross-entropy; the model file is passed via the YAML key pipeline.stage_5.model.architecture without other changes to the loop.
The model takes as input single-channel spectrogram images, with 224×224 pixel dimensions consistent with the spectrogram generation parameters described in Section 2.4. Training uses the Adam optimizer with an initial learning rate of 0.0001. The loss function is binary cross-entropy for the binary classification task and categorical cross-entropy in the multiclass setting, reflecting the sigmoid and softmax output activations respectively.
The training process uses a batch size of 32 and a maximum of 100 epochs. A linear learning-rate warmup is optionally applied during the initial epochs to improve optimization stability. Model performance is evaluated using accuracy during training and complemented by additional metrics (e.g., macro F1-score) computed on the held-out data. The training process incorporates an early stopping mechanism with a patience of 10 epochs and a 5-epoch warmup period— designed to prevent premature termination during initial loss plateaus—while ensuring the restoration of the best-performing weights at the end of the session.
To ensure reproducibility, all random processes are controlled via fixed seeds, and model initialization is re-seeded at each fold.
To conclude the model preparation phase, the proposed toolkit provides a dedicated script, ai_pam_train.py, which automates the management of k-fold cross-validation, individual model training, and in-domain validation, ultimately generating the performance metrics discussed in Section 2.7 and managing the training directives (epochs, early stopping, batch size, etc.). Training hyperparameter recommendations are provided in the Reference Manual (Extended Data, S7.4).
2.7 Stage 6 — Model evaluation and performance measurement
2.7.1 In-domain cross-validation
Once the spectrogram image dataset has been assembled, the framework evaluates model performance through stratified k-fold cross-validation. The full dataset is partitioned into
non-overlapping subsets (folds) of approximately equal size. In each of the
iterations, one fold serves as the held-out test set while the remaining
folds are used for training, including an in-training validation split that governs early stopping. This procedure is repeated until every fold has served once as the test set, yielding
independent performance estimates. In the present study,
and partitioning is performed by random assignment with a fixed seed (seed
), ensuring full reproducibility.
To prevent data leakage from temporally overlapping windows, the cross-validation folds are constructed using StratifiedGroupKFold (scikit-learn), with the source audio filename as the grouping key (group_by: ‘stem’ in the YAML configuration). This ensures that all windows derived from the same recording — including those generated by the 50% temporal overlap or by scroll augmentation — remain within the same fold, providing a conservative estimate of in-domain performance. A diagnostic log reports the window-to-group ratio per class and verifies that no group spans multiple folds (see Extended Data, S7.3.1).
For binary classification, the predicted class is determined by applying a fixed threshold
to the sigmoid output: a sample is assigned to the positive class (vocalization present) if the network output exceeds
, and to the negative class otherwise. In the present study the decision threshold is fixed at θ = 0.5 for the primary evaluation. The sensitivity of cross-domain metrics to threshold choice is examined in Section 3.3.3.
2.7.2 Binary classification metrics
Performance is summarized by five complementary metrics computed per fold and then averaged across folds:
• Accuracy
• Precision
• Recall
• F1-score
• False Discovery Rate
Recall and F1-score are the primary metrics of interest in PAM detection tasks, however which metric should be prioritized depends on the deployment context: for surveys and density estimation, missed detections are usually the more costly error, whereas for the deterrence application targeted here false alarms carry a direct operational cost (Sections 3.2 and 4.2).
2.7.3 Multiclass evaluation metrics
For classification tasks involving
classes, the sigmoid output layer is replaced by a softmax activation, and a sample is assigned to the class with the highest predicted probability. Performance is assessed through per-class and aggregate metrics.
Per-class Precision, Recall, and F1. Each metric is computed independently for each class
:
where
,
, and
are the true positives, false positives, and false negatives for class
, respectively. Per-class recall is particularly diagnostic: in the presence of class imbalance, a model may achieve high overall accuracy while suppressing minority classes entirely (
), a failure mode that is invisible to aggregate metrics alone.
Macro-averaged F1. The unweighted mean F1 across all
classes:
Macro F1 is sensitive to minority-class performance and constitutes the primary aggregate metric when all classes are of equal operational importance, regardless of their support in the training set.
Weighted-averaged F1. The support-weighted mean:
where
is the number of true instances of class
. Weighted F1 reflects aggregate performance on the training distribution and may be misleadingly high when the dominant class performs well while minority classes are suppressed; it is reported for reference alongside Macro F1.
Confusion matrix. The full
matrix
, where
is the number of instances of true class
predicted as class
. Normalized by true class (row-normalized), the confusion matrix identifies systematic confusions between acoustically similar classes and distinguishes genuine misclassification from class suppression due to imbalance.
The practical effect of unbalanced training data and annotation incompleteness on these metrics is illustrated in Section 3.4, where a known-imbalanced multiclass configuration on the Oltremare dataset produces the metric patterns described above.
2.7.4 Cross-validation design for multiclass tasks
For multiclass tasks with unequal class frequencies, stratified k-fold cross-validation is recommended to ensure each fold contains a representative distribution of all classes. For the binary balanced design in Experiment A (Section 3.2), standard and stratified k-fold are equivalent. The stratified design is adopted in Experiment B (Section 3.4).
2.7.5 Cross-dataset generalization
In-domain cross-validation provides an estimate of performance on held-out data drawn from the same recording session and acoustic environment as the training set. This upper bound is an optimistic predictor of deployment performance, since real-world PAM scenarios routinely involve equipment, sites, and species that differ from the training conditions. The framework therefore separates in-domain validation from cross-dataset testing as two distinct evaluation stages. Cross-dataset testing is carried out using ai_pam_crosstest.py, which loads a set of trained fold models and evaluates them on an entirely independent spectrogram dataset.
The evaluation was conducted using the all mode: performance metrics are computed independently for each of the
fold models, providing a granular distribution of results.
2.8 Iterative use of the framework
Although the framework is organized into six formal stages, its execution is intentionally iterative rather than strictly linear. When validation performance is unsatisfactory, the experiment can be reconfigured by modifying the YAML file and re-running the pipeline from Stage 2 onward, or, when needed, from any later stage downstream of the unchanged source recordings. In practice, this allows segmentation parameters, spectrogram settings, preprocessing filters, sampling policies, or training hyperparameters to be adjusted without revisiting the original recordings or repeating the manual labeling process. Since annotation is one of the most demanding and time-consuming steps in PAM workflows, preserving the labeled source material while enabling controlled reprocessing is a major operational advantage.
3. Use cases
3.1 Datasets
3.1.1 Oltremare dataset (in-domain)
Acoustic recordings were collected at the Oltremare marine park (Riccione, Italy) in November 2021, as described in Screpanti et al.28 Seven free-swimming Tursiops truncatus were recorded using a UREC U384K autonomous recorder (Dodotronic/Nauta) equipped with a SQ26–05 hydrophone at a sampling rate of 192 kHz. Annotation was performed by a trained PAM operator following the labeling protocol described in Section 2.2, yielding 3,214 whistle segments (label W), 233 multiple whistle segments (label MW) and 2,545 non-whistle segments (labels NOISE, ECT, BPS, FB). Each annotated segment yields multiple spectrogram windows depending on its duration and the configured overlap. Since the primary focus of the labeling performed on the dataset was the detection of whistles, the other vocalizations (ECT, BPS, FB, and NOISE) were only labeled to a limited extent. In particular, a very small number of feeding buzzes were labeled (only 308 segments). The complete annotated dataset is publicly available.34
3.1.2 DCLDE 2022 cross-test dataset
For cross-domain evaluation of experiment A, the DCLDE Oahu dataset was used, which concerns acoustic data collected through a multi-channel towed array hydrophone during the Hawaiian Islands Cetacean and Ecosystem Assessment Survey (HICEAS) in 2017.35 A single recording containing vocalizations of T. truncatus (Acoustic detection number 46) was identified in the dataset, dated July 15, 2017, from 11:09 PM to 11:24 PM (UTC). A specific track (n. 3) was extracted for testing purposes and annotated independently following the same protocol as the Oltremare dataset, yielding 458 whistle segments. This process resulted in 715 whistle spectrograms (positive class) and 1,054 non-whistle spectrograms (negative class). From this pool, 500 spectrograms were randomly sampled for each class— utilizing a fixed seed of 42—to establish a balanced test set for the cross-domain performance evaluation. No DCLDE material was used during training; the dataset is used exclusively for cross-domain evaluation.
3.2 Validation
Two experiments validate the framework on the Oltremare dataset. Both share the same baseline CNN architecture and training protocol; they differ in the number of classes, the spectrogram frequency range, and the augmentation strategy, all of which are controlled through the YAML configuration file without modifying any code.
Experiment A demonstrates the binary whistle detector through all six pipeline stages, including cross-domain evaluation. The positive class (C1) groups whistle and multiple-whistle annotations (W, MW); the negative class (C0) groups all co-occurring non-whistle vocalizations — background noise, feeding buzzes, burst-pulse sounds, and echolocation click trains (NOISE, FB, BPS, ECT). Defining C0 from the full range of non-target vocalizations, rather than from background noise alone, trains the detector to reject the acoustic events that co-occur with whistles in real recordings, which reduces the false-positive rate under operational conditions. In this operational context, a false positive carries a direct intervention cost, so minimizing the false discovery rate on unseen data is the primary design objective; recall is a secondary metric that quantifies detector sensitivity. Audio was segmented into 0.8 s windows with 50% overlap (Stage 2). Each window was rendered as a 224 × 224 grayscale spectrogram over 3–25 kHz with a 512-sample FFT and 50% STFT overlap (Stage 3), followed by per-image intensity normalization (Stage 4); no image-domain filters were applied. The model was trained under ten-fold stratified group cross-validation, balanced at 4,700 samples per class with seed 42 (Stage 5). Stage 6 evaluated each of the ten fold models independently on the DCLDE 2022 cross-domain test set.
Experiment B demonstrates the multiclass configuration of the same framework. Five mutually exclusive classes were defined from the Oltremare annotations: C0 (NOISE), C1 (W/MW), C2 (BPS), C3 (ECT), and C4 (FB). The frequency crop was widened to 3–96 kHz to capture the broadband structure of impulsive vocalizations. Because the feeding-buzz class (C4) had substantially fewer annotations than the others (~600 segments, from 308 single annotations), scroll augmentation was enabled for C4 alone, translating the analysis window in 0.1 s steps to generate multiple time-shifted copies of each event (augmentation_scroll: true, scroll_step: 0.1). The resulting dataset totaled C0: 8,154; C1: 5,834; C2: 1,792; C3: 2,406; C4: 3,694 windows. Training used 2,000 randomly sampled images per class (C2: 1,792, limited by annotation density) under ten-fold stratified group cross-validation. Stage 6 was not extended to an external dataset; performance is reported for in-domain held-out data only, as the purpose of Experiment B is to demonstrate framework scalability rather than to characterize cross-domain behavior.
Table 2 lists the YAML key parameters for experiment A.
Table 2. Reproducibility reference for Experiment A: YAML parameter paths and values used in all training runs.
| YAML path | Value (Exp. A) | Description |
|---|
| pipeline.stage_2.as_len | 0.8 | Analysis window length (s) |
| pipeline.stage_2.as_overlap | 0.5 | Window overlap fraction |
| pipeline.stage_3.fft_len | 512 | FFT window size |
| pipeline.stage_3.fft_overlap | 0.5 | STFT overlap |
| pipeline.stage_3.image.width_px | 224 | Spectrogram width (px) |
| pipeline.stage_3.image.height_px | 224 | Spectrogram height (px) |
| pipeline.stage_4.image_normalization | true | Per-image normalization |
| pipeline.stage_5.model.architecture | baseline_cnn.py | CNN model file |
| pipeline.stage_5.model.kernel_size | 3 × 3 | Convolutional kernel size |
| pipeline.stage_5.cross_validation.max_samples per class | 4700 | Samples per class (balanced) |
| pipeline.stage_5.cross_validation.k_folds | 10 | Cross-validation folds |
| pipeline.stage_5.cross_validation.group_by | stem | Fold grouping key (leakage protection) |
| pipeline.stage_5.seed | 42 | Random seed |
| pipeline.stage_6.evaluation.threshold | 0.5 | Binary class. Sigmoid θ |
3.3 Experiment A
3.3.1 In-domain performance (Oltremare)
Performance was assessed at two levels. In-domain ten-fold cross-validation on the Oltremare dataset estimates how the detector generalizes to held-out data from the same recording environment. Cross-domain evaluation on DCLDE 2022 — recordings made with different equipment, in the Pacific Ocean, with free-ranging animals — estimates how the same models behave under the domain shift that operational PAM systems routinely encounter.
Table 3 reports summary statistics for both levels;
Table 4 gives the per-class breakdown.
Table 3. Experiment A performance summary, mean ± SD across ten folds.
In-domain: held-out Oltremare data. Cross-domain: DCLDE 2022, evaluated independently with all ten fold models. Precision, recall and FDR refer to the whistle class; macro F1 averages both classes.
| Metric | In-domain (Oltremare) | Cross-domain (DCLDE 2022) |
|---|
| Accuracy
| 0.960 ± 0.017
| 0.760 ± 0.041
|
| Precision
| 0.969 ± 0.015
| 0.999 ± 0.002
|
| Recall
| 0.950 ± 0.028
| 0.520 ± 0.083
|
| Macro F1
| 0.960 ± 0.017
| 0.744 ± 0.049
|
| FDR
| 0.032 ± 0.015
| 0.001 ± 0.002
|
Table 4. Per-class metrics for Experiment A, aggregated across ten folds.
C0 = non-whistle (NOISE, FB, BPS, ECT); C1 = whistle (W, MW).
| Dataset | Class | Precision | Recall | F1 |
|---|
| Oltremare (in-domain)
| C0 (non-whistle)
| 0.952
| 0.969
| 0.960
|
| C1 (whistle)
| 0.969
| 0.950
| 0.959
|
| DCLDE 2022 (cross-domain)
| C0 (non-whistle)
| 0.678
| 1.000
| 0.807
|
| C1(whistle)
| 0.999
| 0.520
| 0.681
|
3.3.2 Cross-domain performance (DCLDE 2022)
On the DCLDE 2022 dataset, the ten fold models maintained near-perfect precision: the mean was 0.999 (± 0.002) with a false discovery rate of 0.001 (± 0.002). Nine of the ten folds produced no false positives; a single fold produced one false positive out of 500 negative test windows (
Table 3). Recall dropped to 0.520 (± 0.083), meaning that roughly half of the DCLDE whistles were detected, while virtually every window the models flagged as a whistle was a true whistle.
This combination — maximal precision with reduced recall — is the expected signature of domain shift. The models were trained on close-range recordings of seven captive animals in a reverberant basin and tested on free-ranging dolphins recorded at sea with a towed array. The whistles that were not recovered are those whose spectrographic appearance departs most from the training distribution. For an edge-deployed deterrent system, where a false positive triggers an unnecessary intervention, a false discovery rate below 0.1% on entirely unseen data is the operationally critical result. The balanced composition of the test set (500 whistle and 500 non-whistle windows) means that these metrics characterize the detector’s discrimination ability under controlled conditions; they do not directly predict the false-alarm rate in operational deployment, where non-target audio vastly outnumbers target vocalizations.
3.3.3 Threshold sensitivity
To assess whether the cross-domain recall of 0.520 at θ = 0.50 reflects a fundamental model limitation or an effect of the decision threshold, the DCLDE evaluation was repeated at 19 threshold values from 0.05 to 0.95 (
Figure 3). At θ = 0.05, mean recall rose to 0.755 while precision remained above 0.95 (FDR = 0.046), indicating that a substantial fraction of DCLDE whistles produce sigmoid outputs above zero but below the default threshold.
Figure 3. Precision-recall analysis for Experiment A cross-domain evaluation (DCLDE 2022 test set, grouped models, 10 folds).
Left panel: mean precision plotted against mean recall across threshold values from θ = 0.05 to θ = 0.95 (step 0.05). Right panel: mean precision, recall, and F1 as a function of decision threshold θ. The dashed vertical line marks θ = 0.50.
Figure 4. Normalized confusion matrix for Experiment B (five-class Oltremare multiclass demonstration).
Rows represent true classes; columns represent predicted classes. Values are the sum of true-class instances assigned to each predicted class, across the 10 stratified folds. On right: Percentage values, normalized per row. Diagonal percentage entries correspond to per-class recall.
The precision-recall curve (
Figure 3, left panel) shows that precision remains above 0.99 for recall values up to ≈ 0.63 (θ = 0.20), then gradually decreases while staying above 0.95 up to recall ≈ 0.76. Beyond this point, the trade-off steepens.
The choice of operating point is application-dependent. For deterrence systems where each false positive carries an operational cost, the default θ = 0.50 maintains near-perfect precision. For survey-oriented monitoring where detection coverage is prioritized, a lower threshold recovers three quarters of the whistles with fewer than 5% false positives. The optimal threshold for a specific deployment should be calibrated on a held-out dataset representative of the target environment, rather than selected from the evaluation data used here.
3.4 Experiment B — Multiclass vocalization classification
Experiment B evaluates the same framework in a five-class configuration, demonstrating that the pipeline scales from binary detection to multiclass classification through configuration alone. Because the purpose is to show framework scalability, evaluation is limited to in-domain cross-validation; a cross-domain test was not conducted for this experiment.
3.4.1 In-domain performance
The multiclass model reached a macro F1 of 0.835 (± 0.026) across the ten stratified folds (
Table 5). The noise (C0) and whistle (C1) classes retained near-identical performance to the binary setting, with F1 values of 0.930 and 0.930 respectively, confirming that expanding the task to five classes does not degrade whistle detection. The feeding-buzz class (C4) achieved an F1 of 0.808 (± 0.041) despite only ~600 original annotated segments; scroll augmentation generated 3,694 training windows from this material, providing sufficient class representation for stable training.
Table 5. Multiclass applicative demonstration (Oltremare, Experiment B): per-class and aggregate performance metrics, mean ± SD across 10 stratified grouped folds.
Values are in percent.
| Class | Label | Precision | Recall | F1 |
|---|
| C0 | NOISE | 90.2 ± 3.6 | 96.1 ± 2.0 | 93.0 ± 1.5 |
| C1 | W/MW | 95.6 ± 4.7 | 90.7 ± 3.1 | 93.0 ± 3.5 |
| C2 | BPS | 78.6 ± 6.0 | 65.4 ± 8.7 | 70.9 ± 5.9 |
| C3 | ECT | 78.5 ± 5.6 | 81.9 ± 6.0 | 79.9 ± 3.3 |
| C4 | FB | 78.1 ± 4.7 | 84.1 ± 6.7 | 80.8 ± 4.1 |
| Macro (unweighted) | | 84.2 ± 2.3 | 83.6 ± 2.4 | 83.5 ± 2.6 |
Lower performance was observed for Burst Pulse Sounds (C2:
) and the Echolocation Click Trains (C3:
). The confusion matrix (Figure 4) provides further insight into these results, revealing significant inter-class confusion between ECT and BPS. To a lesser extent, a similar pattern of misclassification is observable between BPS and Feeding Buzzes.
With the exception of a few cases involving Feeding Buzzes, other vocalizations were rarely misclassified as Whistles. This underlines the high specificity of the Whistle class, likely due to a more refined labeling process and a more distinctive spectral signature in the training dataset.
4. Discussion
Methodological considerations
4.1
CNN-based detection of cetacean vocalizations involves a large number of configurable parameters36 at every stage of the workflow, from annotation conventions to spectrogram rendering and training hyperparameters. The value of a structured pipeline lies in making each of these choices explicit and reproducible, so that their individual and combined effects on detection performance can be studied systematically.
Two design choices in these experiments illustrate how the framework supports deliberate, documented decisions. In Experiment A, the negative class was assembled from the full set of non-whistle vocalizations rather than from background noise alone, making the training task harder but more representative of operational conditions. In Experiment B, scroll augmentation was applied selectively to the data-scarce feeding-buzz class, generating sufficient training material without artificially inflating the other classes. Both choices are encoded in the configuration file and reproduce exactly; neither required modifying any code. In a field where preprocessing decisions are often left implicit, recording them in a machine-readable form is itself a methodological contribution.
During peer review, it was identified that the original cross-validation partitioning operated at the window level, allowing temporally overlapping windows from the same recording to appear in both training and test partitions. This was corrected by implementing grouped cross-validation (StratifiedGroupKFold), with the source audio filename as the grouping key. A leakage diagnostic showed that 97.3% of C0 windows and 69.0% of C1 windows originated from annotated segments longer than the 0.8 s analysis window, producing thousands of overlapping window pairs. Despite this substantial exposure, the corrected in-domain F1 decreased by less than one percentage point (Exp A: −0.8 pp; Exp B: −0.6 pp), indicating that the CNN learned general spectro-temporal features rather than memorizing specific window content. The inter-fold standard deviation increased by a factor of approximately four (Exp A: from 0.004 to 0.017), reflecting the true variability across recording sessions. Running the same experiment with both grouped and ungrouped folds provides an independent diagnostic of CNN feature extraction quality — a use of the framework’s configurable group_by parameter that extends beyond simple leakage prevention.
The YAML configuration should be understood in the context of existing metadata and framework standards. Training toolkits including koogu,21 Ketos,22 AnimalSpot,23 and the PAMGuard25 deep learning module cover model building and inference. Community efforts to standardize acoustic metadata — Tethys,18 the ANSI/ASA S3/SC1.7 standard,19,21 and PAM data management guidelines20 — operate at the annotation and archive level. The ai-pam-pipeline configuration file complements these by recording the full chain of executable parameters — from annotation conventions through spectrogram rendering to cross-validation grouping and cross-domain evaluation — in a single human-readable file that is automatically archived alongside every output. The framework produces standard Keras model files that can be converted to TensorFlow Lite, ONNX, or other inference formats for deployment on external platforms including PAMGuard’s deep learning module. Version 1.1 also includes a C-spectrogram engine that reproduces the exact spectrogram computation of a target edge device, ensuring training-serving consistency for embedded deployments (see Extended Data, S5.3).
Generalization gap and deployment implications
4.2
The separation between in-domain and cross-domain results is the central methodological point of Experiment A. In-domain cross-validation alone would suggest a near-solved problem, with a macro F1 of 0.960. The cross-domain evaluation (F1 of 0.744) reveals the more realistic figure: the same models recover roughly half the whistles in a genuinely independent recording. A pipeline that stops at in-domain validation would systematically overstate deployment performance; the gap measured here — approximately 22 percentage points of macro F1 — quantifies how large that overstatement can be.
The precision-recall analysis (
Figure 3) reveals that the cross-domain recall at the default threshold understates the model’s detection capacity. At θ = 0.05, recall rises to 0.755 while precision remains above 0.95, indicating that a large fraction of DCLDE whistles activate the detector but produce output probabilities below the default threshold. The operating point should therefore be selected based on the acceptable trade-off between false alarms and missed detections for the target application.
In the dimension that matters most for this detector, the cross-domain result is strong. The mean FDR across folds was below 0.1%. For an edge-deployed deterrent system, where every false positive triggers an unnecessary intervention, this outcome is preferable to a higher-recall detector with non-zero false alarms. The balanced-set metrics reported here characterize the detector’s discrimination ability under controlled conditions. In operational deployment, where non-target audio vastly outnumbers vocalizations, metrics such as false positives per hour of continuous recording would provide a more direct estimate of system performance. Such deployment-oriented evaluation is planned as a next step.
The domain shift between training and test environments is unusually large: controlled marine-park basin versus open-ocean towed array in a different ocean basin. That the detector maintains meaningful discrimination under these conditions is an encouraging result in its own right. Systematic investigation of how individual parameter choices affect cross-domain performance — including an N_fft sweep across multiple values — is enabled by the framework’s design and is the subject of ongoing work. A comparison of the CNN detector against established classical approaches on the same datasets is planned as future work.
The framework is what makes this measurement routine. Because every parameter lives in a single configuration file, a practitioner who wants to reduce the generalization gap can modify the training set composition, the spectrogram band, or the class definitions, and re-run only the affected pipeline stages without revisiting annotation or rewriting code. Cross-domain evaluation becomes a standard step of the workflow rather than an exceptional one.
Limitations
4.3
Several limitations bound the scope of these results. First, both experiments use a single species (T. truncatus) and a single training site (Oltremare); the pipeline is designed to generalize through reconfiguration, but that generalization is asserted rather than demonstrated here. The training recordings come from a semi-enclosed marine-park basin with seven resident animals at close range; this controlled environment provides high signal-to-noise ratio but does not represent the acoustic variability of open-ocean conditions.
Second, the cross-domain evaluation used a single DCLDE recording track. It therefore demonstrates transfer to one independent recording but does not characterize performance variability across multiple sites, recording systems, seasons, or acoustic conditions. Cross-domain performance was measured on a balanced subsample. Deployment-oriented metrics such as false positives per hour were not computed. Third, the multiclass results (Experiment B) are purely in-domain; the ECT/BPS confusion pattern described in Section 3.4.1 may change substantially on recordings with different signal-to-noise ratios or individual variation. Finally, despite fixed seeds, small run-to-run numerical differences can arise from multi-threaded CPU floating-point operations; exact bitwise reproduction may require fixing the hardware thread configuration in addition to the seed.
The framework is demonstrated here using single-channel, spectrogram-based CNN classification of Tursiops truncatus vocalizations. However, its YAML-based configuration supports application to other species and acoustically comparable vocalization types without changes to the pipeline code, by adjusting parameters such as class definitions, analysis-window duration, frequency range, STFT settings, image dimensions, and preprocessing operations. Its model plug-in interface also accommodates compatible architectures, including recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional recurrent neural networks (CRNNs), and vision transformers (ViTs), although these alternatives were not evaluated in the present study. Object-detection architectures, pixel-level segmentation methods, multichannel processing, and click detection operating directly on time-domain signals remain outside the framework’s current scope.
5. Conclusions
This study presented ai-pam-pipeline, an open-source six-stage framework that formalizes the complete workflow from raw audio annotation to cross-domain CNN evaluation, with every parameter defined in a single YAML configuration file. Demonstrated on Tursiops truncatus vocalizations, a binary whistle detector achieved a macro F1 of 0.960 in-domain (ten-fold grouped cross-validation) and, on an independent open-ocean dataset, a precision of 0.999 with a macro F1 of 0.744. Precision-recall analysis showed that recall rises to 0.76 at a lower decision threshold while precision remains above 0.95, indicating that the model’s cross-domain discrimination extends beyond the default operating point. A five-class classifier demonstrated scalability, achieving a macro F1 of 0.835 in-domain. The toolkit is released under the Apache 2.0 license (v1.1.0) and archived on Zenodo, enabling exact reproduction of these results and adaptation to other species through configuration alone. The framework was developed within the LIFE DELFI project for operational edge-deployed deterrent systems, and is offered to the broader PAM community as a foundation for reproducible detector development.
Ethics and consent
Ethical approval and consent were not required for this study. The acoustic recordings used in the experiments were collected as part of the LIFE DELFI project (LIFE19 NAT/IT/000942) and are publicly available. Data acquisition was conducted entirely in passive mode: a hydrophone recorder was positioned unobtrusively within the facility, out of direct view of the animals, and no interaction with, handling of, or intervention upon the recorded individuals took place at any stage. The dolphins were free-swimming in their normal environment throughout the recording sessions. No animal experimentation, invasive procedures, or behavioral manipulation of any kind were performed.
Software Implementation and CLI Usage
The following ten steps reproduce Experiment A (binary whistle detection,
) from a clean system with no prior installation. Estimated time: 15–20 minutes for installation and dataset preparation; training time depends on hardware (GPU recommended).
Step 1 — Download the toolkit.
Download the versioned source archive from Zenodo:
https://doi.org/10.5281/zenodo.19860736
Step 2 — Extract and enter the directory.
unzip ai-pam-pipeline-v1.0.zip cd ai-pam-pipeline
Step 3 — Create a virtual environment.
python -m venv.venv source.venv/bin/activate # Linux/macOS
# .venv\Scripts\activate # Windows
Step 4 — Install dependencies.
pip install -r requirements.txt
Core dependencies include TensorFlow, scipy, Pillow, matplotlib, and soundfile. GPU acceleration is available if a CUDA-compatible device is present and the appropriate TensorFlow GPU build is installed (see Section S1).
Step 5 — Download the Oltremare dataset.
Download the annotated Oltremare dataset from SEANOE:
https://doi.org/10.17882/109081
Step 6 — Place audio files.
Copy all.wav recording files into the data/audio/directory:
ai-pam-pipeline/
└── data/
└── audio/
├── 20211120_102109_192.wav
└── …
Step 7 — Place label files.
Copy the corresponding Audacity label files (.txt) into data/labels/. Each label file must share the same stem as its audio file:
ai-pam-pipeline/
└── data/
└── labels/
├── 20211120_102109_192.txt
└── …
Step 8 — Run preprocessing (Stages 1–4).
python ai_pam_preprocessing.py conf/binary_cnn.yaml
This command reads the configuration file conf/binary_cnn.yaml, processes all paired audio/label files, and exports segmented spectrogram images to outputs/spectrograms/. A summary report is printed to the console on completion.
Step 9 — Run training (Stages 5–6).
python ai_pam_train.py conf/binary_cnn.yaml -v
Step 10 — Inspect results.
Open the training summary report: outputs/report/train/summary_report.md.
This file contains cross-fold mean ± SD for all metrics, per-fold detail, and a fold-level confusion matrix. Individual.keras model files and _metrics.json records are available in outputs/models/for further analysis or cross-domain evaluation.
