Methods
The main goal of ODYSSEUS project is to improve security, effectiveness and resilience of border management systems in European Union by using advanced technologies. In a combination of machine learning, artificial intelligence and multi-sensor technologies, the project aims to create a smooth, smart border control and better decision-making processes.
The ODYSSEUS project has five objectives to achieve this goal:
1.
Border Monitoring: The project will improve real-time surveillance in many border scenarios by utilizing drones, X-ray systems and AI-based analytics.
2.
Identity Verification: ODYSSEUS will promote non-intrusive identity checks through multi-modal biometric verification, which will improve detection and accelerate the border crossing process for authorized travellers.
3.
Risk Assessment and Predictive Analysis: The project will use predictive analytics to assess potential dangers so that border agencies can make informed decisions fast on accurate data.
4.
Interoperability and Data Exchange: To improve border management, ODYSSEUS will focus on interoperability to enable data exchange and to be integrated with current systems across EU Member States.
5.
Sustainable and Ethical AI: A key objective of ODYSSEUS is to promote sustainable and ethical AI that respects data privacy and EU regulations, so technological progress is done responsibly and transparently.6
ODYSSEUS solution combines several AI solutions with a tailored web platform to be used by both border authorities and citizens.
The development of the ODYSSEUS solution followed a step-by-step approach, starting from the needs of border guards and ending with full pilot demonstrations in real environments.
The first phase of the project focused on discussions with border authorities. Together, we analysed how current procedures work, what officers need in their daily activities, and where technology could realistically help. These conversations were translated into specific use cases and operational scenarios that guided the entire technical development.
Once the use cases were clear, the team designed each ODYSSEUS technology: people counting, behavioural authentication, UAV inspection, X-ray analysis, and the main platform, based on the requirements of the upcoming EU AI Act. This meant addressing topics such as transparency, data governance, and human oversight. The goal was to ensure that all components were not only technically strong, but also trustworthy and safe to use.
Each tool was developed and tested individually and then integrated step by step into the ODYSSEUS platform. A modular architecture allowed the team to connect cameras, sensors, AI models, and the operator interface in a flexible way. For EU-level systems that cannot be accessed directly (such as EES/ETIAS), realistic mock-ups were created so that the workflow could still be demonstrated.
Before going to the pilot sites, all components were tested in laboratory settings. This included checking accuracy, response time, performance in different conditions, and the explainability of AI decisions.
The complete ODYSSEUS system was then deployed in three different border environments. Each pilot allowed officers to use the technologies in real scenarios: scanning vehicles with UAVs, analysing X-ray images, verifying digital travel credentials, and monitoring crowds or people flow. These pilots are critical for evaluating the system's technical performance, flexibility, and user experience while keeping privacy, ethics, and being in line with legal compliance. Each pilot was designed to explore how ODYSSEUS’s AI-driven technologies can enhance traditional border checks by automating key tasks thus improving border crossings. For example, in rail scenarios, AI models support seamless identity verification using biometrics and behavioural authentication. In maritime and vehicle-focused scenarios, machine learning helps monitor the surroundings using surveillance data and detects anomalies such as unregistered movement or suspicious patterns.
Ethics and consent statements
ODYSSEUS platform and technologies were developed following an ethical-by-design approach, meaning that key values, including privacy, human dignity, fairness, transparency and accountability, were integrated into all the phases of technology development, from design to pilot deployment phases. Such strategy is in line with the GDPR,7 and AI Act8 requirements. This approach was supported by a structured ethics governance model within the project that included the setup of an External Independent Ethics Board (EIEB), a Security Advisory Board (SAB), a Project-level Ethics Advisor and Data Protection Officer.
Ethics opinions for the research with humans and with artificial intelligence for the ODYSSEUS project were issued by Project partners’ as individual declarations (as organisations that do not have an ethics committee) and also by the EIEB board.
A set of documents based on GDPR and AI Act requirements have been developed to cover the data protection and ethics for the organization and successful completion of the planned trials. Considering the different roles that volunteers participants play in the pilots, informed consents for each of the specific roles and specific pilot has been prepared. For assessing ethical aspects and the respect of ODYSSEUS data protection and ethical principles an assessment questionnaire was prepared, to be distributed to the volunteers-participants and collected after the termination of the pilots. The participants to the pilots signed the writen consent before each pilot iteration. The ethics responsible monitored compliance with ethical and data protection standards during the events and performed the evaluation through questionnaires/debriefing sessions focusing on integrity, privacy, and impact post-event.
Respondents to the ethics assessments considered that pilot activities were non-intrusive, courteous, and they were carried out in a manner that did not threaten comfort and well-being. Their perception of privacy and confidentiality was deemed as heavily secured, as well as strong confidence in data handling in accordance with GDPR requirements was expressed.
No children and minors under 18 years of age took part in the pilots.
Overall, the pilots demonstrate how AI and ML can transform border control into a more efficient, responsive, and user-friendly experience for both travellers and authorities without compromising security or fundamental rights.
Building on this methodology, the ODYSSEUS project brought together all these elements into a coherent technological solution. The insights collected from users, the design choices aligned with EU regulations, and the iterative development cycles all contributed to shaping a platform that can support real border operations. In the following section, we present the ODYSSEUS solution in detail - its main components, how they work together, and the role each technology plays within the overall system.
Results - ODYSSEUS solution
This study presents the ODYSSEUS solution, an AI-based system for improving the efficiency and security of border management processes. The system integrates several key technologies, including digital and virtual passports, AI-powered behavioural verification, UAV-assisted vehicle scanning, and privacy-preserving faceless traveller counting with the ODYSSEUS Platform, allowing the system to be used by both travellers and Border Guards. These technologies work together to optimize identity verification at border crossings, allowing travellers to pass without stopping while maintaining strict security standards. It is tested across various border types, as land and sea, offering a comprehensive approach to managing border crossings and risk assessments.
The main functions of the system can be grouped in:
1. Traveller Enrolment and Pre-Registration
i.
Digital Travel Credentials (DTC) Creation: AI validates traveller identity through facial recognition and liveness detection. ML models ensure secure matching between passport data and real-time selfies.
ii.
Pre-registration: ML algorithms analyse travel data and compare it against historical ones and external risk indicators to perform preliminary risk assessment.
2. Identity Verification
i.
Facial Recognition: Advanced AI models match live facial data against stored Digital Travel Credentials (DTC) or passports, reducing manual verification errors.
ii.
Liveness Detection: AI ensures the authenticity of biometric input by distinguishing real users from spoof attempts.
iii.
1:N Face Matching: AI-powered biometric systems compare traveller faces against large datasets in seconds, enabling swift identity verification even for high-traffic scenarios.
3. Vehicle And Train Scanning
i.
X-Ray Inspection: AI analyses high-resolution X-ray images to detect anomalies, threats, or contraband. ML models identify unusual patterns, reducing reliance on human adjudication.
ii.
UAV-Assisted Vehicle Scanning: AI-driven UAV systems count passengers, detect modifications, and perform license plate recognition. ML algorithms process these inputs to flag potential risks in real-time.
4. Risk Assessment
i.
Threat Detection: AI tools process a combination of biometric data, X-ray scans, and behavioural analysis to identify risks or anomalies.
ii.
Behaviour Analysis: ML models track movement patterns, dwell times, and crowd behaviours, providing real-time insights into suspicious activity.
iii.
Decision Support System (DSS): AI aggregates data from multiple sources, generates actionable insights, and flags high-risk individuals or vehicles for further investigation.
5. Data Security and Sharing
i.
Secure Data Storage: AI monitors data access patterns to identify and prevent unauthorized activity. Encryption and anonymization algorithms ensure compliance with privacy standards.
ii.
Digital Wallet: AI automates secure sharing of travel and health documents, ensuring minimal human intervention and seamless integration with border control systems.
6. Administrative Capabilities
i.
User Management: ML models analyse usage patterns to detect unusual activities among authorized users.
ii.
System Monitoring and Reporting: AI generates insights from operational and statistical data, helping administrators optimize processes and enhance security.
The main AI and ML technologies and the ODYSSEUS platform are explained in the following sections.
Traveller Enrolment And DTC Issuance
The ODYSSEUS platform implements several identity verifications for travellers. Verification of electronic passport (eMRTD - electronic Machine-Readable Travel Document) is included for any traveller, but to use full functionality of the platform the users are encouraged to obtain ODYSSEUS Mobile Wallet App and register to the platform. After registration, the users obtain a digital copy of their passport: an ICAO (International Civil Aviation Organization) Digital Travel Credential that will be used as their travel document and as an authentication token to ODYSSEUS platform self-care portal.
Digital Travel Credential is specified by ICAO organization and is a new type of digital credential that in future will at least partially replace current electronic passports.9 The Credential is a cryptographic token and contains similar data as the electronic passport. The Credential may be issued as two components: Virtual Component (DTC-VC) is purely copy of the original passport data. It is a data file that contains original data groups and a digital signature of the original passport (SOD - Document Security Object). The Virtual Component is a simple file that may be transferred by any digital means and does not have any additional security apart from the original authenticity and integrity protections provided by the digital signature. Physical Component (DTC-PC) is a cryptographic token that contains the original data, but the ICAO specification also defines it must have own key pair protected by a security mechanism. Additionally, the specification defines NFC - Near-field communication mechanism that is like the original ICAO eMRTD (electronic Machine-Readable Travel Document) specification. ICAO defines the token in a generic way, but the most natural way to implement it is a mobile App in a handset with a secure element and NFC capabilities.
During registration to the platform the users need to provide their eMRTD, their selfie is matched with the picture from the eMRTD and verified for liveness. Then the DTC is issued with the obtained data and shared with the platform. The users are then required to schedule their travel in advance. They provide the expected dates, place and registration plate of a car they are going to use.
The identity verification flow at the border is as follows:
• The user’s face is scanned by either BCA (Border and Coastguard Agency) terminal or stationary camera and the ODYSSEUS platform attempts to match it to faces of pre-registered users. If the match is found, then the platform assumes previously shared DTC-VC as the travel document.
• If the match is not successful, then the border control officer requests the user to present his ODYSSEUS Wallet App and reads DTC-PC. This also happens in case the face match is passed with inconclusive result.
• If the DTC-PC is not successful for some reason or if the traveller is assumed substantial risk by the BCA officer or the platform DSS then the BCA will fall back to verification of eMRTD and will follow state of the art identity verification flow.
AI-Powered Continuous Behavioural Authentication
AI-powered continuous behavioural authentication system is used to enhance access control for Digital and Virtual Passports within a mobile wallet application used at border identity verification points. The system collects and processes sensor data such as GPS (Global Positioning System) location, accelerometer, gyroscope, magnetic field, Bluetooth, Wi-Fi traces, and screen touch events to extract behavioural patterns that verify users with measurable confidence levels. The system uses multiple models, each tailored to specific behavioural characteristics, to predict user legitimacy. These predictions are then combined through a fusion model, generating a unified risk score that determines whether the individual accessing the mobile wallet is the legitimate owner. Integration is achieved via an SDK (Software Development Kit) embedded in the mobile app, which securely transmits collected data to a backend for processing and inference. The risk score is accessible at both the app level and the ODYSSEUS platform, supporting its Decision Support System (DSS). To aid border authorities in interpreting these scores, the system incorporates explainable AI techniques. Lastly, the system follows privacy-by-design principles, allowing users to control their data, to grant or withdraw consent, stop processing, or delete it anytime.
One of security threats we identified during analysis of the state of the art of border control processes is an impersonation attack, an attack when illegitimate attacker uses passport issued for someone who is similar to him, either when the passport was stolen or even the legitimate user shares the passport willingly. We can expect this type of attack may be even more frequent when DTC issued from eMRTD is used. The situation when the DTC is issued and the eMRTD is not needed anymore may be exploited by criminal organizations that would build libraries of stolen or even legitimate passports that would allow search for best lookalike match for illegitimate travellers.
In the ODYSSEUS project we plan to mitigate the risk by deploying additional biometric verification that would be linked to the original eMRTD but used by the ODYSSEUS Mobile Wallet App that holds the DTC: Continuous Behavioural Authentication.
We consider that the first time the eMRTD instance is used for the first time user’s interactions will be recorded and next time the same eMRTD instance is used the platform will verify the interactions are similar. In case they differ, the ODYSSEUS platform will not prevent the user from traveling but will mark him as substantial risk and advise the BCA officer to high caution.
Continuous Behavioural Authentication uses Machine Learning techniques to ensure the mobile wallet app is secure and accessible only to its legitimate owner. It combines facial recognition and behavioural patterns to continuously verify the user while they interact with the app.
Behavioural biometrics continuously monitor the user’s actions while they interact with the app. The system collects data from GPS for location, motion sensors like accelerometers and gyroscopes, and touch interactions on the screen. It learns unique patterns, such as how the user types, swipes, or navigates through the app. Typing behaviour is analysed based on keypress timing, while touch interactions measure scrolling and swiping patterns. Movement patterns, including frequently visited locations and typical motion while using the app, are also tracked.
The system uses machine learning models to ensure continuous verification:
• Movement Patterns: Random Forest analyses GPS data to identify routines, such as frequently visited locations and travel speeds.
• Device Usage Patterns: XGBoost evaluates touch interactions (swiping, scrolling) using features like scrolling straightness and touch frequency. Typing dynamics are analysed with sequence-to-embedding models that assess keypress timing. Transformers and one-time learning enable effective training with minimal data (e.g., 5 swipes or 140 characters).
• Relevant Actions: SVM (Support Vector Machines) models analyse app navigation, including screen transitions and interaction patterns.
• Fusion Model: A Random Forest-based fusion model combines outputs from all modalities into a unified risk score.
Vehicles Screening
Non-Obtrusive and Safe X-Ray Technologies for Vehicle Scanning
X-ray screening technology for occupied vehicles provides a means for creating non-intrusive penetrative inspection capabilities for cars and passenger vehicles with little to no impact to the flow of traffic through a check point. High-speed imaging, upwards of 10 KPH, shifts the bottleneck in the inspection process from data capture to image analysis and adjudication. With significant numbers of border crossing points into the European Union, and limited numbers of border security personnel to support screening requirements, particularly at peak travel times, other methods are required to aid in the image adjudication process.
The use of machine learning approaches to support the inspection of X-ray data for vehicles is becoming increasingly more prevalent10,11 as more border security and Customs agencies realize the potential that automated inspection capabilities can have in managing concealed threat identification in high-throughput environments. The approach taken borrows extensively from that developed within the medical industry in support of diagnostic X-ray analysis. AI tools, particularly Convolutional Neural Network (CNN) techniques, can help to identify subtle and complex patterns in image data with high reliability.
In support of the ODYSSEUS ambition, algorithms will be deployed on a multimodal X-ray inspection product to assist in both the detection of illicit materials as well as identification of the number of occupants within the vehicle. The inspection platform comprises two inspection modalities. Firstly, transmission X-ray inspection, where X-rays are detected as they pass through vehicles. Such attenuation images probe the metallic nature of objects within the vehicle. Secondly, backscatter X-ray inspection, utilizing the X-rays scattered backwards to identify predominantly organic materials that reside in front, or on top of, otherwise heavily attenuating metallic objects. Backscatter imaging can penetrate the outer shell of a vehicle and readily show materials secreted within the vehicle frame as well as occupants inside the carriage.
Occupancy detection
The occupancy detection algorithm is based on the popular object detection model YOLOX.12 Benefitting from a single-pass of the network, the computational requirements are reduced such that results can be returned in near real time, supporting the high-throughput Concept of Operations (CONOPs) of the X-ray inspection platform. At the same time, the use of Feature Pyramid Networks supports the detection of a range of feature resolutions from varying object sizes – in this case the variety of people sizes and positions within vehicles.
The YOLOX model was trained on 180 stream-of-commerce images, using solely the top-down backscatter image data. YOLOX-L architecture with pre-trained weights was used with fine-tuning for 300 epochs on the data and tested on 101 occupants within 76 vehicles. Out of the 101 occupants, 98 were correctly identified, and only 1 falsely detected, as presented in
Table 1.
Table 1. Various performance metrics of the object detection model used for occupancy detection during vehicle screening.
| Metric |
Value |
|---|
| Total Occupants | 101 |
| True Positives | 98 |
| False Positives | 1 |
| False Negatives | 3 |
| Accuracy | 96% |
| Precision | 99% |
| Prob. Of Detection | 97% |
| False Alarm Rate | 1% |
Figure 1 below provides an example of the YOLOX bounding box result for the detection of a driver within the vehicle scan.
Figure 1. Top-down backscatter X-ray image demonstrating that the occupant has been detected, shown by the brown box, with an accuracy of 95.7% relative to the ground truth which is shown by the green box.
Targeted Illicit material detection
The ability to detect small objects of an illicit nature (weapons, cigarettes, drugs, alcohol etc.) in an X-ray scan of a vehicle requires an object-detection model that caters for small-scale items in densely packed environments.
RetinaNet13 is one of the best one-stage object detection models that has proven to work well with dense and small-scale objects, using a ‘one shot’ approach for higher speed resolution. It comprises a backbone network for computing a convolutional feature map for object detection, and a second subnet for convolutional bounding box regression.
To overcome the issue of low target item prevalence in X-ray scan data, threat image projection (TIP) approaches are taken using images of target items scanned in isolation and projected into the much larger data set of stream-of-commerce data with no illicit items present. The now healthy, true positive data set allows for a highly effective initial raining stage for object detection algorithms; however, fine tuning is still required at the point of deployment to account for site-specific trade flows and vehicle types.
Figure 2 below shows an example of the detection of a handgun in passenger vehicle.
Figure 2. Top-down transmission X-ray image demonstrating that a firearm has been detected, shown by the red box.
The ground truth box for the firearm is shown for comparison by the green box.
UAV-Assisted Vehicle Scanning Based On Thermal And High-Resolution Cameras
As border security remains a top priority for nations worldwide, many are exploring advanced technologies such as Unmanned Aerial Vehicles (UAVs), along with artificial intelligence (AI) and machine learning (ML) techniques, to safeguard against potential threats. UAVs, commonly known as drones, have emerged as viable alternatives to traditional crewed vehicles for extensive surveillance and monitoring tasks.14,15 Aerial surveillance through drones offers superior mobility and a significantly larger range compared to fixed camera systems.16 Through AI-driven data analysis, UAVs can autonomously interpret vast amounts of information in real time, enabling faster and more accurate threat detection, classification, and response. These versatile aircraft, which can be remotely or autonomously piloted, can carry various payloads, such as cameras, sensors, and communications equipment, enhancing their adaptability for different border security needs.17
The integration of ML techniques into UAVs significantly enhances their capabilities and performance, leading to the development of “smart UAVs” that can operate autonomously and adapt to changing environments.18,19 Building on these advancements, the ODYSSEUS UAV (
Figure 3) has been developed as a specialized drone tailored for border security applications. This fully customized platform is designed to support a variety of hardware components, including an onboard processor, to meet the specific operational requirements of border security. It features a four-arm structure, with each arm capable of hosting two motors. Acting as the onboard computer, the NVIDIA Jetson AGX Xavier (NVIDIA, Santa Clara, CA, USA) efficiently handles the execution of multi-sensor data fusion and AI algorithms. Known for its advanced AI processing capabilities within a compact form factor, the Jetson AGX Xavier is ideally suited for mobile robotic applications.20
Figure 3. ODYSSEUS UAV.
The matrices below provide a comprehensive overview, detailing both the UAV's system requirements (
Table 2) and its technical specifications (
Table 3).
Table 2. UAV system requirements.
| Requirements |
Description |
|---|
| Nominal battery voltage: 22.2 V | 2 × 6S LiPo batteries in parallel configuration. |
| Battery charger | Battery charger that can charge 2 LiPo batteries simultaneously (up to 6S). |
| Radio transmitter for manual piloting | Radio transmitter, compatible with the embedded autopilot, suitable for manual piloting and monitoring of power consumption. |
| Ground station | Necessary for the preparation and the execution of the flight. |
Table 3. UAV technical specifications.
| Requirements |
Description |
|---|
| Max diameter | 1.65 m |
| Weight (no payload) | 9.5 kg |
| Expected flight time | 35-40 min |
| Total motor thrust (100%) | 22.8 Kg |
As demonstrated by its technical specifications, the ODYSSEUS UAV can accommodate multiple sensors tailored to the specific needs of border authorities. This adaptability allows for the integration of high-resolution RGB cameras, thermal cameras, or other specialized sensors, enhancing its versatility across various surveillance and monitoring tasks. In summary, the ODYSSEUS UAV represents a significant advancement in UAV technology for border security, combining a fully customized hardware platform with advanced AI and multi-sensor capabilities to deliver a responsive and autonomous surveillance solution.
High-Resolution Cameras
License Plate Recognition
The license plate recognition system on UAVs applies a multi-stage pipeline to achieve high accuracy and efficiency. First, a vehicle detector based on the YOLOv821 architecture detects vehicles in the frame; the detected vehicles are cropped and passed through another YOLOv8-based detector optimized for license plate localization. Such a two-stage detection process guarantees a robust identification even under tough conditions, such as partial occlusions or various angles.
Upon locating the license plate, the segmented picture is processed using the Google Cloud Vision OCR model,22 which extracts alphanumeric information with reliable accuracy. To maintain temporal consistency and track vehicles throughout consecutive frames, the system employs ByteTrack,23 as a robust multi-object tracking algorithm. This tracker assigns a unique identifier to every detected vehicle, enabling the system to perform long-term tracking of multiple license plates, even in dynamic scenarios such as vehicle movement at border crossing points.
The combination of YOLOv8 detectors for vehicle and license plate detection with Google Cloud Vision OCR and ByteTrack ensures real-time recognition and accurate tracking. This way, the approach offers reassuring support to law enforcement and border security forces by automating the monitoring of vehicle registrations and smoothing out the detection of suspicious or unauthorized activities.
Identifying Car Properties
The system uses a tracker based on YOLOv8 to detect and track vehicles across video frames. Experiments are being done by using even lighter variants, such as YOLOv8s for faster inference.24 Every tracked vehicle maintains a buffer of 20 frames to ensure diverse appearances under varying lighting conditions and angles. These buffered frames are pre-processed through lightweight compression to reduce computational needs.
The pre-processed frames are then analysed by a decision-making AI agent formed on the GPT-4o architecture. This agent returns as a result a structured output highlighting two attributes: the colour and the vehicle type.25 This technique also yields interpretable outputs specifying the criteria for its decision. This explainability makes the model predictions transparent, which is crucial for operational trust in security applications.
The combination of robust tracking, diverse data collection, and a sophisticated GPT-4o-based decision agent will enable precise identification of vehicle attributes. Information integral to cross-referencing with vehicle databases and the identification of anomalies - such as unauthorized modifications or mismatches in reported vehicle details - is obtained. By leveraging this system, border security operations are enhanced with detailed and reliable vehicle profiling.
Thermal Cameras for People Detection
Thermal cameras mounted on UAVs can detect and monitor human occupants in vehicles. As there was no pre-trained model suitable for this application at the beginning, a unique dataset was developed by manually annotating real footage captured from drones. This was achieved using a two-stage process involving dataset creation and model training.
Vehicle Detection: The first dataset consisted of annotated vehicles in thermal imagery. A YOLOv1026 and a YOLOv8 model were both trained on this dataset for robust vehicle detection under thermal conditions.
Human Detection within Vehicles: The second dataset was built by annotating humans visible through open car windows. For this, a dedicated YOLOv8 model was trained to reliably detect human occupants inside the vehicles.27
Given the challenges posed by variations in detections across frames, such as fluctuating numbers of detected people, the system uses a specialized approach for updating the state of each car: Confidence Score Tracking for every detected car and its human occupants, the system maintains the confidence scores returned by the Human Detection Model to ensure the reliability of updates.
Incremental Updates Only: To prevent inconsistencies from temporary occlusions or false negatives, the detected number of occupants is updated only in the case of an increase. This serves as a conservative approach wherein no occupant can be mistakenly missed in the post-processing.
This forms an effective solution for vehicle human-occupancy monitoring using this two-stage detection pipeline with intelligent update criteria. The capability is important in viewing unauthorized transportation of human subjects, especially hidden in secret compartments, and improves situational awareness for border security operations.
SEAMLESS BORDER CROSSING
To allow for the robust and accurate recognition of on-the-move persons, the ODYSSEUS seamless border crossing system relies on a pipeline of several biometric algorithms related to identity recognition. This pipeline encompasses all processes needed to process seamlessly captured images into identity decisions, including the detection of faces, their alignment, the assessment of image quality, the detection of potential attacks, and the templification and matching processes. These are described in detail throughout this section.
Capture, detection, and alignment of faces
The first of such components is the capture system, where continuous streams from high-quality cameras are used to acquire images of each border crosser. Frames from these streams are processed by a face detection algorithm, inspired on generic object detectors28 but trained specifically for human faces. Regarding this specific task, the major problems to overcome are the variations in pose, illumination, expression, and occlusions that may distort or hide information needed by the models to recognize a region of the image as a face. A good face detector should be robust to such noise and missing information and consistently offer accurate face detections, as a human would do.
This is especially relevant in seamless scenarios such as the ODYSSEUS target application. Naturally moving people are more prone to posing variations, occlusions, and changing expressions than individuals who are stationary and actively interacting with a capture device. Great examples of state-of-the-art face detectors that have successfully tackled such problems are RetinaFace29 and MTCNN.30 These have shown competitive performance even in considerably adverse scenarios. Alignment is performed over each detected face, following the proposal in ARCFACE,31 to enhance the posterior templification and matching process.
Face quality assessment
After the capture and detection of faces, it is important to safeguard the accuracy of the system's decisions with thorough quality evaluation. The first step of quality evaluation is the assessment of overall quality features of the face. This can be done by computing individual scores for each of the several captured face images (following the relevant international standards32) such as image sharpness, pose estimation, lighting uniformity, and face occlusions (e.g., glasses or hair). Recent research has also proposed new ways to avoid the strenuous computation of individual quality features by proposing unified quality assessment methodologies. The Open Source Face Image Quality methodology (OFIQ)33 is one of the most promising, using a single residual network MAGFACE34 to offer a single unified score of image and face quality. The authors state their methodology weighs the features defined in the international quality standards by their relative importance to obtain single estimations of overall quality with a direct relation to the expected matching score. Ensuring proper filtering of faces based on face quality allows the system to focus solely on valid faces for matching, thereby reducing the processing load on the pipeline. This improves efficiency in both processing power and time.
This first step of quality assessment is extremely important, since facial features need to be sufficiently detailed and sharp to be used for comparison, later, by the templification and matching components. In fact, quality assessment is not only essential for the live captures taken on-site, but also for the reference face images from identity documents or the pre-enrolment stage. Although it is expected that identity document photographs will adhere to high standards of image quality (namely those set by ICAO32) it is important to ensure it through proper assessment within our system.
Fraud detection
A second step of image quality evaluation relates to potential attacks and fraud attempts. The main one for the live or pre-enrolment captures is presentation attacks: attackers may use face photographs, videos, paper or silicone masks of authorized individuals to impersonate them while crossing a border. In traditional borders, under intense surveillance by border guards, this possibility is negligible. However, as we target new seamless border crossing scenarios, with fewer constraints and less active human surveillance, it is paramount to verify the captured face images to assess if the face is real or a presentation attack. Here, research has focused on improving the current approaches, mostly deep learning-based, towards higher robustness with new and more sophisticated attacks like high-quality and detailed synthetic masks.35,36
In the case of identity document images, it is important to detect potential morphing attacks. Morphing attacks consist of the combination of two face images from different individuals, one of which is an authorized subject. The resulting image retains identity information and relevant visual details from both faces to fool both algorithms and border guards into accepting identity claims from either of the subjects. A passport with a good, morphed image can be successfully used by two different individuals, which poses a serious security threat. To fight it, algorithms based on single image ALPHANET37 and differential methods SYNTHMORPH38 can be used to analyse document photos and ensure they are real and unadulterated face images, before moving on to the proper recognition stage.
Templification and Matching
After quality assessment and attack detection, the system can confidently proceed to the recognition of the detected subjects. Templification entails the computation of a vector of numeric features (template) that represents the face. This process is highly relevant, since the face image, even after alignment and quality assessment, will always carry some intrasubject variability and capture noise that pose a significant threat to the subsequent matching process. The templification process is conducted using a deep learning model (such as ARCFACE31 and ELASTICFACE39) which is trained following mixed objective functions with instance and siamese loss (such as triplet loss40 or ArcFaceLoss31) During training, the model learns to extract templates so that they retain most of the identity information within the input face image, while discarding most of the undesired variability and noise.
The matching process involves the comparison of a pair of templates. One of these templates (probe) is the one extracted from the live image, captured on-site, while the second template (reference) can be the one computed from the passport image or the pre-enrolment process. Templates can be compared using simple distance or similar metrics such as the cosine similarity or the Euclidean distance. On an identity verification scenario (1:1), the result of it will be a representation of how likely it is that both templates' identities match and can be thresholded to obtain a 0/1 (no match/match) decision by defining an operating point. On an identification scenario (1:N), with a gallery of multiple reference images, matching is conducted between the probe and each available reference, and the best score is selected, provided it meets a minimum threshold. It is thus possible to fulfil the intent of accurately recognizing individuals on both scenarios of comparison between live vs. document image (1:1) and live vs. pre-enrolment gallery (1:N).
Gallery management is another crucial optimization, as larger gallery sizes increase the probability of error. By implementing a system that can populate, and segregate galleries based on predefined business rules or the expected behaviour of subjects, the system can reduce both identification time and error rates.
Final remarks on seamless border crossing
As described, this pipeline is composed of several algorithms and components, a majority of which will be essential on most seamless biometric application scenarios. Thus, there are natural concerns over the processing requirements of such a comprehensive pipeline, since a seamless border biometric solution must be able to perform recognition in near real-time, ensuring the natural movement of border crossers, avoiding delays, and ensuring the highest possible throughput rates. Hence, during the development of this part of the ODYSSEUS solution, there was a concerted effort of balancing throughput with precision, by carefully designing and tuning each algorithm to be as lightweight as possible making each algorithm in the pipeline as lightweight as possible while retaining (at least) appropriate levels of accuracy, thus providing a secure and seamless experience to travellers.
Another concern lies with the fairness of the developed algorithms. Bias and fairness in biometric recognition is currently a pressing issue and a trending topic among researchers and industry. It has been reported that biometric algorithms may behave differently depending on demographic aspects of the user, such as their ethnicity, race, gender, or age.41,42 This is attributed to the data used for training the algorithms and can be fought by improving the diversity of the training datasets. As such, the models developed and integrated into the ODYSSEUS system were trained with carefully selected datasets to ensure demographic diversity and representation, while conducting comprehensive assessments to detect and avoid any biases, to ensure equitable performance with all users.
Faceless GDPR Compliant Travellers Counting
Effective management of crowds has become crucial in terms of security. As a result, the need for efficient and privacy-preserving travellers’ counting technology has become more critical, especially at border crossings. The ODYSSEUS project introduces a novel approach, leveraging advanced machine learning (ML) for crowd counting. At the core of this technology is a focus on unauthorized human presence detection in restricted areas, raising alerts for border authorities. Additionally, crowd-related metrics, such as detection and monitoring of unattended or abandoned objects, and real time number estimation of individuals, are provided in travellers used predefined areas, circumstances indicating security risk possibilities. These capabilities of precise crowd management are vital for seamless border crossing.43
Over recent years, the advancement of the Convolutional Neural Networks (CNN) has yielded significant improvement in crowd counting accuracy and robustness.44 We could distinguish the crowd counting techniques in traditional approach and CNN-based approach. On the one hand the traditional methods can be categorized as:
• Detection based: This approach tries to recognize each object of the crowd, directly in the image. Features, such as head or body, are recognized based on feature detection and usually are combined with tracking algorithms to follow individuals as they move throughout the area. Based on the detector, these methods can be divided into monolithic detection and part-based detection. In monolithic detection approaches, images of full body appearances of individuals are used to train a classifier based on features extracted of the full body. Monolithic approaches have low quality results in crowded scenes. In part-based approaches, instead of using full body features to train the classifier, only part of the body is used (such as the head), increasing the quality. The surveys45–47 describe various detection-based approaches.
• Regression based: This approach consists of two main steps: low-level feature extraction and regression modelling. In the first step, a set of features such as edge feature, colours, texture, and gradient features are extracted and are useful for capturing local information, reducing dimensionality, providing robustness, and finally enabling faster image processing. Once these features are extracted, a variety of regression techniques, such as linear regression, piecewise linear regression, ridge regression, Gaussian process regression, and neural network are used to calculate a mapping from low-level feature to crowd counting. In46,48 surveys, the most significant regression approaches are mentioned briefly.
• Density based: This approach, instead of focusing on individual counting, as detection or regression-based approaches do, is trying to directly estimate the density of a scene. With this approach, the challenging task of individual object detection is avoided. Furthermore, the problem is transformed to an estimation of a continuous density function who is integral over any image region gives the object count within that region. More details can be found in.48,49
On the other hand, the last decade, a rapid evolution of the crowd counting accuracy is observed, since modern deep learning models are increasingly used.50,51 CNN-based models are adept at generating density maps, which provide not only the head count but also spatial information about crowd distribution. Recent CNN approaches for crowd counting can be classified by network properties and training strategies,45,46,49:
• Basic CNN: Only basic CNN layers are used.
• Context aware CNN: Both global and local contextual details of an image are combined in CNN framework to reduce the estimation errors.
• Scale aware CNN: Multi-column and multi-resolution like architecture is used to provide robustness to scale.
• Multi-task CNN: Along with crowd-counting and density, estimation combines crowd velocity estimation and foreground-background subtraction.
The approaches based on training methods can be classified into:
• Patch-based inference: The CNNs are trained using patches cropped from the input images. During the prediction phase, a sliding window is run over the test image and predictions are obtained for each window and finally aggregated to obtain total count in the image.
• Whole image-based inference: Methods in this category perform a whole-image based inference. These methods avoid computationally expensive sliding windows.
Concluding,
Figure 4 (source46:) depicts the evolution of crowd counting methods overtime. Traditional methods are highlighted with the green-shaded area, and the modern CNN-based density estimation and crowd-counting models are within the red-shaded background.
Figure 4. Evolution of crowd counting techniques.
DECISION MAKING – MULTI-MODAL FUSION AND DECISION SUPPORT SYSTEM EQUIPPED WITH XAI
The Multi-Modal Fusion and Decision Support System in the ODYSSEUS platform implements sophisticated eXplainable AI (XAI) methodologies, which will guarantee transparency, accountability, and trustworthiness in the decision-making process. Considering the heterogeneous nature of the data and model outputs, the implementation of XAI is customized to handle three different cases, namely:
Tabular Data Explanations
The one relevant to tabular data predictions, for instance, deciding whether a traveller is legitimate or not, will be LIME52 and SHAP.53 These algorithms show which features contributed positively or negatively to each prediction. This interpretability is important for border agents, enabling them to understand which variables have driven an automated decision. This approach builds trust in AI-driven systems. For example, attributes like travel history and behavioural patterns are analysed to highlight their impact on classification outcomes.
Detector-Based Explanations
In applications where object detectors are used, for example, vehicle and human detection, dedicated XAI algorithms were designed. These algorithms stem from the principles of Layer-wise Relevance Propagation (LRP)54 and EigenGradCAM,55 adapted to suit the unique demands of UAV-based applications. These methods offer localized explanations by underlining regions of the input images that most contributed to the obtained detection results. Such a representation allows the operators to validate the detections, gain insight from false positives or negatives, and consequently tune operational protocols.
Reasoning Outputs by Large Language Models
When the outputs come from Large Language Models, such as in the case of car property identification-colour and car type-specialized agents are designed to provide reasoning for the decisions taken. These agents assess the criteria set upon each property, such as patterns in captured lighting and angles across several frames, before generating well-detailed reports that explain the logical sense behind the Large Language Model predictions. The approach would ensure accountability, as it bridges the gap between the LLM outputs and the operational requirement by providing a clear rationale for each result.
By integrating these XAI techniques across various components of the ODYSSEUS platform, the system not only improves decision-making but also boosts operator confidence in using automated systems. This multi-faced explainability fits well with the commitment of the project to ethics and transparency in AI applications dealing with border management and it reinforces the human-in-the-loop element by giving the operators the abilities to interpret, validate, and intervene in decisions with AI inducement, where appropriate.56
ODYSSEUS PLATFORM
ODYSSEUS platform is a technical solution that combines machine learning, artificial intelligence, and multi-sensor capabilities to improve actual border management systems. This scalable and adaptable platform offers a thorough approach to border security, guaranteeing operational effectiveness as well as its compliance with legal and ethical regulations.
The system leverages ML/AI for:
•
Threat Detection: AI analyses images (e.g., from high-speed rail or drive-through systems) to identify risks, anomalies, and passenger counts.
•
UAV-Assisted Scanning: Advanced AI algorithms support UAV-based preliminary scanning for identity verification, license plate recognition, and detecting vehicle modifications.
•
Real-Time Insights: ML models process X-ray data for anomaly detection and passenger estimation, integrating results into the ODYSSEUS risk assessment engine.
AI algorithms are optimized for on-board processors in UAVs, ensuring efficient computation at the edge level. The architecture highlights a robust integration of ML and AI to enhance border security, operational efficiency, and decision-making precision.
The platform comprises three main levels - Edge, Backend, and User Interface—each playing a critical role in integrating ML and AI capabilities for efficient border management. There are also specialized components, which even placed on one of the three levels, are emphasized here for their specific role. The levels are:
1.
Edge Level (Data Acquisition Layer). This layer focuses on data collection from various sources:
•
IoT Sensors and Devices: Cameras (CCTV, UAV, thermal, and RGB), X-ray systems, digital passport readers, and NFC-enabled devices.
•
Mobile Devices: Used for traveller enrolment, DTC (Digital Travel Credential) creation, and pre-registration.
•
External Sources and Databases: Integration with national and EU databases for validation and risk assessment.
•
UAV Systems: For advanced vehicle monitoring, including passenger counting, license plate recognition, and anomaly detection.
2.
Platform Level (Data Management Layer). This is the core of the solution, where data is processed, stored, and analysed:
•
AI and ML Tools: For facial recognition, behaviour analysis, crowd monitoring, threat detection, and risk assessment.
•
Decision Support System (DSS): Centralized system offering real-time insights for border authorities.
•
Data Storage and Persistence: Secure and scalable storage for personal and operational data.
•
Audit and Monitoring Mechanisms: To ensure transparency and regulatory compliance.
•
Communication Gateway: For seamless data transfer between edge devices and the cloud.
3.
User Interface Level (Application Layer). This layer facilitates interaction with the system for travellers, border officers, and administrators:
•
Mobile and Web Applications: Allow pre-registration, enrolment, and DTC handling.
•
Border Control Touchpoints: Equipped with NFC readers, biometric verification tools, and display systems for real-time data access.
•
Administrative Portals: For system configuration, reporting, and role-based user management.
The deployment adapts to land, sea, and train border crossing scenarios, with consistent private cloud components supporting tailored local configurations:
•
Edge Devices: Integration of AI-ready hardware like UAV-assisted scanners for anomaly detection and identity verification.
•
Private Cloud Resources: High-performance clusters (management and worker nodes) facilitate large-scale AI computation.
