Objective

{Page 1} ... this paper aims to provide a comprehensive survey on applications, challenges, and opportunities of GAI in unmanned vehicle swarms.

1. Present overview of unmanned vehicles and vehicle swarms, including use cases and existing issues.
2. How Generative AI (GAI) enhance unmanned vehicle swarms.
3. Review on applications and challenges of GAI in unmanned vehicle swarms with insights and discussions.
4. Discuss open issues of GAI in unmanned vehicles swarms + potential research directions.

Introduction

UVs can be:

• Aerial (UAV)
• Ground (UGV)
• Surface (USV) (refers to water)
• Underwater (UUV)

Each vehicle may have its own:

• sensor
• processor
• communication capability

AI and robotics adopted to coordinate behaviour and perform complex tasks:

• autonomous navigation,
• self-organisation,
• failure management,
• etc.

{Page 2} ... generative AI (GAI) has been widely adopted in the literature recently due to its groundbreaking abilities in understanding, capturing, and generating the distribution of complex and high-dimensional data

• Applications of conventional AI (deep learning, deep reinforcement learning, and federated learning in UAV-based networks). (lahmeri2021)
• Applications of machine learning (ML) in operations and communications of UAVs. (kurunathan2023)
• AI-enabled UAV optimization methods in IoT networks, focusing on AI for UAV communications, swarm routing and networking, and collision avoidance. (cheng2023)

Unmanned Vehicle Swarms

{Page 2} ... primarily based on the degree of autonomy, which ranges from fully autonomous systems, capable of independent operation based on pre-programmed protocols and real-time data, to semi-autonomous systems that require some level of human intervention.

{Page 2} ... can also be classified by the hierarchical structure, which includes single-layered or multi-layered configurations

Generative AI

{Page 4} ... unlike discriminative models that focus on classification or prediction, GAI models are adept at interpreting instructions and generating tangible outputs

{Page 5} ... enabling them to not only replicate but also innovate within the learned frameworks

Is what GAI generates considered innovation?

Generative Adversarial Networks (GAN)

• Training two networks (generator and discriminator) simultaneously:
  • generator produces data to mimic reality.
  • discriminator classifies.
• Nash Equilibrium
• Issues:
  • achieving local Nash equilibria
  • mode collapse
  • vanishing gradients
• Solutions:
  • unrolled GAN
  • mini-batch discrimination
  • historical averaging
  • feature matching
  • two timescale update rule
  • self-attention GAN
• Useful for swarms:
  • efficiency in sample generation -> needed for rapid decision-making and computational constraints
  • capacity to learn rich and diverse latent spaces

Variational Autoencoder (VAE)

• Deep latent space generative models.
• Learn distribution of data to generate new, meaningful data with intra-class variations.
• Two interlinked but independently parametrised components (like GAN):
  • encoder: provides estimation of posterior over latent variables to assist decoder update parameters during "expectation maximisation" step of learning.
  • decoder: forms framework that assists encoder to learn meaningful data representations (may include class-labels).

{Page 6} ... training of VAEs involves optimizing the Evidence Lower Bound (ELBO), which balances reconstruction accuracy and the similarity of the latent space distribution with the target distribution

Page {6} ... VAEs are valuable for their ability to increase the variance of a dataset, particularly in domains with limited ideal training samples ... the samples generated by VAEs tend to be of lower quality compared to GANs

• Representation learning: transforming raw data into more advanced training data representation. VAEs automate this by learning mappings from high-dimensional space to meaninful low-dimenstional embeddings.

{Page 6} ... VAEs can mitigate issues like mode collapse, making them a more stable choice for generating training simulations

Generative Diffusion Models

• Unlike GAN and VAE, utilise a two-stage process of forward and reverse diffusion.

{Page 6} ... A diffusion model is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time

• A Markov chain is a series of events where the probability of each event depends only on the state of the previous event.

• Transitions between states in the Markov chain (both forward and reverse process) are governed by trained (parametrised) neural networks.

• Variation inference a method to approximate complex probability distributions that are difficult to compute directly.

• They run for a fixed number of steps, and the output becomes more refined with each step of the reverse diffusion process.

• Denoised Diffusion Probabilistic Models (DDPM) used in DALL-E and Stable Diffusion

• Noise-Conditioned Score Network (NCSN) used in Deepfake

• Stochastic Differential Equations (SDE) applied in models like DiffFlow

{Page 7} ... GDMs can be particularly useful in generating highly detailed and diverse environmental simulations for training UV systems. Unlike GANs that often suffer from mode collapse, or VAEs that sometimes generate low quality images, GDMs can produce samples with a high level of detail and variation. This is crucial for UV swarms to train in realistic and varied conditions. ... the iterative process of GDMs that involves multiple steps of noise addition and removal, results in a slower and less efficient sample generation

Transformer Models (LLMs)

• Similar to VAE with encoder-decoder design.
  • encoder reads the input (like a sentence) and figures out how the different words are related to each other.
  • decoder generates the output (like a translation of the sentence) one word at a time, paying special attention to certain parts of the input that help it make the best translation.
• Transformers are powerful tools for working with UV swarms because they excel at understanding sequences of data (like sensor inputs) and identifying important relationships within that data.
• This helps the UVs make better decisions quickly, since they can consider a wide range of information all at once.
• Additionally, transformers can be used to create realistic simulations for training UV swarms, potentially making them more effective in real-world situations.

Graphical Normalising Flow (GNF)

• Transform a simple probability distribution into a more complex one through sequence of invertible and differentiable mappings.

{Page 7} ... This transformation allows for the evaluation of the density of a sample by reverting it back to the original simple distribution and calculating the product of the density of the inverse-transformed sample and the associated change in volume induced by the sequence of inverse transformations

{Page 8} ... GNF leverages Bayesian networks to identify relationships among time series components and perform density estimation in the low-density regions of a distribution where anomalies typically occur. Comparing to other GAI models, the key advantage of NF models lies in their capability to provide exact likelihood estimation, which offers a precise, stable, and efficient solution for identifying and responding to various anomalies in UV systems

• Uses Bayesian networks to understand how different parts of a time series (data over time) relate to each other.
• It focuses on finding anomalies, which are unusual data points that often exist in areas where there isn't much other data.
• The main advantage of GNF (and similar NF models) is their ability to calculate the exact likelihood of a data point. This means identifying anomalies becomes very reliable and precise.

Solutions to UVS utilising GAI

State Estimation

{Page 8} ... Therefore, the primary objective of state estimation is to deduce the distribution over state variables based on the available time observations

{Page 9} ... The versatility of GAI in state estimation for UV swarms is evident in two aspects: ability of generating missing information through adversarial mechanisms and ability of fusing varied data sources for comprehensive state analysis. These ability enables more accurate states estimation in complex operational scenarios

Environmental Perception

{Page 9} ... typically refers to the ability of the vehicle to perceive and understand its surrounding environment in real-time

{Page 9} ... Besides improving UVs' accuracy by enhancing remote sensing resolution, a more common application of GAI is to generate synthetic datasets, which indicates the challenge of reduced model accuracy caused by insufficient data

• Trajectory GAN (TraGAN) for generating realistic lane change trajectories from highway traffic data
• DeepRoad tests input validation in autonomous driving systems.

{Page 10} ... An interesting aspect of these technologies is their ability to process and interpret complex visual inputs, providing a level of contextual understanding that closely resembles human perception.

Level of Autonomy

{Page 10} ... The level of autonomy represents the ability of an UV can operate independently while solely relying on its onboard sensors, algorithms, and computational resources. In UV swarms, the level of autonomy depends on various factors such as the type and complexity of the task, the ability to plan and execute route

{Page 11} ... GAI's generative capabilities are applied in multiple aspects, from generating new trajectories to refining routing strategies and imitating expert agent routing behaviors in diverse scenarios. These diverse applications demonstrate dynamic and adaptable solutions crucial for UVs to efficiently and independently navigate and operate in complex and changing environments.

Task/Resource Allocation

{Page 12} ... GAI introduces effective approaches that enhance efficiency and adaptability of these systems.

{Page 12} ... In conclusion, GAI substantially advances the field of task and resource allocation in multi-agent UV swarms. From creat- ing vivid simulation environments for the allocation algorithm to explore on, to iteratively adjust the allocation strategy and break a rough intent to detail tasks, GAI demonstrates a strong ability to handle dynamic environments and various challenges.

Network Coverage and Peer-to-Peer communication

{Page 13-14} ... GAI can either generate positioning strategies directly or act as an encoder to enhance traditional algorithms by capturing spatial information. For efficiency, GAI acts as a framework that uses semantic information to reduce data transmission while maintaining communication by guided generation.

{Page 13-14} ... However, while these developments represent a leap forward in managing UV swarms, there remain areas for further exploration. For example, the authors in open the question of integrating other modalities for more efficient communication. This suggests an opportunity for future research to investigate the incorporation of multimodal data processing in UV networks. Such explorations could significantly enhance the adaptability of these technologies to diverse network topology and environmental conditions. Additionally, the potential of GAI to facilitate autonomous decision-making within UV swarm deployments presents a promising avenue for advancing the field. By expanding the scope of GAI applications, researchers can further optimize UVs for a variety of complex real-world scenarios.

Security/Privacy

{Page 14} ... an interesting potential application is to utilize GAI's ability to generate fake data or simulate communication activities to act as a honeypot to mislead potential attackers and reinforce system security

Vehicle Safety and Fault Detection

{Page 15} ... Research in this field aims to enhance the overall reliability and safety of UV operations by developing methodologies and technologies that enable these systems to effectively identify and rectify potential faults before they impact vehicle performance or safety.

{Page 16} ... Thus, integrating GAI into UV swarm fault management strategies offers an advanced method to enhance safety. For example, by analyzing real-time sensor data and understanding the intricacies of swarm dynamics, Transformers are able to make context-aware decisions to identify the safest termination points for compromised UVs accurately

Open Issues and Future Research Directions

Scalability

{Page 17} ... as the number of UVs increases, coordinating movements and communications between them become much more complex due to various factors like congestion, latency, signal interference, and limited communication range.

{Page 17} ... One potential direction is to design distributed GAI architectures based on federated learning for collaborations between different groups of unmanned swarms, instead of relying on a single server/swarm leader.

Adaptive GAI

{Page 17} ... it is essential to develop adaptive GAI approaches to further reduce the system latency when retraining GAI models under new environmental conditions, especially in large-scale unmanned swarms

{Page 17} ... transfer learning aims to transfer knowledge learned in a source environment to facilitate the learning process of similar tasks in new environments while meta-learning aims to learn how to learn to speed up the learning process in new conditions.

• This is something Willian suggested, to train a model against regular operating data from other scenarios and using this pre-trained model as the starting point.

AI-native UV Networks

{Page 17} In large-scale and heterogeneous UV networks, the energy efficiency, latency and security issues are the most critical for coordination locally and globally. In this operational circum- stance, semantic communication (SemCom) with knowledge base (KB) shared among UVs is a viable solution to minimize the transmission overhead

3-D Interference Control

• Large UV networks face lots of communication problems. Their 3D movement patterns cause signal interference, making it hard for them to talk to each other.
• Each robot has limited power and resources, which makes solving this problem even trickier.
• Because the robots are always moving, the interference changes unpredictably. GAI approaches seem to be a good way to manage signal interference and keep robots connected in this complex situation.

Security and Privacy

{Page 17} ... Nevertheless, GAI, and AI in general, are vulnerable to adversarial attacks in which attackers try to disrupt the training process of AI models by poisoning training data, evading trained models, and extracting model information. More dangerously, attackers can also leverage GAI to generate "fake" data that is hard to distinguish.

{Page 17} ... One potential solution is to leverage GAI to recover poisoned training data. Integrating deep reinforcement learning with human feedback can also be useful in defeating adversarial attacks. Finally, federated learning can also be adopted to enhance the privacy of unmanned swarms since data of vehicles (e.g., their sensing data and locations) will not be shared with others.