Introduction

Perturbations take three forms:

1. Natural failures of hardware or software - due to bugs, aging, misconfigurations, resource contentions in shared environments, downtime due to planned upgrades, etc.
2. Maliciously induced failures or security attacks - from outside the system
3. Unexpected inputs - either through errors or unpredictability

Outcome of perturbation that is not handled can be a hard failure (crash or hang) or soft failure (i.e. missing a deadline for a latency-sensitive application).

Autonomous Systems as Application Drivers

Distributed resilience in multi-agent drones for medical deliveries

Problem and Current State - A multiple drone system that is responsible for transporting essential supplies to a population affected by a natural disaster or medical supplies to a population where the surface transportation infrastructure is poor.

Consensus-based algorithms assume that all agents are cooperative, sharing their state values with neighbors and adhering to a common protocol to achieve network objectives. However, the autonomous systems using these algorithms face challenges due to their operation in open and potentially adversarial environments, making them susceptible to a range of disruptions.

While distributed algorithms are generally robust against failures of individual nodes or links due to their decentralized nature, their reliance on local coordination also makes them vulnerable to cyber attacks targeting specific agents. Therefore, ensuring resilience is crucial for the successful application of autonomous multi-agent swarms in critical areas.

Cooperative autonomous rescue with active adversary

Consider application of artificial intelligence (AI) in military contexts, particularly focusing on operational challenges similar to those in civilian rescue operations, such as dealing with dirty and sparse data. It highlights a hypothetical military rescue scenario in an urban environment to underscore the complexities AI systems face, including rapid environmental changes, unstable structures, and the need for situational understanding (SU) in the presence of anti-access and area denial (A2AD) tactics.

These challenges necessitate advanced AI capabilities for data fusion from diverse sensors, autonomous coordination, and navigating through physically compromised structures without reliable communication links. The passage also mentions ongoing research efforts aimed at improving sensor technologies, sensor fusion, robotic perception, and the integration of these technologies into complex systems capable of performing in dynamic and uncertain environments.

Addressing these challenges is crucial not only for military operations but also for enhancing the efficacy of civilian rescue missions in disaster scenarios, thereby reducing risks to human rescuers and shortening rescue timelines.

Resilience by Design

Attacks against building blocks of autonomous systems

Current State

Recent research has shown that adversarial examples, specifically designed to fool machine learning models, pose a genuine threat to real-world systems. Even when an attacker has limited knowledge of a model's inner workings (a black-box scenario), they can successfully craft adversarial examples. Techniques include training a substitute model using synthetic data or approximating the target model's decision-making process.

This highlights the transferability of adversarial examples, where attacks designed for one model often work against others. A significant focus lies in developing physically realizable adversarial examples that target autonomous vehicles. These must remain effective despite variations in depth, angle, and lighting conditions that self-driving cars encounter.

Researchers are adapting attack methods to include image transformations, making adversarial examples more resilient to the challenges of real-world environments.

Desired End State and Ways to Get There

The goal for autonomous systems driven by machine learning is to be robust against both intentionally crafted adversarial examples and unexpected, "out-of-distribution" inputs that fall outside the system's training data. Current open-world machine learning techniques, which attempt to identify out-of-distribution samples, have proven vulnerable to adversarial attacks.

Instead, a promising approach focuses on detecting whether hidden layers within the model are processing unusual data. While defense strategies against adversarial examples are developing, many are tailored to specific machine learning models and assume the attacker's strategy is known.

An ideal approach would make machine learning algorithms fundamentally resilient to both intentional attacks and unexpected inputs. Key factors in developing such resiliency include:

• the ease of detecting attacks,
• effectively using secret information (like cryptographic keys) to protect the model, and
• increasing the computational cost required for effective attacks, even if the attacker has full system knowledge.

Resilient ML Algorithms

Current State

Machine learning (ML) is increasingly used in critical systems like self-driving cars, but ML models can behave unexpectedly due to issues like biased training data. Attackers can exploit these flaws, or accidents can occur even without malicious intent, as seen in the Tesla autopilot example. Traditional ML testing often can't cover enough scenarios to be truly effective in real-world systems.

Researchers are developing new testing and verification tools to systematically find potential problems in ML systems. The key idea is "whitebox" testing, which uses the internal workings of the ML model to guide the process. The authors of this text have been creating such tools with a focus on analyzing both static and dynamic aspects of ML systems.

These analysis methods work well for traditional software, but ML systems pose unique challenges. Unlike hand-coded logic, ML system logic is learned from data. ML is highly non-linear, and it can even be hard to define expected behaviors in complex autonomous systems. Therefore, tools focusing on analyzing desirable/undesirable ranges of behavior can be beneficial.

The authors have spent the last few years developing testing and verification tools specifically for the complexities of deep learning (DL) systems. Because specifying exact behavior for these systems is difficult, their tools focus on verifying properties like robustness to slight input changes (ex: a change in lighting shouldn't alter image classification). They balance scalability, completeness, and soundness in their tool design. These tools have found numerous issues in a variety of real-world ML applications and demonstrate impressive performance gains over existing solutions. The researchers note that their work is already starting to be adopted by both industry and other research teams.

Desired End State and Ways to Get There

While initial results in the field of deep neural network (DNN) verification are encouraging, there are significant challenges yet to be solved. Current tools excel at verifying properties for specific test samples, but there's an assumption that those guarantees will hold for unseen data. To make tools more robust, techniques like interval analysis could be adapted to reason about entire distributions of possible inputs, making fewer assumptions about specific samples.

Expanding the scope of DNN verification is crucial. This includes developing tools that can handle a wider variety of safety properties, support different neural network architectures (like Recurrent Neural Networks, or RNNs), and work with a broader range of activation functions (such as Sigmoid and tanh).

TO BE CONTINUED