AI + Safety

From May 2018, the EU General Data Protection Regulation (GDPR) became enforceable, and applies to all companies and organizations that "offer goods or services to, or monitor the behaviour of, EU data subjects." Many consider GDPR to be the first major international legal step by which AI is held accountable and must provide an explanation for decisions (see, e.g., T. Revell, 2018).

The metric of efficacy of AI and ML algorithms has previously been based on predicative power alone. However, we now require that they also adhere to human ethical principles and that erroneous predictions do not lead to unintended negative consequences. Google Brain researchers have published a good discussion on:

Microsoft discusses a set of ethical principles that should act as a foundation for development of AI-powered solutions to ensure that humans are put at the centre in:

The US Defence Advanced Research Projects Agency (DARPA) highlights that AI needs to be explainable to humans in order to create the appropriate level of trust, and runs a special project on this: the XAI project - Explainable Artificial Intelligence.

However, one aspect of ML and AI that is often not given appropriate attention is beautifully expressed by Cathy O'Neil:

Big Data processes codify the past. They do not invent the future.

Cathy O'Neil
"Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy", 2017

In her aptly named book "Weapons of Math Destruction", she exemplifies many of the aspects discussed by Google Brain and Microsoft. She explains how algorithms - which, in theory, should be blind to prejudice, judge according to objective rules, and eliminate biases - actually might do the opposite.

Since these algorithmic models are based on historical data, they not only incorporate previous discriminations and biases - they reinforce them. She explains how these algorithmic "Weapons of math destruction" are used to grant (or deny) loans, to set parole dates or sentence durations, and to evaluate workers, teachers, or students, and how vicious feedback loops are created. Students from poor neighbourhoods may, for example, be denied loans that could have helped them obtain an education that could, in turn, have brought them out of poverty.

One of the problems is that many of these models confuse correlation with causation, and to base decisions on correlations alone is one of the major pitfalls in the analysis of data (see, e.g., J. Pearl, 2010a).

Since many of the ML and AI models used today are opaque, incontestable, unaccountable, and unregulated, Cathy O'Neil calls on modelers to take more responsibility for their algorithms.

This is a position that DNV supports, and although the GDPR is aimed at protecting natural person(s), it is prudent to apply the same principles to engineering systems.

As illustrated above, the field of AI safety is broad, and an active area of research. This position paper complements the discussion by focusing on complicating factors and possible solutions when AI and ML are utilized within a safety context (i.e., to inform or make decisions in safety-critical systems - for example controlling industrial assets). Rather than considering the general safety of AI, here we discuss the implications of using AI and ML in low-probability and high-consequence scenarios.

All models are wrong, but some are useful.

George Edward Pelham Box
"Robustness in the strategy of scientific model building", 1979

In their 1987 book Computer Simulation of Liquids, Allen and Tildesley illustrated the connection between experiments (and experience), theory and computer simulations, for how to gain and improve knowledge of a real system. Similar connections between real-world observations and data-driven models are equally relevant, and Figure 5 illustrates these connections (based on the original figure by Allen and Tildesley).

Figure 1: The connection between experiments (and experience), theory, computer simulations, and data-driven models.

They argued that the more difficult and interesting the problem, the more desirable it becomes to have as close to an exact solution to the problem as possible. For complex problems, this may only be tractable through computer simulations, and, as the computational power and finesse in modelling have increased, the engineering community has been able to develop extremely accurate and strongly predictive simulation models. The engineers recognized computer experiments (simulations) as a way of connecting the microscopic details of a system to the macroscopic properties of experimental interest.

Similarly, the revolution of Big Data analytics has shown the efficacy of data-driven models when predicting individual outcomes based on large collections of observed data. This is also a good example of a bridge between individual (microscopic) entities and the real-world macroscopic observations.

Learning occurs for both us, humans, and AI or ML algorithms, on the basis of a cycle where we observe real-world data, construct a model, use the model to predict a real-world outcome, and finally improve our models by comparing the predictions with actual observations. This learning process should be done in accordance with the principles of the scientific method: allowing the models to be falsifiable, and continuously testing and improving them. In this way the current collection of knowledge is advanced.

However, the aphorism of George Box that all models are wrong, but some are useful is still important when assessing the confidence in the results from our models.

Data-driven decisions are based on three principles in data science: predictability, computability, and stability (B. Yu, 2017). In addition, and particularly important for safety-critical systems, the consequences of erroneous predictions need to be assessed in a decision context.

Prediction is a requisite if an AI or ML algorithm is going to be used in a decision context. Together with cross validation (CV), it makes a human-interpretable validation of the applied model for a certain application. In order to have good prediction accuracy, it is important that the model is developed based on sufficient relevant data from the data-generating process (DGP), (i.e., the process that generates the data that one wants to predict). It is also important that the DGP is computationally feasible, i.e., that an algorithm exists that can learn the DGP from a finite set of training data (see, e.g., A. Blum and R.L. Rivest, 1988).

If the above requisites are met, and an erroneous prediction has a sufficiently negative consequence to be unwanted, the last principle is stability. This is the major problem with safety-critical decisions based on data-driven models. For valid predictions, the future scenarios need to be stable with respect to the data from which the prediction model was trained. At the same time, data related to high-consequence scenarios are, thankfully, scarce. In line with Cathy O'Neil's argument above, this makes ordinary data-driven methods inappropriate for decision making in the context of unwanted and rare scenarios.

In a recent position paper by Google AI researchers, D. Sculley et al., 2018, it is argued that "the rate of empirical advancement may not have been matched by consistent increase in the level of empirical rigor across the [ML] field". They point to the fact that "empirical studies have become challenges to be “won” [e.g. as competitions on Kaggle], rather a process for developing insight and understanding". In addition to current practices, they suggest a set of standards for empirical evaluations that should be applied when using ML:

Tuning Methodology:

Tuning of all key hyperparameters should be performed for all models including baselines via grid search or guided optimization, and results shared as part of publication.

Sliced Analysis:

Performance measures such as accuracy or AUC [Area under curve of Receiver operating characteristic (ROC)] on a full test set may mask important effects, such as quality improving in one area but degrading in another. Breaking down performance measures by different dimensions or categories of the data is a critical piece of full empirical analysis.

Ablation Studies:

Full ablation studies of all changes from prior baselines should be included, testing each component change in isolation and a select number in combination.

Sanity Checks and Counterfactuals:

Understanding of model behavior should be informed by intentional sanity checks, such as analysis on counter-factual or counter-usual data outside of the test distribution. How well does a model perform on images with different backgrounds, or on data from users with different demographic distributions?

At Least One Negative Result:

Because the No Free Lunch Theorem still applies, we believe that it is important for researchers to find and report areas where a new method does not perform better than previous baselines. Papers that only show wins are potentially suspect and may be rejected for that reason alone.

D. Sculley et al., 2018
Winner's curse? On pace, progress, and empirical rigor

These standards are crucial when applying ML and AI in safety-critical systems to ensure that the models are trained on sufficient and relevant data to be able to generalize their predictions to rare and potentially catastrophic scenarios.

In addition to these standards, DNV argues that ML and AI model predictions should also adhere to established scientific (physical or causal) knowledge about the system (or DGP) as outlined in section Guide the AI - impose constraints.

Many of the engineering systems that have been developed, and will be developed, may result in serious consequences if they fail (e.g., do not operate as intended). These systems range from a single individual relying on the anti-lock braking system and traction control of a car, through hundreds of people blissfully unaware of the collision-avoidance systems at work in commercial airplanes, to emergency shut-down systems of nuclear power plants.

Historically, the safety of such systems has been addressed by careful scrutiny of potential scenarios to which the system can be subjected, and cautious engineering of the capacity of these systems. And, although many processes have been automated, it is people who oversee the entire system and thus hold the ultimate responsibility. But this is no longer the case.

As the complexity of our engineering systems increases, and more and more of our systems are interconnected and controlled by computers, our human minds have become hard pressed to cope with, and understand, this enormous and dynamic complexity. In fact, it seems likely that we will be unable to apply human oversight to many of these systems at the timescale required to ensure safe operation. Thus:

Machines need to make safety-critical decisions in real-time, and we, the industry, have the ultimate responsibility for designing artificially intelligent systems that are safe!

Assurance of high-risk and safety-critical systems that rely on ML methods is receiving greater attention (see current status and, e.g., A. Brandsæter and K.E. Knutsen, 2018). C. Agrell et al., 2018 highlights three critical challenges when applying ML methods to high-risk and low-probability scenarios:

The first two points are in direct contrast to the typical ML applications that are in use today (e.g., non-consequential recommendation engines etc.). The last point, on lack of transparency in the ML models, relates to how easy it is to quantify the model discrepancy and the ability to falsify models that do not comply with observations. Together, these will affect the approach to validation and quality assurance of high-risk and safety-critical systems under ML or AI influence.

DNV argues the paramount importance of combining causal and data-driven models to ensure safe AI decisions. Just as the physics-based and causal models increase the confidence and accuracy of ML models, it is equally important that we manage to utilize data-driven methods to increase the robustness with respect to stochastic variations. The following discussion is thus limited to the class of problems to which we have access, at least in principle, to causal knowledge - how the physics work, and the associated consequences.

However, two aspects of physics-based models make them ill suited for practical use in a real-time, risk-based decision context:

These challenges are at the heart of making sure that systems influenced by AI are safe.

SpaceX learning cycle

On Tuesday February 6th 2018, SpaceX successfully launched the world's biggest rocket, Falcon Heavy, and landed the two first-stage side boosters safely back on the launch pad in the upright position - ready for refurbishing and new flights.

No full-system test was possible prior to the launch. The team of engineers relied on computer models, together with previous experience of similar systems and advanced ML models, to simulate how the launch would play out and to determine how to control the actual boosters back to the launch pad. This is a perfect example of what can be achieved through extensive use of models based on both causal knowledge and data-driven methods, making autonomous real-time adjustments (see L. Blackmore, 2016 for details on the precision landing of Falcon 9 first stages).

Note, however, that safe landing of the first stages was not achieved on the very first attempt. The video below illustrates well how SpaceX utilized the continuous learning cycle presented in Figure 1 to achieve their ambitious goals.

The Falcon Heavy core booster was supposed to land upright on a drone ship in the Atlantic, but missed by 90 meters and crashed into the ocean (watch on YouTube).

This illustrates two important aspects. First, even when we have designed and tested something thoroughly and in extreme detail, there is always an element of stochastic variation that is difficult to foresee. Six days after the launch Musk tweeted "Not enough ignition fluid to light the outer two engines after several three engine relights. Fix is pretty obvious." There are many possible reasons as to why the core booster had too little ignition fluid. It could be on the provisioning side - not filling the fuel tanks sufficiently before launch; or it could be related to environmental loads - maybe winds caused the controls to use more fuel than anticipated; or it could be any other of a multitude of reasons. This underpins that:

It is "easy" to make something work, but it can be near impossible to ensure that it will not fail!

When we arrive at this acknowledgment, the second aspect follows logically. When our system might fail, we need to understand and address the associated risk. That is why the Falcon Heavy was launched (east bound to utilize Earth's rotation) out of cape Canaveral Air Force Station (map) on the east cost of Florida. This ensured that any failures resulting in a crash of the massive rockets would mean that only oceans would be hit.

It is obvious that hazardous systems must be designed with safety and risk in mind - which we have done for decades, if not centuries. However:

As we move towards more autonomous systems with AI agents making safety-critical decisions, it is paramount that the data-driven models also treat uncertainties with rigour and incorporate risk measures in the decision making process.

To enable real-time decisions for safety-critical systems, DNV argues that the industry must combine its extensive causal knowledge with more data-driven approaches, and, at the same time, address both uncertainty and risk. DNV suggests the following steps to facilitate the assurance of AI-influenced systems.

Let us consider a complex computer model , which we treat as a deterministic black-box function. This function maps a set of inputs to an output .

If the evaluation of is expensive, e.g., such that evaluating at any point takes such a long time that it is cumbersome to achieve within the time frame of a certain decision context, we cannot evaluate it practically as often as we need. The solution is to replace the expensive with a cheap and fast approximation, often referred to as a surrogate model.

Most ML algorithms create such surrogate models - fast approximations - based on a large set of training data. However, in addition to providing a fast approximation of a real process, A. O'Hagan, 2006 argues that to be able to perform reasonably good and efficient uncertainty analysis (UA) and sensitivity analysis (SA) of complex computer models, the surrogate model should also exhibit two other important traits. First, it should be possible to interpolate, i.e., ensure that all approximations match the observations exactly at the training points, and secondly it should be able to provide an uncertainty estimate of the prediction on all points that are not part of the training set (this is not typically available for many ML algorithms). Such a surrogate model is referred to as an emulator.

Surrogate model

A surrogate model is an approximation , established from a finite set of evaluations .

Emulator

An emulator is a surrogate model that can:

• Interpolate. I.e. for all training data .
• Provide uncertainty estimates on for all data not in the training set .

For safety-critical systems, it is evident that not only being able to predict the outcome of a certain scenario, but also being able to acquire information on the confidence and uncertainty in that prediction is essential for safe operations and high-quality decision support.

Uncertainty Quantification (UQ) associated with use of complex (computer) models has been an active area of research in the Statistics community since the 1980s, and has had more recent interest from the Applied Mathematics community. Kennedy and O'Hagan, 2001 is one of the fundamental papers on UQ in complex mathematical (computer) models that addresses all sources of uncertainty encountered when using models to predict real world scenarios.

Uncertainty quantification related to such a function can be described by four types of UQ problems:

Forward UQ

Uncertainty analysis. For a stochastic , study the distribution of .

Sensitivity analysis. Study how responds to changes in .

Inverse UQ

Calibration. Estimate some from noisy observations of the real process .

Bias correction. Quantify model discrepancy.

Understanding the uncertainty of our approximations is how we establish confidence in our systems.

In the following sections, we outline how such emulators can be utilized with respect to real-time decision support and design of experiments to ensure safe designs. We also discuss how we can impose physics-based constraints on data driven models, and why this is worth doing.

Note that expensive or dangerous physical tests can be simulated using virtual models, and both real-world data and simulated data can be used to establish fast-running emulators, including uncertainty estimates. If we are able to replace the expensive and/or dangerous physical tests with simulations and construct fast-running emulators, we can guide our exploration of scenarios virtually that are not feasible to explore in the real world, and simultaneously obtain information on uncertainty related to the approximation.

Design-of-experiments (DOE) is a field of study that has its roots back in 1747 when James Lind carried out a systematic clinical trial to compare remedies for scurvy in British sailors. From this early beginning, the field has developed significantly, and, in 1935, the English statistician Ronald A. Fisher published one of the seminal books on the topic, The Design of Experiments.

Despite the science of design-of-experiments being a well-established field of study, modellers have a tendency to prefer non-optimal designs. A. Saltelli and P. Annoni, 2010 explain the tendency of modellers to prefer one-at-a-time (OAT) experimental designs, highlighting the aspects of:

• The baseline being a "safe" starting point where the model properties are well known (i.e., the best estimate).
• Changing one factor at the time ensures that whatever effect is observed on the output can be attributed to this factor alone.
• A non-zero effect implies influence, i.e., OAT do not make Type I errors (false positives).

These aspects make OAT approaches intuitive. However, to be a valid approach for UQ and SA, it must be assumed that all the probability density functions for the uncertain factors have sharp peaks on this multidimensional best-estimate point. This is not usually the case.

A. Saltelli and P. Annoni, 2010 also show how the curse-of-dimensionality proves that an OAT approach is inefficient with respect to exploring only a tiny fraction of a multidimensional input space. An OAT approach also has a tendency to miss important features of a response (see Example: Design-of-experiments below).

As illustrated in the examples above, there are several different methods to explore an unknown response space. The last two methods focus on reducing the uncertainty in the emulator prediction. Other algorithms have been developed e.g., for efficient optimization of global minima or maxima or contour estimation (see, e.g., J. Sacks et al., 1989, D.R. Jones et al., 1998, and P. Ranjan et al., 2008). These are just a few examples of the multitude of methods that exist.

For high-risk systems, it is prudent to include the consequence and risk perspective as well. The following sections describe and exemplify how we think that experiments (either physical or virtual) should be planned and designed. We need to combine different approaches and utilize the best of data-driven models and our phenomenological knowledge of both the physical processes and the associated consequences and risks.

From a safety perspective, prudent exploration of the operational space should focus on the limit between inconsequential and critical scenarios. We want to be able to have an effective way to find contour(s) in the operational space that distinguish safe scenarios from unsafe scenarios. This means that we want to focus our testing efforts close to these contours. Expending test effort inside a region that is clearly a failure set is wasteful and potentially dangerous. Expending test efforts in a region that is clearly inconsequential is, from a safety perspective, also wasteful.

Thus, in order to be able to incorporate a risk perspective when exploring, we need to have a measure for both the consequence at different operational states, as well as a measure for the uncertainty associated with each operational state. This is where the added attributes of an emulator make it superior to ordinary ML surrogate models.

An example of an AL algorithm with a risk-based objective is illustrated below, where a risk measure has been included by comparing the estimated response against a capacity distribution.

In order to increase confidence in the safe use of ML and AI in general engineering applications, Agrell et al., 2018 advocate adopting or developing models with certain attributes:

1. Flexibility: The model should be flexible enough to represent a broad class of functions.
2. Constrainability: It should be possible to impose constraints on the model based on phenomenological knowledge.
3. Probabilistic inference: The model output should be represented by a distribution at any point of the model range.

For engineering purposes, an ML model should be flexible enough to capture the often complex behaviour of a physical process - without prior knowledge of the function form (i.e., linear, polynomial, exponential, sinusodial, etc.). This flexibility is achieved by most non-parametric ML models, as well as popular parametric models, through various techniques (see, e.g., T. Hastie et al., 2009 for details) .

The possibility of imposing constraints on a model can help to make an opaque model more transparent. By a transparent model, we mean a model that has a relationship between inputs and output that can be understood by humans, and the characteristic properties and limitations of the model can be understood without explicit computation (e.g., a linear regression with linear basis). In contrast, advanced ML methods are often considered opaque, i.e., black-box models. These models are so complex that a human cannot understand the relationship between the model's inputs and its output.

The scientific method is based on the principle that any model, or hypothesis, should be falsifiable. All models, including ML models, are based on a set of assumptions. To be able to falsify a model or its assumptions, it must, in principle, be possible to make an observation that would show the assumption to be false. Transparency is one way of falsifying a model; however, many ML models are opaque, and falsification is usually based on measures of poor prediction accuracy (e.g., tests to check for overfitting - see T. Hastie et al., 2009 for details).

For complex and high-risk engineering systems, it might be difficult to obtain the observations that are needed to assess the validity of the model in regions with serious consequences from erroneous predictions.

However, if constraints based on phenomenological knowledge can be imposed, the problem of overfitting has the potential to be significantly reduced. Known physical constraints would increase confidence in extrapolation outside the data on which the model was trained, and this would increase the robustness of the model and its performance for application on future data. Figure 15 illustrates three relevant constraints that are often applicable to physical systems based on differential inequalities.

Figure 15: Example functions fitted to three observations for no constraints, an upper bounded function, a monotonic function, and a convex function.

The three constraints in Figure 15 can be described mathematically by:

(bounded), (monotone), and (convex).

A large variation in unconstrained functions can potentially be fitted to these observations and, if the model is opaque, the only way to characterize and assess the model fully is through exhaustive evaluation for all possible inputs. By imposing an upper bound, the analyst ensures that the prediction will never exceed a certain value, and the response space is reduced.

Where the bound constraint limits the response space, the monotone and convex constraints will limit the form of possible functions, and can, depending on the observed points, be very strong constraints. The models can be further restricted by imposing combinations of multiple constraints. This can be thought of as putting a black-box model inside a white box, enabling deduction of bounds on the model properties through the imposed constraints. This is an area of recent research (see, e.g., H. Maatouk, X. Bay, 2017 and T. Yu, 2007).

Where outputs from ML models are to be used for risk and reliability applications, probabilistic inference is essential. Model predictions in the form of fixed values and best estimates are not applicable. The danger of expressing risk through expected values is that it may hide valuable information about potentially catastrophic scenarios. Modern definitions of risk are therefore usually related to a distribution over potential outcomes. A common metric for quantifying the uncertainty of an ML model is the probability used to describe model accuracy (e.g., the fraction of correctly classified dogs in a set of animal images). However, the fact that a model has an acceptable accuracy 99% of the time might be meaningless if an erroneous prediction in the remaining 1% is associated with severe consequences. The traditional approach to compensating for this has been to scrutinize the model by human quality control measures to ensure an acceptable maximum prediction error - usually on the conservative side. This might no longer be feasible for higher dimensional models, which, more often than not, are opaque. One way to mitigate this is to use models that can estimate the uncertainty of their predictions on their own. This is why Bayesian methods have an advantage over other ML methods, when ML is used either to evaluate risk or to make predictions for high-risk systems. A relevant podcast on AI decision making and Bayesian methods can be found on Data Skeptic - AI Decision Making.

References