Towards a better understanding of why machine learning models make the decisions they do, and why it matters<\/h2>\n<\/p>\n\n\n\n![\"Rui](\"https:\/\/miro.medium.com\/fit\/c\/96\/96\/2*jDgtP6RejFrTtokacnDpog.jpeg\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\nPhoto by Chris Ried<\/a> on Unsplash<\/a><\/figcaption><\/figure>\nModel explainability is one of the most important problems in machine learning today. It\u2019s often the case that certain \u201cblack box\u201d models such as deep neural networks are deployed to production and are running critical systems from everything in your workplace security cameras to your smartphone. It\u2019s a scary thought that not even the developers of these algorithms understand why exactly the algorithms make the decisions they do \u2014 or even worse, how to prevent an adversary from exploiting them.<\/p>\n
While there are many challenges facing the designer of a \u201cblack box\u201d algorithm, it\u2019s not completely hopeless. There are actually many different ways to illuminate the decisions a model makes. It\u2019s even possible to understand which features are the most salient in a model\u2019s predictions.<\/p>\n
In this article, I give a comprehensive overview of model explainability for deeper models in machine learning. I hope to explain how deeper models more traditionally considered \u201cblack boxes\u201d can actually be surprisingly explainable. We use model-agnostic methods to apply interpretability to all different kinds of black box models.<\/p>\n
A partial dependence plot shows the effect of a feature on the outcome of a ML model.<\/p>\nPartial dependence equation for regression<\/figcaption><\/figure>\nPartial dependence works by marginalizing the machine learning model output over the distribution of the features we are not interested in (denoted by features in set C). This makes it such that the partial dependence function shows the relationship between the features we do care about (which we denote by buying in set S) and the predicted outcome. By marginalizing over the other features, we get a function that depends only on features in S. This makes it easy to understand how varying a specific feature influences model predictions. For example, here are 3 PDP plots for Temperature, Humidity and Wind Speed as relating to predicted bike sales by a linear model.<\/p>\nPDP\u2019s for temperature, humidity and wind speed for a regression problem where the number of bikes is the outcome. From these plots, it\u2019s clear to see that temperature is an important determiner of how many bikes arre rented, and the hotter it is, the more bikes are rented.<\/figcaption><\/figure>\nPDP\u2019s can even be used for categorical features. Here\u2019s one for the effect of season on bike rental.<\/p>\nPartial dependence plot for the effect of seasons on bike rentals<\/figcaption><\/figure>\nFor classification, the partial dependence plot displays the probability for a certain class given different values for features. A good way to deal with multi-class problems is to have one PDP per class.<\/p>\n
The partial dependence plot method is useful because it is global. It makes a point about the global relationship between a certain feature and a target outcome across all values of that feature.<\/p>\n
Advantages<\/strong><\/p>\nPartial Dependence Plots are highly intuitive. The partial dependence function for a feature at a value represents the average prediction if we have all data points assume that feature value.<\/p>\n
Disadvantages<\/strong><\/p>\nYou can really only model a maximum of two features using the partial dependence function.<\/p>\n
Assumption of independence: You are assuming the features that you are plotting are not correlated with any other features. For example, if you are predicting blood pressure off of height and weight, you have to assume that height is not correlated with weight. The reason this is the case is that you have to average over the marginal distribution of weight if you are plotting height (or vice-versa). This means, for example, that you can have very small weights for someone who is quite tall, which you probably not see in your actual dataset.<\/p>\n
Great! I want to implement a PDP for my model. Where do I start?<\/strong><\/p>\nHere\u2019s an implementation with with scikit-learn<\/a>.<\/p>\nPermutation feature importance is a way to measure the importance of a feature by calculating change in a model\u2019s prediction error after permuting the feature. A feature is \u201cimportant\u201d if permuting its values increases the model error, and \u201cunimportant\u201d if permuting the values leaves the model error unchanged.<\/p>\n
The algorithm works as follows:<\/p>\n
input: a model f, feature matrix X, target vector Y and error measure L(y, f)<\/span>1. Estimate the original model error e\u2070 = L(Y, f(x))<\/span>2. For each feature j:<\/span>- Generate feature matrix X' by permuting feature j in the original feature matrix X.<\/span>- Estimate the new error e\u00b9=L(Y, f(X')) based off the model's predictions for the new data X'<\/span>- Calculate the permutation feature importance FI=e\u00b9\/e\u2070. You can also use e\u00b9-e\u2070.<\/span>3. Sort the features by descending FI.<\/span><\/pre>\nAfter you have sorted the features by descending FI, you can plot the results. Here is the permutation feature importance plot for the bike rentals problem.<\/p>\nPermutation feature importance plot for bike rentals. You can clearly see that the model sees temperature and days since 2011 as the most important features.<\/figcaption><\/figure>\nAdvantages<\/strong><\/p>\nInterpretability: Feature importance is just how much the error increases when a feature is distorted. This is easy to explain and visualize.<\/p>\n
Permutation feature importance provides global insight into the model\u2019s behavior.<\/p>\n
Permutation feature importance does not require training a new model or retraining an existing model, simply shuffling features around.<\/p>\n
Disadvantages<\/strong><\/p>\nIt\u2019s not clear whether you should use training or test data for your plot.<\/p>\n
If features are correlated, you can get unrealistic samples after permuting features, biasing the outcome.<\/p>\n
Adding a correlated feature to your model can decrease the importance of another feature.<\/p>\n
Great! I want to implement Permutation Feature Importance for my model. Where do I start?<\/strong><\/p>\nHere\u2019s an implementation with the eli5 model in Python.<\/a><\/p>\nALE plots are a faster and unbiased alternative to partial dependence plots. They measure how features influence the prediction of a model. Because they are unbiased, they handle correlated features much better than PDP\u2019s do.<\/p>\n
If features of a machine learning model are correlated, the partial dependence plot cannot be trusted, because you can generate samples that are very unlikely in reality by varying a single feature. ALE plots solve this problem by calculating \u2013 also based on the conditional distribution of the features \u2013 differences in predictions instead of averages. One way to interpret this is by thinking of the ALE as saying<\/p>\n
\n\u201cLet me show you how the model predictions change in a small \u201cwindow\u201d of the feature.\u201d<\/p>\n<\/blockquote>\n
Here\u2019s a visual interpretation of what is going on in an ALE plot.<\/p>\nALE plot calculation. Divide x1 into \u201cwindows\u201d. For all the points in each window, calculate the difference in prediction when we replace each point with the upper and lower bounds of the window.<\/figcaption><\/figure>\nThis can also be done with two features.<\/p>\n2D-ALE plot. Same general idea as the 1D plot, but instead of using the upper and lower \u201cwindow\u201d bounds, you calculate the difference in prediction over the four corners of the square in the grid.<\/figcaption><\/figure>\nOnce you have computed the differences in predictions over each window, you can generate an ALE plot.<\/p>\nALE plot for bike rentals. Clearly, temperature has a strong effect on bike rental prediction, but you can also see that if humidity is above a certain point, it has a strong effect on bike rental prediction as well.<\/figcaption><\/figure>\nALE plots can also be done for categorical features.<\/p>\nALE plot for month and bike rentals. January and March seem to have little effect on the number of bikes rented, but December and November seem to have a large negative effect.<\/figcaption><\/figure>\nAdvantages<\/strong><\/p>\nALE plots are unbiased, meaning they work with correlated features.<\/p>\n
ALE plots are computationally fast to compute.<\/p>\n
The interpretation of the ALE plot is clear.<\/p>\n
Disadvantages<\/strong><\/p>\nThe implementation of ALE plots is complicated and difficult to understand.<\/p>\n
Interpretation still remains difficult if features are strongly correlated.<\/p>\n
Second-order or 2D ALE plots can be hard to interpret.<\/p>\n
Generally, it is better to use ALE\u2019s over PDP\u2019s, especially if you expect correlated features.<\/p>\n
Great! I want to implement ALE\u2019s for my model. Where do I start?<\/strong><\/p>\nHere\u2019s a library that provides an ALE implementation.<\/a><\/p>\nIndividual Conditional Expectation (ICE) plots display one line per data point. It produces a plot that shows how the model\u2019s prediction for a data point changes as a feature varies across all data points in a set. For the plot below, you can see the ICE plots for varying temperature, humidity and wind speed across all instances in the training set bike rental data.<\/p>\nICE plot for bike-sharing<\/figcaption><\/figure>\nLooking at this plot, you may ask yourself: what is the point of looking at an ICE plot instead of a PDP? It seems much less interpretable.<\/p>\n
PDPs can only show you what the average relationship between what a feature and a prediction looks like. This only works well if the interactions between the features for which the PDP is calculated and the other features are uncorrelated, but in the case of strong, correlated interactions, the ICE plot will be more insightful.<\/p>\n
Advantages<\/strong><\/p>\nLike PDP plots, ICE plots are very intuitive to understand.<\/p>\n
ICE plots can uncover heterogeneous relationships better than PDP plots can.<\/p>\n
Disadvantages<\/strong><\/p>\nICE curves can only display one feature at a time.<\/p>\n
The plots generated by this method can be hard to read and overcrowded.<\/p>\n
Great! I want to implement ICE for my model. Where do I start?<\/strong><\/p>\nHere\u2019s an overview of interpretability with an ICE implementation.<\/a><\/p>\nA surrogate model is an interpretable model (such as a decision tree or linear model) that is trained to approximate the predictions of a black box. We can understand the black box better by interpreting the surrogate model\u2019s decisions.<\/p>\n
The algorithm for generating a surrogate model is straightforward.<\/p>\n
1. Select a dataset X that you can run your black box model on.<\/span>2. For the selected dataset X, get the predictions of your black box model. <\/span>3. Select an interpretable model type (e.g. linear model or decision tree)<\/span>4. Train the interpretable model on the dataset X and the black box's predictions. <\/span>5. Measure how well the surrogate model replicates the predictions of the black box model. <\/span>6. Interpret the surrogate model.<\/span><\/pre>\nOne way to measure how well the surrogate replicates the black box through the R-squared metric:<\/p>\n
The R-squared metric is a way to measure the variance captured by the surrogate model. An R-squared value close to 1 implies the surrogate model captures the variance well, and close to 0 implies that it is capturing very little variance, and not explaining the black box model well.<\/p>\n
Advantages<\/strong><\/p>\nThis approach is intuitive: you are learning what the black box model thinks is important by approximating it.<\/p>\n
Easy to measure: It\u2019s clear how well the interpretable model performs in approximating the black box through the R-squared metric.<\/p>\n
Disadvantages<\/strong><\/p>\nThe linear model may not approximate the black box model well.<\/p>\n
You are drawing conclusions about the black box model and not the actual data, as you are using the black box\u2019s model predictions as labels without seeing the ground truth.<\/p>\n
Even if you do approximate the black box model well, the explainability of the \u201cinterpretable\u201d model may not actually represent what the black box model has learned.<\/p>\n
It may be difficult to explain the interpretable model.<\/p>\n
Great! I want to implement a surrogate model.Where do I start?<\/strong><\/p>\nHere\u2019s an overview of interpretability with a surrogate model implementation.<\/a><\/p>\nAs machine learning becomes more prominent in daily life, the future of interpretability is more important than ever before. I believe a few trends will categorize the future of interpretability, and this will shape how we interact with AI models in the future.<\/p>\n
Model agnostic interpretability focus<\/strong><\/p>\nAll the trends in deep learning research point to the fact that deep networks are not saturating with our current computing and data limits. It\u2019s important to realize that as our models get deeper and deeper in everything from image recognition to text generation, there is a need for methods that can provide interpretability across all types of models. The generalizability aspect will become more and more useful the more machine learning takes a hold in different fields. The methods I discussed in this blog post are a start, but we need to take interpretability more seriously as a whole to better understand why the machine learning systems powering our day-to-day are making the decisions they do.<\/p>\n
Models that explain themselves<\/strong><\/p>\nOne trend that I have not seen take hold in most ML systems that I believe will exist in the future is the idea of a self-explainable model. Most systems today simply make a decision with reasons that are opaque to the user. In the future, I believe that will change. If a self-driving car makes a decision to stop, we will know why. If Alexa cannot understand our sentence, it will tell us in specific detail what went wrong and how we can phrase our query more clearly. With models that explain themselves, we can better understand how the ML systems in our lives work.<\/p>\n
Increased model scrutiny<\/strong><\/p>\nFinally, I believe that we as a society have pushed black-box model scrutiny under the rug. We do not understand the decisions that our models are making, and that doesn\u2019t seem to be bothering anyone in particular. This will have to change in the future. Engineers and Data Scientists will be held accountable as models start to make mistakes, and this will lead to a trend where we examine the decisions our model makes with the same rigor we would a dataset that the model is trained on.<\/p>\n
I hope you enjoyed this post. I certainly found it illuminating to write, and I hope it helps you with your studies or research in the field of machine learning.<\/p>\n
Works Cited:<\/strong><\/p>\nMost of the examples here inspired from the excellent Interpretable Machine Learning<\/a> book.<\/p>\n(Molnar, Christoph. \u201cInterpretable machine learning. A Guide for Making Black Box Models Explainable\u201d, 2019. https:\/\/christophm.github.io\/interpretable-ml-book\/)<\/p>\n
I highly encourage you to buy it if you wish to further your knowledge in the topic.<\/p>\n<\/div>\n
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\nModel explainability is one of the most important problems in machine learning today. It\u2019s often the case that certain \u201cblack box\u201d models such as deep neural networks are deployed to production and are running critical systems from everything in your workplace security cameras to your smartphone. It\u2019s a scary thought that not even the developers of these algorithms understand why exactly the algorithms make the decisions they do \u2014 or even worse, how to prevent an adversary from exploiting them.<\/p>\n
While there are many challenges facing the designer of a \u201cblack box\u201d algorithm, it\u2019s not completely hopeless. There are actually many different ways to illuminate the decisions a model makes. It\u2019s even possible to understand which features are the most salient in a model\u2019s predictions.<\/p>\n
In this article, I give a comprehensive overview of model explainability for deeper models in machine learning. I hope to explain how deeper models more traditionally considered \u201cblack boxes\u201d can actually be surprisingly explainable. We use model-agnostic methods to apply interpretability to all different kinds of black box models.<\/p>\n
A partial dependence plot shows the effect of a feature on the outcome of a ML model.<\/p>\n Partial dependence works by marginalizing the machine learning model output over the distribution of the features we are not interested in (denoted by features in set C). This makes it such that the partial dependence function shows the relationship between the features we do care about (which we denote by buying in set S) and the predicted outcome. By marginalizing over the other features, we get a function that depends only on features in S. This makes it easy to understand how varying a specific feature influences model predictions. For example, here are 3 PDP plots for Temperature, Humidity and Wind Speed as relating to predicted bike sales by a linear model.<\/p>\n PDP\u2019s can even be used for categorical features. Here\u2019s one for the effect of season on bike rental.<\/p>\n For classification, the partial dependence plot displays the probability for a certain class given different values for features. A good way to deal with multi-class problems is to have one PDP per class.<\/p>\n The partial dependence plot method is useful because it is global. It makes a point about the global relationship between a certain feature and a target outcome across all values of that feature.<\/p>\n Advantages<\/strong><\/p>\n Partial Dependence Plots are highly intuitive. The partial dependence function for a feature at a value represents the average prediction if we have all data points assume that feature value.<\/p>\n Disadvantages<\/strong><\/p>\n You can really only model a maximum of two features using the partial dependence function.<\/p>\n Assumption of independence: You are assuming the features that you are plotting are not correlated with any other features. For example, if you are predicting blood pressure off of height and weight, you have to assume that height is not correlated with weight. The reason this is the case is that you have to average over the marginal distribution of weight if you are plotting height (or vice-versa). This means, for example, that you can have very small weights for someone who is quite tall, which you probably not see in your actual dataset.<\/p>\n Great! I want to implement a PDP for my model. Where do I start?<\/strong><\/p>\n Here\u2019s an implementation with with scikit-learn<\/a>.<\/p>\n Permutation feature importance is a way to measure the importance of a feature by calculating change in a model\u2019s prediction error after permuting the feature. A feature is \u201cimportant\u201d if permuting its values increases the model error, and \u201cunimportant\u201d if permuting the values leaves the model error unchanged.<\/p>\n The algorithm works as follows:<\/p>\n After you have sorted the features by descending FI, you can plot the results. Here is the permutation feature importance plot for the bike rentals problem.<\/p>\n Advantages<\/strong><\/p>\n Interpretability: Feature importance is just how much the error increases when a feature is distorted. This is easy to explain and visualize.<\/p>\n Permutation feature importance provides global insight into the model\u2019s behavior.<\/p>\n Permutation feature importance does not require training a new model or retraining an existing model, simply shuffling features around.<\/p>\n Disadvantages<\/strong><\/p>\n It\u2019s not clear whether you should use training or test data for your plot.<\/p>\n If features are correlated, you can get unrealistic samples after permuting features, biasing the outcome.<\/p>\n Adding a correlated feature to your model can decrease the importance of another feature.<\/p>\n Great! I want to implement Permutation Feature Importance for my model. Where do I start?<\/strong><\/p>\n Here\u2019s an implementation with the eli5 model in Python.<\/a><\/p>\n ALE plots are a faster and unbiased alternative to partial dependence plots. They measure how features influence the prediction of a model. Because they are unbiased, they handle correlated features much better than PDP\u2019s do.<\/p>\n If features of a machine learning model are correlated, the partial dependence plot cannot be trusted, because you can generate samples that are very unlikely in reality by varying a single feature. ALE plots solve this problem by calculating \u2013 also based on the conditional distribution of the features \u2013 differences in predictions instead of averages. One way to interpret this is by thinking of the ALE as saying<\/p>\n \u201cLet me show you how the model predictions change in a small \u201cwindow\u201d of the feature.\u201d<\/p>\n<\/blockquote>\n Here\u2019s a visual interpretation of what is going on in an ALE plot.<\/p>\n This can also be done with two features.<\/p>\n Once you have computed the differences in predictions over each window, you can generate an ALE plot.<\/p>\n ALE plots can also be done for categorical features.<\/p>\n Advantages<\/strong><\/p>\n ALE plots are unbiased, meaning they work with correlated features.<\/p>\n ALE plots are computationally fast to compute.<\/p>\n The interpretation of the ALE plot is clear.<\/p>\n Disadvantages<\/strong><\/p>\n The implementation of ALE plots is complicated and difficult to understand.<\/p>\n Interpretation still remains difficult if features are strongly correlated.<\/p>\n Second-order or 2D ALE plots can be hard to interpret.<\/p>\n Generally, it is better to use ALE\u2019s over PDP\u2019s, especially if you expect correlated features.<\/p>\n Great! I want to implement ALE\u2019s for my model. Where do I start?<\/strong><\/p>\n Here\u2019s a library that provides an ALE implementation.<\/a><\/p>\n Individual Conditional Expectation (ICE) plots display one line per data point. It produces a plot that shows how the model\u2019s prediction for a data point changes as a feature varies across all data points in a set. For the plot below, you can see the ICE plots for varying temperature, humidity and wind speed across all instances in the training set bike rental data.<\/p>\n Looking at this plot, you may ask yourself: what is the point of looking at an ICE plot instead of a PDP? It seems much less interpretable.<\/p>\n PDPs can only show you what the average relationship between what a feature and a prediction looks like. This only works well if the interactions between the features for which the PDP is calculated and the other features are uncorrelated, but in the case of strong, correlated interactions, the ICE plot will be more insightful.<\/p>\n Advantages<\/strong><\/p>\n Like PDP plots, ICE plots are very intuitive to understand.<\/p>\n ICE plots can uncover heterogeneous relationships better than PDP plots can.<\/p>\n Disadvantages<\/strong><\/p>\n ICE curves can only display one feature at a time.<\/p>\n The plots generated by this method can be hard to read and overcrowded.<\/p>\n Great! I want to implement ICE for my model. Where do I start?<\/strong><\/p>\n Here\u2019s an overview of interpretability with an ICE implementation.<\/a><\/p>\n A surrogate model is an interpretable model (such as a decision tree or linear model) that is trained to approximate the predictions of a black box. We can understand the black box better by interpreting the surrogate model\u2019s decisions.<\/p>\n The algorithm for generating a surrogate model is straightforward.<\/p>\n One way to measure how well the surrogate replicates the black box through the R-squared metric:<\/p>\n The R-squared metric is a way to measure the variance captured by the surrogate model. An R-squared value close to 1 implies the surrogate model captures the variance well, and close to 0 implies that it is capturing very little variance, and not explaining the black box model well.<\/p>\n Advantages<\/strong><\/p>\n This approach is intuitive: you are learning what the black box model thinks is important by approximating it.<\/p>\n Easy to measure: It\u2019s clear how well the interpretable model performs in approximating the black box through the R-squared metric.<\/p>\n Disadvantages<\/strong><\/p>\n The linear model may not approximate the black box model well.<\/p>\n You are drawing conclusions about the black box model and not the actual data, as you are using the black box\u2019s model predictions as labels without seeing the ground truth.<\/p>\n Even if you do approximate the black box model well, the explainability of the \u201cinterpretable\u201d model may not actually represent what the black box model has learned.<\/p>\n It may be difficult to explain the interpretable model.<\/p>\n Great! I want to implement a surrogate model.Where do I start?<\/strong><\/p>\n Here\u2019s an overview of interpretability with a surrogate model implementation.<\/a><\/p>\n As machine learning becomes more prominent in daily life, the future of interpretability is more important than ever before. I believe a few trends will categorize the future of interpretability, and this will shape how we interact with AI models in the future.<\/p>\n Model agnostic interpretability focus<\/strong><\/p>\n All the trends in deep learning research point to the fact that deep networks are not saturating with our current computing and data limits. It\u2019s important to realize that as our models get deeper and deeper in everything from image recognition to text generation, there is a need for methods that can provide interpretability across all types of models. The generalizability aspect will become more and more useful the more machine learning takes a hold in different fields. The methods I discussed in this blog post are a start, but we need to take interpretability more seriously as a whole to better understand why the machine learning systems powering our day-to-day are making the decisions they do.<\/p>\n Models that explain themselves<\/strong><\/p>\n One trend that I have not seen take hold in most ML systems that I believe will exist in the future is the idea of a self-explainable model. Most systems today simply make a decision with reasons that are opaque to the user. In the future, I believe that will change. If a self-driving car makes a decision to stop, we will know why. If Alexa cannot understand our sentence, it will tell us in specific detail what went wrong and how we can phrase our query more clearly. With models that explain themselves, we can better understand how the ML systems in our lives work.<\/p>\n Increased model scrutiny<\/strong><\/p>\n Finally, I believe that we as a society have pushed black-box model scrutiny under the rug. We do not understand the decisions that our models are making, and that doesn\u2019t seem to be bothering anyone in particular. This will have to change in the future. Engineers and Data Scientists will be held accountable as models start to make mistakes, and this will lead to a trend where we examine the decisions our model makes with the same rigor we would a dataset that the model is trained on.<\/p>\n I hope you enjoyed this post. I certainly found it illuminating to write, and I hope it helps you with your studies or research in the field of machine learning.<\/p>\n Works Cited:<\/strong><\/p>\n Most of the examples here inspired from the excellent Interpretable Machine Learning<\/a> book.<\/p>\n (Molnar, Christoph. \u201cInterpretable machine learning. A Guide for Making Black Box Models Explainable\u201d, 2019. https:\/\/christophm.github.io\/interpretable-ml-book\/)<\/p>\n I highly encourage you to buy it if you wish to further your knowledge in the topic.<\/p>\n<\/div>\ninput: a model f, feature matrix X, target vector Y and error measure L(y, f)<\/span>1. Estimate the original model error e\u2070 = L(Y, f(x))<\/span>2. For each feature j:<\/span>- Generate feature matrix X' by permuting feature j in the original feature matrix X.<\/span>- Estimate the new error e\u00b9=L(Y, f(X')) based off the model's predictions for the new data X'<\/span>- Calculate the permutation feature importance FI=e\u00b9\/e\u2070. You can also use e\u00b9-e\u2070.<\/span>3. Sort the features by descending FI.<\/span><\/pre>\n\n
1. Select a dataset X that you can run your black box model on.<\/span>2. For the selected dataset X, get the predictions of your black box model. <\/span>3. Select an interpretable model type (e.g. linear model or decision tree)<\/span>4. Train the interpretable model on the dataset X and the black box's predictions. <\/span>5. Measure how well the surrogate model replicates the predictions of the black box model. <\/span>6. Interpret the surrogate model.<\/span><\/pre>\n