[ Google / ICLR 2017 / Paper Summary ] Gradients of Counterfactuals

Evaluation / Debugging Network / Discussion

In summary, in this section the authors performed additional experiments to evaluate Integral Gradients. (Such as Pixel ablations or comparing the bounding box of the highest active gradients) And when compared to pure gradient, integral gradient gave Superior results. (It you wish to see more examples please click here.)

In a setting where, bar for precision is high, such as medical diagnosis it is very important to know what is going on within the network. And to more accurately know what features contribute to which classes, and integral gradients can be used as a tool to gain more insights.

Finally, the authors discusses some limitation of this approach.

a. Inability to capture Feature interactions → The model can perform some operation that combines certain features together. Important scores have no way to represent these combinations.

b. Feature correlations → If similar feature occurs multiple times the model can assign weights to either one of them. (Or both). But those weights might not be human-intelligible.