DeepExplainer doesn't work with keras concatenate layers #305

ellehoej · 2018-10-29T13:45:38Z

Hi,
Thanks for some great work.
I'm getting an error (see below) when using DeepExplainer (SHAP 0.24, TF 1.8, Keras 2.2.4) on a keras model that uses "concatenate" through the Keras functional API.

The model is simply:

x_input = Input(shape=(days_input, n_features_input))

x_d = Flatten()(x_input)

x_d1 = Dense(256, use_bias=False)(x_d)
x_d1 = BatchNormalization()(x_d1)
x_dl = PReLU()(x_d1)
x_d1 = Dropout(0.5)(x_d1)

x_d3 = Dense(128, use_bias=False)(x_d1)
x_d3 = BatchNormalization()(x_d3)
x_d3 = PReLU()(x_d3)
x_d3 = Dropout(0.5)(x_d3)

x_d3 = concatenate([x_d1, x_d3])

dense = Dense(1,activation='sigmoid')

x = dense(x_d3)

model = Model(inputs=x_input, outputs=x)
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mse'])

Generating the DeepExplainer works:

e = shap.DeepExplainer(model, train_X[:20])

But when generating shap_values:

shap_values = e.shap_values(test_X)

I get the following error:

InvalidArgumentError                      Traceback (most recent call last)
C:\Anaconda\envs\keras_updated\lib\site-packages\tensorflow\python\framework\ops.py in get_attr(self, name)
   2326         with c_api_util.tf_buffer() as buf:
-> 2327           c_api.TF_OperationGetAttrValueProto(self._c_op, name, buf)
   2328           data = c_api.TF_GetBuffer(buf)

InvalidArgumentError: Operation 'concatenate_1/concat' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
C:\Anaconda\envs\keras_updated\lib\site-packages\tensorflow\python\ops\gradients_impl.py in _MaybeCompile(scope, op, func, grad_fn)
    379     try:
--> 380       xla_compile = op.get_attr("_XlaCompile")
    381       xla_separate_compiled_gradients = op.get_attr(

C:\Anaconda\envs\keras_updated\lib\site-packages\tensorflow\python\framework\ops.py in get_attr(self, name)
   2330         # Convert to ValueError for backwards compatibility.
-> 2331         raise ValueError(str(e))
   2332       x = attr_value_pb2.AttrValue()

ValueError: Operation 'concatenate_1/concat' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

AssertionError                            Traceback (most recent call last)
<ipython-input-23-9f62380f1037> in <module>()
----> 1 shap_values = e.shap_values(test_X)

C:\Anaconda\envs\keras_updated\lib\site-packages\shap\explainers\deep.py in shap_values(self, X, ranked_outputs, output_rank_order)
    290                 # run attribution computation graph
    291                 feature_ind = model_output_ranks[j,i]
--> 292                 sample_phis = self.run(self.phi_symbolic(feature_ind), self.model_inputs, joint_input)
    293 
    294                 # assign the attributions to the right part of the output arrays

C:\Anaconda\envs\keras_updated\lib\site-packages\shap\explainers\deep.py in phi_symbolic(self, i)
    202             try:
    203                 out = self.model_output[:,i] if self.multi_output else self.model_output
--> 204                 self.phi_symbolics[i] = tf.gradients(out, self.model_inputs)
    205 
    206             finally:

C:\Anaconda\envs\keras_updated\lib\site-packages\tensorflow\python\ops\gradients_impl.py in gradients(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients)
    492   with ops.get_default_graph()._lock:  # pylint: disable=protected-access
    493     return _GradientsHelper(ys, xs, grad_ys, name, colocate_gradients_with_ops,
--> 494                             gate_gradients, aggregation_method, stop_gradients)
    495 
    496 

C:\Anaconda\envs\keras_updated\lib\site-packages\tensorflow\python\ops\gradients_impl.py in _GradientsHelper(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients)
    634                 # functions.
    635                 in_grads = _MaybeCompile(grad_scope, op, func_call,
--> 636                                          lambda: grad_fn(op, *out_grads))
    637               else:
    638                 # For function call ops, we add a 'SymbolicGradient'

C:\Anaconda\envs\keras_updated\lib\site-packages\tensorflow\python\ops\gradients_impl.py in _MaybeCompile(scope, op, func, grad_fn)
    383       xla_scope = op.get_attr("_XlaScope").decode()
    384     except ValueError:
--> 385       return grad_fn()  # Exit early
    386 
    387   if not xla_compile:

C:\Anaconda\envs\keras_updated\lib\site-packages\tensorflow\python\ops\gradients_impl.py in <lambda>()
    634                 # functions.
    635                 in_grads = _MaybeCompile(grad_scope, op, func_call,
--> 636                                          lambda: grad_fn(op, *out_grads))
    637               else:
    638                 # For function call ops, we add a 'SymbolicGradient'

C:\Anaconda\envs\keras_updated\lib\site-packages\shap\explainers\deep.py in custom_grad(self, op, *grads)
    315         """ Passes a gradient op creation request to the correct handler.
    316         """
--> 317         return op_handlers[op.type](self, op, *grads)
    318 
    319 

C:\Anaconda\envs\keras_updated\lib\site-packages\shap\explainers\deep.py in handler(explainer, op, *grads)
    515 def linearity_1d(input_ind):
    516     def handler(explainer, op, *grads):
--> 517         return linearity_1d_handler(input_ind, explainer, op, *grads)
    518     return handler
    519 

C:\Anaconda\envs\keras_updated\lib\site-packages\shap\explainers\deep.py in linearity_1d_handler(input_ind, explainer, op, *grads)
    522     for i in range(len(op.inputs)):
    523         if i != input_ind:
--> 524             assert not explainer._variable_inputs(op)[i], str(i) + "th input to " + op.name + " cannot vary!"
    525 
    526     return explainer.orig_grads[op.type](op, *grads)

AssertionError: 1th input to concatenate_1/concat cannot vary!

The text was updated successfully, but these errors were encountered:

slundberg · 2018-11-02T23:42:17Z

The 1th input is the second (axis) argument so it looks like you have an axis that depends on your inputs...is that supposed to be true? Typically people don't choose the axis they concat with based on the model inputs.

LasseRegin · 2018-11-05T09:25:14Z

I think you misread the notation around concatenate @slundberg .

I get the same error using the following network:

# Define inputs
input_features = Input(shape=(n_features_input,))
input_embeddings = Input(shape=(1,))

# Define embeddings
x_embeddings = Embedding(input_dim=n_stations, output_dim=embedding_size)(input_embeddings)
x_embeddings = Reshape((embedding_size, ))(x_embeddings)

# Concatenate
x = Concatenate(axis=1)([input_features, x_embeddings])

# Layers
x = Dense(512, kernel_initializer='normal', activation='relu')(x)
x = Dense(256, kernel_initializer='normal', activation='relu')(x)
output = Dense(1, activation='linear')(x)

# Compile model
model = Model(
    inputs=[input_features, input_stations, input_vis_codes], 
    outputs=output)
model.compile(
    loss='mean_squared_error', 
    optimizer=keras.optimizers.Adam(lr=learning_rate),
    metrics=['mse'])

The final error line (same as @ellehoej ):

AssertionError: 1th input to concatenate_10/concat cannot vary!

Is the Concatenate layer just not supported in shap?

I am using

Tensorflow 1.8 backend
Keras 2.2.4
Python 3.6

slundberg · 2018-11-08T18:31:16Z

There must be an issue with the assertion then, causing Concatenate to break. I will look into it as soon as I can, it could be that the assertion is checking the wrong input.

…

On Mon, Nov 5, 2018 at 1:25 AM Lasse Regin Nielsen ***@***.***> wrote: I think you misread the notation @slundberg <https://github.com/slundberg> . # Define inputs input_features = Input(shape=(n_features_input,)) input_embeddings = Input(shape=(1,)) # Define embeddings x_embeddings = Embedding(input_dim=n_stations, output_dim=embedding_size)(input_embeddings) x_embeddings = Reshape((embedding_size, ))(x_embeddings) # Concatenate x = Concatenate(axis=1)([input_features, x_embeddings]) # Layers x = Dense(512, kernel_initializer='normal', activation='relu')(x) x = Dense(256, kernel_initializer='normal', activation='relu')(x) output = Dense(1, activation='linear')(x) # Compile model model = Model( inputs=[input_features, input_stations, input_vis_codes], outputs=output) model.compile( loss='mean_squared_error', optimizer=keras.optimizers.Adam(lr=learning_rate), metrics=['mse']) and I get the same error as @ellehoej <https://github.com/ellehoej> AssertionError: 1th input to concatenate_10/concat cannot vary! Is the Concatenate layer just not supported in shap? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#305 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADkTxW8ACQmBfb2X8txlMw19ZeaQf1M7ks5usAP8gaJpZM4X_NKx> .

LasseRegin · 2018-11-08T18:32:24Z

Thank you very much @slundberg ! Let me know if you need anything from me.

slundberg · 2018-11-09T17:06:10Z

@LasseRegin do you happen to already have a complete notebook that demonstrates the problem? If not I can build one.

LasseRegin · 2018-11-09T17:30:30Z

I created a new (simplified) Notebook demonstrating the problem. It can be found here:
https://gist.github.com/LasseRegin/a3de5fea9e6c3a499df52973574ed405

ellehoej · 2018-11-09T17:31:46Z

Hi @slundberg,
Unfortunately we are not able to share the data, but here's another example:

#Define input from training set.
n_features_input = train_X_features.shape[1]
x_features = Input(shape=(n_features_input,), name ='Features_input')


x_1 = Dense(256, use_bias=False)(x_features)
x_1 = BatchNormalization()(x_1)
x_l = PReLU()(x_1)
x_1 = Dropout(0.5)(x_1)


x_2 = Dense(256, use_bias=False)(x_1)
x_2 = BatchNormalization()(x_2)
x_2 = PReLU()(x_2)
x_2 = Dropout(0.5)(x_2)

x_3 = concatenate([x_1, x_2])

dense = Dense(1,activation='sigmoid')
x = dense(x_3)

model = Model(inputs=x_features, outputs=x)

adam = keras.optimizers.Adam(lr=3e-4)
model.compile(loss='mean_squared_error', optimizer=adam, metrics=['mse'])
model.summary()

The model summary is then:

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Features_input (InputLayer)     (None, 64)           0                                            
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 256)          16384       Features_input[0][0]             
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 256)          1024        dense_1[0][0]                    
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 256)          0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 256)          65536       dropout_1[0][0]                  
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 256)          1024        dense_2[0][0]                    
__________________________________________________________________________________________________
p_re_lu_2 (PReLU)               (None, 256)          256         batch_normalization_2[0][0]      
__________________________________________________________________________________________________
dropout_2 (Dropout)             (None, 256)          0           p_re_lu_2[0][0]                  
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 512)          0           dropout_1[0][0]                  
                                                                 dropout_2[0][0]                  
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 1)            513         concatenate_1[0][0]              
==================================================================================================
Total params: 84,737
Trainable params: 83,713
Non-trainable params: 1,024

background = SHAP_train_X_scaled[::100]
explainer = shap.DeepExplainer(model, background)
X_shap = SHAP_train_X_scaled[::1000].values
shap_values = explainer.shap_values(X_shap)

Gives us the error.

Best regards
Mads

ellehoej · 2018-11-09T17:33:19Z

Excellent, @LasseRegin!

slundberg · 2018-11-09T19:50:07Z

Thanks @LasseRegin for the setup! I found and fixed the problem.

LasseRegin · 2018-11-09T21:17:21Z

That is great @slundberg ! Thank you very much!

ellehoej · 2018-11-12T07:47:24Z

Thanks for the quick fix and for a great package in general @slundberg , it works in our end as well now.

slundberg added the bug Indicates an unexpected problem or unintended behaviour label Nov 9, 2018

slundberg closed this as completed in 890d85b Nov 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DeepExplainer doesn't work with keras concatenate layers #305

DeepExplainer doesn't work with keras concatenate layers #305

DeepExplainer doesn't work with keras concatenate layers #305

DeepExplainer doesn't work with keras concatenate layers #305

Comments