Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test TestSparseL1Descent::test_attack_strength in tests_tf\test_attacks.py #1142

Closed
loopylangur opened this issue Jan 18, 2020 · 6 comments
Closed

Flaky test TestSparseL1Descent::test_attack_strength in tests_tf\test_attacks.py #1142

loopylangur opened this issue Jan 18, 2020 · 6 comments

Comments

@loopylangur
Copy link
Contributor

Describe the bug
The TestSparseL1Descent::test_attack_strength test in tests_tf\test_attacks.py fails intermittently with the following error:

>     self.assertLess(np.mean(orig_labs == new_labs), 0.4)
E     AssertionError: 0.4 not less than 0.4

tests_tf/test_attacks.py:854: AssertionError

This failied 1 out of 30 times for me. Should this be changed to assertLessEqual instead?

To Reproduce
This can be reprodcuced by setting the following seed in the setup function:

class TestSparseL1Descent(CleverHansTest):
  def setUp(self):
    super(TestSparseL1Descent, self).setUp()
    tf.set_random_seed(6450233306813956982)

Expected behavior
Test should always pass

Screenshots
NA
Full error log:

============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.7.5, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: cleverhans
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From cleverhans/cleverhans/utils_tf.py:345: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

collected 1 item

tests_tf/test_attacks.py WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:684: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:685: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-01-18 12:28:06.439957: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2020-01-18 12:28:06.447021: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2993350000 Hz
2020-01-18 12:28:06.447928: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5625c6bd7410 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-18 12:28:06.447945: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-23
OMP: Info #156: KMP_AFFINITY: 24 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 12 cores/pkg x 2 threads/core (12 total cores)
OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to package 0 core 0 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to package 0 core 1 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to package 0 core 2 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to package 0 core 3 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 4 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 16 maps to package 0 core 4 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 5 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 17 maps to package 0 core 5 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 8 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 18 maps to package 0 core 8 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 9 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 19 maps to package 0 core 9 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 10 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 20 maps to package 0 core 10 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 11 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 21 maps to package 0 core 11 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 12 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 22 maps to package 0 core 12 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 13 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 23 maps to package 0 core 13 thread 1 
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31503 thread 0 bound to OS proc set 0
2020-01-18 12:28:06.455630: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
[INFO 2020-01-18 12:28:06,456 cleverhans] Constructing new graph for attack SparseL1Descent
WARNING:tensorflow:From cleverhans/cleverhans/attacks/attack.py:141: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From cleverhans/cleverhans/utils_tf.py:749: The name tf.assert_greater_equal is deprecated. Please use tf.compat.v1.assert_greater_equal instead.

WARNING:tensorflow:From cleverhans/cleverhans/utils_tf.py:740: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.

WARNING:tensorflow:From cleverhans/cleverhans/utils_tf.py:448: The name tf.random_gamma is deprecated. Please use tf.random.gamma instead.

WARNING:tensorflow:From cleverhans/cleverhans/utils_tf.py:404: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:49: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:49: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From cleverhans/cleverhans/attacks/sparse_l1_descent.py:96: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From cleverhans/cleverhans/compat.py:79: calling softmax_cross_entropy_with_logits_v2_helper (from tensorflow.python.ops.nn_ops) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31595 thread 1 bound to OS proc set 1
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31596 thread 2 bound to OS proc set 2
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31598 thread 3 bound to OS proc set 3
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31601 thread 6 bound to OS proc set 6
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31599 thread 4 bound to OS proc set 4
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31600 thread 5 bound to OS proc set 5
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31602 thread 7 bound to OS proc set 7
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31603 thread 8 bound to OS proc set 8
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31604 thread 9 bound to OS proc set 9
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31606 thread 11 bound to OS proc set 11
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31605 thread 10 bound to OS proc set 10
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31607 thread 12 bound to OS proc set 12
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31608 thread 13 bound to OS proc set 13
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31609 thread 14 bound to OS proc set 14
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31611 thread 16 bound to OS proc set 16
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31610 thread 15 bound to OS proc set 15
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31612 thread 17 bound to OS proc set 17
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31613 thread 18 bound to OS proc set 18
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31615 thread 20 bound to OS proc set 20
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31614 thread 19 bound to OS proc set 19
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31616 thread 21 bound to OS proc set 21
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31617 thread 22 bound to OS proc set 22
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31618 thread 23 bound to OS proc set 23
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31619 thread 24 bound to OS proc set 0
OMP: Info #250: KMP_AFFINITY: pid 31503 tid 31620 thread 25 bound to OS proc set 1

==================================

test_attacks.TestSparseL1Descent.test_attack_strength took 0.7115423679351807 seconds
F

=================================================================================================== FAILURES ====================================================================================================
___________________________________________________________________________________ TestSparseL1Descent.test_attack_strength ____________________________________________________________________________________

self = <test_attacks.TestSparseL1Descent testMethod=test_attack_strength>

    def test_attack_strength(self):
      """
      Without clipped gradients, we achieve
      np.mean(orig_labels == new_labels) == 0.31.
      """
      x_val = np.random.rand(100, 2)
      x_val = np.array(x_val, dtype=np.float32)
    
      # sanity checks turned off because this test initializes outside
      # the valid range.
      x_adv = self.attack.generate_np(x_val, eps=10.0, rand_init=True,
                                      clip_min=0.5, clip_max=0.7,
                                      nb_iter=10, sanity_checks=False)
    
      orig_labs = np.argmax(self.sess.run(self.model.get_logits(x_val)), axis=1)
      new_labs = np.argmax(self.sess.run(self.model.get_logits(x_adv)), axis=1)
>     self.assertLess(np.mean(orig_labs == new_labs), 0.4)
E     AssertionError: 0.4 not less than 0.4

tests_tf/test_attacks.py:854: AssertionError
----------------------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------------------
WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:684: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:685: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

INFO     cleverhans:attack.py:127 Constructing new graph for attack SparseL1Descent
WARNING  tensorflow:module_wrapper.py:139 From cleverhans/cleverhans/attacks/attack.py:141: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/cleverhans/utils_tf.py:749: The name tf.assert_greater_equal is deprecated. Please use tf.compat.v1.assert_greater_equal instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/cleverhans/utils_tf.py:740: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/cleverhans/utils_tf.py:448: The name tf.random_gamma is deprecated. Please use tf.random.gamma instead.

WARNING  tensorflow:deprecation.py:323 From cleverhans/cleverhans/utils_tf.py:404: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:49: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:49: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING  tensorflow:deprecation.py:323 From cleverhans/cleverhans/attacks/sparse_l1_descent.py:96: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING  tensorflow:deprecation.py:506 From cleverhans/cleverhans/compat.py:79: calling softmax_cross_entropy_with_logits_v2_helper (from tensorflow.python.ops.nn_ops) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
=============================================================================================== 1 failed in 2.40s ===============================================================================================

System configuration

Ubuntu 16.04
python 3.7
numpy==1.17.4
tensorflow==1.15.0
torch==1.3.1 
tensorflow-probability==0.8.0
@ftramer
Copy link
Contributor

ftramer commented Mar 27, 2020

Thanks for noticing this. I guess I didn't choose the bounds well enough.
I think changing 0.4 -> 0.5 should be yield a robust enough test.

@loopylangur
Copy link
Contributor Author

Ok, I can create a PR to change it..

do you mind look at #1145 ?

@ftramer
Copy link
Contributor

ftramer commented Mar 27, 2020

Yeah that seems to be a similar issue. These type of probabilistic tests are problematic in general.
Maybe they should be run with a fixed seed @npapernot? But even then, you might get some non-determinism from the GPU.

@loopylangur loopylangur mentioned this issue Mar 27, 2020
Merged
@loopylangur
Copy link
Contributor Author

Raised #1154

@jonasguan
Copy link
Collaborator

Thanks @loopylangur , I merged your PR.

@loopylangur
Copy link
Contributor Author

Thanks @jonasguan!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ftramer @jonasguan @loopylangur