Skip to content
This repository has been archived by the owner on Jan 3, 2023. It is now read-only.

tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported #271

Open
tiogoGit opened this issue Aug 27, 2019 · 10 comments
Open

tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported #271

tiogoGit opened this issue Aug 27, 2019 · 10 comments

Comments

@tiogoGit
Copy link

I've looked into question #235 But that couldn't solve this problem. I once succeeded in testing Cryptonets. However, after that I will always fail to do
NGRAPH_ENCRYPT_DATA=1 \ NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \ NGRAPH_TF_BACKEND=HE_SEAL \ python test.py --batch_size=4096
I've tried rebuilding from master and changing HE_SEAL to HE_SEAL_CKKS. I also tried restarting the PC. None of them worked.
The result of failure is
`(venv-tf-py3) mugi@mugi-Precision-5820-Tower:~/he-transformer/examples/MNIST/Cryptonets$ NGRAPH_ENCRYPT_DATA=1 \

NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json
NGRAPH_TF_BACKEND=HE_SEAL
python test.py --batch_size=4096
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2019-08-27 15:06:01.623635: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2019-08-27 15:06:01.624328: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x19d0b90 executing computations on platform Host. Devices:
2019-08-27 15:06:01.624366: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
Traceback (most recent call last):
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 90, in
test_mnist_cnn(FLAGS)
File "test.py", line 58, in test_mnist_cnn
(x_train, y_train, x_test, y_test) = load_mnist_data()
File "/home/mugi/he-transformer/examples/MNIST/mnist_util.py", line 29, in load_mnist_data
y_test = tf.one_hot(y_test, 10).eval()
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 731, in eval
return _eval_using_default_session(self, feed_dict, self.graph, session)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5579, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported`

I've built from master and also built python after that. The version message of my PC is:

Operating system: Ubuntu 18.04.
CMake = 3.10.2
Compiler: g++ version = 8.3.0, clang = 6.0.0
python3.6
virtualenv v16.1.0
bazel v0.25.2
@fboemer
Copy link
Collaborator

fboemer commented Aug 27, 2019 •
edited
Loading

Does the make install step succeed? We expect libhe_seal_backend.so to appear in
$HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib
and
$HE_TRANSFORMER/build/external/lib

@tiogoGit
Copy link
Author

Does the make install step succeed? We expect he_seal_backend.so to appear in
$HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib
and
$HE_TRANSFORMER/build/external/lib

No, there is only libhe_seal_backend.so. And I installed it again but there is still no he_seal_backend.so. However the install seemed ok. The make install ends with this:
[ 90%] Linking CXX executable unit-test [ 91%] Built target unit-test [100%] Built target pybind11 Install the project... -- Install configuration: "" -- Installing: /home/mugi/he-transformer/build/external/lib/libhe_seal_backend.so -- Set runtime path of "/home/mugi/he-transformer/build/external/lib/libhe_seal_backend.so" to "$ORIGIN"

What's strange is that, this time only $HE_TRANSFORMER/build/external/lib contains he_seal_backend.so
this time. And $HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib contains no .so file.
I can run NGRAPH_ENCRYPT_DATA=1 \ NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \ NGRAPH_TF_BACKEND=HE_SEAL \ python test.py --batch_size=4096
While the result is still unacceptable.
| Encryption parameters : | scheme: CKKS | poly_modulus_degree: 8192 | coeff_modulus size: 180 (30 + 24 + 24 + 24 + 24 + 24 + 30) bits \ [INFO] 2019-08-28T04:09:15z src/seal/he_seal_backend.cpp 87 Scale 1.67608e+07 [INFO] 2019-08-28T04:09:15z src/seal/he_seal_executable.cpp 462 Encrypting data [INFO] 2019-08-28T04:09:15z src/seal/he_seal_executable.cpp 465 Batching data with batch size 4096 total time(s) 10.469 Error count: 3679 of 4096 elements. Accuracy: 0.101806640625

At the same time, Crypptonets-ReLU is alright:
/ | Encryption parameters : | scheme: CKKS | poly_modulus_degree: 2048 | coeff_modulus size: 54 (54) bits \ [INFO] 2019-08-28T04:32:34z src/seal/he_seal_backend.cpp 87 Scale 8.38861e+06 [INFO] 2019-08-28T04:32:34z src/seal/he_seal_executable.cpp 462 Encrypting data [INFO] 2019-08-28T04:32:34z src/seal/he_seal_executable.cpp 465 Batching data with batch size 1024 [WARN] 2019-08-28T04:32:34z src/seal/he_seal_executable.cpp 1384 Performing Relu without client is not privacy-preserving [WARN] 2019-08-28T04:32:35z src/seal/he_seal_executable.cpp 1384 Performing Relu without client is not privacy-preserving total time(s) 0.754 y_pred [7 2 1 ... 7 5 4] Error count 84 of 1024 elements. Accuracy: 0.917969
Just like #266 . I think there is something wrong in make install, but don't know how to correct it.
Could you tell me why?

@fboemer
Copy link
Collaborator

fboemer commented Aug 29, 2019

So there are two orthogonal issues here:

  1. The install; if the CryptoNets-Relu example is working, it seems the install worked okay. After the install, we expect
    *$HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib to contain libhe_seal_backend.so
  • $HE_TRANSFORMER/build/external/lib to contain libhe_seal_backend.so
  1. Accuracy. Try larger parameters, e.g. he_seal_ckks_config_N13_L8.json. I do see high accuracy (~97.5%) with the currents weights using the current master branch.

@fboemer
Copy link
Collaborator

fboemer commented Aug 30, 2019 •
edited
Loading

As it turns out, HE_SEAL is not supported seems to be thrown whenever he-transformer throws an error, rather than printing the more descriptive reason. This is a reported issue with the ngraph-bridge.

Could you confirm NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json exists? (i.e. $HE_TRANSFORMER is defined)

@tiogoGit
Copy link
Author

Yes, it exists.

@cjbumgardner
Copy link

cjbumgardner commented Sep 17, 2019 •
edited
Loading

I'll chime in and say I'm have the same problem as the poster. I also have HE_TRANSFORMER defined, and the configs file mentioned exists. Also, make install succeeded. Although, even after a few make install runs, I don't get "libhe_seal_backend.so" in the virtual environment bin; it is in "external/lib", though. There is no "he_seal_backend.so" anywhere. Here's output for folder contents:

ubuntu@******:~/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib$ ls
python2.7 python3.6

ubuntu@******:~/he-transformer/build/external/lib$ ls
cmake libiomp5.so libngraph_bridge.so libprotobuf-lite.la libprotobuf.a libprotobuf.so.20.0.1 libprotoc.so.20 libtbb.so.2
libcpu_backend.so libmkldnn.so libngraph_test_util.a libprotobuf-lite.so libprotobuf.la libprotoc.a libprotoc.so.20.0.1 pkgconfig
libhe_seal_backend.so libmklml_intel.so libnop_backend.so libprotobuf-lite.so.20 libprotobuf.so libprotoc.la libseal.a
libinterpreter_backend.so libngraph.so libprotobuf-lite.a libprotobuf-lite.so.20.0.1 libprotobuf.so.20 libprotoc.so libtbb.so

@fboemer
Copy link
Collaborator

fboemer commented Sep 19, 2019

Thank you, @tiogoGit , @cjbumgardner for your reports. Here's more details on what the installation looks like at each step. Note, there is no he_seal_backend.so, just libhe_seal_backend.so, and build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib should be empty after all. Please let me know which, if any, of the below steps fails.

  1. cmake ..
  2. make install
  • $HE_TRANSFORMER/build/external/lib/libhe_seal_backend.so should exist
  • $HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/lib/libhe_seal_backend.so should exist, and be a soft link to $HE_TRANSFORMER/build/external/lib/libhe_seal_backend.so
  1. You should be able to run ./test/unit-test
  2. source external/venv-tf-py3/bin/activate
  3. NGRAPH_TF_BACKEND=HE_SEAL python $HE_TRANSFORMER/examples/ax.py should work and print NGraph using backend: HE_SEAL This indicates the installation worked as expected
  4. cd $HE_TRANSFORMER/examples/MNIST/Cryptonets
  5. NGRAPH_VOPS=all NGRAPH_TF_BACKEND=HE_SEAL python test.py runs inference on unencrypted data using HE_SEAL backend
  6. NGRAPH_ENCRYPT_DATA=1 \ NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \ NGRAPH_TF_BACKEND=HE_SEAL \ python test.py --batch_size=4096 NGRAPH_VOPS=all NGRAPH_TF_BACKEND=HE_SEAL python test.py should result in ~89% accuracy (higher accuracy can be found with larger parameters, or perhaps re-training the model)

@cjbumgardner
Copy link

cjbumgardner commented Sep 21, 2019 via email

@fboemer
Copy link
Collaborator

fboemer commented Sep 24, 2019 •
edited
Loading

Thanks @cjbumgardner for your detailed report again. A few notes:

  1. Once merged, shrestha/backend_error_message_fixes tensorflow/ngraph-bridge#229 will hopefully return a more useful error message
  2. I wouldn't recommend using older versions. There have been many improvements in the functionality and runtime in recent months. We used to support two backends, HE_SEAL_CKKS, and HE_SEAL_BFV. The former has been renamed HE_SEAL, the latter is no longer supported.
  3. TensorFlow version installed: 1.12.0 suggests you may be using an older version. We expect TensorFlow version 1.14
  4. Would you also try
python -c "import ngraph_bridge; print(ngraph_bridge.list_backends())"

We expect HE_SEAL to appear in the list. If so, this suggests the package is installed correctly, there is just some error when creating backend instance.

@cjbumgardner
Copy link

cjbumgardner commented Sep 29, 2019 via email

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fboemer @cjbumgardner @tiogoGit