Training issue

[Slejpnir]

When trying to run dp_train the following error arises:

Traceback (most recent call last):
  File "/bin/dp_train", line 16, in <module>
    from deepmd.Model import NNPModel
  File "/bin/../lib/deepmd/Model.py", line 21, in <module>
    op_module = tf.load_op_library(module_path + "libop_abi.so")
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /lib/deepmd/libop_abi.so: undefined symbol: _ZN10tensorflow14kernel_factory17OpKernelRegistrar12InitInternalEPKNS_9KernelDefEN4absl11string_viewEPFPNS_8OpKernelEPNS_20OpKernelConstructionEE

The versions of python and C++ Tensorflow are the same. I've also tried -DTF_GOOGLE_BIN=true option.

Could you please give me some suggestion?

[dingye18]

You can try command

ldd /lib/deepmd/libop_abi.so

to check if you linked the right library for libop_abi.so.

[Slejpnir]

The output is:

linux-vdso.so.1 (0x00007fffe874a000)
        libtensorflow_framework.so => not found
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3138a10000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3138670000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f3138440000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3138220000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3137e20000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3139000000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3137c00000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f31379e0000)

But there is libtensorflow_framework.so in the $tensorflow_root/lib folder.

[dingye18]

You need to add the path of libtensorflow_framework.so to environment variable $LD_LIBRARY_PATH. Try this command, and rerun your program.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$tensorflow_root/lib

[Slejpnir]

Thanks a lot!
But now the new error:

Traceback (most recent call last):
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/slejpnir/venv/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/slejpnir/venv/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: /home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _ZN6google8protobuf8internal26fixed_address_empty_stringE

During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/bin/dp_train", line 9, in <module>
    import tensorflow as tf
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/slejpnir/venv/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/slejpnir/venv/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: /home/slejpnir/venv/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _ZN6google8protobuf8internal26fixed_address_empty_stringE


Failed to load the native TensorFlow runtime.

[dingye18]

OK. Now you have the same problem with me before.
My solution is to compile tensorflow again. But I did not compile the C++ version again. You need to compile the pip package of tensorflow with

bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

and install the pip package (Do not need to configure again). Find more details from this link.
And try again after you installed the tensorflow(self-compiled) python package.
BTW, remove the old tensorflow package from site-packages first.

[Slejpnir]

Thanks, it has started working finally.

[vijendra037]

I follow same instruction as my error was same.
_pywrap_tensorflow_internal.so is present in tenserflow folder.
/home/vkumar/venv/lib/python3.6/sitepackages/tensorflow/python/_pywrap_tensorflow_internal.so

but I don't understand your last solution.
how to remove tensorflow package from the site packages. Do I need to remove by conda uninstall tensorflow or manually by rm -rf command?

I run given command:
"bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package"
Bazel didin't build. I got and error output.

Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 204.926s, Critical Path: 60.52s
INFO: 3863 processes: 3863 local.
FAILED: Build did NOT complete successfully.

please reply.