Skip to content
This repository has been archived by the owner on Jun 10, 2021. It is now read-only.

Boost and Eigen 3.3 on Ubuntu 16.04 LTS #22

Open
loretoparisi opened this issue Jun 20, 2017 · 15 comments
Open

Boost and Eigen 3.3 on Ubuntu 16.04 LTS #22

loretoparisi opened this issue Jun 20, 2017 · 15 comments

Comments

@loretoparisi
Copy link

loretoparisi commented Jun 20, 2017 •
edited
Loading

I have successfully installed both Boost and Eigen on Ubuntu 16.04 LTS:

Selecting previously unselected package libboost1.58-dev:amd64.
(Reading database ... 48598 files and directories currently installed.)
Preparing to unpack .../libboost1.58-dev_1.58.0+dfsg-5ubuntu3.1_amd64.deb ...
Unpacking libboost1.58-dev:amd64 (1.58.0+dfsg-5ubuntu3.1) ...
Selecting previously unselected package libboost-dev:amd64.
Preparing to unpack .../libboost-dev_1.58.0.1ubuntu1_amd64.deb ...
Unpacking libboost-dev:amd64 (1.58.0.1ubuntu1) ...
Selecting previously unselected package pkg-config.
Preparing to unpack .../pkg-config_0.29.1-0ubuntu1_amd64.deb ...
Unpacking pkg-config (0.29.1-0ubuntu1) ...
Selecting previously unselected package libeigen3-dev.
Preparing to unpack .../libeigen3-dev_3.3~beta1-2_all.deb ...
Unpacking libeigen3-dev (3.3~beta1-2) ...
Setting up libboost1.58-dev:amd64 (1.58.0+dfsg-5ubuntu3.1) ...
Setting up libboost-dev:amd64 (1.58.0.1ubuntu1) ...
Setting up pkg-config (0.29.1-0ubuntu1) ...
Setting up libeigen3-dev (3.3~beta1-2) ...

so libeigen3-dev (3.3~beta1-2) and libboost1.58-dev:amd64 (1.58.0+dfsg-5ubuntu3.1).

but when I compile

Step 24/29 : RUN git clone https://github.com/OpenNMT/CTranslate.git &&   cd CTranslate &&   git submodule update --init &&   mkdir build &&   cd build &&   cmake -DCMAKE_CXX_FLAGS="-march=native" .. &&   make
 ---> Running in bd9def302cf0
Cloning into 'CTranslate'...
Submodule 'lib/tokenizer' (https://github.com/OpenNMT/Tokenizer.git) registered for path 'lib/tokenizer'
Cloning into 'lib/tokenizer'...
Submodule path 'lib/tokenizer': checked out '0d90868e52fce98b1eda968ef23db3668b6f0db9'
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: Release
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp  
-- Could NOT find Boost
-- Eigen3 version 3.2.92 found in /usr/include/eigen3, but at least version 3.3 is required

the cmake does not detect boost or eigen3 in the system default folder.

Since I'm building on docker there are not previous packages installed, so it should find the ones I have installed via apt-get:

# OpenNMT dependencies: boot, eigen, intel mkl
RUN apt-get update && apt-get install -y \
  libboost-dev \
  libeigen3-dev -t xenial

where -t xenial is necessary to install the latest 3.3-beta.
@loretoparisi
Copy link
Author

loretoparisi commented Jun 20, 2017 •
edited
Loading

[UPDATE]

I have found a solution for eigen3:

RUN curl http://security.ubuntu.com/ubuntu/pool/universe/e/eigen3/libeigen3-dev_3.3.2-1_all.deb -o libeigen3-dev_3.3.2-1_all.deb && dpkg -i libeigen3-dev_3.3.2-1_all.deb

so using dpkg solves the install dir issue:

-- Could NOT find Boost
-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3")
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5") 
-- Could NOT find MKL (missing:  MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY)

The problem with boost is still present, so of course it still exits with error later on.
Reading the apt-get logs I can see that it installs some packages of boost but not all the recommended ones:

Reading state information...
The following additional packages will be installed:
  libboost1.58-dev
Suggested packages:
  libboost-doc libboost1.58-doc libboost-atomic1.58-dev
  libboost-chrono1.58-dev libboost-context1.58-dev libboost-coroutine1.58-dev
  libboost-date-time1.58-dev libboost-exception1.58-dev
  libboost-filesystem1.58-dev libboost-graph1.58-dev
  libboost-graph-parallel1.58-dev libboost-iostreams1.58-dev
  libboost-locale1.58-dev libboost-log1.58-dev libboost-math1.58-dev
  libboost-mpi1.58-dev libboost-mpi-python1.58-dev
  libboost-program-options1.58-dev libboost-python1.58-dev
  libboost-random1.58-dev libboost-regex1.58-dev
  libboost-serialization1.58-dev libboost-signals1.58-dev
  libboost-system1.58-dev libboost-test1.58-dev libboost-thread1.58-dev
  libboost-timer1.58-dev libboost-wave1.58-dev libboost1.58-tools-dev
  libmpfrc++-dev libntl-dev
The following NEW packages will be installed:
  libboost-dev libboost1.58-dev pkg-config
0 upgraded, 3 newly installed, 0 to remove and 9 not upgraded.

Not sure if this can be the issue here.

@loretoparisi loretoparisi changed the title Boot and Eigen 3.3 on Ubuntu 16.04 LTS Boost and Eigen 3.3 on Ubuntu 16.04 LTS Jun 20, 2017
@loretoparisi
Copy link
Author

loretoparisi commented Jun 20, 2017 •
edited
Loading

[UPDATE]

Ok this is my last finding to solve the boost issue:

# OpenNMT dependencies: boot, eigen, intel mkl
RUN apt-get update && apt-get install -y \
  libboost-dev \
  libboost-all-dev \
  pkg-config

It seems that libboost-all-dev will install all needed dependencies not only the boost library alone, so that now I came out with

-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3") 
... 
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5") 
-- Could NOT find MKL (missing:  MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY) 
-- Boost version: 1.58.0

I just have some complains about the architecture from nvcc when building

[ 27%] Building C object lib/TH/CMakeFiles/TH.dir/THGeneral.c.o
[ 30%] Building C object lib/TH/CMakeFiles/TH.dir/THFile.c.o
[ 33%] Building C object lib/TH/CMakeFiles/TH.dir/THDiskFile.c.o
[ 36%] Linking C shared library libTH.so
[ 36%] Built target TH
[ 39%] Building NVCC (Device) object CMakeFiles/onmt.dir/src/cuda/onmt_generated_Kernels.cu.o
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).

and the missing Intel MKL, but this is a separate thing.

@guillaumekln
Copy link
Collaborator

guillaumekln commented Jun 20, 2017 •
edited
Loading

If you want to make the nvcc warnings quiet, you can pass this flag to cmake:

-DCUDA_NVCC_FLAGS="-gencode=arch=compute_61,code=compute_61"

where 61 is the compute capability of your GPU.

Intel MKL is optional.

@loretoparisi
Copy link
Author

@guillaumekln thanks it make sense, by example when I have built the Dockerfile for darkflow I add to change this manually in the Cmake.

Somewhere I have used a command to detected the architecture automatically, I'm searching this right now...

@loretoparisi
Copy link
Author

Closing this since it has been solved btw.

@loretoparisi
Copy link
Author

loretoparisi commented Jun 20, 2017 •
edited
Loading

@guillaumekln re-opening since I get

root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# echo "hello this is me" | ./translate --model /root/nmt/onmt_baseline_wmt15-all.en-de_epoch13_7.19_release.t7 --beam_size 5 --batch_size 1 -
Illegal instruction (core dumped)

where the built file was

root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# file translate 
translate: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9116d7fc8728d04bb935c6781256aa9c486e6358, not stripped

My cmake options were

cmake -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&

I have added the DWITH_OPENMP and it seems to work

cmake -DWITH_OPENMP=OFF -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&

@loretoparisi loretoparisi reopened this Jun 20, 2017
@guillaumekln
Copy link
Collaborator

What is your g++ --version?

@loretoparisi
Copy link
Author

@guillaumekln

0.0.90.190 ubuntu@ai-tools-nmt-i-04fc6623dcd3531f8:~$ g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)

@guillaumekln
Copy link
Collaborator

Not sure about this.

  • Are you compiling and testing on the same machine?
  • Are you compiling with a compiler other than g++?

@loretoparisi
Copy link
Author

loretoparisi commented Aug 3, 2017 •
edited
Loading

@guillaumekln I think the problem were the linked libraries

root@ac0117818001:/mxmlib/CTranslate/build/cli# ldd translate 
    linux-vdso.so.1 =>  (0x00007fff2bbda000)
    libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)
    libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007f7cab9b9000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7cab637000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7cab421000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cab057000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7caae3a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7caac36000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7caaa2d000)
    libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007f7ca7a12000)
    libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
    libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7ca72e7000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f7ca70c5000)
    /lib64/ld-linux-x86-64.so.2 (0x0000557438c78000)

So I think I was missing

libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)

or libboost compiled on a different machine.

Question: no way to get rid of boost and eigen i.e. not using them?

@guillaumekln
Copy link
Collaborator

Why would you like to drop these dependencies?

@loretoparisi
Copy link
Author

@guillaumekln I wonder if it would be possibile to make the binaries standalone i.e. like fasttext does. This would be great and make everything more portable. Thank you!

@guillaumekln
Copy link
Collaborator

Did you try to compile with static libraries?

cmake -DBUILD_SHARED_LIBS=OFF [...]

@loretoparisi
Copy link
Author

@guillaumekln thanks now I have only the boost dependency:

root@e195516272c0:/mxmlib/CTranslate/build/cli# ldd translate 
	linux-vdso.so.1 =>  (0x00007ffc8f1cc000)
	libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007ffa56042000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffa55e24000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffa55c20000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffa55a18000)
	libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007ffa529fc000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffa5267a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffa52371000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffa5215a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa51d91000)
	/lib64/ld-linux-x86-64.so.2 (0x00005637afd00000)

@guillaumekln
Copy link
Collaborator

boost is only used for managing command line options. It should be easy to write your own translation client without this dependency.

See cli/translate.cc.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@loretoparisi @guillaumekln