Month: April 2018

Tensorflow is a continuous pain to get working on any Linux other than a recent Ubuntu. Unfortunately most of my production systems are on some variant of CentOS 6…

Here’s what I currently do to get Tensorflow 1.7.0 working on it. These instructions are brief and not absolutely step by step, but you can probably figure out what to do from them, and certainly I should be able to look back at these notes and figure out what I was doing. I’m not going to do things like explicitly tell you when to cd or mkdir a new place to drop things in. There aren’t proper patchfiles. These are scribbled notes as I went along getting bits from various links around the place.

1. Download source and install to some –prefix=/location a recent binutils – 2.30 at time of writing
2. Download source and install to some –prefix=/location a recent GCC (I used 7.3.0 initially but CUDA doesn’t like GCC 6 or later, so you probably want a v5.4.0 compiler, but not v5.5 or later as that breaks stuff too! v4.9.4 didn’t work so just go with v5.4.0!) – note also there’s some references below to 7.3.0 which I’m too lazy to fix.
3. yum install java-1.8.0-openjdk-devel
4. Grab a suitably new Python (I used an Anaconda installation)
5. Get bazel (I used v0.11.1) – make sure it’s the -dist.zip download
   Unpack bazel
6. Patch bazel – in tools/cpp/unix_cc_configure.bzl comment out
   “-B/usr/bin”,
   – it’ll be prefixed by a comment like
   # Always have -B/usr/bin, see https://github.com/bazelbuild/bazel/issues/760.
   which is totally rubbish, as that very link will tell you in one of the github comments.
7. Set $PATH to add python, gcc, and binutils bin locations. Set $INCLUDE to add gcc and binutils include location. Set $LD_LIBRARY_PATH and LIBRARY_PATH to add gcc lib64 location and binutils lib location. Set $JAVA_HOME to /usr/lib/jvm/java-1.8.0.
8. Backup if wanted, then rm -rf ~/.cache and mkdir /tmp/someplace and ln -s /tmp/someplace ~/.cache (prevents NFS headaches with bazel)
9. In bazelsrc directory do ./compile.sh, then copy resulting output/bazel somewhere and add that somewhere to $PATH
10. git clone https://github.com/tensorflow/tensorflow.git
    (and checkout a specific version if wanted)
11. Patch tensorflow –
    1. tensorflow/tensorflow.bzl:

       def tf_extension_linkopts():
       
        return ["-lrt"] # No extension link opts

       (add “-lrt” inside []s)

       def tf_cc_shared_object(
       
        name,
       
        srcs=[],
       
        deps=[],
       
        linkopts=["-lrt"],

       (ditto)

        args += [src.path]
       
        outputs = [ctx.outputs.cc_out, ctx.outputs.py_out]
       
        ctx.action(
       
       use_default_shell_env = True,
       
        executable=ctx.executable._swig,
       
        arguments=args,

       (adding use_default_shell_env=True,)

    2. tensorflow/third_party/gpus/crosstool/CROSSTOOL_clang.tpl, tensorflow/third_party/gpus/crosstool/CROSSTOOL_nvcc.tpl, tensorflow/third_party/sycl/crosstool/CROSSTOOL.tpl, tensorflow/third_party/toolchains/cpus/arm/CROSSTOOL.tpl, tensorflow/third_party/toolchains/gpus/crosstool/CROSSTOOL,
       change “-B/usr/bin” to “-B/path/to/binutils/bin”
    3. tensorflow/third_party/gpus/crosstool/CROSSTOOL_nvcc.tpl
       change
       linker_flag: “-Wl,-no-as-needed”
       to
       linker_flag: “-Wl,-R/share/apps/gnu/gcc/7.3.0/lib64,-no-as-needed”
       Change /usr/bin paths for ar, compat-ld, ld, nm, objcopy, objdump, strip to installed binutils and cpp to installed gcc
       After
       cxx_builtin_include_directory: “/”
       add
       cxx_builtin_include_directory: “/path/to/gcc/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include”
       cxx_builtin_include_directory: “/path/to/gcc/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include-fixed”
       cxx_builtin_include_directory: “/path/to/include/c++/7.3.0”
12. Configure tensorflow – most defaults should be ok especially if it picks your python up correctly. Notable exception – answer no to jemalloc question.
13. bazel build –config=opt //tensorflow/tools/pip_package:build_pip_package may also need –config=cuda
14. bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
15. Find .whl file in /tmp/tensorflow_pkg, copy it somewhere safe and pip install tensorflow-….longname.whl

Note – –verbose_failures is a useful flag at the bazel build stage.

(You can comment below but I’m unlikely to offer much help. I’ve had enough of this.)