如何快速安装深度学习框架如何快速安装深度学习框架

安装Pytorch 首先参考Pytorch官网利用pip的方式安装pytorch包：

pip install http://download.pytorch.org/whl/cpu/torch-0.4.0-cp27-cp27mu-linux_x86_64.whl pip install torchvision # if the above command does not work, then you have python 2.7 UCS2, use this command pip install http://download.pytorch.org/whl/cpu/torch-0.4.0-cp27-cp27m-linux_x86_64.whl

文章图片

安装之后测试import torch发现报错“ImportError: /lib64/libgcc_s.so.1: version `GCC_4.2.0’ not found (required by /opt/compiler/gcc-4.8.2/lib/libstdc++.so.6)”，这个问题通常是系统版本较低，因此libgcc的版本较低，我们可以查看下系统的libstdc++中的gcc版本：

> strings /lib64/libgcc_s.so.1 | grep GCC GCC_3.0 GCC_3.3 GCC_3.3.1 GCC_3.4 GCC_3.4.2 GCC_3.4.4

通常遇到这种问题，我们只能自己重新编译Pytorch，因此pip库中编译是默认依赖的是系统的/lib64的动态链接库，而重新编译的代价实在是大，所以有什么简单的方法吗？？？
很高兴的说当然有！！！我们换个角度想一下，是不是将Pytorch依赖的/lib64/libgcc_s.so.1换个高版本的就可以？
所以，我首先想到的是通过设置LD_LIBRARY_PATH来改变运行时依赖库，但是将高版本的gcc库放进去之后，遗憾的发现系统的很多命令用不了了……
那么还有没有别的办法：仅仅修改Pytorch依赖的运行库，而不修改其他的？经过不断查找，终于找到一种方式，通过patchelf的方式修改Pytorch的动态链接库搜索路径。
首先我们看下Pytorch的so库依赖的路径：

> readelf -d ./build/lib/python2.7/site-packages/torch/_C.soDynamic section at offset 0xb165a0 contains 30 entries: TagTypeName/Value 0x0000000000000001 (NEEDED)Shared library: [libshm.so] 0x0000000000000001 (NEEDED)Shared library: [libATen.so] 0x0000000000000001 (NEEDED)Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED)Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED)Shared library: [libc.so.6] 0x0000000000000001 (NEEDED)Shared library: [ld-linux-x86-64.so.2] 0x000000000000001d (RUNPATH)Library runpath: [$ORIGIN:$ORIGIN/lib] 0x000000000000000c (INIT)0x1e2b90 0x000000000000000d (FINI)0x900acc 0x0000000000000019 (INIT_ARRAY)0xcf2640 0x000000000000001b (INIT_ARRAYSZ)864 (bytes) 0x000000000000001a (FINI_ARRAY)0xcf29a0 0x000000000000001c (FINI_ARRAYSZ)8 (bytes) 0x000000006ffffef5 (GNU_HASH)0xe55d78 0x0000000000000005 (STRTAB)0xf66268 0x0000000000000006 (SYMTAB)0x157c8 0x000000000000000a (STRSZ)1023503 (bytes) 0x000000000000000b (SYMENT)24 (bytes) 0x0000000000000003 (PLTGOT)0xd1a000 0x0000000000000002 (PLTRELSZ)69528 (bytes) 0x0000000000000014 (PLTREL)RELA 0x0000000000000017 (JMPREL)0x1d1bf8 0x0000000000000007 (RELA)0x162610 0x0000000000000008 (RELASZ)456168 (bytes) 0x0000000000000009 (RELAENT)24 (bytes) 0x000000006ffffffe (VERNEED)0x162550 0x000000006fffffff (VERNEEDNUM)4 0x000000006ffffff0 (VERSYM)0x15bf24 0x000000006ffffff9 (RELACOUNT)4787 0x0000000000000000 (NULL)0x0

这里可以看到依赖的RUNPATH=$ORIGIN:$ORIGIN/lib，ORIGIN表示当前路径，那么接下来就该下RUNPATH就OK了：

patchelf--set-rpath '$ORIGIN:$ORIGIN/lib:/opt/compiler/gcc-4.8.2/lib64:/opt/compiler/gcc-4.8.2/lib'./build/lib/python2.7/site-packages/torch/_C.so

然后再来尝试一下：

> build/bin/python Python 2.7.15 (default, May 30 2018, 10:28:45) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import torch Traceback (most recent call last): File "", line 1, in File "/home/disk6/python/build/lib/python2.7/site-packages/torch/__init__.py", line 278, in import torch.multiprocessing File "/home/disk6/python/build/lib/python2.7/site-packages/torch/multiprocessing/__init__.py", line 33, in from .queue import Queue, SimpleQueue File "/home/disk6/python/build/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 3, in import multiprocessing.queues File "/home/disk6/python/build/lib/python2.7/multiprocessing/queues.py", line 48, in from .synchronize import Lock, BoundedSemaphore, Semaphore, Condition File "/home/disk6/python/build/lib/python2.7/multiprocessing/synchronize.py", line 59, in " function, see issue 3770.") ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

这个问题可以参考multiprocessing sem_open 3770这篇文章解决：
1.需要root权限部分的操作：

修改/etc/fstab 增加： tmpfs /dev/shm tmpfs defaults 0 0 mount /dev/shm chmod 777 /dev/shm

2.不需要root权限部分的操作:

1、重新安装python 从 python 官网下载源码，解压；进入解压后的python文件夹；执行： ./configure --prefix=/home/disk6/python/build make && make install2、重新安装pip 安装 setuptools： wget --no-check-certificate http://pypi.python.org/packages/source/s/setuptools/setuptools-2.0.tar.gz tar -xzvf setuptools-2.0.tar.gz cd setuptools-2.0 /home/disk6/python/build/bin/python setup.py install 安装pip： wget --no-check-certificate https://pypi.python.org/packages/41/27/9a8d24e1b55bd8c85e4d022da2922cb206f183e2d18fee4e320c9547e751/pip-8.1.1.tar.gz#md5=6b86f11841e89c8241d689956ba99ed7 tar -xzf pip-8.1.1.tar.gz cd pip-8.1.1 /home/disk6/python/build/bin/python setup.py install

安装TensorFlow 按照官网安装，报错

> build/bin/python Python 2.7.15 (default, May 30 2018, 10:28:45) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow Traceback (most recent call last): File "", line 1, in File "/home/disk6/python/build/lib/python2.7/site-packages/tensorflow/__init__.py", line 24, in from tensorflow.python import pywrap_tensorflow# pylint: disable=unused-import File "/home/disk6/python/build/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/home/disk6/python/build/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/home/disk6/python/build/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "/home/disk6/python/build/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/disk6/python/build/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) ImportError: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.17' not found (required by /home/disk6/python/build/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)Failed to load the native TensorFlow runtime.See https://www.tensorflow.org/install/install_sources#common_installation_problemsfor some common reasons and solutions.Include the entire stack trace above this error message when asking for help.

同上，使用patchelf lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so即可。

patchelf--set-rpath '$ORIGIN/:$ORIGIN/..:/opt/compiler/gcc-4.8.2/lib64:/opt/compiler/gcc-4.8.2/lib'/home/disk6/python/build/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so

安装PaddlePaddle 使用pip install paddlepaddle安装，运行import paddle报错：

> build/bin/python Python 2.7.15 (default, May 30 2018, 10:28:45) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import paddle Traceback (most recent call last): File "", line 1, in File "/home/disk6/python/build/lib/python2.7/site-packages/paddle/__init__.py", line 25, in import dataset File "/home/disk6/python/build/lib/python2.7/site-packages/paddle/dataset/__init__.py", line 25, in import sentiment File "/home/disk6/python/build/lib/python2.7/site-packages/paddle/dataset/sentiment.py", line 26, in import nltk File "/home/disk6/python/build/lib/python2.7/site-packages/nltk/__init__.py", line 137, in from nltk.stem import * File "/home/disk6/python/build/lib/python2.7/site-packages/nltk/stem/__init__.py", line 29, in from nltk.stem.snowball import SnowballStemmer File "/home/disk6/python/build/lib/python2.7/site-packages/nltk/stem/snowball.py", line 32, in from nltk.corpus import stopwords File "/home/disk6/python/build/lib/python2.7/site-packages/nltk/corpus/__init__.py", line 66, in from nltk.corpus.reader import * File "/home/disk6/python/build/lib/python2.7/site-packages/nltk/corpus/reader/__init__.py", line 105, in from nltk.corpus.reader.panlex_lite import * File "/home/disk6/python/build/lib/python2.7/site-packages/nltk/corpus/reader/panlex_lite.py", line 15, in import sqlite3 File "/home/disk6/python/build/lib/python2.7/sqlite3/__init__.py", line 24, in from dbapi2 import * File "/home/disk6/python/build/lib/python2.7/sqlite3/dbapi2.py", line 28, in from _sqlite3 import * ImportError: No module named _sqlite3

【如何快速安装深度学习框架】解决办法，重新编译python with sqlite模块。