nnpack.md 5.41 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144
### NNPACK for Multi-Core CPU Support in TVM
[NNPACK](https://github.com/Maratyszcza/NNPACK) is an acceleration package
for neural network computations, which can run on x86-64, ARMv7, or ARM64 architecture CPUs.
Using NNPACK, higher-level libraries like _MXNet_ can speed up
the execution on multi-core CPU computers, including laptops and mobile devices.

***Note***: AS TVM already has natively tuned schedules, NNPACK is here mainly for reference and comparison purpose. 
For regular use prefer native tuned TVM implementation.

_TVM_ supports NNPACK for forward propagation (inference only) in convolution, max-pooling, and fully-connected layers.
In this document, we give a high level overview of how to use NNPACK with _TVM_.

### Conditions
The underlying implementation of NNPACK utilizes several acceleration methods,
including [fft](https://arxiv.org/abs/1312.5851) and [winograd](https://arxiv.org/abs/1509.09308).
These algorithms work better on some special `batch size`, `kernel size`, and `stride` settings than on other,
so depending on the context, not all convolution, max-pooling, or fully-connected layers can be powered by NNPACK.
When favorable conditions for running NNPACKS are not met,

NNPACK only supports Linux and OS X systems. Windows is not supported at present.
The following table explains under which conditions NNPACK will work.

| operation      | conditions |
|:---------      |:---------- |
|convolution     |2d convolution `and` no-bias=False `and` dilate=(1,1) `and` num_group=1 `and` batch-size = 1 or batch-size > 1 && stride = (1,1);|
|pooling         | max-pooling `and` kernel=(2,2) `and` stride=(2,2) `and` pooling_convention=full    |
|fully-connected| without any restrictions |

### Build/Install LLVM
LLVM is required for CPU codegen that needs LLVM.
Since LLVM takes long time to build from source, you can download pre-built version of LLVM from [LLVM Download Page](http://releases.llvm.org/download.html).
For llvm 4.0 you can do the following step : 

```bash
# Add llvm repository in apt source list
echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-4.0 main" >> /etc/apt/sources.list

# Update apt source list
apt-get update
# Install clang and full llvm
apt-get install -y \
    clang-4.0 \
    clang-4.0-doc \
    libclang-common-4.0-dev \
    libclang-4.0-dev \
    libclang1-4.0 \
    libclang1-4.0-dbg \
    libllvm-4.0-ocaml-dev \
    libllvm4.0 \
    libllvm4.0-dbg \
    lldb-4.0 \
    llvm-4.0 \
    llvm-4.0-dev \
    llvm-4.0-doc \
    llvm-4.0-examples \
    llvm-4.0-runtime \
    clang-format-4.0 \
    python-clang-4.0 \
    libfuzzer-4.0-dev
```

### Build/Install NNPACK

If the trained model meets some conditions of using NNPACK,
you can build TVM with NNPACK support.
Follow these simple steps:  
* Build NNPACK shared library with the following commands. _TVM_ will link NNPACK dynamically.

Note: The following NNPACK installation instructions have been tested on Ubuntu 16.04.

#### Build [Ninja](https://ninja-build.org/)

NNPACK need a recent version of Ninja. So we need to install ninja from source.
```bash
git clone git://github.com/ninja-build/ninja.git
cd ninja
./configure.py --bootstrap
```

Set the environment variable PATH to tell bash where to find the ninja executable. For example, assume we cloned ninja on the home directory ~. then we can added the following line in ~/.bashrc. 
```bash
export PATH="${PATH}:~/ninja"
```

#### Build [NNPACK](https://github.com/Maratyszcza/NNPACK)

The new CMAKE version of NNPACK download [Peach](https://github.com/Maratyszcza/PeachPy) and other dependencies alone

```bash
git clone --recursive https://github.com/Maratyszcza/NNPACK.git
cd NNPACK
# Add PIC option in CFLAG and CXXFLAG to build NNPACK shared library
sed -i "s|gnu99|gnu99 -fPIC|g" CMakeLists.txt
sed -i "s|gnu++11|gnu++11 -fPIC|g" CMakeLists.txt
mkdir build
cd build
# Generate ninja build rule and add shared library in configuration
cmake -G Ninja -D BUILD_SHARED_LIBS=ON ..
ninja
sudo ninja install

# Add NNPACK lib folder in your ldconfig
echo "/usr/local/lib" > /etc/ld.so.conf.d/nnpack.conf
sudo ldconfig
```

### Build TVM with NNPACK support

```bash
git clone --recursive https://github.com/dmlc/tvm
```

* Set `USE_NNPACK = 1` in config.mk.
* Set `NNPACK_PATH` to the $(YOUR_NNPACK_INSTALL_PATH)
* Set `LLVM_CONFIG = llvm-config-4.0` depending of llvm version installed

after configuration use `make` to build TVM

```bash
make
make install
```

#### Python Package Installation

The python package for [tvm](https://github.com/dmlc/tvm) depends of [topi](https://github.com/dmlc/tvm/tree/master/topi).
The tvm python package is located at `tvm/python` and topi python package is located in `tvm/topi/python` folder.
There are several ways to install the package, in all these cases the TVM library and TOPI must be present in the python env:

1. Set the environment variable PYTHONPATH to tell python where to find the libraries. For example, assume we cloned tvm on the home directory ~. then we can added the following line in ~/.bashrc. It is recommended for developers who may change the codes. The changes will be immediately reflected once you pulled the code and rebuild the project (no need to call setup again)

```bash
export PYTHONPATH=/path/to/tvm/python:/path/to/tvm/topi/python:${PYTHONPATH}
```

2. Install tvm and topi python bindings by setup.py:

```bash
# install tvm package for the current user
cd topi/python
python setup.py install --user; 
cd ../../python
python setup.py install --user; 
```