# Performance Benchmark

## Results

See results on wiki page https://github.com/dmlc/tvm/wiki/Benchmark

## How to Reproduce

### ARM CPU
We use RPC infrastructure in TVM to make device management easy. So you need to use it for reproducing benchmark results.

1. Start an RPC Tracker on the host machine
```bash
python3 -m tvm.exec.rpc_tracker
```

2. Register devices to the tracker
* For Linux device
  * Build tvm runtime on your device [Help](https://docs.tvm.ai/tutorials/nnvm/deploy_model_on_rasp.html#build-tvm-runtime-on-device)
  * Register your device to tracker by
  ```bash
  python3 -m tvm.exec.rpc_sever --tracker=[HOST_IP]:9190 --key=[DEVICE_KEY]
  ```
  replace `[HOST_IP]` with the IP address of the host machine, `[DEVICE_KEY]` with the name of device.
  
  E.g. Here is an example command for RK3399,
  `python3 -m tvm.exec.rpc_sever --tracker=10.77.1.123:9190 --key=rk3399`, where 10.77.1.123 is the IP address of the tracker.

* For Android device
   * Build and install tvm RPC apk on your device [Help](https://github.com/dmlc/tvm/tree/master/apps/android_rpc).
     Make sure you can pass the android rpc test. Then you have alreadly known how to register.

3. Verify the device registration  
  We can query all registered devices by
  ```bash
  python3 -m tvm.exec.query_rpc_tracker
  ```
  You should be able to find your devices in `Queue Status`. Make sure the registration is correct before going ahead.

  For our test environment, one sample output can be 
  ```bash
  Queue Status                
  ------------------------------
  key            free    pending    
  ------------------------------
  mate10pro      1       0   
  p20pro         2       0  
  pixel2         2       0 
  rk3399         2       0
  rasp3b         8       0
  ```

 4. Run benchmark  
  We did auto-tuning for Huawei P20/Mate10 Pro, Google Pixel2, Raspberry Pi3 and Firefly-RK3399,
  and release pre-tuned parameters in [this repo](https://github.com/uwsaml/tvm-distro).
  During compilation, TVM will download these operator parameters automatically.

  ```bash
  python3 arm_cpu_imagenet_bench.py --device rasp3b --rpc-key rasp3b
  python3 arm_cpu_imagenet_bench.py --device rk3399 --rpc-key rk3399
  python3 arm_cpu_imagenet_bench.py --device pixel2 --rpc-key pixel2
  python3 arm_cpu_imagenet_bench.py --device p20pro --rpc-key p20pro
  python3 arm_cpu_imagenet_bench.py --device mate10pro --rpc-key mate10pro  
  ```

  If your device has a same SoC of the above device, you can reuse these parameters
  (e.g. use `llvm -device=arm_cpu -mode=rk3399 -target=aarch64-linux-gnu` as target).
  Otherwise, you need to tune for your own device, please follow this 
  [tutorial](https://docs.tvm.ai/tutorials/autotvm/tune_nnvm_arm.html).

